Piping programs in shell

When I started to write this thread of articles on shell scripting, I did not know where it would lead me. I just started writing to try and cover what I knew about scripting, and how I have applied it. During this, I have written about:

• Conditions (or tests)
• Variables
• Bare basics

The next bit I want to cover is about the toy block like structure of this type of language. You can build programs on top of other programs, and then "pipe" their output from one to another. Piping (also known as the verticle bar - "|" ) is a way of chaining output from one program in as the input for another program. Here is the shell script:

$ cat new.sh
#!/usr/bin/sh

echo This is a find on the test directory:
find test/ -print

echo ""
echo This is the same find, being passed through to grep to only find stuff ending with txt
find test/ -print | grep txt$

The first line defines this as a shell script. The second line prints the words "This is a find on the test directory" to the screen. The third line runs the find program on the test directory, and outputs the contents to the screen. The forth line line prints a blank line. The fifth line prints the line "This is the same find, being passed through to grep to only find stuff ending with txt". The sixth line is similar tot he third line, but before outputting the data to the screen, passes it via the grep filter (by the pipe character) to only print out lines that end with txt. I saved the file as new.sh, and running it produces:

$ ./new.sh
This is a find on the test directory:
test/
test/CheckThis.pl
test/servers.txt
test/CheckThat.pl
test/CheckSomethingElse.pl
test/web_pages.txt

This is the same find, being passed through to grep to only find stuff ending with txt
test/servers.txt
test/web_pages.txt

As a side note, while this script is straight forward, it introduces two new programs - being find and grep. Find, as its name suggests, finds files on a filesystem (it behaves differently on a Windows system). Grep is different. It is a filtering tool based on regular expressions. A regular expressions is a way to match strings. I could write an entire book on how to master regular expressions - there is so many combinations and permutations on how to write them. If you are interested in doing more with regular expressions, have a look at the Mastering Regular Expressions site.

Conditional scripting

When you break programming down, regardless what language or syntax it is written or developed in, the basis of a piece of code is how it handles a condition.

What happens when you have something? How do you handle it? What do you do with it?

The absolute basic conditional statement in shell scripting is the "if" statement. The basics of the if is that "if the conditions are met, then do something". To be fancier, you can also add "else". Here is an example:

#!/usr/bin/sh

variable1=Hello;
variable2=Goodbye;

echo "Let's start!";
if [ "$variable1" = "$variable2" ]; then
  echo "The salutations are the same!";
else
  echo "The salutations are not the same!";
fi

variable2=Hello;
echo "";
echo "Let's try again!";
if [ "$variable1" = "$variable2" ]; then
  echo "The salutations are the same!";
else
  echo "The salutations are not the same!";
fi

This simple script sets up 2 variables, and the compares them. In the second paragraph, it changes the second variable to be the same as the first variable, and runs the same if statement test on them again. The output looks like:

$ ./test.sh
Let's start!
The salutations are not the same!

Let's try again!
The salutations are the same!

While not very exciting, you can do many different types of data. Keep in mind that comparing text is different from comparing data. More comparison tests can be found here.

Lastly, I would like to mention that you can put a shell script into a type of "debug" mode where it will output more verbose information while it is running. This is by changing the first line of the script from:

#!/usr/bin/sh

to

#!/usr/bin/sh -x

Now the output looks like:

$ ./test.sh
variable1=Hello
variable2=Goodbye
+ echo Let's start!
Let's start!
+ [ Hello = Goodbye ]
+ echo The salutations are not the same!
The salutations are not the same!
variable2=Hello
+ echo

+ echo Let's try again!
Let's try again!
+ [ Hello = Hello ]
+ echo The salutations are the same!
The salutations are the same!

Good luck with your scripting, and I welcome any comments or queries.

Basic variable creation in shell

In my last article on basic shell scripting, I did the universal hello world program. This time around, I will discuss making variables.

A variable is a symbolic name for information. You can call a variable nearly anything in unix shell scripting. There are some exceptions though. Here is the simple program outlining variables:

#!/usr/bin/sh

salutation=Hi;
name=Fred;

echo $salutation $name;
echo "";
echo salutation name;

The first line describes this program as a shell script. The second line assigns the value of "Hi" to the variable of salutation. The third line assigned the value of "Fred" to the variable of name. The forth line outputs the two variable values. This is done by using the "$" in front of the variable name. The fifth line prints out a blank line. The sixth line is only for comparison of what the fourth line would have looked like if it did not have the "$". I have saved the contents of the file to a file called "test.sh" and made it executable. The output is:

$ ./test.sh
Hi Fred

salutation name

That is as easy as variables can get. You can store text and letters in variables, as well as strings. With some clever delimiting, you can also turn these variables into mini lists - similar to arrays in other more advanced forms of programming.

I will attempt to write up how to do some basic comparisons in my next article on scripting. I am also welcome to suggestions via the comments of this post.

Basic shell scripting tutorial

Kitty Kat has asked me to teach her some basic shell scripting tips. She may be interested in writing her own scripts to automate functions or tasks on the Linux machine we both use. I thought that this was a good idea, but while I was giving her some pointers as what to do, I'd share it with anyone else who may be interested.

Therefore, I'll start with the universal "hello world" program. The first part is to determine if you have a shell to use, and if you can access the echo command. From you command line (assuming it is a "$"), type in the following (without the "$"):

$ which sh
$ which echo

The response from your system should look something like this:

$ which sh
/usr/bin/sh
$ which echo
/usr/bin/echo

Variations of the path are acceptable, but something like this means there is something wrong:

$ which echo
which: no echo in (/usr/local/bin:/usr/bin:/bin)

This means that either echo is not installed on your system, or that echo is not in the PATH that your command line interface is aware of. Assuming that everything is correct, open up a text editor that you are comfortable with, and (based on the information above), type in the following:

#!/usr/bin/sh

# Hello World Program
echo "Hello World!";

Save it as a file called "helloWorld.sh", and quit out of the editor. If your which commands returned different variables from the ones used in this example, adjust your helloWorld.sh script accordingly.

By default, most Linux systems do not make scripts automatically executable. To make it executable, type the following:

$ chmod u+x helloWorld.sh

To run the script, type the following:

$ ./helloWorld.sh

The output should be:

$ ./helloWorld.sh
Hello World!

In breaking down the simple script, the first line ("#!/usr/bin/sh") tells the command line that is is a shell script. Changing it would invoke different scripting engines (for example, "#!/usr/bin/perl" would mean the contents of the script should be a perl script). The next line is a comment line - and this is designated by the use of a hash symbol ("#") preceding the other text on the line. Comments are used to improve readability, but are totally optional. The third line is the line that displays the words.

I hope to expand this article into a few more that cover a few other things to do in the shell. Just look for the "Shell Scripting" tag I will use for this series of articles.