Piping programs in shell

When I started to write this thread of articles on shell scripting, I did not know where it would lead me. I just started writing to try and cover what I knew about scripting, and how I have applied it. During this, I have written about:

• Conditions (or tests)
• Variables
• Bare basics

The next bit I want to cover is about the toy block like structure of this type of language. You can build programs on top of other programs, and then "pipe" their output from one to another. Piping (also known as the verticle bar - "|" ) is a way of chaining output from one program in as the input for another program. Here is the shell script:

$ cat new.sh
#!/usr/bin/sh

echo This is a find on the test directory:
find test/ -print

echo ""
echo This is the same find, being passed through to grep to only find stuff ending with txt
find test/ -print | grep txt$

The first line defines this as a shell script. The second line prints the words "This is a find on the test directory" to the screen. The third line runs the find program on the test directory, and outputs the contents to the screen. The forth line line prints a blank line. The fifth line prints the line "This is the same find, being passed through to grep to only find stuff ending with txt". The sixth line is similar tot he third line, but before outputting the data to the screen, passes it via the grep filter (by the pipe character) to only print out lines that end with txt. I saved the file as new.sh, and running it produces:

$ ./new.sh
This is a find on the test directory:
test/
test/CheckThis.pl
test/servers.txt
test/CheckThat.pl
test/CheckSomethingElse.pl
test/web_pages.txt

This is the same find, being passed through to grep to only find stuff ending with txt
test/servers.txt
test/web_pages.txt

As a side note, while this script is straight forward, it introduces two new programs - being find and grep. Find, as its name suggests, finds files on a filesystem (it behaves differently on a Windows system). Grep is different. It is a filtering tool based on regular expressions. A regular expressions is a way to match strings. I could write an entire book on how to master regular expressions - there is so many combinations and permutations on how to write them. If you are interested in doing more with regular expressions, have a look at the Mastering Regular Expressions site.