Removing comments and blank lines out of files

Sometimes I have the need to read configuration files, but I do not want to see the comments in them, or the blank lines - I just want to see the information that is relevant. If you have ever had to deal with a "default" configuration file for the Apache Web Server, or the Squid Proxy Server, then you know what I mean. The default files will usually have big sections of comments, to be used as examples or reminders of features that you can switch on or off.

I find this next line useful in removing comments and blank lines:

grep -v ^# file_name|grep -v '^[[:space:]]*$'

For example, if the file is called web.txt and result of running the line over the file is:

[ ric_man @ mymachine ~/test ] 506 $ cat web.txt
### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# ServerType is either inetd, or standalone. Inetd mode is only supported on
# Unix platforms.
#
ServerType standalone

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE! If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation
# (available at );
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/var/www"

#
# The LockFile directive sets the path to the lockfile used when Apache
# is compiled with either USE_FCNTL_SERIALIZED_ACCEPT or
# USE_FLOCK_SERIALIZED_ACCEPT. This directive should normally be left at
# its default value. The main reason for changing it is if the logs
# directory is NFS mounted, since the lockfile MUST BE STORED ON A LOCAL
# DISK. The PID of the main server process is automatically appended to
# the filename.
#
#LockFile logs/accept.lock
[ ric_man @ mymachine ~/test ] 507 $ grep -v ^# web.txt|grep -v '^[[:space:]]*$'
ServerType standalone
ServerRoot "/var/www"

Just in case you are confused with what I have done above - I have outputted the contents of the file first, and then run the command over it. This is to illustrate the amount of removing that the line does.

Finding text within files recursively

Occasionally, I need to search for text strings within files within directories on a UNIX / Linux command line. I find that the following is able to assist me in finding the correct line:

find . -exec grep -i search_string {} /dev/null \;

Here is an example of it in action:

[ ric_man @ mymachine ~/test ] 521 $ ls -alR
.:
total 11K
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:12 ./
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:09 ../
-rwx------+ 1 ric_man None 1.2K 2011-01-17 15:03 OBS_bulk.txt*
-rwxr-xr-x 1 ric_man None 291 2011-01-17 15:41 convert.sh*
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:11 dir1/
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:12 dir2/
-rw-r--r-- 1 ric_man None 10 2011-01-27 10:12 fileA.txt
-rw-r--r-- 1 ric_man None 83 2010-12-23 13:55 input.txt
-rwxr-xr-x 1 ric_man None 1.1K 2010-12-23 14:03 runme.sh*

./dir1:
total 2.0K
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:11 ./
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:12 ../
-rw-r--r-- 1 ric_man None 39 2011-01-27 10:11 file1.txt
-rw-r--r-- 1 ric_man None 39 2011-01-27 10:11 file2.txt

./dir2:
total 1.0K
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:12 ./
drwxr-xr-x+ 1 ric_man None 0 2011-01-27 10:12 ../
-rw-r--r-- 1 ric_man None 18 2011-01-27 10:12 fred.conf
[ ric_man @ mymachine ~/test ] 522 $ find . -exec grep -i snack {} /dev/null \;
./dir1/file1.txt:Here is the string to be found - snack
./dir1/file2.txt:Here is the string to be found - snack
./dir2/fred.conf:Snack found here!
./fileA.txt:Snack me!

In Microsoft Windows environments, I think there's a search facility, but it is slow, and sometimes inaccurate.

Finding out users in groups

In a UNIX / Linux environment, it may be useful to determine who is in a group. The following line will determine who is in a group, and pull out their information based on the finger protocol:

for i in $(cat /etc/group|grep -i insert_group_name_here|cut -d ":" -f 4|sed 's/,/ /g'); do echo "################"; echo "";finger $i; echo"";done

This looks like:

$ for i in $(cat /etc/group|grep -i mygroup|cut -d ":" -f 4|sed 's/,/ /g'); do echo "################"; echo "";finger $i; echo"";done
################

Login name: smithjrr In real life: 6116
Site Info: A/E/John Smith
Directory: /home/smithjrr Shell: /usr/bin/ksh
No Plan.

################

Login name: mywasguy In real life: 6116
Site Info: C/K/L/Web App Server Admin
Directory: /home/mywasguy Shell: /usr/bin/ksh
No Plan.

################

Login name: ric In real life: 6116
Site Info: E/I/ric_man
Directory: /home/ric Shell: /usr/bin/ksh
On since Jan 25 11:17:22 on pts/0
No Plan.

Your results may vary depending on how the information is stored.

Copying and renaming files in Linux / UNIX

I had a problem of multiple locations containing the same file names, but wanting to copy all those files into the one directory to back them up. The file structure looked something like this:

./somewhere/install1/conf/webserver1.conf
./somewhere/install1/conf/webserver2.conf
./somewhere/install1/conf/webserver3.conf
./somewhere/install1/conf/webserver4.conf
./somewhere/install2/conf/webserver1.conf
./somewhere/install2/conf/webserver2.conf
./somewhere/install2/conf/webserver3.conf
./somewhere/install2/conf/webserver4.conf

If I was to copy the files over into the same location, some of the files would overwrite files which had a similar name. So, with the following line, I manipulated the target filename to retain the original location of the file:

$ for i in $(find . -name webserver*.conf | grep -v backup | grep -v Trace | grep -v LocalConfig); do cp $i /tmp/mylocation/$(echo $i|sed 's/\//_/g'|sed 's/^\._//g'); done

The final result is:

/tmp/mylocation/somewhere_install1_conf_webserver1.conf
/tmp/mylocation/somewhere_install1_conf_webserver2.conf
/tmp/mylocation/somewhere_install1_conf_webserver3.conf
/tmp/mylocation/somewhere_install1_conf_webserver4.conf
/tmp/mylocation/somewhere_install2_conf_webserver1.conf
/tmp/mylocation/somewhere_install2_conf_webserver2.conf
/tmp/mylocation/somewhere_install2_conf_webserver3.conf
/tmp/mylocation/somewhere_install2_conf_webserver4.conf

While this could have been made into a script, I have not done it in a full script. Therefore it is not apart of the shell scripting tutorial series I was writing for Kitty Kat.

I have also noticed that a few people are reading this, and making suggestions to me via email. Please feel free to share your comments on this and all the articles on this blog, by using the comments link at the bottom of every blog posting.

Tuna Niçoise salad at home

I appreciate a good Tuna Niçoise salad, and will venture out to Pancake Parlour to get one. While I have ordered it at many different restaurants, and had some ones that are "fancier" than others, I do like the version that the Pancake Parlour produce. It is one of their staples of their menu, and with their Doncaster store (in Melbourne) being open 24 hours a day, I know where I can always find one - although I have never ordered one for breakfast. I find it enjoyable in both winter and summer, but am most likely to consume it more in the warmer months of the year. It can be served cold, or with fresh seared tuna on top. The ingredients are relatively simple:

• Blanched green beans
• Boiled Eggs
• Onions (red are preferable for appearance)
• Lettuce mix (variation of different leaves, but can be done with Butter Lettuce or Iceberg Lettuce)
• Capers
• Boiled potatoes
• Cherry tomatoes
• Olives
• Canned or fresh pan seared tuna
• Optional: Capsicum (I prefer the red ones as they a mild sweetness to the dish)
• Optional: Anchovies (although I do not think they suit this dish - but are well suited towards pizza)

Cut them up and arrange on a plate, then drizzle with a dressing made from:

• Lemon juice
• White wine vinegar
• Parsley
• Garlic
• Virgin Olive Oil
• Salt
• Pepper

You may have noticed that I have not specified any quantities. This was done on purpose. I find this dish best when it is customized for the diner. If they like more tuna, then add more tuna. If they do not like boiled eggs, then omit them from the dish. The result looks something like: This was KittyKat's first attempt at making this dish - and it looked and tasted excellent - especially seeing how she is not a qualified chef. Good food will always be produced when the person making puts their heart into the effort of making it. Following a recipe and not experimenting also helps. I will be looking forward to having this dish again in the future!

The liquid poured Pearl coloured shoes

Back in 1989, the first of many Batman films was released at the cinemas - starring Michael Keaton as Batman. He continued to star in one more film sequel, and was later replaced by Val Kilmer, as well as George Clooney. The role is now being filled by Christian Bale. For good or bad, the films were visually interesting to watch. One aspect of the film I do remember the most was the suit Batman wore when he was out fighting crime. Unlike the lycra-like suit worn by Adam West in the 1960s Batman television show, the modern-day versions of Batman had a suit which replicated an athletic physique. The suit would look like it was made in one casting, out of some type of plastic or rubber substance, and while flexible, it also held the look of rippling muscles.

This leads me to this next shoe - the Nike Air Foamposite Pro. This shoe is rock solid. The foamposite material makes them heavy and very sturdy. Although they are not directly related, this shoe reminds me of the Batman suit used by the modern day incarnations. This shoe has the appearance that it is made in one casting - even though there are visible stitching on the toe and the heel.

I cannot do a proper review of these, besides a visual collage below, but I do suggest having a look at DJ Delz's videos on Youtube. In his words: "Shout out to DJ Delz!" for being a thorough reviewer of sneakers, and he will usually film a small segment where he will have them "on feet".

The key points of the video is that the shoe is highly noticeable, and is very tight until they are broken in. Also, wearing the sneakers will not cause them to crease. These shoes are similar to the Nike Air Foamposite I, which was released in 1997 and we initially worn by Anfernee Hardaway - a guard that started off with much promise for a stellar career, but was crippled by knee injuries. The main differences between the Foamposite Pro and the Foamposite Is are that there is a large Nike swoosh on the Pro, the Is have a "1 cent" logo on them (in reference to Hardaway's nickname "Penny"), and the sole is slightly different.

These shoes appear to be the inspiration for the look of the Nike Air Max Wavy shoes, although the material that both shoes are constructed from is vastly different.

My thanks for these shoes goes out to the team at crazyforsneaker.com. If you are after a pair for yourself, please do not hesitate in contacting them.

Basic shell scripting tutorial

Kitty Kat has asked me to teach her some basic shell scripting tips. She may be interested in writing her own scripts to automate functions or tasks on the Linux machine we both use. I thought that this was a good idea, but while I was giving her some pointers as what to do, I'd share it with anyone else who may be interested.

Therefore, I'll start with the universal "hello world" program. The first part is to determine if you have a shell to use, and if you can access the echo command. From you command line (assuming it is a "$"), type in the following (without the "$"):

$ which sh
$ which echo

The response from your system should look something like this:

$ which sh
/usr/bin/sh
$ which echo
/usr/bin/echo

Variations of the path are acceptable, but something like this means there is something wrong:

$ which echo
which: no echo in (/usr/local/bin:/usr/bin:/bin)

This means that either echo is not installed on your system, or that echo is not in the PATH that your command line interface is aware of. Assuming that everything is correct, open up a text editor that you are comfortable with, and (based on the information above), type in the following:

#!/usr/bin/sh

# Hello World Program
echo "Hello World!";

Save it as a file called "helloWorld.sh", and quit out of the editor. If your which commands returned different variables from the ones used in this example, adjust your helloWorld.sh script accordingly.

By default, most Linux systems do not make scripts automatically executable. To make it executable, type the following:

$ chmod u+x helloWorld.sh

To run the script, type the following:

$ ./helloWorld.sh

The output should be:

$ ./helloWorld.sh
Hello World!

In breaking down the simple script, the first line ("#!/usr/bin/sh") tells the command line that is is a shell script. Changing it would invoke different scripting engines (for example, "#!/usr/bin/perl" would mean the contents of the script should be a perl script). The next line is a comment line - and this is designated by the use of a hash symbol ("#") preceding the other text on the line. Comments are used to improve readability, but are totally optional. The third line is the line that displays the words.

I hope to expand this article into a few more that cover a few other things to do in the shell. Just look for the "Shell Scripting" tag I will use for this series of articles.

Starting a new 2011

Everyone still here?

I am back now for 2011, and I am hoping you are all here too. There is a lot of interesting things going on for 2011, even though it is a few days old. I know of many major changes in a few people around me, and I am sure you have a similar scenario going on in your life. I am predicting there will be a few great changes in my life as well.

So if you wish to continue reading, I am hoping to deliver more personal observations on Information Technology, Freedom, Open Source, Linux, Football, sneakers, and whatever...