Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev	Chapter 12. External Filters, Programs and Commands	Next

12.4. Text Processing Commands

Commands affecting text and text files

sort

File sorter, often used as a filter in a pipe. This command sorts a text stream or file forwards or backwards, or according to various keys or character positions. Using the -m option, it merges presorted input files. The info page lists its many capabilities and options. See Example 10-9, Example 10-10, and Example A-8.

tsort

Topological sort, reading in pairs of whitespace-separated strings and sorting according to input patterns.

uniq

This filter removes duplicate lines from a sorted file. It is often seen in a pipe coupled with sort.
1 cat list-1 list-2 list-3 | sort | uniq > final.list 2 # Concatenates the list files, 3 # sorts them, 4 # removes duplicate lines, 5 # and finally writes the result to an output file.

The useful -c option prefixes each line of the input file with its number of occurrences.

bash$ cat testfile This line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times. bash$ uniq -c testfile 1 This line occurs only once. 2 This line occurs twice. 3 This line occurs three times. bash$ sort testfile | uniq -c | sort -nr 3 This line occurs three times. 2 This line occurs twice. 1 This line occurs only once.

The sort INPUTFILE | uniq -c | sort -nr command string produces a frequency of occurrence listing on the INPUTFILE file (the -nr options to sort cause a reverse numerical sort). This template finds use in analysis of log files and dictionary lists, and wherever the lexical structure of a document needs to be examined.

Example 12-11. Word Frequency Analysis

   1 #!/bin/bash
   2 # wf.sh: Crude word frequency analysis on a text file.
   3 # This is a more efficient version of the "wf2.sh" script.
   4 
   5 
   6 # Check for input file on command line.
   7 ARGS=1
   8 E_BADARGS=65
   9 E_NOFILE=66
  10 
  11 if [ $# -ne "$ARGS" ]  # Correct number of arguments passed to script?
  12 then
  13   echo "Usage: `basename $0` filename"
  14   exit $E_BADARGS
  15 fi
  16 
  17 if [ ! -f "$1" ]       # Check if file exists.
  18 then
  19   echo "File \"$1\" does not exist."
  20   exit $E_NOFILE
  21 fi
  22 
  23 
  24 
  25 ########################################################
  26 # main ()
  27 sed -e 's/\.//g'  -e 's/\,//g' -e 's/ /\
  28 /g' "$1" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr
  29 #                           =========================
  30 #                            Frequency of occurrence
  31 
  32 #  Filter out periods and commas, and
  33 #+ change space between words to linefeed,
  34 #+ then shift characters to lowercase, and
  35 #+ finally prefix occurrence count and sort numerically.
  36 
  37 #  Arun Giridhar suggests modifying the above to:
  38 #  . . . | sort | uniq -c | sort +1 [-f] | sort +0 -nr
  39 #  This adds a secondary sort key, so instances of
  40 #+ equal occurrence are sorted alphabetically.
  41 #  As he explains it:
  42 #  "This is effectively a radix sort, first on the
  43 #+ least significant column
  44 #+ (word or string, optionally case-insensitive)
  45 #+ and last on the most significant column (frequency)."
  46 ########################################################
  47 
  48 exit 0
  49 
  50 # Exercises:
  51 # ---------
  52 # 1) Add 'sed' commands to filter out other punctuation,
  53 #+   such as semicolons.
  54 # 2) Modify to also filter out multiple spaces and other whitespace.

expand, unexpand

The expand filter converts tabs to spaces. It is often used in a pipe.

The unexpand filter converts spaces to tabs. This reverses the effect of expand.

cut

A tool for extracting fields from files. It is similar to the print $N command set in awk, but more limited. It may be simpler to use cut in a script than awk. Particularly important are the -d (delimiter) and -f (field specifier) options.

Using cut to obtain a listing of the mounted filesystems:
1 cat /etc/mtab | cut -d ' ' -f1,2

Using cut to list the OS and kernel version:
1 uname -a | cut -d" " -f1,3,11,12

Using cut to extract message headers from an e-mail folder:
bash$ grep '^Subject:' read-messages | cut -c10-80 Re: Linux suitable for mission-critical apps? MAKE MILLIONS WORKING AT HOME!!! Spam complaint Re: Spam complaint

Using cut to parse a file:
1 # List all the users in /etc/passwd. 2 3 FILENAME=/etc/passwd 4 5 for user in $(cut -d: -f1 $FILENAME) 6 do 7 echo $user 8 done 9 10 # Thanks, Oleg Philon for suggesting this.

cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename

See also Example 12-34.

tail

lists the end of a file to stdout (the default is 10 lines). Commonly used to keep track of changes to a system logfile, using the -f option, which outputs lines appended to the file.

Example 12-14. Using tail to monitor the system log

   1 #!/bin/bash
   2 
   3 filename=sys.log
   4 
   5 cat /dev/null > $filename; echo "Creating / cleaning out file."
   6 #  Creates file if it does not already exist,
   7 #+ and truncates it to zero length if it does.
   8 #  : > filename   and   > filename also work.
   9 
  10 tail /var/log/messages > $filename  
  11 # /var/log/messages must have world read permission for this to work.
  12 
  13 echo "$filename contains tail end of system log."
  14 
  15 exit 0

See also Example 12-5, Example 12-34 and Example 30-6.

grep

A multi-purpose file search tool that uses Regular Expressions. It was originally a command/filter in the venerable ed line editor: g/re/p -- global - regular expression - print.

grep pattern [file...]

Search the target file(s) for occurrences of pattern, where pattern may be literal text or a Regular Expression.

bash$ grep '[rst]ystem.$' osinfo.txt The GPL governs the distribution of the Linux operating system.

If no target file(s) specified, grep works as a filter on stdout, as in a pipe.

bash$ ps ax | grep clock 765 tty1 S 0:00 xclock 901 pts/1 S 0:00 grep clock

The -i option causes a case-insensitive search.

The -w option matches only whole words.

The -l option lists only the files in which matches were found, but not the matching lines.

The -r (recursive) option searches files in the current working directory and all subdirectories below it.

The -n option lists the matching lines, together with line numbers.

bash$ grep -n Linux osinfo.txt 2:This is a file containing information about Linux. 6:The GPL governs the distribution of the Linux operating system.

The -v (or --invert-match) option filters out matches.
1 grep pattern1 *.txt | grep -v pattern2 2 3 # Matches all lines in "*.txt" files containing "pattern1", 4 # but ***not*** "pattern2".

The -c (--count) option gives a numerical count of matches, rather than actually listing the matches.
1 grep -c txt *.sgml # (number of occurrences of "txt" in "*.sgml" files) 2 3 4 # grep -cz . 5 # ^ dot 6 # means count (-c) zero-separated (-z) items matching "." 7 # that is, non-empty ones (containing at least 1 character). 8 # 9 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz . # 4 10 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '$' # 5 11 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '^' # 5 12 # 13 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -c '$' # 9 14 # By default, newline chars (\n) separate items to match. 15 16 # Note that the -z option is GNU "grep" specific. 17 18 19 # Thanks, S.C.

When invoked with more than one target file given, grep specifies which file contains matches.

bash$ grep Linux osinfo.txt misc.txt osinfo.txt:This is a file containing information about Linux. osinfo.txt:The GPL governs the distribution of the Linux operating system. misc.txt:The Linux operating system is steadily gaining in popularity.

To force grep to show the filename when searching only one target file, simply give /dev/null as the second file.

bash$ grep Linux osinfo.txt /dev/null osinfo.txt:This is a file containing information about Linux. osinfo.txt:The GPL governs the distribution of the Linux operating system.

If there is a successful match, grep returns an exit status of 0, which makes it useful in a condition test in a script, especially in combination with the -q option to suppress output.
1 SUCCESS=0 # if grep lookup succeeds 2 word=Linux 3 filename=data.file 4 5 grep -q "$word" "$filename" # The "-q" option causes nothing to echo to stdout. 6 7 if [ $? -eq $SUCCESS ] 8 # if grep -q "$word" "$filename" can replace lines 5 - 7. 9 then 10 echo "$word found in $filename" 11 else 12 echo "$word not found in $filename" 13 fi

Example 30-6 demonstrates how to use grep to search for a word pattern in a system logfile.

Example 12-15. Emulating "grep" in a script

   1 #!/bin/bash
   2 # grp.sh: Very crude reimplementation of 'grep'.
   3 
   4 E_BADARGS=65
   5 
   6 if [ -z "$1" ]    # Check for argument to script.
   7 then
   8   echo "Usage: `basename $0` pattern"
   9   exit $E_BADARGS
  10 fi  
  11 
  12 echo
  13 
  14 for file in *     # Traverse all files in $PWD.
  15 do
  16   output=$(sed -n /"$1"/p $file)  # Command substitution.
  17 
  18   if [ ! -z "$output" ]           # What happens if "$output" is not quoted?
  19   then
  20     echo -n "$file: "
  21     echo $output
  22   fi              #  sed -ne "/$1/s|^|${file}: |p"  is equivalent to above.
  23 
  24   echo
  25 done  
  26 
  27 echo
  28 
  29 exit 0
  30 
  31 # Exercises:
  32 # ---------
  33 # 1) Add newlines to output, if more than one match in any given file.
  34 # 2) Add features.

How can grep search for two (or more) separate patterns? What if you want grep to display all lines in a file or files that contain both "pattern1" and "pattern2"?

One method is to pipe the result of grep pattern1 to grep pattern2.

For example, given the following file:

1 # Filename: tstfile 2 3 This is a sample file. 4 This is an ordinary text file. 5 This file does not contain any unusual text. 6 This file is not unusual. 7 Here is some text.

Now, let's search this file for lines containing both "file" and "test" . . .

 bash$ grep file tstfile
 # Filename: tstfile
 This is a sample file.
 This is an ordinary text file.
 This file does not contain any unusual text.
 This file is not unusual.
 
 bash$ grep file tstfile | grep text
 This is an ordinary text file.
 This file does not contain any unusual text.

egrep (extended grep) is the same as grep -E. This uses a somewhat different, extended set of Regular Expressions, which can make the search a bit more flexible.

fgrep (fast grep) is the same as grep -F. It does a literal string search (no regular expressions), which usually speeds things up a bit.

agrep (approximate grep) extends the capabilities of grep to approximate matching. The search string may differ by a specified number of characters from the resulting matches. This utility is not part of the core Linux distribution.

To search compressed files, use zgrep, zegrep, or zfgrep. These also work on non-compressed files, though slower than plain grep, egrep, fgrep. They are handy for searching through a mixed set of files, some compressed, some not.

To search bzipped files, use bzgrep.

look

The command look works like grep, but does a lookup on a "dictionary", a sorted word list. By default, look searches for a match in /usr/dict/words, but a different dictionary file may be specified.

Example 12-16. Checking words in a list for validity

   1 #!/bin/bash
   2 # lookup: Does a dictionary lookup on each word in a data file.
   3 
   4 file=words.data  # Data file from which to read words to test.
   5 
   6 echo
   7 
   8 while [ "$word" != end ]  # Last word in data file.
   9 do
  10   read word      # From data file, because of redirection at end of loop.
  11   look $word > /dev/null  # Don't want to display lines in dictionary file.
  12   lookup=$?      # Exit status of 'look' command.
  13 
  14   if [ "$lookup" -eq 0 ]
  15   then
  16     echo "\"$word\" is valid."
  17   else
  18     echo "\"$word\" is invalid."
  19   fi  
  20 
  21 done <"$file"    # Redirects stdin to $file, so "reads" come from there.
  22 
  23 echo
  24 
  25 exit 0
  26 
  27 # ----------------------------------------------------------------
  28 # Code below line will not execute because of "exit" command above.
  29 
  30 
  31 # Stephane Chazelas proposes the following, more concise alternative:
  32 
  33 while read word && [[ $word != end ]]
  34 do if look "$word" > /dev/null
  35    then echo "\"$word\" is valid."
  36    else echo "\"$word\" is invalid."
  37    fi
  38 done <"$file"
  39 
  40 exit 0

sed, awk

Scripting languages especially suited for parsing text files and command output. May be embedded singly or in combination in pipes and shell scripts.

sed

Non-interactive "stream editor", permits using many ex commands in batch mode. It finds many uses in shell scripts.

awk

Programmable file extractor and formatter, good for manipulating and/or extracting fields (columns) in structured text files. Its syntax is similar to C.

wc

wc gives a "word count" on a file or I/O stream:
bash $ wc /usr/doc/sed-3.02/README 20 127 838 /usr/doc/sed-3.02/README [20 lines 127 words 838 characters]

wc -w gives only the word count.

wc -l gives only the line count.

wc -c gives only the character count.

wc -L gives only the length of the longest line.

Using wc to count how many .txt files are in current working directory:
1 $ ls *.txt | wc -l 2 # Will work as long as none of the "*.txt" files have a linefeed in their name. 3 4 # Alternative ways of doing this are: 5 # find . -maxdepth 1 -name \*.txt -print0 | grep -cz . 6 # (shopt -s nullglob; set -- *.txt; echo $#) 7 8 # Thanks, S.C.

Using wc to total up the size of all the files whose names begin with letters in the range d - h
bash$ wc [d-h]* | grep total | awk '{print $3}' 71832

Using wc to count the instances of the word "Linux" in the main source file for this book.
bash$ grep Linux abs-book.sgml | wc -l 50

See also Example 12-34 and Example 16-8.

Certain commands include some of the functionality of wc as options.
1 ... | grep foo | wc -l 2 # This frequently used construct can be more concisely rendered. 3 4 ... | grep -c foo 5 # Just use the "-c" (or "--count") option of grep. 6 7 # Thanks, S.C.

tr

character translation filter.

Must use quoting and/or brackets, as appropriate. Quotes prevent the shell from reinterpreting the special characters in tr command sequences. Brackets should be quoted to prevent expansion by the shell.

Either tr "A-Z" "*" <filename or tr A-Z \* <filename changes all the uppercase letters in filename to asterisks (writes to stdout). On some systems this may not work, but tr A-Z '[**]' will.

The -d option deletes a range of characters.
1 echo "abcdef" # abcdef 2 echo "abcdef" | tr -d b-d # aef 3 4 5 tr -d 0-9 <filename 6 # Deletes all digits from the file "filename".

The --squeeze-repeats (or -s) option deletes all but the first instance of a string of consecutive characters. This option is useful for removing excess whitespace.
bash$ echo "XXXXX" | tr --squeeze-repeats 'X' X

The -c "complement" option inverts the character set to match. With this option, tr acts only upon those characters not matching the specified set.

bash$ echo "acfdeb123" | tr -c b-d + +c+d+b++++

Note that tr recognizes POSIX character classes. [1]

bash$ echo "abcd2ef1" | tr '[:alpha:]' - ----2--1

Example 12-17. toupper: Transforms a file to all uppercase.

   1 #!/bin/bash
   2 # Changes a file to all uppercase.
   3 
   4 E_BADARGS=65
   5 
   6 if [ -z "$1" ]  # Standard check for command line arg.
   7 then
   8   echo "Usage: `basename $0` filename"
   9   exit $E_BADARGS
  10 fi  
  11 
  12 tr a-z A-Z <"$1"
  13 
  14 # Same effect as above, but using POSIX character set notation:
  15 #        tr '[:lower:]' '[:upper:]' <"$1"
  16 # Thanks, S.C.
  17 
  18 exit 0

Example 12-18. lowercase: Changes all filenames in working directory to lowercase.

   1 #! /bin/bash
   2 #
   3 # Changes every filename in working directory to all lowercase.
   4 #
   5 # Inspired by a script of John Dubois,
   6 # which was translated into into Bash by Chet Ramey,
   7 # and considerably simplified by Mendel Cooper, author of this document.
   8 
   9 
  10 for filename in *                # Traverse all files in directory.
  11 do
  12    fname=`basename $filename`
  13    n=`echo $fname | tr A-Z a-z`  # Change name to lowercase.
  14    if [ "$fname" != "$n" ]       # Rename only files not already lowercase.
  15    then
  16      mv $fname $n
  17    fi  
  18 done   
  19 
  20 exit 0
  21 
  22 
  23 # Code below this line will not execute because of "exit".
  24 #--------------------------------------------------------#
  25 # To run it, delete script above line.
  26 
  27 # The above script will not work on filenames containing blanks or newlines.
  28 
  29 # Stephane Chazelas therefore suggests the following alternative:
  30 
  31 
  32 for filename in *    # Not necessary to use basename,
  33                      # since "*" won't return any file containing "/".
  34 do n=`echo "$filename/" | tr '[:upper:]' '[:lower:]'`
  35 #                             POSIX char set notation.
  36 #                    Slash added so that trailing newlines are not
  37 #                    removed by command substitution.
  38    # Variable substitution:
  39    n=${n%/}          # Removes trailing slash, added above, from filename.
  40    [[ $filename == $n ]] || mv "$filename" "$n"
  41                      # Checks if filename already lowercase.
  42 done
  43 
  44 exit 0

Example 12-19. Du: DOS to UNIX text file conversion.

   1 #!/bin/bash
   2 # Du.sh: DOS to UNIX text file converter.
   3 
   4 E_WRONGARGS=65
   5 
   6 if [ -z "$1" ]
   7 then
   8   echo "Usage: `basename $0` filename-to-convert"
   9   exit $E_WRONGARGS
  10 fi
  11 
  12 NEWFILENAME=$1.unx
  13 
  14 CR='\015'  # Carriage return.
  15            # 015 is octal ASCII code for CR.
  16            # Lines in a DOS text file end in CR-LF.
  17            # Lines in a UNIX text file end in LF only.
  18 
  19 tr -d $CR < $1 > $NEWFILENAME
  20 # Delete CR's and write to new file.
  21 
  22 echo "Original DOS text file is \"$1\"."
  23 echo "Converted UNIX text file is \"$NEWFILENAME\"."
  24 
  25 exit 0
  26 
  27 # Exercise:
  28 # --------
  29 # Change the above script to convert from UNIX to DOS.

Example 12-20. rot13: rot13, ultra-weak encryption.

   1 #!/bin/bash
   2 # rot13.sh: Classic rot13 algorithm,
   3 #           encryption that might fool a 3-year old.
   4 
   5 # Usage: ./rot13.sh filename
   6 # or     ./rot13.sh <filename
   7 # or     ./rot13.sh and supply keyboard input (stdin)
   8 
   9 cat "$@" | tr 'a-zA-Z' 'n-za-mN-ZA-M'   # "a" goes to "n", "b" to "o", etc.
  10 #  The 'cat "$@"' construction
  11 #+ permits getting input either from stdin or from files.
  12 
  13 exit 0

Example 12-21. Generating "Crypto-Quote" Puzzles

   1 #!/bin/bash
   2 # crypto-quote.sh: Encrypt quotes
   3 
   4 #  Will encrypt famous quotes in a simple monoalphabetic substitution.
   5 #  The result is similar to the "Crypto Quote" puzzles
   6 #+ seen in the Op Ed pages of the Sunday paper.
   7 
   8 
   9 key=ETAOINSHRDLUBCFGJMQPVWZYXK
  10 # The "key" is nothing more than a scrambled alphabet.
  11 # Changing the "key" changes the encryption.
  12 
  13 # The 'cat "$@"' construction gets input either from stdin or from files.
  14 # If using stdin, terminate input with a Control-D.
  15 # Otherwise, specify filename as command-line parameter.
  16 
  17 cat "$@" | tr "a-z" "A-Z" | tr "A-Z" "$key"
  18 #        |  to uppercase  |     encrypt       
  19 # Will work on lowercase, uppercase, or mixed-case quotes.
  20 # Passes non-alphabetic characters through unchanged.
  21 
  22 
  23 # Try this script with something like:
  24 # "Nothing so needs reforming as other people's habits."
  25 # --Mark Twain
  26 #
  27 # Output is:
  28 # "CFPHRCS QF CIIOQ MINFMBRCS EQ FPHIM GIFGUI'Q HETRPQ."
  29 # --BEML PZERC
  30 
  31 # To reverse the encryption:
  32 # cat "$@" | tr "$key" "A-Z"
  33 
  34 
  35 #  This simple-minded cipher can be broken by an average 12-year old
  36 #+ using only pencil and paper.
  37 
  38 exit 0
  39 
  40 #  Exercise:
  41 #  --------
  42 #  Modify the script so that it will either encrypt or decrypt,
  43 #+ depending on command-line argument(s).

fold

A filter that wraps lines of input to a specified width. This is especially useful with the -s option, which breaks lines at word spaces (see Example 12-22 and Example A-1).

fmt

Simple-minded file formatter, used as a filter in a pipe to "wrap" long lines of text output.

Example 12-22. Formatted file listing.

   1 #!/bin/bash
   2 
   3 WIDTH=40                    # 40 columns wide.
   4 
   5 b=`ls /usr/local/bin`       # Get a file listing...
   6 
   7 echo $b | fmt -w $WIDTH
   8 
   9 # Could also have been done by
  10 #    echo $b | fold - -s -w $WIDTH
  11  
  12 exit 0

Prev	Home	Next
Time / Date Commands	Up	File and Archiving Commands

12.4. Text Processing Commands

Notes