9.2. Manipulating Strings

Bash supports a surprising number of string manipulation operations. Unfortunately, these tools lack a unified focus. Some are a subset of parameter substitution, and others fall under the functionality of the UNIX expr command. This results in inconsistent command syntax and overlap of functionality, not to mention confusion.

String Length

${#string}

expr length $string

expr "$string" : '.*'

   1 stringZ=abcABC123ABCabc
   2 
   3 echo ${#stringZ}                 # 15
   4 echo `expr length $stringZ`      # 15
   5 echo `expr "$stringZ" : '.*'`    # 15


Example 9-10. Inserting a blank line between paragraphs in a text file

   1 #!/bin/bash
   2 # paragraph-space.sh
   3 
   4 # Inserts a blank line between paragraphs of a single-spaced text file.
   5 # Usage: $0 <FILENAME
   6 
   7 MINLEN=45        # May need to change this value.
   8 #  Assume lines shorter than $MINLEN characters
   9 #+ terminate a paragraph.
  10 
  11 while read line  # For as many lines as the input file has...
  12 do
  13   echo "$line"   # Output the line itself.
  14 
  15   len=${#line}
  16   if [ "$len" -lt "$MINLEN" ]
  17     then echo    # Add a blank line after short line.
  18   fi  
  19 done
  20 
  21 exit 0

Length of Matching Substring at Beginning of String

expr match "$string" '$substring'

$substring is a regular expression.

expr "$string" : '$substring'

$substring is a regular expression.

   1 stringZ=abcABC123ABCabc
   2 #       |------|
   3 
   4 echo `expr match "$stringZ" 'abc[A-Z]*.2'`   # 8
   5 echo `expr "$stringZ" : 'abc[A-Z]*.2'`       # 8

Index

expr index $string $substring

Numerical position in $string of first character in $substring that matches.

   1 stringZ=abcABC123ABCabc
   2 echo `expr index "$stringZ" C12`             # 6
   3                                              # C position.
   4 
   5 echo `expr index "$stringZ" 1c`              # 3
   6 # 'c' (in #3 position) matches before '1'.

This is the near equivalent of strchr() in C.

Substring Extraction

${string:position}

Extracts substring from $string at $position.

If the $string parameter is "*" or "@", then this extracts the positional parameters, [1] starting at $position.

${string:position:length}

Extracts $length characters of substring from $string at $position.

   1 stringZ=abcABC123ABCabc
   2 #       0123456789.....
   3 #       0-based indexing.
   4 
   5 echo ${stringZ:0}                            # abcABC123ABCabc
   6 echo ${stringZ:1}                            # bcABC123ABCabc
   7 echo ${stringZ:7}                            # 23ABCabc
   8 
   9 echo ${stringZ:7:3}                          # 23A
  10                                              # Three characters of substring.
  11 
  12 
  13 
  14 # Is it possible to index from the right end of the string?
  15     
  16 echo ${stringZ:-4}                           # abcABC123ABCabc
  17 # Defaults to full string, as in ${parameter:-default}.
  18 # However . . .
  19 
  20 echo ${stringZ:(-4)}                         # Cabc 
  21 echo ${stringZ: -4}                          # Cabc
  22 # Now, it works.
  23 # Parentheses or added space "escape" the position parameter.
  24 
  25 # Thank you, Dan Jacobson, for pointing this out.

If the $string parameter is "*" or "@", then this extracts a maximum of $length positional parameters, starting at $position.

   1 echo ${*:2}          # Echoes second and following positional parameters.
   2 echo ${@:2}          # Same as above.
   3 
   4 echo ${*:2:3}        # Echoes three positional parameters, starting at second.

expr substr $string $position $length

Extracts $length characters from $string starting at $position.

   1 stringZ=abcABC123ABCabc
   2 #       123456789......
   3 #       1-based indexing.
   4 
   5 echo `expr substr $stringZ 1 2`              # ab
   6 echo `expr substr $stringZ 4 3`              # ABC

expr match "$string" '\($substring\)'

Extracts $substring at beginning of $string, where $substring is a regular expression.

expr "$string" : '\($substring\)'

Extracts $substring at beginning of $string, where $substring is a regular expression.

   1 stringZ=abcABC123ABCabc
   2 #       =======	    
   3 
   4 echo `expr match "$stringZ" '\(.[b-c]*[A-Z]..[0-9]\)'`   # abcABC1
   5 echo `expr "$stringZ" : '\(.[b-c]*[A-Z]..[0-9]\)'`       # abcABC1
   6 echo `expr "$stringZ" : '\(.......\)'`                   # abcABC1
   7 # All of the above forms give an identical result.

expr match "$string" '.*\($substring\)'

Extracts $substring at end of $string, where $substring is a regular expression.

expr "$string" : '.*\($substring\)'

Extracts $substring at end of $string, where $substring is a regular expression.

   1 stringZ=abcABC123ABCabc
   2 #                ======
   3 
   4 echo `expr match "$stringZ" '.*\([A-C][A-C][A-C][a-c]*\)'`    # ABCabc
   5 echo `expr "$stringZ" : '.*\(......\)'`                       # ABCabc

Substring Removal

${string#substring}

Strips shortest match of $substring from front of $string.

${string##substring}

Strips longest match of $substring from front of $string.

   1 stringZ=abcABC123ABCabc
   2 #       |----|
   3 #       |----------|
   4 
   5 echo ${stringZ#a*C}      # 123ABCabc
   6 # Strip out shortest match between 'a' and 'C'.
   7 
   8 echo ${stringZ##a*C}     # abc
   9 # Strip out longest match between 'a' and 'C'.

${string%substring}

Strips shortest match of $substring from back of $string.

${string%%substring}

Strips longest match of $substring from back of $string.

   1 stringZ=abcABC123ABCabc
   2 #                    ||
   3 #        |------------|
   4 
   5 echo ${stringZ%b*c}      # abcABC123ABCa
   6 # Strip out shortest match between 'b' and 'c', from back of $stringZ.
   7 
   8 echo ${stringZ%%b*c}     # a
   9 # Strip out longest match between 'b' and 'c', from back of $stringZ.


Example 9-11. Converting graphic file formats, with filename change

   1 #!/bin/bash
   2 #  cvt.sh:
   3 #  Converts all the MacPaint image files in a directory to "pbm" format.
   4 
   5 #  Uses the "macptopbm" binary from the "netpbm" package,
   6 #+ which is maintained by Brian Henderson (bryanh@giraffe-data.com).
   7 #  Netpbm is a standard part of most Linux distros.
   8 
   9 OPERATION=macptopbm
  10 SUFFIX=pbm          # New filename suffix. 
  11 
  12 if [ -n "$1" ]
  13 then
  14   directory=$1      # If directory name given as a script argument...
  15 else
  16   directory=$PWD    # Otherwise use current working directory.
  17 fi  
  18   
  19 #  Assumes all files in the target directory are MacPaint image files,
  20 # + with a ".mac" suffix.
  21 
  22 for file in $directory/*    # Filename globbing.
  23 do
  24   filename=${file%.*c}      #  Strip ".mac" suffix off filename
  25                             #+ ('.*c' matches everything
  26 			    #+ between '.' and 'c', inclusive).
  27   $OPERATION $file > "$filename.$SUFFIX"
  28                             # Redirect conversion to new filename.
  29   rm -f $file               # Delete original files after converting.   
  30   echo "$filename.$SUFFIX"  # Log what is happening to stdout.
  31 done
  32 
  33 exit 0
  34 
  35 # Exercise:
  36 # --------
  37 #  As it stands, this script converts *all* the files in the current
  38 #+ working directory.
  39 #  Modify it to work *only* on files with a ".mac" suffix.

Substring Replacement

${string/substring/replacement}

Replace first match of $substring with $replacement.

${string//substring/replacement}

Replace all matches of $substring with $replacement.

   1 stringZ=abcABC123ABCabc
   2 
   3 echo ${stringZ/abc/xyz}           # xyzABC123ABCabc
   4                                   # Replaces first match of 'abc' with 'xyz'.
   5 
   6 echo ${stringZ//abc/xyz}          # xyzABC123ABCxyz
   7                                   # Replaces all matches of 'abc' with # 'xyz'.

${string/#substring/replacement}

If $substring matches front end of $string, substitute $replacement for $substring.

${string/%substring/replacement}

If $substring matches back end of $string, substitute $replacement for $substring.

   1 stringZ=abcABC123ABCabc
   2 
   3 echo ${stringZ/#abc/XYZ}          # XYZABC123ABCabc
   4                                   # Replaces front-end match of 'abc' with 'XYZ'.
   5 
   6 echo ${stringZ/%abc/XYZ}          # abcABC123ABCXYZ
   7                                   # Replaces back-end match of 'abc' with 'XYZ'.

9.2.1. Manipulating strings using awk

A Bash script may invoke the string manipulation facilities of awk as an alternative to using its built-in operations.


Example 9-12. Alternate ways of extracting substrings

   1 #!/bin/bash
   2 # substring-extraction.sh
   3 
   4 String=23skidoo1
   5 #      012345678    Bash
   6 #      123456789    awk
   7 # Note different string indexing system:
   8 # Bash numbers first character of string as '0'.
   9 # Awk  numbers first character of string as '1'.
  10 
  11 echo ${String:2:4} # position 3 (0-1-2), 4 characters long
  12                                          # skid
  13 
  14 # The awk equivalent of ${string:pos:length} is substr(string,pos,length).
  15 echo | awk '
  16 { print substr("'"${String}"'",3,4)      # skid
  17 }
  18 '
  19 #  Piping an empty "echo" to awk gives it dummy input,
  20 #+ and thus makes it unnecessary to supply a filename.
  21 
  22 exit 0

9.2.2. Further Discussion

For more on string manipulation in scripts, refer to Section 9.3 and the relevant section of the expr command listing. For script examples, see:

  1. Example 12-9

  2. Example 9-15

  3. Example 9-16

  4. Example 9-17

  5. Example 9-19

Notes

[1]

This applies to either command line arguments or parameters passed to a function.