Chapter 17. Here Documents

A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or command, such as ftp, telnet, or ex.

   1 COMMAND <<InputComesFromHERE
   2 ...
   3 InputComesFromHERE

A "limit string" delineates (frames) the command list. The special symbol << designates the limit string. This has the effect of redirecting the output of a file into the stdin of the program or command. It is similar to interactive-program < command-file, where command-file contains
   1 command #1
   2 command #2
   3 ...

The "here document" alternative looks like this:

   1 #!/bin/bash
   2 interactive-program <<LimitString
   3 command #1
   4 command #2
   5 ...
   6 LimitString

Choose a limit string sufficiently unusual that it will not occur anywhere in the command list and confuse matters.

Note that here documents may sometimes be used to good effect with non-interactive utilities and commands.


Example 17-1. dummyfile: Creates a 2-line dummy file

   1 #!/bin/bash
   2 
   3 # Non-interactive use of 'vi' to edit a file.
   4 # Emulates 'sed'.
   5 
   6 E_BADARGS=65
   7 
   8 if [ -z "$1" ]
   9 then
  10   echo "Usage: `basename $0` filename"
  11   exit $E_BADARGS
  12 fi
  13 
  14 TARGETFILE=$1
  15 
  16 # Insert 2 lines in file, then save.
  17 #--------Begin here document-----------#
  18 vi $TARGETFILE <<x23LimitStringx23
  19 i
  20 This is line 1 of the example file.
  21 This is line 2 of the example file.
  22 ^[
  23 ZZ
  24 x23LimitStringx23
  25 #----------End here document-----------#
  26 
  27 #  Note that ^[ above is a literal escape
  28 #+ typed by Control-V <Esc>.
  29 
  30 #  Bram Moolenaar points out that this may not work with 'vim',
  31 #+ because of possible problems with terminal interaction.
  32 
  33 exit 0

The above script could just as effectively have been implemented with ex, rather than vi. Here documents containing a list of ex commands are common enough to form their own category, known as ex scripts.


Example 17-2. broadcast: Sends message to everyone logged in

   1 #!/bin/bash
   2 
   3 wall <<zzz23EndOfMessagezzz23
   4 E-mail your noontime orders for pizza to the system administrator.
   5     (Add an extra dollar for anchovy or mushroom topping.)
   6 # Additional message text goes here.
   7 # Note: 'wall' prints comment lines.
   8 zzz23EndOfMessagezzz23
   9 
  10 # Could have been done more efficiently by
  11 #         wall <message-file
  12 # However, embedding the message template in a script saves work.
  13 
  14 exit 0


Example 17-3. Multi-line message using cat

   1 #!/bin/bash
   2 
   3 #  'echo' is fine for printing single line messages,
   4 #+  but somewhat problematic for for message blocks.
   5 #   A 'cat' here document overcomes this limitation.
   6 
   7 cat <<End-of-message
   8 -------------------------------------
   9 This is line 1 of the message.
  10 This is line 2 of the message.
  11 This is line 3 of the message.
  12 This is line 4 of the message.
  13 This is the last line of the message.
  14 -------------------------------------
  15 End-of-message
  16 
  17 #  Replacing line 7, above, with
  18 #+   cat > $Newfile <<End-of-message
  19 #+       ^^^^^^^^^^
  20 #+ writes the output to the file $Newfile, rather than to stdout.
  21 
  22 exit 0
  23 
  24 
  25 #--------------------------------------------
  26 # Code below disabled, due to "exit 0" above.
  27 
  28 # S.C. points out that the following also works.
  29 echo "-------------------------------------
  30 This is line 1 of the message.
  31 This is line 2 of the message.
  32 This is line 3 of the message.
  33 This is line 4 of the message.
  34 This is the last line of the message.
  35 -------------------------------------"
  36 # However, text may not include double quotes unless they are escaped.

The - option to mark a here document limit string (<<-LimitString) suppresses leading tabs (but not spaces) in the output. This may be useful in making a script more readable.


Example 17-4. Multi-line message, with tabs suppressed

   1 #!/bin/bash
   2 # Same as previous example, but...
   3 
   4 #  The - option to a here document <<-
   5 #+ suppresses leading tabs in the body of the document,
   6 #+ but *not* spaces.
   7 
   8 cat <<-ENDOFMESSAGE
   9 	This is line 1 of the message.
  10 	This is line 2 of the message.
  11 	This is line 3 of the message.
  12 	This is line 4 of the message.
  13 	This is the last line of the message.
  14 ENDOFMESSAGE
  15 # The output of the script will be flush left.
  16 # Leading tab in each line will not show.
  17 
  18 # Above 5 lines of "message" prefaced by a tab, not spaces.
  19 # Spaces not affected by   <<-  .
  20 
  21 # Note that this option has no effect on *embedded* tabs.
  22 
  23 exit 0

A here document supports parameter and command substitution. It is therefore possible to pass different parameters to the body of the here document, changing its output accordingly.


Example 17-5. Here document with parameter substitution

   1 #!/bin/bash
   2 # Another 'cat' here document, using parameter substitution.
   3 
   4 # Try it with no command line parameters,   ./scriptname
   5 # Try it with one command line parameter,   ./scriptname Mortimer
   6 # Try it with one two-word quoted command line parameter,
   7 #                           ./scriptname "Mortimer Jones"
   8 
   9 CMDLINEPARAM=1     # Expect at least command line parameter.
  10 
  11 if [ $# -ge $CMDLINEPARAM ]
  12 then
  13   NAME=$1          # If more than one command line param,
  14                    # then just take the first.
  15 else
  16   NAME="John Doe"  # Default, if no command line parameter.
  17 fi  
  18 
  19 RESPONDENT="the author of this fine script"  
  20   
  21 
  22 cat <<Endofmessage
  23 
  24 Hello, there, $NAME.
  25 Greetings to you, $NAME, from $RESPONDENT.
  26 
  27 # This comment shows up in the output (why?).
  28 
  29 Endofmessage
  30 
  31 # Note that the blank lines show up in the output.
  32 # So does the "comment".
  33 
  34 exit 0

This is a useful script containing a here document with parameter substitution.


Example 17-6. Upload a file pair to "Sunsite" incoming directory

   1 #!/bin/bash
   2 # upload.sh
   3 
   4 #  Upload file pair (Filename.lsm, Filename.tar.gz)
   5 #+ to incoming directory at Sunsite/UNC (ibiblio.org).
   6 #  Filename.tar.gz is the tarball itself.
   7 #  Filename.lsm is the descriptor file.
   8 #  Sunsite requires "lsm" file, otherwise will bounce contributions.
   9 
  10 
  11 E_ARGERROR=65
  12 
  13 if [ -z "$1" ]
  14 then
  15   echo "Usage: `basename $0` Filename-to-upload"
  16   exit $E_ARGERROR
  17 fi  
  18 
  19 
  20 Filename=`basename $1`           # Strips pathname out of file name.
  21 
  22 Server="ibiblio.org"
  23 Directory="/incoming/Linux"
  24 #  These need not be hard-coded into script,
  25 #+ but may instead be changed to command line argument.
  26 
  27 Password="your.e-mail.address"   # Change above to suit.
  28 
  29 ftp -n $Server <<End-Of-Session
  30 # -n option disables auto-logon
  31 
  32 user anonymous "$Password"
  33 binary
  34 bell                             # Ring 'bell' after each file transfer.
  35 cd $Directory
  36 put "$Filename.lsm"
  37 put "$Filename.tar.gz"
  38 bye
  39 End-Of-Session
  40 
  41 exit 0

Quoting or escaping the "limit string" at the head of a here document disables parameter substitution within its body.


Example 17-7. Parameter substitution turned off

   1 #!/bin/bash
   2 #  A 'cat' here document, but with parameter substitution disabled.
   3 
   4 NAME="John Doe"
   5 RESPONDENT="the author of this fine script"  
   6 
   7 cat <<'Endofmessage'
   8 
   9 Hello, there, $NAME.
  10 Greetings to you, $NAME, from $RESPONDENT.
  11 
  12 Endofmessage
  13 
  14 #  No parameter substitution when the "limit string" is quoted or escaped.
  15 #  Either of the following at the head of the here document would have the same effect.
  16 #  cat <<"Endofmessage"
  17 #  cat <<\Endofmessage
  18 
  19 exit 0

Disabling parameter substitution permits outputting literal text. Generating scripts or even program code is one use for this.


Example 17-8. A script that generates another script

   1 #!/bin/bash
   2 # generate-script.sh
   3 # Based on an idea by Albert Reiner.
   4 
   5 OUTFILE=generated.sh         # Name of the file to generate.
   6 
   7 
   8 # -----------------------------------------------------------
   9 # 'Here document containing the body of the generated script.
  10 (
  11 cat <<'EOF'
  12 #!/bin/bash
  13 
  14 echo "This is a generated shell script."
  15 #  Note that since we are inside a subshell,
  16 #+ we can't access variables in the "outside" script.
  17 
  18 echo "Generated file will be named: $OUTFILE"
  19 #  Above line will not work as normally expected
  20 #+ because parameter expansion has been disabled.
  21 #  Instead, the result is literal output.
  22 
  23 a=7
  24 b=3
  25 
  26 let "c = $a * $b"
  27 echo "c = $c"
  28 
  29 exit 0
  30 EOF
  31 ) > $OUTFILE
  32 # -----------------------------------------------------------
  33 
  34 #  Quoting the 'limit string' prevents variable expansion
  35 #+ within the body of the above 'here document.'
  36 #  This permits outputting literal strings in the output file.
  37 
  38 if [ -f "$OUTFILE" ]
  39 then
  40   chmod 755 $OUTFILE
  41   # Make the generated file executable.
  42 else
  43   echo "Problem in creating file: \"$OUTFILE\""
  44 fi
  45 
  46 #  This method can also be used for generating
  47 #+ C programs, Perl programs, Python programs, Makefiles,
  48 #+ and the like.
  49 
  50 exit 0

It is possible to set a variable from the output of a here document.
   1 variable=$(cat <<SETVAR
   2 This variable
   3 runs over multiple lines.
   4 SETVAR)
   5 
   6 echo "$variable"

A here document can supply input to a function in the same script.


Example 17-9. Here documents and functions

   1 #!/bin/bash
   2 # here-function.sh
   3 
   4 GetPersonalData ()
   5 {
   6   read firstname
   7   read lastname
   8   read address
   9   read city 
  10   read state 
  11   read zipcode
  12 } # This certainly looks like an interactive function, but...
  13 
  14 
  15 # Supply input to the above function.
  16 GetPersonalData <<RECORD001
  17 Bozo
  18 Bozeman
  19 2726 Nondescript Dr.
  20 Baltimore
  21 MD
  22 21226
  23 RECORD001
  24 
  25 
  26 echo
  27 echo "$firstname $lastname"
  28 echo "$address"
  29 echo "$city, $state $zipcode"
  30 echo
  31 
  32 exit 0

It is possible to use : as a dummy command accepting output from a here document. This, in effect, creates an "anonymous" here document.


Example 17-10. "Anonymous" Here Document

   1 #!/bin/bash
   2 
   3 : <<TESTVARIABLES
   4 ${HOSTNAME?}${USER?}${MAIL?}  # Print error message if one of the variables not set.
   5 TESTVARIABLES
   6 
   7 exit 0

Tip

A variation of the above technique permits "commenting out" blocks of code.


Example 17-11. Commenting out a block of code

   1 #!/bin/bash
   2 # commentblock.sh
   3 
   4 : <<COMMENTBLOCK
   5 echo "This line will not echo."
   6 This is a comment line missing the "#" prefix.
   7 This is another comment line missing the "#" prefix.
   8 
   9 &*@!!++=
  10 The above line will cause no error message,
  11 because the Bash interpreter will ignore it.
  12 COMMENTBLOCK
  13 
  14 echo "Exit value of above \"COMMENTBLOCK\" is $?."   # 0
  15 # No error shown.
  16 
  17 
  18 #  The above technique also comes in useful for commenting out
  19 #+ a block of working code for debugging purposes.
  20 #  This saves having to put a "#" at the beginning of each line,
  21 #+ then having to go back and delete each "#" later.
  22 
  23 : <<DEBUGXXX
  24 for file in *
  25 do
  26  cat "$file"
  27 done
  28 DEBUGXXX
  29 
  30 exit 0

Tip

Yet another twist of this nifty trick makes "self-documenting" scripts possible.


Example 17-12. A self-documenting script

   1 #!/bin/bash
   2 # self-document.sh: self-documenting script
   3 # Modification of "colm.sh".
   4 
   5 DOC_REQUEST=70
   6 
   7 if [ "$1" = "-h"  -o "$1" = "--help" ]     # Request help.
   8 then
   9   echo; echo "Usage: $0 [directory-name]"; echo
  10   sed --silent -e '/DOCUMENTATIONXX$/,/^DOCUMENTATIONXX$/p' "$0" |
  11   sed -e '/DOCUMENTATIONXX$/d'; exit $DOC_REQUEST; fi
  12 
  13 
  14 : <<DOCUMENTATIONXX
  15 List the statistics of a specified directory in tabular format.
  16 ---------------------------------------------------------------
  17 The command line parameter gives the directory to be listed.
  18 If no directory specified or directory specified cannot be read,
  19 then list the current working directory.
  20 
  21 DOCUMENTATIONXX
  22 
  23 if [ -z "$1" -o ! -r "$1" ]
  24 then
  25   directory=.
  26 else
  27   directory="$1"
  28 fi  
  29 
  30 echo "Listing of "$directory":"; echo
  31 (printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \
  32 ; ls -l "$directory" | sed 1d) | column -t
  33 
  34 exit 0

See also Example A-25 for an excellent example of a self-documenting script.

Note

Here documents create temporary files, but these files are deleted after opening and are not accessible to any other process.

 bash$ bash -c 'lsof -a -p $$ -d0' << EOF
 > EOF
 lsof    1213 bozo    0r   REG    3,5    0 30386 /tmp/t1213-0-sh (deleted)
 	      

Caution

Some utilities will not work inside a here document.

Warning

The closing limit string, on the final line of a here document, must start in the first character position. There can be no leading whitespace. Trailing whitespace after the limit string likewise causes unexpected behavior. The whitespace prevents the limit string from being recognized.

   1 #!/bin/bash
   2 
   3 echo "----------------------------------------------------------------------"
   4 
   5 cat <<LimitString
   6 echo "This is line 1 of the message inside the here document."
   7 echo "This is line 2 of the message inside the here document."
   8 echo "This is the final line of the message inside the here document."
   9      LimitString
  10 #^^^^Indented limit string. Error! This script will not behave as expected.
  11 
  12 echo "----------------------------------------------------------------------"
  13 
  14 #  These comments are outside the 'here document',
  15 #+ and should not echo.
  16 
  17 echo "Outside the here document."
  18 
  19 exit 0
  20 
  21 echo "This line had better not echo."  # Follows an 'exit' command.

For those tasks too complex for a "here document", consider using the expect scripting language, which is specifically tailored for feeding input into interactive programs.

17.1. Here Strings

A here string can be considered as a stripped-down form of here document. It consists of nothing more than COMMAND <<<$WORD, where $WORD is expanded and fed to the stdin of COMMAND.


Example 17-13. Prepending a line to a file

   1 #!/bin/bash
   2 # prepend.sh: Add text at beginning of file.
   3 #
   4 #  Example contributed by Kenny Stauffer,
   5 #  and slightly modified by document author.
   6 
   7 
   8 E_NOSUCHFILE=65
   9 
  10 read -p "File: " file   # -p arg to 'read' displays prompt.
  11 if [ ! -e "$file" ]
  12 then   # Bail out if no such file.
  13   echo "File $file not found."
  14   exit $E_NOSUCHFILE
  15 fi
  16 
  17 read -p "Title: " title
  18 cat - $file <<<$title > $file.new
  19 
  20 echo "Modified file is $file.new"
  21 
  22 exit 0
  23 
  24 # from "man bash"
  25 # Here Strings
  26 # 	A variant of here documents, the format is:
  27 # 
  28 # 		<<<word
  29 # 
  30 # 	The word is expanded and supplied to the command on its standard input.

Exercise: Find other uses for here strings.