|
|
Advanced Bash-Scripting GuideAdvanced Bash-Scripting GuideAn in-depth exploration of the art of shell scriptingMendel Cooper4.1
08 October 2006
| Revision History |
|---|
| Revision 3.9 | 15 May 2006 | Revised by: mc | | 'SPICEBERRY' release: Minor Update. | | Revision 4.0 | 18 Jun 2006 | Revised by: mc | | 'WINTERBERRY' release: Major Update. | | Revision 4.1 | 08 Oct 2006 | Revised by: mc | | 'WAXBERRY' release: Minor Update. |
This tutorial assumes no previous knowledge of
scripting or programming, but progresses rapidly toward an
intermediate/advanced level of instruction . . . all
the while sneaking in little snippets of UNIX® wisdom and lore. It
serves as a textbook, a manual for self-study, and a reference and
source of knowledge on shell scripting techniques. The exercises
and heavily-commented examples invite active reader participation,
under the premise that the only way to really learn
scripting is to write scripts. This book is suitable for classroom use as a general
introduction to programming concepts.
The latest update of this document, as an archived, bzip2-ed "tarball"
including both the SGML source and rendered HTML, may
be downloaded from the author's home site. A pdf
version is also available. See the change
log for a revision history.
DedicationFor Anita, the source of all the magic - List of Examples
- 2-1. cleanup: A script to clean up the log
files in /var/log
- 2-2. cleanup: An improved clean-up
script
- 2-3. cleanup: An enhanced
and generalized version of above scripts.
- 3-1. Code blocks and I/O redirection
- 3-2. Saving the results of a code block to a file
- 3-3. Running a loop in the background
- 3-4. Backup of all files changed in last day
- 4-1. Variable assignment and substitution
- 4-2. Plain Variable Assignment
- 4-3. Variable Assignment, plain and fancy
- 4-4. Integer or string?
- 4-5. Positional Parameters
- 4-6. wh, whois domain name lookup
- 4-7. Using shift
- 5-1. Echoing Weird Variables
- 5-2. Escaped Characters
- 6-1. exit / exit status
- 6-2. Negating a condition using !
- 7-1. What is truth?
- 7-2. Equivalence of test,
/usr/bin/test, [ ],
and /usr/bin/[
- 7-3. Arithmetic Tests using (( ))
- 7-4. Testing for broken links
- 7-5. Arithmetic and string comparisons
- 7-6. Testing whether a string is null
- 7-7. zmore
- 8-1. Greatest common divisor
- 8-2. Using Arithmetic Operations
- 8-3. Compound Condition Tests Using && and ||
- 8-4. Representation of numerical constants
- 9-1. $IFS and whitespace
- 9-2. Timed Input
- 9-3. Once more, timed input
- 9-4. Timed read
- 9-5. Am I root?
- 9-6. arglist: Listing arguments with $* and $@
- 9-7. Inconsistent $* and $@ behavior
- 9-8. $* and $@ when
$IFS is empty
- 9-9. Underscore variable
- 9-10. Inserting a blank line between paragraphs in a text file
- 9-11. Converting graphic file formats, with filename change
- 9-12. Converting streaming audio files to
ogg
- 9-13. Emulating getopt
- 9-14. Alternate ways of extracting substrings
- 9-15. Using parameter substitution and error messages
- 9-16. Parameter substitution and "usage" messages
- 9-17. Length of a variable
- 9-18. Pattern matching in parameter substitution
- 9-19. Renaming file extensions:
- 9-20. Using pattern matching to parse arbitrary strings
- 9-21. Matching patterns at prefix or suffix of string
- 9-22. Using declare to type variables
- 9-23. Indirect References
- 9-24. Passing an indirect reference to awk
- 9-25. Generating random numbers
- 9-26. Picking a random card from a deck
- 9-27. Random between values
- 9-28. Rolling a single die with RANDOM
- 9-29. Reseeding RANDOM
- 9-30. Pseudorandom numbers, using awk
- 9-31. C-type manipulation of variables
- 10-1. Simple for loops
- 10-2. for loop with two parameters in each
[list] element
- 10-3. Fileinfo: operating on a file list
contained in a variable
- 10-4. Operating on files with a for loop
- 10-5. Missing in [list] in a
for loop
- 10-6. Generating the [list] in a for
loop with command substitution
- 10-7. A grep replacement
for binary files
- 10-8. Listing all users on the system
- 10-9. Checking all the binaries in a directory for
authorship
- 10-10. Listing the symbolic
links in a directory
- 10-11. Symbolic links in a directory, saved to a file
- 10-12. A C-like for loop
- 10-13. Using efax in batch mode
- 10-14. Simple while loop
- 10-15. Another while loop
- 10-16. while loop with multiple conditions
- 10-17. C-like syntax in a while loop
- 10-18. until loop
- 10-19. Nested Loop
- 10-20. Effects of break and
continue in a loop
- 10-21. Breaking out of multiple loop levels
- 10-22. Continuing at a higher loop level
- 10-23. Using "continue N" in an actual task
- 10-24. Using case
- 10-25. Creating menus using case
- 10-26. Using command substitution to generate the
case variable
- 10-27. Simple string matching
- 10-28. Checking for alphabetic input
- 10-29. Creating menus using select
- 10-30. Creating menus using select in a function
- 11-1. Stupid script tricks
- 11-2. Generating a variable from a loop
- 11-3. Finding anagrams
- 14-1. A script that forks off multiple instances of itself
- 14-2. printf in action
- 14-3. Variable assignment, using read
- 14-4. What happens when read has no
variable
- 14-5. Multi-line input to read
- 14-6. Detecting the arrow keys
- 14-7. Using read with
file redirection
- 14-8. Problems reading from a pipe
- 14-9. Changing the current working directory
- 14-10. Letting "let" do arithmetic.
- 14-11. Showing the effect of eval
- 14-12. Forcing a log-off
- 14-13. A version of "rot13"
- 14-14. Using eval to force variable
substitution in a Perl script
- 14-15. Using set with positional
parameters
- 14-16. Reversing the positional parameters
- 14-17. Reassigning the positional parameters
- 14-18. "Unsetting" a variable
- 14-19. Using export to pass a variable to an
embedded awk script
- 14-20. Using getopts to read the
options/arguments passed to a script
- 14-21. "Including" a data file
- 14-22. A (useless) script that sources itself
- 14-23. Effects of exec
- 14-24. A script that exec's itself
- 14-25. Waiting for a process to finish before proceeding
- 14-26. A script that kills itself
- 15-1. Using ls to create a table of contents
for burning a CDR disk
- 15-2. Hello or Good-bye
- 15-3. Badname, eliminate file names
in current directory containing bad characters and whitespace.
- 15-4. Deleting a file by its inode
number
- 15-5. Logfile: Using xargs to monitor system log
- 15-6. Copying files in current directory to another
- 15-7. Killing processes by name
- 15-8. Word frequency analysis
using xargs
- 15-9. Using expr
- 15-10. Using date
- 15-11. Word Frequency Analysis
- 15-12. Which files are scripts?
- 15-13. Generating 10-digit random numbers
- 15-14. Using tail to monitor the system log
- 15-15. Emulating "grep" in a script
- 15-16. Looking up definitions in Webster's 1913 Dictionary
- 15-17. Checking words in a list for validity
- 15-18. toupper: Transforms a file to all uppercase.
- 15-19. lowercase: Changes all filenames in working directory to lowercase.
- 15-20. Du: DOS to UNIX text file conversion.
- 15-21. rot13: rot13, ultra-weak encryption.
- 15-22. Generating "Crypto-Quote" Puzzles
- 15-23. Formatted file listing.
- 15-24. Using column to format a directory
listing
- 15-25. nl: A self-numbering script.
- 15-26. manview: Viewing formatted manpages
- 15-27. Using cpio to move a directory tree
- 15-28. Unpacking an rpm archive
- 15-29. Stripping comments from C program files
- 15-30. Exploring /usr/X11R6/bin
- 15-31. An "improved"
strings command
- 15-32. Using cmp to compare two files
within a script.
- 15-33. basename and dirname
- 15-34. Checking file integrity
- 15-35. Uudecoding encoded files
- 15-36. Finding out where to report a spammer
- 15-37. Analyzing a spam domain
- 15-38. Getting a stock quote
- 15-39. Updating FC4
- 15-40. Using ssh
- 15-41. A script that mails itself
- 15-42. Monthly Payment on a Mortgage
- 15-43. Base Conversion
- 15-44. Invoking bc using a "here
document"
- 15-45. Calculating PI
- 15-46. Converting a decimal number to hexadecimal
- 15-47. Factoring
- 15-48. Calculating the hypotenuse of a triangle
- 15-49. Using seq to generate loop arguments
- 15-50. Letter Count"
- 15-51. Using getopt to parse command-line
options
- 15-52. A script that copies itself
- 15-53. Exercising dd
- 15-54. Capturing Keystrokes
- 15-55. Securely deleting a file
- 15-56. Filename generator
- 15-57. Converting meters to miles
- 15-58. Using m4
- 16-1. Setting a new password
- 16-2. Setting an erase character
- 16-3. secret password:
Turning off terminal echoing
- 16-4. Keypress detection
- 16-5. Checking a remote server for
identd
- 16-6. pidof helps kill a process
- 16-7. Checking a CD image
- 16-8. Creating a filesystem in a file
- 16-9. Adding a new hard drive
- 16-10. Using umask to hide an output file
from prying eyes
- 16-11. killall, from /etc/rc.d/init.d
- 18-1. broadcast: Sends message to everyone logged in
- 18-2. dummyfile: Creates a 2-line dummy file
- 18-3. Multi-line message using cat
- 18-4. Multi-line message, with tabs suppressed
- 18-5. Here document with parameter substitution
- 18-6. Upload a file pair to "Sunsite" incoming
directory
- 18-7. Parameter substitution turned off
- 18-8. A script that generates another script
- 18-9. Here documents and functions
- 18-10. "Anonymous" Here Document
- 18-11. Commenting out a block of code
- 18-12. A self-documenting script
- 18-13. Prepending a line to a file
- 18-14. Parsing a mailbox
- 19-1. Redirecting stdin using
exec
- 19-2. Redirecting stdout using
exec
- 19-3. Redirecting both stdin and
stdout in the same script with
exec
- 19-4. Avoiding a subshell
- 19-5. Redirected while loop
- 19-6. Alternate form of redirected while loop
- 19-7. Redirected until loop
- 19-8. Redirected for loop
- 19-9. Redirected for loop (both
stdin and stdout
redirected)
- 19-10. Redirected if/then test
- 19-11. Data file "names.data" for above examples
- 19-12. Logging events
- 20-1. Variable scope in a subshell
- 20-2. List User Profiles
- 20-3. Running parallel processes in subshells
- 21-1. Running a script in restricted mode
- 23-1. Simple functions
- 23-2. Function Taking Parameters
- 23-3. Functions and command-line args passed to the script
- 23-4. Passing an indirect reference to a function
- 23-5. Dereferencing a parameter passed to a function
- 23-6. Again, dereferencing a parameter passed to a function
- 23-7. Maximum of two numbers
- 23-8. Converting numbers to Roman numerals
- 23-9. Testing large return values in a function
- 23-10. Comparing two large integers
- 23-11. Real name from username
- 23-12. Local variable visibility
- 23-13. Recursion, using a local variable
- 23-14. The Towers of Hanoi
- 24-1. Aliases within a script
- 24-2. unalias: Setting and unsetting an alias
- 25-1. Using an "and list" to test for command-line arguments
- 25-2. Another command-line arg test using an "and list"
- 25-3. Using "or lists" in combination with an "and list"
- 26-1. Simple array usage
- 26-2. Formatting a poem
- 26-3. Various array operations
- 26-4. String operations on arrays
- 26-5. Loading the contents of a script into an array
- 26-6. Some special properties of arrays
- 26-7. Of empty arrays and empty elements
- 26-8. Initializing arrays
- 26-9. Copying and concatenating arrays
- 26-10. More on concatenating arrays
- 26-11. An old friend:
The Bubble Sort
- 26-12. Embedded arrays and indirect references
- 26-13. Complex array application:
Sieve of Eratosthenes
- 26-14. Emulating a push-down stack
- 26-15. Complex array application:
Exploring a weird mathematical series
- 26-16. Simulating a two-dimensional array, then tilting it
- 27-1. Using /dev/tcp for troubleshooting
- 27-2. Finding the process associated with a PID
- 27-3. On-line connect status
- 28-1. Hiding the cookie jar
- 28-2. Setting up a swapfile using /dev/zero
- 28-3. Creating a ramdisk
- 29-1. A buggy script
- 29-2. Missing keyword
- 29-3. test24, another buggy script
- 29-4. Testing a condition with an "assert"
- 29-5. Trapping at exit
- 29-6. Cleaning up after Control-C
- 29-7. Tracing a variable
- 29-8. Running multiple processes (on an SMP box)
- 31-1. Numerical and string comparison are not equivalent
- 31-2. Subshell Pitfalls
- 31-3. Piping the output of echo to a read
- 33-1. shell wrapper
- 33-2. A slightly more complex shell wrapper
- 33-3. A generic shell wrapper that writes to a logfile
- 33-4. A shell wrapper around an awk script
- 33-5. A shell wrapper around another awk script
- 33-6. Perl embedded in a Bash script
- 33-7. Bash and Perl scripts combined
- 33-8. A (useless) script that recursively calls itself
- 33-9. A (useful) script that recursively calls itself
- 33-10. Another (useful) script that recursively calls itself
- 33-11. A "colorized" address database
- 33-12. Drawing a box
- 33-13. Echoing colored text
- 33-14. A "horserace" game
- 33-15. Return value trickery
- 33-16. Even more return value trickery
- 33-17. Passing and returning arrays
- 33-18. Fun with anagrams
- 33-19. Widgets invoked from a shell script
- 34-1. String expansion
- 34-2. Indirect variable references - the new way
- 34-3. Simple database application, using indirect variable
referencing
- 34-4. Using arrays and other miscellaneous trickery
to deal four random hands from a deck of cards
- A-1. mailformat: Formatting an e-mail message
- A-2. rn: A simple-minded file rename utility
- A-3. blank-rename: renames filenames containing
blanks
- A-4. encryptedpw: Uploading to an ftp site,
using a locally encrypted password
- A-5. copy-cd: Copying a data CD
- A-6. Collatz series
- A-7. days-between: Calculate number of days
between two dates
- A-8. Make a "dictionary"
- A-9. Soundex conversion
- A-10. "Game of Life"
- A-11. Data file for "Game of Life"
- A-12. behead: Removing mail and news message headers
- A-13. ftpget: Downloading files via ftp
- A-14. password: Generating random
8-character passwords
- A-15. fifo: Making daily backups, using named pipes
- A-16. Generating prime numbers using the modulo operator
- A-17. tree: Displaying a directory tree
- A-18. string functions: C-like string functions
- A-19. Directory information
- A-20. Object-oriented database
- A-21. Library of hash functions
- A-22. Colorizing text using hash functions
- A-23. More on hash functions
- A-24. Mounting USB keychain storage devices
- A-25. Preserving weblogs
- A-26. Protecting literal strings
- A-27. Unprotecting literal strings
- A-28. Spammer Identification
- A-29. Spammer Hunt
- A-30. Making wget easier to use
- A-31. A "podcasting" script
- A-32. Nightly backup to a firewire HD
- A-33. An expanded cd command
- A-34. Basics Reviewed
- C-1. Counting Letter Occurrences
- K-1. Sample .bashrc file
- L-1. VIEWDATA.BAT: DOS Batch File
- L-2. viewdata.sh: Shell Script Conversion of VIEWDATA.BAT
- P-1. Print the server environment
Part 1. IntroductionThe shell is a command interpreter. More than just the
insulating layer between the operating system kernel and the user,
it's also a fairly powerful programming language. A shell program,
called a script, is an easy-to-use tool for
building applications by "gluing" together system
calls, tools, utilities, and compiled binaries. Virtually the
entire repertoire of UNIX commands, utilities, and tools is
available for invocation by a shell script. If that were
not enough, internal shell commands, such as testing and loop
constructs, give additional power and flexibility to scripts.
Shell scripts lend themselves exceptionally well to administrative
system tasks and other routine repetitive jobs not requiring the
bells and whistles of a full-blown tightly structured programming
language.
Chapter 1. Why Shell Programming?| | No programming language is perfect. There is not even a single
best language; there are only languages well suited or perhaps poorly
suited for particular purposes. | | | Herbert Mayer |
A working knowledge of shell scripting is essential to anyone
wishing to become reasonably proficient at system administration,
even if they do not anticipate ever having to actually write a
script. Consider that as a Linux machine boots up, it executes the
shell scripts in /etc/rc.d
to restore the system configuration and set up services. A detailed
understanding of these startup scripts is important for analyzing
the behavior of a system, and possibly modifying it. Writing shell scripts is not hard to learn, since the scripts
can be built in bite-sized sections and there is only a fairly
small set of shell-specific operators and options
to learn. The syntax is simple and straightforward, similar to
that of invoking and chaining together utilities at the command
line, and there are only a few "rules" to learn.
Most short scripts work right the first time, and debugging even
the longer ones is straightforward. A shell script is a "quick and dirty" method of
prototyping a complex application. Getting even a limited subset
of the functionality to work in a shell script is often a useful
first stage in project development. This way, the structure of
the application can be tested and played with,
and the major pitfalls found before proceeding
to the final coding in C,
C++, Java, or Perl. Shell scripting hearkens back to the classic UNIX philosophy
of breaking complex projects into simpler subtasks, of chaining
together components and utilities. Many consider this a better,
or at least more esthetically pleasing approach to problem solving
than using one of the new generation of high powered all-in-one
languages, such as Perl, which attempt to
be all things to all people, but at the cost of forcing you to
alter your thinking processes to fit the tool. When not to use shell scripts
Resource-intensive tasks, especially where speed is
a factor (sorting, hashing, etc.) Procedures involving heavy-duty math operations,
especially floating point arithmetic, arbitrary
precision calculations, or complex numbers (use
C++ or FORTRAN
instead) Cross-platform portability required (use C or Java instead) Complex applications, where structured programming is
a necessity (need type-checking of variables, function
prototypes, etc.) Mission-critical applications upon which you are betting the
ranch, or the future of the company Situations where security is important, where you need to
guarantee the integrity of your system and protect against
intrusion, cracking, and vandalism Project consists of subcomponents with interlocking
dependencies Extensive file operations required (Bash is limited to
serial file access, and that only in a particularly clumsy
and inefficient line-by-line fashion) Need native support for multi-dimensional arrays Need data structures, such as linked lists or trees Need to generate or manipulate graphics or GUIs Need direct access to system hardware Need port or socket I/O Need to use libraries or interface with legacy code Proprietary, closed-source applications (shell scripts put the
source code right out in the open for all the world to see)
If any of the above applies, consider a more powerful scripting
language -- perhaps Perl,
Tcl, Python,
Ruby -- or possibly a high-level
compiled language such as C,
C++, or Java. Even
then, prototyping the application as a shell script might still
be a useful development step. We will be using Bash, an acronym for
"Bourne-Again shell" and a pun on Stephen Bourne's
now classic Bourne shell. Bash has become
a de facto standard for shell
scripting on all flavors of UNIX. Most of the principles this
book covers apply equally well to scripting with other shells,
such as the Korn Shell, from which Bash
derives some of its features,
and the C Shell and its variants. (Note that
C Shell programming is not recommended due to
certain inherent problems, as pointed out in an October, 1993 Usenet
post by Tom Christiansen.) What follows is a tutorial on shell scripting. It relies
heavily on examples to illustrate various features of the shell.
The example scripts work -- they've been tested, insofar as was
possible -- and some of them are even useful in real life. The
reader can play with the actual working code of the examples
in the source archive (scriptname.sh or
scriptname.bash),
give them execute permission (chmod
u+rx scriptname), then run them
to see what happens. Should the source archive
not be available, then cut-and-paste from the HTML,
pdf,
or text
rendered versions. Be aware that some of the scripts presented here
introduce features before they are explained, and this may require
the reader to temporarily skip ahead for enlightenment. Unless otherwise noted, the author of this
book wrote the example scripts that follow.
Chapter 2. Starting Off With a Sha-Bang| | Shell programming is a 1950s juke box . . . | | | Larry Wall |
In the simplest case, a script is nothing more than a list of system
commands stored in a file. At the very least, this saves the
effort of retyping that particular sequence of commands each time
it is invoked. Example 2-1. cleanup: A script to clean up the log
files in /var/log # Cleanup
# Run as root, of course.
cd /var/log
cat /dev/null > messages
cat /dev/null > wtmp
echo "Logs cleaned up." |
There is nothing unusual here, only a set of commands that
could just as easily be invoked one by one from the command line on
the console or in an xterm. The advantages of
placing the commands in a script go beyond not having to retype them
time and again. The script becomes a tool,
and can easily be modified or customized for a particular
application. Example 2-2. cleanup: An improved clean-up
script #!/bin/bash
# Proper header for a Bash script.
# Cleanup, version 2
# Run as root, of course.
# Insert code here to print error message and exit if not root.
LOG_DIR=/var/log
# Variables are better than hard-coded values.
cd $LOG_DIR
cat /dev/null > messages
cat /dev/null > wtmp
echo "Logs cleaned up."
exit # The right and proper method of "exiting" from a script. |
Now that's beginning to look like a real script. But we can
go even farther . . . Example 2-3. cleanup: An enhanced
and generalized version of above scripts. #!/bin/bash
# Cleanup, version 3
# Warning:
# -------
# This script uses quite a number of features that will be explained
#+ later on.
# By the time you've finished the first half of the book,
#+ there should be nothing mysterious about it.
LOG_DIR=/var/log
ROOT_UID=0 # Only users with $UID 0 have root privileges.
LINES=50 # Default number of lines saved.
E_XCD=66 # Can't change directory?
E_NOTROOT=67 # Non-root exit error.
# Run as root, of course.
if [ "$UID" -ne "$ROOT_UID" ]
then
echo "Must be root to run this script."
exit $E_NOTROOT
fi
if [ -n "$1" ]
# Test if command line argument present (non-empty).
then
lines=$1
else
lines=$LINES # Default, if not specified on command line.
fi
# Stephane Chazelas suggests the following,
#+ as a better way of checking command line arguments,
#+ but this is still a bit advanced for this stage of the tutorial.
#
# E_WRONGARGS=65 # Non-numerical argument (bad arg format)
#
# case "$1" in
# "" ) lines=50;;
# *[!0-9]*) echo "Usage: `basename $0` file-to-cleanup"; exit $E_WRONGARGS;;
# * ) lines=$1;;
# esac
#
#* Skip ahead to "Loops" chapter to decipher all this.
cd $LOG_DIR
if [ `pwd` != "$LOG_DIR" ] # or if [ "$PWD" != "$LOG_DIR" ]
# Not in /var/log?
then
echo "Can't change to $LOG_DIR."
exit $E_XCD
fi # Doublecheck if in right directory, before messing with log file.
# far more efficient is:
#
# cd /var/log || {
# echo "Cannot change to necessary directory." >&2
# exit $E_XCD;
# }
tail -n $lines messages > mesg.temp # Saves last section of message log file.
mv mesg.temp messages # Becomes new log directory.
# cat /dev/null > messages
#* No longer needed, as the above method is safer.
cat /dev/null > wtmp # ': > wtmp' and '> wtmp' have the same effect.
echo "Logs cleaned up."
exit 0
# A zero return value from the script upon exit
#+ indicates success to the shell. |
Since you may not wish to wipe out the entire system log,
this version of the script keeps the last section of the message
log intact. You will constantly discover ways of refining previously
written scripts for increased effectiveness. The
sha-bang
( #!) at the head of a script
tells your system that this file is a set of commands to be fed
to the command interpreter indicated. The
#! is actually a two-byte
magic number, a special marker that
designates a file type, or in this case an executable shell
script (type man magic for more
details on this fascinating topic). Immediately following
the sha-bang is a path
name. This is the path to the program that interprets
the commands in the script, whether it be a shell, a programming
language, or a utility. This command interpreter then executes
the commands in the script, starting at the top (line following
the sha-bang line), ignoring comments.
#!/bin/sh
#!/bin/bash
#!/usr/bin/perl
#!/usr/bin/tcl
#!/bin/sed -f
#!/usr/awk -f |
Each of the above script header lines calls a different command
interpreter, be it /bin/sh, the default shell
(bash in a Linux system) or otherwise.
Using #!/bin/sh, the default Bourne shell
in most commercial variants of UNIX, makes the script portable to non-Linux machines,
though you sacrifice Bash-specific features.
The script will, however, conform to the
POSIX
sh standard. Note that the path given at the "sha-bang" must
be correct, otherwise an error message -- usually "Command
not found" -- will be the only result of running the
script. #! can be omitted if the script consists only
of a set of generic system commands, using no internal
shell directives. The second example, above, requires the
initial #!, since the variable assignment line,
lines=50, uses a shell-specific construct.
Note again that #!/bin/sh invokes the default
shell interpreter, which defaults to /bin/bash
on a Linux machine.  | This tutorial encourages a modular approach
to constructing a script. Make note of and collect
"boilerplate" code snippets that might be useful
in future scripts. Eventually you can build quite an extensive
library of nifty routines. As an example, the following script
prolog tests whether the script has been invoked with the correct
number of parameters. E_WRONG_ARGS=65
script_parameters="-a -h -m -z"
# -a = all, -h = help, etc.
if [ $# -ne $Number_of_expected_args ]
then
echo "Usage: `basename $0` $script_parameters"
# `basename $0` is the script's filename.
exit $E_WRONG_ARGS
fi |
Many times, you will write a script that carries out one
particular task. The first script in this chapter is an
example of this. Later, it might occur to you to generalize
the script to do other, similar tasks. Replacing the literal
("hard-wired") constants by variables is a step in
that direction, as is replacing repetitive code blocks by functions. |
2.1. Invoking the scriptHaving written the script, you can invoke it by sh
scriptname,
or alternatively bash scriptname. (Not
recommended is using sh <scriptname,
since this effectively disables reading from
stdin within the script.) Much more
convenient is to make the script itself directly executable with
a chmod.
- Either:
chmod 555 scriptname (gives
everyone read/execute permission)
- or
chmod +rx scriptname (gives
everyone read/execute permission) chmod
u+rx scriptname (gives only the
script owner read/execute permission)
Having made the script executable, you may now test it by
./scriptname.
If it begins with a "sha-bang" line, invoking the
script calls the correct command interpreter to run it. As a final step, after testing and debugging,
you would likely want to move it to /usr/local/bin (as
root, of course), to make the script
available to yourself and all other users as a system-wide
executable. The script could then be invoked by simply typing
scriptname [ENTER] from the
command line.
Chapter 3. Special CharactersSpecial Characters Found In
Scripts and Elsewhere - #
# This line is a comment. |
Comments may also occur following the end of a command. echo "A comment will follow." # Comment here.
# ^ Note whitespace before # |
Comments may also follow whitespace at the beginning
of a line. # A tab precedes this comment. |
 | A command may not follow a comment on the
same line. There is no method of terminating the comment,
in order for "live code" to begin on the same
line. Use a new line for the next command. |
 | Of course, an escaped # in an
echo statement does
not begin a comment. Likewise, a
# appears in certain parameter
substitution constructs and in numerical constant expressions.
echo "The # here does not begin a comment."
echo 'The # here does not begin a comment.'
echo The \# here does not begin a comment.
echo The # here begins a comment.
echo ${PATH#*:} # Parameter substitution, not a comment.
echo $(( 2#101011 )) # Base conversion, not a comment.
# Thanks, S.C. |
The standard quoting and
escape characters (" ' \) escape the #.
|
Certain pattern matching
operations also use the #. - ;
echo hello; echo there
if [ -x "$filename" ]; then # Note that "if" and "then" need separation.
# Why?
echo "File $filename exists."; cp $filename $filename.bak
else
echo "File $filename not found."; touch $filename
fi; echo "File test complete." |
Note that the ";" sometimes
needs to be escaped. - ;;
case "$variable" in
abc) echo "\$variable = abc" ;;
xyz) echo "\$variable = xyz" ;;
esac |
- .
- .
When considering directory names, a single
dot represents the current working directory,
and two dots denote the parent
directory. bash$ pwd
/home/bozo/projects
bash$ cd .
bash$ pwd
/home/bozo/projects
bash$ cd ..
bash$ pwd
/home/bozo/
|
The dot often appears as the
destination (directory) of a file movement command. bash$ cp /home/bozo/current_work/junk/* .
|
- .
- "
- '
- ,
- \
\X
"escapes" the character
X. This has the effect of
"quoting" X, equivalent
to 'X'. The \ may
be used to quote " and ',
so they are expressed literally. See Chapter 5 for an in-depth explanation
of escaped characters. - /
This is also the division arithmetic operator. - `
- :
Endless loop: while :
do
operation-1
operation-2
...
operation-n
done
# Same as:
# while true
# do
# ...
# done |
Placeholder in if/then test: if condition
then : # Do nothing and branch ahead
else
take-some-action
fi |
Provide a placeholder where a binary operation is
expected, see Example 8-2 and default parameters. : ${username=`whoami`}
# ${username=`whoami`} Gives an error without the leading :
# unless "username" is a command or builtin... |
Provide a placeholder where a command is expected in a
here document. See Example 18-10. Evaluate string of variables using
parameter substitution
(as in Example 9-15).
: ${HOSTNAME?} ${USER?} ${MAIL?}
# Prints error message
#+ if one or more of essential environmental variables not set. |
Variable expansion / substring
replacement. In combination with the > redirection operator,
truncates a file to zero length, without changing its
permissions. If the file did not previously exist,
creates it.
: > data.xxx # File "data.xxx" now empty.
# Same effect as cat /dev/null >data.xxx
# However, this does not fork a new process, since ":" is a builtin. |
See also Example 15-14.In combination with the >>
redirection operator, has no effect on a pre-existing
target file (: >> target_file).
If the file did not previously exist, creates it.  | This applies to regular files, not pipes,
symlinks, and certain special files. |
May be used to begin a comment line, although this is not
recommended. Using # for a comment turns
off error checking for the remainder of that line, so
almost anything may appear in a comment. However,
this is not the case with
:.
: This is a comment that generates an error, ( if [ $x -eq 3] ). |
The ":" also serves as a field
separator, in /etc/passwd, and in the $PATH variable.
bash$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin:/usr/games |
- !
In a different context, the !
also appears in indirect variable
references. In yet another context, from the command
line, the ! invokes the
Bash history mechanism (see Appendix J). Note that within a script,
the history mechanism is disabled. - *
bash$ echo *
abs-book.sgml add-drive.sh agram.sh alias.sh
|
The * also represents any number
(or zero) characters in a regular expression. - *
A double asterisk, **, is the exponentiation
operator. - ?
In a double
parentheses construct, the ? serves
as a C-style trinary operator. See Example 9-31. In a parameter
substitution expression, the ?
tests whether a variable has been
set. - ?
- $
A $ prefixing a variable name
indicates the value the variable
holds. - $
- ${}
- $*, $@
- $?
- $$
- ()
 | A listing of commands within
parentheses starts a subshell. Variables inside parentheses, within the subshell, are not
visible to the rest of the script. The parent process,
the script, cannot read variables
created in the child process, the subshell.
a=123
( a=321; )
echo "a = $a" # a = 123
# "a" within parentheses acts like a local variable. |
|
- {xxx,yyy,zzz,...}
A command may act upon a comma-separated list of file specs within
braces.
Filename expansion (globbing)
applies to the file specs between the braces.  | No spaces allowed within the braces
unless the spaces are quoted or escaped. echo {file1,file2}\ :{\ A," B",' C'} file1 : A file1 : B file1 : C file2 : A file2 : B file2 : C |
- {a..z}
The {a..z}
extended brace
expansion construction is a feature introduced
in version 3 of
Bash. - {}
bash$ { local a;
a=123; }
bash: local: can only be used in a
function
| a=123
{ a=321; }
echo "a = $a" # a = 321 (value inside code block)
# Thanks, S.C. |
The code block enclosed in braces may have I/O redirected to and from
it. Example 3-1. Code blocks and I/O redirection #!/bin/bash
# Reading lines in /etc/fstab.
File=/etc/fstab
{
read line1
read line2
} < $File
echo "First line in $File is:"
echo "$line1"
echo
echo "Second line in $File is:"
echo "$line2"
exit 0
# Now, how do you parse the separate fields of each line?
# Hint: use awk. |
Example 3-2. Saving the results of a code block to a file #!/bin/bash
# rpm-check.sh
# Queries an rpm file for description, listing, and whether it can be installed.
# Saves output to a file.
#
# This script illustrates using a code block.
SUCCESS=0
E_NOARGS=65
if [ -z "$1" ]
then
echo "Usage: `basename $0` rpm-file"
exit $E_NOARGS
fi
{
echo
echo "Archive Description:"
rpm -qpi $1 # Query description.
echo
echo "Archive Listing:"
rpm -qpl $1 # Query listing.
echo
rpm -i --test $1 # Query whether rpm file can be installed.
if [ "$?" -eq $SUCCESS ]
then
echo "$1 can be installed."
else
echo "$1 cannot be installed."
fi
echo
} > "$1.test" # Redirects output of everything in block to file.
echo "Results of rpm test in file $1.test"
# See rpm man page for explanation of options.
exit 0 |
 | Unlike a command group within (parentheses),
as above, a code block enclosed by {braces} will
not normally launch a subshell.
|
- {}
ls . | xargs -i -t cp ./{} $1
# ^^ ^^
# From "ex42.sh" (copydir.sh) example. |
- {} \;
 | The ";" ends
the -exec option of a
find command sequence. It needs
to be escaped to protect it from interpretation by the
shell. |
- [ ]
Test expression between
[ ]. Note that [
is part of the shell builtin test (and a synonym for it),
not a link to the external command
/usr/bin/test. - [[ ]]
Test expression between [[ ]]. This is a shell
keyword. See the discussion on the [[ ... ]] construct. - [ ]
In the context of an array,
brackets set off the numbering of each element of that array.
Array[1]=slot_1
echo ${Array[1]} |
- [ ]
As part of a regular
expression, brackets delineate a range of characters to
match. - (( ))
Expand and evaluate integer expression between
(( )). See the discussion on the (( ... )) construct. - > &> >& >> < <>
scriptname >filename redirects the output of
scriptname to file
filename. Overwrite
filename if it already exists. command &>filename redirects
both the stdout and the
stderr of command
to filename. command >&2 redirects
stdout of command
to stderr. scriptname >>filename appends
the output of scriptname
to file filename. If
filename does not already exist,
it is created. [i]<>filename
opens file filename for reading
and writing, and assigns file
descriptor i to it. If
filename does not exist, it is
created. (command)> <(command) In a different context,
the "<" and
">" characters act
as string comparison
operators. In yet another context,
the "<" and
">" characters act
as integer comparison
operators. See also Example 15-9. - <<
- <<<
- <, >
- \<, \>
bash$ grep '\<the\>' textfile - |
echo ls -l | sh
# Passes the output of "echo ls -l" to the shell,
#+ with the same result as a simple "ls -l".
cat *.lst | sort | uniq
# Merges and sorts all ".lst" files, then deletes duplicate lines. |
The output of a command or commands
may be piped to a script.
#!/bin/bash
# uppercase.sh : Changes input to uppercase.
tr 'a-z' 'A-Z'
# Letter ranges must be quoted
#+ to prevent filename generation from single-letter filenames.
exit 0 |
Now, let us pipe the output of ls -l to this
script.
bash$ ls -l | ./uppercase.sh
-RW-RW-R-- 1 BOZO BOZO 109 APR 7 19:49 1.TXT
-RW-RW-R-- 1 BOZO BOZO 109 APR 14 16:48 2.TXT
-RW-R--R-- 1 BOZO BOZO 725 APR 20 20:56 DATA-FILE
|
 | The stdout of each process in
a pipe must be read as the stdin
of the next. If this is not the case, the data stream
will block, and the pipe will not
behave as expected.
cat file1 file2 | ls -l | sort
# The output from "cat file1 file2" disappears. |
A pipe runs as a child
process, and therefore cannot alter script
variables.
variable="initial_value"
echo "new_value" | read variable
echo "variable = $variable" # variable = initial_value |
If one of the commands in the pipe
aborts, this prematurely terminates execution of the
pipe. Called a broken pipe, this
condition sends a SIGPIPE signal. |
- >|
- ||
- &
bash$ sleep 10 &
[1] 850
[1]+ Done sleep 10
|
Within a script, commands and even loops may run in the
background. Example 3-3. Running a loop in the background #!/bin/bash
# background-loop.sh
for i in 1 2 3 4 5 6 7 8 9 10 # First loop.
do
echo -n "$i "
done & # Run this loop in background.
# Will sometimes execute after second loop.
echo # This 'echo' sometimes will not display.
for i in 11 12 13 14 15 16 17 18 19 20 # Second loop.
do
echo -n "$i "
done
echo # This 'echo' sometimes will not display.
# ======================================================
# The expected output from the script:
# 1 2 3 4 5 6 7 8 9 10
# 11 12 13 14 15 16 17 18 19 20
# Sometimes, though, you get:
# 11 12 13 14 15 16 17 18 19 20
# 1 2 3 4 5 6 7 8 9 10 bozo $
# (The second 'echo' doesn't execute. Why?)
# Occasionally also:
# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
# (The first 'echo' doesn't execute. Why?)
# Very rarely something like:
# 11 12 13 1 2 3 4 5 6 7 8 9 10 14 15 16 17 18 19 20
# The foreground loop preempts the background one.
exit 0
# Nasimuddin Ansari suggests adding sleep 1
#+ after the echo -n "$i" in lines 6 and 14,
#+ for some real fun. |
 | A command run in the background within a
script may cause the script to hang, waiting
for a keystroke. Fortunately, there is a remedy for this. |
- &&
- -
COMMAND -[Option1][Option2][...] ls -al sort -dfu $filename set -- $variable if [ $file1 -ot $file2 ]
then
echo "File $file1 is older than $file2."
fi
if [ "$a" -eq "$b" ]
then
echo "$a is equal to $b."
fi
if [ "$c" -eq 24 -a "$d" -eq 47 ]
then
echo "$c equals 24 and $d equals 47."
fi |
- -
(cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -)
# Move entire file tree from one directory to another
# [courtesy Alan Cox <a.cox@swansea.ac.uk>, with a minor change]
# 1) cd /source/directory Source directory, where the files to be moved are.
# 2) && "And-list": if the 'cd' operation successful, then execute the next command.
# 3) tar cf - . The 'c' option 'tar' archiving command creates a new archive,
# the 'f' (file) option, followed by '-' designates the target file as stdout,
# and do it in current directory tree ('.').
# 4) | Piped to...
# 5) ( ... ) a subshell
# 6) cd /dest/directory Change to the destination directory.
# 7) && "And-list", as above
# 8) tar xpvf - Unarchive ('x'), preserve ownership and file permissions ('p'),
# and send verbose messages to stdout ('v'),
# reading data from stdin ('f' followed by '-').
#
# Note that 'x' is a command, and 'p', 'v', 'f' are options.
# Whew!
# More elegant than, but equivalent to:
# cd source/directory
# tar cf - . | (cd ../dest/directory; tar xpvf -)
#
# Also having same effect:
# cp -a /source/directory/* /dest/directory
# Or:
# cp -a /source/directory/* /source/directory/.[^.]* /dest/directory
# If there are hidden files in /source/directory. |
bunzip2 -c linux-2.6.16.tar.bz2 | tar xvf -
# --uncompress tar file-- | --then pass it to "tar"--
# If "tar" has not been patched to handle "bunzip2",
#+ this needs to be done in two discrete steps, using a pipe.
# The purpose of the exercise is to unarchive "bzipped" kernel source. |
Note that in this context the "-" is not
itself a Bash operator, but rather an option recognized by
certain UNIX utilities that write to
stdout, such as tar,
cat, etc. bash$ echo "whatever" | cat -
whatever |
Where a filename is expected,
- redirects output to
stdout (sometimes seen with
tar cf), or accepts input from
stdin, rather than from a file. This
is a method of using a file-oriented utility as a filter
in a pipe. bash$ file
Usage: file [-bciknvzL] [-f namefile] [-m magicfiles] file...
|
By itself on the command line, file fails with an error message.
Add a "-" for a more useful result. This causes the
shell to await user input.
bash$ file -
abc
standard input: ASCII text
bash$ file -
#!/bin/bash
standard input: Bourne-Again shell script text executable
|
Now the command accepts input from stdin
and analyzes it.
The "-" can be used to pipe
stdout to other commands. This permits
such stunts as prepending lines
to a file. Using diff to
compare a file with a section
of another: grep Linux file1 | diff file2 - Finally, a real-world example using
- with tar. Example 3-4. Backup of all files changed in last day #!/bin/bash
# Backs up all files in current directory modified within last 24 hours
#+ in a "tarball" (tarred and gzipped file).
BACKUPFILE=backup-$(date +%m-%d-%Y)
# Embeds date in backup filename.
# Thanks, Joshua Tschida, for the idea.
archive=${1:-$BACKUPFILE}
# If no backup-archive filename specified on command line,
#+ it will default to "backup-MM-DD-YYYY.tar.gz."
tar cvf - `find . -mtime -1 -type f -print` > $archive.tar
gzip $archive.tar
echo "Directory $PWD backed up in archive file \"$archive.tar.gz\"."
# Stephane Chazelas points out that the above code will fail
#+ if there are too many files found
#+ or if any filenames contain blank characters.
# He suggests the following alternatives:
# -------------------------------------------------------------------
# find . -mtime -1 -type f -print0 | xargs -0 tar rvf "$archive.tar"
# using the GNU version of "find".
# find . -mtime -1 -type f -exec tar rvf "$archive.tar" '{}' \;
# portable to other UNIX flavors, but much slower.
# -------------------------------------------------------------------
exit 0 |
 | Filenames beginning with
"-" may cause problems when coupled with the
"-" redirection operator. A script should
check for this and add an appropriate prefix to such
filenames, for example ./-FILENAME,
$PWD/-FILENAME, or
$PATHNAME/-FILENAME. If the value of a variable begins with a
-, this may likewise create
problems.
var="-n"
echo $var
# Has the effect of "echo -n", and outputs nothing. |
|
- -
 | Do not confuse the "-" used in this
sense with the "-" redirection
operator just discussed. The interpretation of the
"-" depends on the context in which it
appears. |
- -
- =
In a different context,
the "=" is a string comparison
operator. - +
In a different context,
the + is a Regular
Expression operator. - +
Certain commands and builtins use the
+ to enable certain options and the
- to disable them. - %
In a different context,
the % is a pattern
matching operator. - ~
- ~+
- ~-
- =~
- ^
- Control Characters
Control characters are not normally useful inside a
script. Ctl-B Backspace (nondestructive). Ctl-C Break. Terminate a foreground job.
Ctl-D Log out from a shell (similar to
exit). "EOF" (end of file). This also
terminates input from stdin. When typing text on the console or in an
xterm window,
Ctl-D erases the character under the
cursor. When there are no characters present,
Ctl-D logs out of the session, as
expected. In an xterm window, this has the effect of closing
the window. Ctl-G "BEL" (beep). On some old-time teletype
terminals, this would actually ring a bell. Ctl-H "Rubout" (destructive backspace). Erases
characters the cursor backs over while backspacing. #!/bin/bash
# Embedding Ctl-H in a string.
a="^H^H" # Two Ctl-H's (backspaces).
echo "abcdef" # abcdef
echo -n "abcdef$a " # abcd f
# Space at end ^ ^ Backspaces twice.
echo -n "abcdef$a" # abcdef
# No space at end Doesn't backspace (why?).
# Results may not be quite as expected.
echo; echo |
Ctl-I Horizontal tab. Ctl-J Newline (line feed). In a script, may also be expressed
in octal notation -- '\012' or in hexadecimal -- '\x0a'. Ctl-K Vertical tab. When typing text on the console or in an
xterm window,
Ctl-K erases from the character
under the cursor to end of line. Within a script,
Ctl-K may behave differently,
as in Lee Lee Maschmeyer's example, below. Ctl-L Formfeed (clear the terminal screen). In a terminal,
this has the same effect as the clear command. When sent
to a printer, a Ctl-L causes
an advance to end of the paper sheet. Ctl-M Carriage return. #!/bin/bash
# Thank you, Lee Maschmeyer, for this example.
read -n 1 -s -p $'Control-M leaves cursor at beginning of this line. Press Enter. \x0d'
# Of course, '0d' is the hex equivalent of Control-M.
echo >&2 # The '-s' makes anything typed silent,
#+ so it is necessary to go to new line explicitly.
read -n 1 -s -p $'Control-J leaves cursor on next line. \x0a'
# '0a' is the hex equivalent of Control-J, linefeed.
echo >&2
###
read -n 1 -s -p $'And Control-K\x0bgoes straight down.'
echo >&2 # Control-K is vertical tab.
# A better example of the effect of a vertical tab is:
var=$'\x0aThis is the bottom line\x0bThis is the top line\x0a'
echo "$var"
# This works the same way as the above example. However:
echo "$var" | col
# This causes the right end of the line to be higher than the left end.
# It also explains why we started and ended with a line feed --
#+ to avoid a garbled screen.
# As Lee Maschmeyer explains:
# --------------------------
# In the [first vertical tab example] . . . the vertical tab
#+ makes the printing go straight down without a carriage return.
# This is true only on devices, such as the Linux console,
#+ that can't go "backward."
# The real purpose of VT is to go straight UP, not down.
# It can be used to print superscripts on a printer.
# The col utility can be used to emulate the proper behavior of VT.
exit 0 |
Ctl-Q Resume (XON). This resumes stdin in a terminal. Ctl-S Suspend (XOFF). This freezes stdin in a terminal.
(Use Ctl-Q to restore input.) Ctl-U Erase a line of input, from the cursor backward to
beginning of line. In some settings,
Ctl-U erases the entire
line of input, regardless of cursor
position. Ctl-V When inputting text, Ctl-V
permits inserting control characters. For example, the
following two are equivalent:
echo -e '\x0a'
echo <Ctl-V><Ctl-J> |
Ctl-V is primarily useful from
within a text editor. Ctl-W When typing text on the console or in an xterm window,
Ctl-W erases from the character
under the cursor backwards to the first instance of
whitespace. In some settings, Ctl-W
erases backwards to first non-alphanumeric character. Ctl-Z Pause a foreground job.
- Whitespace
Blank lines have no effect on the action of a script,
and are therefore useful for visually separating functional
sections. $IFS, the special variable
separating fields of input to certain commands, defaults
to whitespace. To preserve whitespace within a string or in a variable,
use quoting.
Chapter 4. Introduction to Variables and ParametersVariables |