"Linux Gazette...making Linux just a little more fun! "

When running a single box with tried and true software, tracking the versions of software that you use may be a no-brainer. That is to say, you use whatever Red Hat, Debian, or Sun provided (yes, I will touch on non-Linux issues here) if you could find or build the necessary package. But wait: what if you have been running the same machine for years and you simply must have the latest Emacs? What if you are developing your own software and don't want to create RPMs, or Debian dpkg each time you pause at a version? What if you don't trust that software package written by a 14 year old in that far away country with an unstable government? In short, what if you are heeding Obi-Wan Kenobe's advice, and using the source? How do you make it easy to rip out those configuration files, man pages, binaries, and libraries that you may want to replace in the future?

Well, when you think about it a little bit, Unix has sort of provided the raw materials to do that, in the form of a symbolic link or symlink . Symlinks are a powerful tool because they allow you to configure software so that its implementation does not necessarily connect directly to it's interface (sound familiar?). I might be playing a little loose with the definitions, but that really is what is being done when, for example, postfix mimics sendmail. The implementation, that is postfix, is presenting the same interface as sendmail, which has become a de facto standard interface to the Unix mail transport agent (MTA).

In the case of symlinks, you might have a program /opt/bin/new_cat linked to /opt/bin/cat. So if you looked at the link, you'd know right away what version was being run, but it would still seem to be the same familiar program. In this way the actual program being run can change as a better implementation (algorithm, etc.) is developed. Yes, environmental variables, as used in scripts, allow this, but try retrofitting all the variables that point to a program after the fact. It might not be so easy. Symlinks may be the answer. For example symlinks are typically used to ease the building of motif from source via the lndir utility. Of course this symlinking stuff could get out of hand, and should not be abused, but you get the idea. What the folks at the GNU project did was write a little Perl script that automates that entire process of symlinking the code you are using to the interface that you want to present to the user. Note that hard links are subtly different, because there is no differentiation between the original file and the link (really a second name since they share inodes, and hence are identical). I find hard links to be of minimal use, because it becomes too easy to lose track of which filename should be deleted and which should be kept.

Introducing Stow

Right away I want to emphasize that stow is not a replacement for a full package management database, but it does allow one to get many of the benefits of a complex package management system from a humble Perl script. As an aside, there is a package that will allow source to be entered into a Slackware, RPM, or Debian package database, called checkinstall . As an example I will go through the steps to install stow, then the steps to install a mail (MUA) replacement called nail . This is a good example because it includes multiple files so that you can see how one might encounter inadvertant collisions with previous versions. Also, nail a great enhancement to standard Berkeley mail, since it allows sending binary attachments on the command line, while offering the same base functionality.

Stow is so simple to install that really no in depth discussion is needed. It should work if you have Perl 5.005 or later (this version is stock on Solaris 8 AFAIK). Simply download the source from the GNU website or a local mirror, extract to a source directory with tar xzf and repeat the familiar ./configure , make, and make install sequence. Despite appearances, nothing is compiled, but a few things like the manual still need to get built. The make install step will place stow into the /usr/local/bin directory. This is the default location, and I chose this setting to simplify this discussion. The reasons will hopefully become apparent by the end of this article. The location of the installed stow executable is shown on the last line of the sample output below. I used the type command, but you could also use which or perhaps whereis.

[zippy@mybox zippy]$ cd src/
[zippy@mybox src]$ gunzip -c ../stow-1.3.3.tar.gz | tar xf -
[zippy@mybox src]$ ll
total 8
drwxrwxr-x    2 zippy   zippy       4096 Jan  6 06:19 stow-1.3.3
[zippy@mybox stow-1.3.3]$ ./configure 
creating cache ./config.cache
checking for a BSD compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for mawk... no
checking for gawk... gawk
checking whether make sets ${MAKE}... yes
checking for a BSD compatible install... /usr/bin/install -c
checking for perl... /usr/bin/perl
updating cache ./config.cache
creating ./config.status
creating Makefile
creating stow
[zippy@mybox stow-1.3.3]$ make
make: Nothing to be done for `all'.
[zippy@mybox stow-1.3.3]$ sudo make install
make[1]: Entering directory `/home/zippy/src/stow-1.3.3'
/bin/sh ./mkinstalldirs /usr/local/bin
 /usr/bin/install -c stow /usr/local/bin/stow
/bin/sh ./mkinstalldirs /usr/local/info
 /usr/bin/install -c -m 644 ./stow.info /usr/local/info/stow.info
/bin/sh ./mkinstalldirs /usr/local/man/man8
 /usr/bin/install -c -m 644 ./stow.8 /usr/local/man/man8/stow.8
make[1]: Leaving directory `/home/zippy/src/stow-1.3.3'
[zippy@mybox stow-1.3.3]$ type stow
stow is /usr/local/bin/stow
      

At this point stow is installed under /usr/local/bin. Make sure to include this directory your $PATH

Under The Hood

To describe stow, one first needs to understand the configure script, because these two scripts work together, with configure building all the software components, and installing them on your machine. The configure script is a marvelous convenience. It sniffs the system, checking for various prerequisite software. The results of these tests are used to design a set of Makefiles which will build and install your software to fit your system configuration. There are many options to configure, in fact there are alternate versions of this script as well, but for our purposes the options of greatest interest is the --prefix argument. Note a second argument, the --exec-prefix allows some finer tuning of the actual installation process, but this option will not be discussed in much detail.

So now we understand that configure builds the scripts that build the code, and that the location of the installed code may be specified via configure's --prefix command-- line argument. It turns out that if you pick a single special spot to install all source code, stow can then cleanly automate the creation of symlinks to the installed code in such a way that the source tree is readily evident, and can be replaced and removed. For example, invoking the configure script as ./configure --prefix=/opt/stow/foo-1.2.1 will install your package under /opt/stow/foo-1.2.1

I'm Still Confused. What is this prefix and exec-prefix stuff?

Feel free to skip this section, and come back to it later, after you have digested the rest of this article. Once you are comfortable with the notion of an actual install location being separate from the apparent location of a program you can consider the parts of the puzzle that don't fit the this ideal scenario. Imagine the case of installing software across multiple machines where everything is installed in a symlinked directory tree isolated from the apparent location (found in the $PATH, or $MANPATH). Depending on your intentions, this might not be what you want. Consider the situation where an application might be built for multiple architectures, for example source code could be built for Solaris and linux systems as follows (assuming an identical cross mounted source trees, but separate build directories):

sun$ cd sunsparc
sun$ ../foolib-1.1/configure --prefix=/usr/local \
> --exec-prefix=/usr/local/sunsparc
sun$ make
sun$ make install

sun$ ssh pengie
pengie$ cd linux
pengie$ ../foolib-1.1/configure --prefix=/usr/local \
> --exec-prefix=/usr/local/linux
pengie$ make
pengie$ make install

The bottom line is that the developer has to decide which files are architecture dependant, and which are not, and you might not agree with her. Obviously documentation, and possibly configuration files could be considered architecture independent. Still, if you use stow, you are free to remove symlinks by "unstowing" files. Since this does upgrading will not overwrite the old source, instead it will only break the links, and you can hand copy configuration files back. Just "restow" the package and try again you get the upgrade right. Personally, I don't use the --exec-prefix option much, preferring instead to manually link the (hopefully) few configuration files that I want to treat specially, fixing broken links after upgrading. So far I think it's been a good approach for the simple situations I've encountered.

Installing Software With Stow

When I first started using stow a few years ago, I had some frustration with it because I had already started setting up the system (an HP-UX server) without it. There were frequent collisions with info files and manpages, ironically this was encountered the most with emacs. Naturally, following what is going on is easier for simple packages. The MUA software nail, is about as simple as you can get, since it consists of the executable, the documentation, and the config files (while you might want to link to /etc BTW).

[zippy@mybox src]$ gunzip -c ../nail-9.29.tar.gz  | tar xf -
[zippy@mybox src]$ cd nail-9.29/
[zippy@mybox nail-9.29]$ ./configure --prefix=/opt/stow/nail-9.29
creating cache ./config.cache
checking for a BSD compatible install... /usr/bin/install -c
checking for iswprint... yes
...
..... lots of stuff ...
updating cache ./config.cache
creating ./config.status
creating Makefile
creating config.h
[zippy@mybox nail-9.29]$ 
      

What we are doing here is telling configure to put the files under /opt/stow/nail-9.29 but (implicit as far as stow is concerned) that the installed package will appear to be under /opt for run time files. ( If you're curious, you can look at the generated Makefile to see that the prefix variable is set via the --prefix option).

[zippy@mybox nail-9.29]$ 
[zippy@mybox nail-9.29]$ make             
gcc -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -c version.c
gcc -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -c aux.c
... more stuff ...
gcc -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -c tty.c
gcc -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -c vars.c
gcc  -g -O2  -o nail  version.o aux.o base64.o cmd1.o cmd2.o \
cmd3.o cmdtab.o collect.o dotlock.o edit.o fio.o getname.o \
head.o v7.local.o lex.o list.o main.o mime.o names.o popen.o \
quit.o send.o sendout.o smtp.o strings.o temp.o tty.o vars.o  
[zippy@mybox nail-9.29]$
      

Now that we have compiled everything, we can install the software.

[zippy@mybox nail-9.29]$ sudo make install
make[1]: Entering directory `/home/zippy/src/nail-9.29'
/bin/sh ./mkinstalldirs /opt/stow/nail-9.29/bin
mkdir /opt/stow
mkdir /opt/stow/nail-9.29
mkdir /opt/stow/nail-9.29/bin
  /usr/bin/install -c  nail /opt/stow/nail-9.29/bin/nail
/bin/sh ./mkinstalldirs /opt/stow/nail-9.29/man/man1
mkdir /opt/stow/nail-9.29/man
mkdir /opt/stow/nail-9.29/man/man1
 /usr/bin/install -c -m 644 ./nail.1 /opt/stow/nail-9.29/man/man1/nail.1
test -f /etc/nail.rc || \
        { /bin/sh ./mkinstalldirs /etc; \
        /usr/bin/install -c -m 644 ./nail.rc /etc/nail.rc; }
make[1]: Leaving directory `/home/zippy/src/nail-9.29'
[zippy@mybox nail-9.29]$
      

So it's apparent from the previous listing that the file was tucked under /opt/stow/nail-9.29 as desired. Stow then assumes that all the subdirectories of the package are to be symlinked to their corresponding locations under --prefix (or ${prefix} if you look in the Makefile), so that /opt/stow/nail-9.29/bin becomes /opt/bin Similarly /opt/stow/nail-9.29/man/man1 becomes /opt/man/man1 etc. This convention makes it very easy to isolate files used from the install locations. The only step left is to actually create the symlinks by running stow.

[zippy@mybox nail-9.29]$ cd /opt/stow/
[zippy@mybox stow]$ sudo stow -vv nail-9.29/
Stowing package nail-9.29...
Stowing contents of nail-9.29
Stowing directory nail-9.29/bin
LINK /opt/bin to stow/nail-9.29/bin
Stowing directory nail-9.29/man
LINK /opt/man to stow/nail-9.29/man
[zippy@mybox stow]$ ls -ltr /opt/ 
[zippy@mybox stow]$ ls -ltr /opt 
total 4
drwxr-xr-x    3 root     root         4096 Jan  9 16:33 stow
lrwxrwxrwx    1 root     root           18 Jan  9 16:33 man -> stow/nail-9.29/man
lrwxrwxrwx    1 root     root           18 Jan  9 16:33 bin -> stow/nail-9.29/bin
stow/nail-9.29/bin
[zippy@mybox stow]$ PATH=/opt/bin:$PATH type nail
nail is /opt/bin/nail
      

Some explanation may be in order here, I cd'd to the stow directory (${prefix}/stow by default), and simply typed stow -vv plus the name of the subdirectory to recursively symlink. The -vv simply adds verbose output for illustrative purposes. So now all that needs to be done is to modify the $PATH variable, and your files are installed. Stow has created all the necessary links. Note that to uninstall the files (thus breaking the links) simply unstow them. This will disconnect (unlink) the installed binaries, but will not delete any files, so it's really quite a useful safety net.

[zippy@mybox stow]$ pwd
/opt/stow
[zippy@mybox stow]$ ls -l
total 4
drwxr-xr-x    4 root     root         4096 Jan  9 16:33 nail-9.29
[zippy@mybox stow]$ sudo stow -Dvv nail-9.29/
Unstowing in /opt
UNLINK /opt/bin
UNLINK /opt/man
[zippy@mybox stow]$

And all the installed files are neatly out of the way. Of course to restow the files you simply repeat the previous commands. This may seem like a lot of extra work, but once you get in the habit of using it, and experience the convenience of being able to unlink and entire package you'll find it's worth it. Finally, you might want to install nail yourself, and use it, possibly via an alias or shell function, as a mail replacement. But that could be an entire article in itself.

Happy hacking!

References

1. GNU stow
   Maintained by Guillaume Morin
   http://www.gnu.org/software/stow/stow.html
   GNU stow entry on Savannah
   http://savannah.gnu.org/projects/stow
2. Checkinstall
   by Itzo http://freshmeat.net/projects/checkinstall/
3. Nail, a replacement for the mail MUA
   by Gunnar Ritter http://omnibus.ruf.uni-freiburg.de/~gritter/
4. Linux Filesystem Hierarchy Standard, (FHS)
   Maintained by freestandards.org http://www.pathname.com/fhs/
5. GNU Autoconf, Automake, and libtool
   By Gary V. Vaughan, Ben Elliston, Tom Tromey, and Ian Lance Taylor
   offers an excellent review of the concepts behind exec-prefix options to the configure script.
   http://sources.redhat.com/autobook/ ISBN 1-57870-190-2
   

Allan Peda

Allan has been enjoying Linux since about 1995, discovering Perl shortly thereafter. Currently he works as a programmer analyst at Rockefeller University, and does part time Linux consulting work in the NYC area. He enjoys surfing and sailing, and dreams of owning a charter boat in tranquilo Costa Rica.

---

Copyright © 2002, Allan Peda.
Copying license http://www.linuxgazette.net/copying.html
Published in Issue 75 of Linux Gazette, February 2002

---