% ls -laR root
root:
total 36
drwxrwxrwx   4 root     wheel      14336 Jun 24 06:25 ../
drwxr-xr-x   5 root     bin         2048 Sep  5  1996 usr/
root/usr:
total 20
drwxr-xr-x   5 root     bin         2048 Sep  5  1996 ./
drwxr-xr-x   3 jlarke   wheel       2048 Jul 12  1996 ../
drwxr-xr-x   3 root     wheel       2048 Jul 12  1996 X11R6/
drwxr-xr-x   2 root     bin         2048 Jul 15  1997 bin/
drwxr-xr-x   2 root     wheel       2048 Mar 18  1997 sbin/
root/usr/X11R6:
total 12
drwxr-xr-x   3 root     wheel       2048 Jul 12  1996 ./
drwxr-xr-x   5 root     bin         2048 Sep  5  1996 ../
drwxr-xr-x   2 root     wheel       2048 Jun  3  1997 bin/
root/usr/X11R6/bin:
total 716
drwxr-xr-x   2 root     wheel       2048 Jun  3  1997 ./
drwxr-xr-x   3 root     wheel       2048 Jul 12  1996 ../
-rwxr-xr-x   1 root     wheel     361556 Jul 25  1996 xdm*
root/usr/bin:
total 2228
drwxr-xr-x   2 root     bin         2048 Jul 15  1997 ./
drwxr-xr-x   5 root     bin         2048 Sep  5  1996 ../
-r-xr-xr-x   1 root     bin      1136292 Jun  7  1997 login*
root/usr/sbin:
total 1560
drwxr-xr-x   2 root     wheel       2048 Mar 18  1997 ./
drwxr-xr-x   5 root     bin         2048 Sep  5  1996 ../
-r-xr-xr-x   1 bin      bin       427648 Sep  5  1996 in.ftpd*
-r-xr-xr-x   1 root     wheel     366012 Mar 18  1997 in.rexecd*

Generally speaking (although this may vary in other implementations), directories on the local disk but not in a template are ignored by Synctree.

Presas and Posas

Sometimes copying a file isn't enough. For example, if we're distributing a new daemon, we want to stop an old version and start the new. Presas (pre-sync activities) and posas (post-sync activities) take care of this.

Presas and posas are keyed to a particular file and run before or after any changes to that file are made. At LS&A, we don't have much call for presas, because Synctree is able to update a running program without crashing it. We just use a posa to stop the old version and start the new after the update.

For example, Listing 7 shows an SI file entry for sendmail.

The posa line tells it to run that script after making changes to sendmail. When Synctree runs a posa or presa, it gives it two command-line arguments: the reason why the file had to be changed, and the name of the file. So the command line might look like Listing 8. meaning that sendmail was updated because its md5 hash changed. The script is very simple; see Listing 9.

{
is /usr/lib/sendmail
like /srv/template-server/LSA/root/usr/lib/sendmail
st_mode 104751
st_uid 0
st_gid 3
st_size 401640
md5 473c1c1400281a032473c1f121d0049f
trigger 4f2
posa /usr/private/admin/synctree/posas/sendmail
}

/usr/private/admin/synctree/posas/sendmail MD5SYNC /usr/lib/sendmail

#!/bin/sh
# Restart sendmail if the binary was synced.
. /usr/private/admin/synctree/posas/posa.env
echo "/etc/init.d/sendmail stop"
/etc/init.d/sendmail stop
echo "/etc/init.d/sendmail start"
/etc/init.d/sendmail start

{
is /usr/lib/sendmail
like /srv/template-server/LSA/root/usr/lib/sendmail
st_mode 104751
st_uid 0
st_gid 3
st_size 401640
md5 473c1c1400281a032473c1f121d0049f
trigger 4f2
}

CTRL Files

Sometimes Statinfo doesn't (or can't) get all the information about a program. For example, there's no way it could tell what posa was associated with a file just by examining the file. CTRL files allow administrators to manually override all or part of an SI file. They look just like SI files, but they're usually typed by hand, and they usually have parentheses instead of curly braces. When Synctree sees a curly brace, that stanza replaces any previous stanza for that IS file. Parentheses tell it to keep the old data, and only replace the specified fields.

In the previous section, we saw an SI file for sendmail that was generated by merging an SI file and a CTRL file. The SI file entry would look like Listing 10, and the CTRL file entry is shown in Listing 11.

(
is /usr/lib/sendmail
st_mode 104751
posa /usr/private/admin/synctree/posas/sendmail
)

Organization (Classes)

Once you have the ability to create and use a system image, you run into the fact that no single image works for everyone. Each department, or sometimes each machine, has unique needs. So the authors created the notion of classes.

A class is a named set of files. Each department in LS&A that uses Synctree has a class. Berkeley-style printing is installed with a class, as is wuftpd. We also use them to track changes to the system-one class, SOL251, is Sun's OS image with vendor patches and AFS installed. The LSA class represents all the changes we've made to suit the College's needs. If I notice that 'rdist' doesn't match the system I used to work on, I can easily find out that the LSA class replaces it with a different version.

Each class has its own SI file. Synctree reads the SI file and builds a database of files to compare and reconcile, keyed by the IS entry in the SI file.

Each machine belongs to a specific set of classes, determined by the CLASS= line in /etc/hostconfig. Every machine has its own class, which is appended to the list of classes specified in the machine's /etc/hostconfig file. The order of the classes determines which files take precedence over others. Classes later in the list win out over those listed earlier.

For example, let's say my /etc/hostconfig file includes Listing 12. Both the SOL251 class and the AFSMODS class include versions of the /bin/login program. Synctree first reads the SI file for the SOL251 class, which looks like Listing 13. and then, later on, reads the SI file for AFSMODS, which would include lines shown in Listing 14.

CLASS=SOL251:LSA:PRINT:NPI-CDDI:ODS4:AFSMODS:WUFTP242UM:PSC:PSCSOL

{
is /usr/bin/login
like /srv/template-server/.SOL251-sun4m_55/root/usr/bin/login
st_mode 104555
st_uid 0
st_gid 2
st_size 28800
md5 635130471cda6d64d5f2a6667dc8ecfd
trigger 4f2
}

{
is /usr/bin/login
like /srv/template-server/AFSMODS/root/usr/bin/login
st_mode 104555
st_uid 0
st_gid 2
st_size 1136292
md5 3ac4528006a8ee5d0685e1e25d789673
trigger 4f2
}

When Synctree sees the second entry for the same IS file, it automatically replaces the first one with the new version. When it has read all the SI and CTRL files for all the classes listed (plus the host's own class), it has a complete image of the system built up, and it's ready to start comparing and updating.

It may sound complicated, but the upshot of it is that this one program takes care of all the synchronization chores - all the administrator of a machine has to do is run the command.

Installation Procedures

LS&A tries to use vendor installation procedures to install Synctree, and then let Synctree install the rest of our changes. We've only implemented this philosophy for Solaris thus far, but see no reason that it couldn't be done for other platforms.

Syncing a Machine

Just running the Synctree binary doesn't get us very far. It provides the comparison engine, but we use wrapper scripts to tell it which SI files to use, authenticate to Kerberos, and handle logging what it does.

0 3 * * * /usr/private/admin/synctree/bin/run_synctasks 2>&1

If you want to run Synctree by hand, just run 'syncnode'. You can also use the '-norun' switch if you want to see what Synctree would do, without actually making any changes.

Making Changes to a Template

Before we change Synctree templates, we generally test the change on a particular machine first. This prevents the nightmare scenario where a change which breaks machines happens on hundreds of machines at 3 am.

Once we know that a modification works, we need to determine which class to put it in, and update that class.

Generally, we put changes in the most general template that makes sense. If we have a file that goes on every machine in a department, we don't put it in each machine's class; we put it in a class that every machine syncs to.

Updates of a class's version of a file are done with the 'cisync' command. The basic use is just

cisync <filename>

Cisync will ask which class to add the file to, or we can specify it on the command line with the '-c' flag.

cisync -c MATH <filename>

Once we've cisync'd all the files, we need to make sure the SI files get updated to match. At LS&A, the college runs a cron job that will take care of this at 2 am each night. If you want to sync machines before then, you need to run the mktemplate command.

mktemplateSI <CLASSNAME>

Creating a New Class

The command to create a new class is 'mkclass'. At LS&A, running AFS, you need to be in the AFS system:administrators PTS group for the lsa.umich.edu cell. Class names can be no longer than eight characters long and, by convention, are in all caps. The length limit is a product of AFS' limits on volume names, since each class gets its own volume.

mkclass CYRUS152

Commonly Used Classes

For Solaris, classes of general interest at the College of LS&A include:

• [TEST,TEST4c,TEST4m,TEST4u]: Test classes for new patches, updated software, etc. If it succeeds here, it gets rolled out to production.
• [SOL251]: Stock Solaris with AFS installed, no customizations. Some files deleted in order to link them into AFS.
• [LSA]: LS&A changes to the stock Solaris install.
• [AFSMODS]: AFS-aware login, xdm, xlock, etc.
• [AFSSERVER]: Useful AFS server mods, afs fstype, tcpwrappers, etc.
• [FTP242B13]: Wu-ftpd version 2.4.2b13 with AFS mods, and .principals support.
• [MAILSERV]: Programs needed to support SMTP, IMAP2bis, POP3, and KPOP services.
• [MAILSERV2]: Like MAILSERV, but keeps inboxes in hashed directories - i.e., /var/ mail/r/o/root/INBOX.
• [PERL]: Install Perl 5.004 in /usr/private/.
• [PRINT]: BSD-style printing commands.
• [SAMBA]: Samba 1.9.17p2 with AFS and NT hashed password support.

Other Synctree Utilities

Precedence

Precedence helps system administrators determine what Synctree class(es) a given file comes from. The simplest form:

# precedence /etc/inet/inetd.conf

% precedence /etc/inet/inetd.conf
/srv/template-server/MATH/root/etc/inet/inetd.conf
/srv/template-server/LSA/root/etc/inet/inetd.conf
/srv/template-server/SOL251/root/etc/inet/inetd.conf

mksynclist

mksynclist list Synctree clients and/or classes. Its original purpose was to provide a list of all the possible targets for mktemplateSI, but it can also be used to list all the machines that sync to a certain class.

whosyncs

whosyncs displays a list of all machines and classes that contain 'filename'

cisync

cisync copies a file from the local machine to Synctree classes. Since it uses Synctree to make the copy, administrators can be sure that the mode, type, and ownership of the file will be the same in the template, as on the local disk.

cosync

cosync gets a particular file out of Synctree. Used when you only want one or two files, and a full Synctree run would be too expensive.

addclass

addclass installs a new Synctree class on a machine.

Normally, the way to do this would be to add the class to the /etc/hostconfig file and run a syncnode. However, if you only want to add the functionality provided by the new class, there's no need to check all the files on the entire machine against the template.

rmclass

rmclass removes Synctree classes when you no longer want them on your machine.

Taking a class out of a machine's hostconfig file makes sure that the class will not sync again, but rmclass goes the extra step of removing files installed by the class.

In the interests of being careful, the output of rmclass is a list of shell commands, which you can either pipe into sh for execution or save to a file for examination.

The Big Picture: A Usage Scenario

At the University of Michigan's College of Literature, Sciences, and the Arts, Synctree and the core classes are maintained by a small group of sysadmins employed by the College. Most machine installs are done by sysadmins employed by various departments within the College. With Synctree in place, the departmental sysadmin's activities tend to revolve around it.

For example, when a new desktop machine arrives from Sun, the sysadmin uses Jumpstart to load the OS and Synctree. When the machine reboots, it has guessed (from its fully qualified domain name) which classes to install. The sysadmin double-checks this list, inputs her AFS password and the machine syncs. When it reboots, it will be a fully configured and operational workstation, already tailored to her department's needs. This operation - which can be quite tedious without automation - has taken only five to fifteen minutes of her time.

If the machine was a server, she might have added classes to install imapd and qpopper. She might have to generate a sendmail.cf file, configure DNS, or perform several other tweaks. The advantage of Synctree is that she will *not* have to worry about getting out of date versions of her mail software, an insecure version of BIND, or any of the other common bugaboos for sysadmins. She will not have to build any software from source, or get permission from a higher authority. In fact, she could deploy an entire department's worth of machines, and the expert sysadmins working for the College would never have to worry.

Comparison to Other System Configuration Systems

A detailed comparison to all systems would be impossible. However, with so much prior art in the field, a little elaboration on Synctree's unique advantages seem appropriate.

Many of the systems in common use (notably rdist(1)) require a server to initiate a sync. In environments where not every sysadmin trusts every other, such systems require additional automation scripts to be written in order to manage privileges. Synctree uses file system permissions to control access to the templates, and only the root user on the client machine can order a sync. Only root on the client can control which classes the client syncs to. The only parties the client needs to trust are those with administrator access to the file system the templates live on - in AFS, those in the system:administrator group.

Systems that rely on a central server also face performance bottlenecks. Even high-end machines can only carry on an rdist to a certain number of machines. Using rdists' binary comparison mode for security drops this down even further. The only central service required by Synctree is a file server. Synctree downloads roughly five megabytes of SI files from the server, and all further work (except for copying files from AFS) happens entirely on the client. This means that much smaller demands are placed on the file server than would be placed on the rdist server. Furthermore, the use of precomputed SI files for the template, instead of forcing the server to examine each file in the template as it checks them, provides an enormous savings.

Many systems depend on complete images being distributed. Synctree, like GNU cfgengine, allows images to be overlaid by one another, to any desired resolution. As a result, a client can build up a picture in memory of the finished system before any changes are made. Using overlays without the ability to resolve conflicts in memory, "thrashes" the machine and versions of the same file are replaced by one another. For example, we once used an AFSMODS Synctree class to install an AFS Kerberos-aware login program. The default Solaris login was superceded by this version while the SI files were being merged in memory. Rdist would require two different dists, with a window of time between them when the wrong version was installed on the client.

Synctree is modular and open. The only configuration file is /etc/hostconfig, where a machine's classes are listed. Otherwise, almost all changes can be made by simply copying files. Every step of Synctree's abstraction, from the initial "I type syncnode and it works" level down to watching file-by-file comparison, can be observed and understood by anyone who can read a man page and shell scripts. Other systems that hide more of their functionality, or depend on more complex grammar, can be more difficult to learn and troubleshoot.

Finally, we argue that basing the system on a file server has fundamental architectural superiorities. If the goal of a configuration system is to distribute the correct version of several hundred files to hundreds or thousands of clients, it makes sense to rely on software that has been designed for providing files on that scale. Other common concerns for configuration systems, such as security and scalability, have already been addressed and solved for file systems. Finally, many large sites already have file servers scaled to match their clients. Using this existing resource can save quite a bit of worry later on.

Reflection, Surprise, Terror, for the Future

We at the College of LS&A have been working on an update to this system that will allow Synctree to take software packages that are normally on the network, and install them on the system's local hard drive. The motivation for this, is that you usually get a new system today, where the hard drive will hold the OS around four times over. In many cases, you end up with at least a Gigabyte of space left over which may be wasted.

This idea will pull traffic off of the network and put it onto the hard drive of the local machine. By doing this, we will end up with a faster system all around. The machines with the large hard drives will load applications locally, not off the network, and the other machines, with smaller hard-drives, will no longer have to contend with the others over that network bandwidth to get at the application.

One obviously desirable feature for Synctree would be a way to sync partial files. For example, a class for installing imapd would work best if it could also add the appropriate lines to /etc/services and /etc/inetd.conf. We've discussed doing this with either M4 macros or an implementation of the GNU cfgengine, but have yet to do any actual planning or work toward this goal.

Existing Synctree systems assume that AFS is available for file distribution. While the authors rather like AFS, we understand that not everyone has it or wants it. Synctree could easily be adapted to synccp, krcp, or any other secure client-initiated copying system. The major bottleneck is that such a system requires acls, so that clients may copy down files that are only readable by root, but cannot make changes *as* root, or (probably) grab the server's /etc/shadow file. If anyone is interested in working on this, Jason Larke may be able to provide some support.

Acknowledgments

Thanks to Rik Farrow for his support, and for prodding John into thinking that this presentation would be of interest to anyone. Also, thanks to all the UofM Unix Admins that gave the encouragement to sit down and type up this paper.

Availability

Synctree is available for non-commercial use, and is the property of the Regents of the University of Michigan, Ann Arbor, MI. Synctree (v.2) can be sampled from: ftp.math.lsa.umich.edu/pub/Synctree/.

Author Information

John Lockard is the Systems Administrator for the Department of Mathematics in the College of LS&A at the University of Michigan. He is thinking of switching to the Emacs paintball team. After some thinking, is really glad that Jason talked him out of titling the paper MS-SMS (Multi System-Synctree Management System). John can be reached by mail at:
Department of Mathematics
2072 East Hall - B746
525 East University Avenue
Ann Arbor, Michigan 48109-1109
He can be reached by email at <jlockard@umich.edu>.

Jason Larke is a Systems Administrator at ANS Communications in Ann Arbor, Michigan. He was the lead Unix Administrator for the College of LS&A previously being a member of The University of Michigan's fabled F-Five. He will never forgive himself for missing the Emacs vs Vi paintball game. Jason can be reached by mail at:
ANS CO+RE
880 Technology Drive
Ann Arbor, Michigan 48108
He can be reached by email at <jlarke@ans.net>.

References

[Anderson] Anderson, Paul, "Toward a High-Level Machine Configuration System," LISA VIII Proceedings, 1994.

[Evard] Evard, Remy, "An Analysis of UNIX Machine Configuration," LISA XI Proceedings, 1997.

[Fisk] Fisk, Michael, "Automating the Administration of Heterogeneous LANs," Tenth USENIX System Administration Conference, 1996.

[Harlander] Harlander, Dr. Magnus, "Central System Administration in a Heterogeneous Unix Environment: GeNUAdmin," LISA VIII Proceedings, 1994.

[Hideyo] Hideyo, Imazu, "OMNICONF - Making OS Upgrades and Disk Crash Recovery Easier," LISA VIII Proceedings, 1994.

[Shaddock] Shaddock, Michael E. and Mitchell, Michael C. and Harrison, Helen E., "How to Upgrade 1500 Workstations on Saturday, and Still Have Time to Mow the Yard on Sunday," LISA IX Proceedings, 1995.