Twelfth Systems Administration Conference (LISA '98)
December 6-11, 1998
Refereed Papers Track
Invited Talks Track
The Great Certification Debate
Invited Talks Track II
Advanced Topics Workshop
SAGE Community Meeting and Candidates Forum
The Evolution of Open Source Software
Eric Allman, Sendmail, Inc.
Summary by Carolyn M. Hennings
The original author of sendmail, Eric Allman, has been actively
involved in all aspects of software development. He provided an
authoritative view of the history, current state, and possible future
of what we now call open-source software.
Allman reminded us that the first implementations of computer systems
were mainframes that required special facilities and staff. These
systems came with the source code for the operating system, and all
configuration information was in the source code. The next systems to
become available were minicomputers designed originally for lab
environments. These systems were less expensive and required less
support, but the users were more sophisticated. Some commercial
software started coming on the scene, but most software was free, and
the source code was available. With both the mainframe and minicomputer
systems, the hardware blueprints were available.
The advent of microcomputers changed the characteristics of the
computer-user community. Microcomputers were affordable and available
to the hobbyist. As the hardware became more and more available and
software became easier and easier to use, the demand for systems
increased. With the availability of cheap, reliable hardware, the need
for blueprints and specifications decreased. At the same time, the
demand and need for knowledge about the internal workings of the
computer software decreased. These factors led us to our current
situation of cheap hardware and expensive software.
The recent movement toward open-source software has been built on the
desire and contributions of avid technologists who were frustrated by
the inability to access, understand, and optimize the software. The
expansion of the Internet and the resulting ability quickly and easily
to share code has provided a forum for this community to develop
software on "Internet time."
Allman referred to "The Cathedral and The Bazaar" article
<http://www.tuxedo.org/~esr/writings/cathedral-bazaar/> by Eric S. Raymond comparing the
commercial centralized software-development process to a cathedral and
the decentralized development of open-source software on the Internet
to a bazaar. Allman suggested that a moderator is necessary for truly
successful open-source software development. The "bazaar" analogy does
not extend to this level. He described a number of new company models
supporting the open source philosophy in different ways:
- Give away old source code, sell new.
- Develop and give away source code, sell service.
- Market and give away source code, sell service.
- Develop and give away base set of source code, sell ease-of-use
extensions, other tools, and service.
- Develop and give away parts, sell most as closed source.
- Develop and give away source, place restrictions on the
subsequent use of the code.
Allman's answer to the question, "Why give it away?" is that
"religious" arguments don't make good business. Sendmail, Inc.
continues giving away the source code in order to maintain market
share, ensure universal interoperability, and improve the quality of
the code. He commented that the volunteer individuals likely to support
and contribute to open-source code development are naturally focused on
the "interesting" and "cool" work. However, commercially viable
software requires development that might be less exciting, such as GUIs
for configuration and/or management tools. His proposal is to develop
and sell these extensions as a commercial package, while continuing to
release the base software in the open-source arena.
Allman predicted that features-based code will tend toward open source,
while algorithm-based code will generally stay closed. He also
suggested that open-
protocol-based code will tend toward open source and that,
subsequently, mission-critical applications will tend toward open
REFEREED PAPERS TRACK
Summary by Kurt Dillard
Dan Farmer, Earthlink Network; Brad Powell, Sun Microsystems, Inc.;
Matthew Archibald, KLA-Tencor
Titan is a freely available host-based tool that can be used to audit
or improve the security of a UNIX system. It started as a Bourne Shell
script to reconfigure various daemons. Checks for verifying
configurations were added, and over time Titan became an effective tool
for auditing computers. The authors made it clear that this is a
powerful tool not designed for the weak or timid sysadmin. Using it
incorrectly, you could easily render a system unusable or even
unbootable. For the SA willing to put in the time to learn Titan
thoroughly, it can save a great deal of time while helping to verify
and maintain security across multiple hosts. The authors also made it
clear that Titan is not the be-all and end-all of information-systems
security; it is designed to be only part of the overall infrastructure.
Titan now runs on most versions of Solaris, but it shouldn't be too
difficult to port the scripts to other flavors of UNIX. By editing the
scripts you can reconfigure Titan so that it performs auditing and
configuration changes appropriate to the type of host you are running
it on and the security policies that your network requires.
Infrastructure: A Prerequisite for Effective Security
Bill Fithen, Steve Kalinowski, Jeff Carpenter, and Jed Pickel, CERT
The authors started their presentation with some scary data compiled by
CERT. A 1997 survey shows that 50% of systems were not kept up to date
with security patches after they were compromised. One site
appeared in 35 incidents between 1997 and 1998; the site was used for
password sniffing and probing of other sites in many of those cases.
Ten of the 35 incidents involved root compromise of the host. In
another break-in, 20-25 hosts were compromised. All of these systems
needed to be rebuilt, but the site's administrator said that they
didn't have enough resources to do so. The authors set out to improve
infrastructure manageability at CERT by creating an easily maintained
system of distributing software packages. The result is SAFARI, a
centralized repository of 900 collections of software for multiple
versions of UNIX. Using SAFARI, a sysadmin can build new systems from
scratch and update existing systems with patches and new packages.
SAFARI includes flexible version controls so that developers and admins
can easily post and retrieve software packages from the same central
SSU: Extending SSH for Secure Root Administration
Christopher Thorpe, Yahoo!, Inc.
Christopher Thorpe needed to create a low-cost method of allowing
distributed access to privileged operations. As a system administrator
at Harvard University, he had 200 systems to manage, and many of the
students, staff, and faculty required root access to one or more
processes on those systems. A method was needed to allow these users to
execute certain processes as root in a secure environment. SSU combines
SSH and Perl to create a system that allows this by combining RSA key
pairs with those commands. Unfortunately, users need a separate key for
each command that they need access to, but the solution works at
Harvard because most of the users need to execute only a few commands
with root access. A side-benefit of SSU is that everyone is now using
SSH for all of their console connections, making all network activity
Session: Pushing Users and Scripts Around
Summary by Allen Supynuk
System Management with NetScript
Apratim Purakayastha and Ajay Mohindra, IBM T.J. Watson Research
NetScript is a BASIC-like (could also be Perl- or Tcl-like) scripting
language for remote administration of heterogeneous systems (UNIX,
Wintel, and soon PDAs like the PalmPilot). It includes neat features
like parallel scripts, isolation, and disconnected operation.
Single Sign-On and the System Administrator
Michael Fleming Grubb and Rob Carter, Duke University
This presentation did a nice job of covering the major interpretations
and issues involved in single sign-on, but, not surprisingly, was a bit
short on satisfying solutions. The paper is worth reading for anyone
interested in this holiest of grails.
Session: Storage Performance
Summary by Carolyn M. Hennings
Using Gigabit Ethernet to Backup Six Terabytes
W. Curtis Preston, Collective Technologies
Curtis Preston presented his paper and some interesting additional
information. He talked about two different backup systems he
implemented and some of the things he learned along the way. One
conclusion from these experiences was that the limitation in backing up
this amount of data in a reasonable timeframe was really in the
network. He suggested that private "storage-area networks" will be the
Configuring Database Systems
Christopher R. Page, Millennium Pharmaceuticals
Christopher Page described relational databases from a
system-management viewpoint. Relational databases reside on computer
systems that are managed by system administrators. The system
administrator needs to be knowledgeable about how the database works
with, and uses, the operating system. Page began with information about
the relational-database architecture. From a database user's
perspective, data is conceptualized in tables; it is accessed and
controlled through the Structured Query Language (SQL), and data
manipulation is transaction-oriented. From the relational-database
server perspective, it is an operating system on top of the native
operating system, it manages the steps in processing requests, and it
maintains four different file types (data, log, temporary, and control
files). Issues that system administrators need to be aware of and work
with the database administrators on include: setting up and monitoring
memory structures; optimizing "Intimate Shared Memory" (ISM); and
configuring swap space, CPU, and network usage. Page noted that some
key things to be aware of are data-block size, raw versus filesystem
disk usage, and synchronous I/O.
Session: Distributed Computing
Summary by Carolyn M. Hennings
Design and Implementation of an Administration System for
Distributed Web Server
C. S. Yang and M. Y. Luo, National Sun Yat-Sen University
This presentation, winner of the Best Student Paper award, described a
system for managing distributed Web servers. The components of the
system include a control interface, a controller, broker, agent, and
remote console GUI. The control interface is used by system
administrators for turning distribution on and off, adding and removing
nodes, managing Web content, and reading statistical information. The
controller is a Java application that runs in the background on the
distributor and responds to system-administration requests. The broker
is a standalone Java app on each server that consists of an agent and
monitoring thread. The agent performs the delegated task on requested
nodes such as adding and deleting files, searching for a file, and
analyzing log files. The remote console is used for executing
management operations and interacting with the controller. The
presenters described the system as extensible and applicable to any Web
Summary by John Talbot
MRTG -- The Multi Router Traffic Grapher
Tobias Oetiker, Swiss Federal Institute of Technology
MRTG is one of the more valuable free tools in use today in thousands
(a very conservative guesstimate) of network and Internet sites around
the globe. Tobias Oetiker, MRTG's creator, maintainer, and chief cook
and bottlewasher, originally created MRTG in the summer of 1994 using
his free time on a hobby project to analyze network traffic at the
Montfort University. In the short time since then, MRTG has established
its place as an essential tool for network monitoring.
Unfortunately Oetiker is ending his support of MRTG because of personal
and professional time constraints. He delivered this news with a touch
of sarcasm and modesty by stating that he "sometimes wonder[s] if no
one can program" since it is "just a Perl script." Oetiker has done an
excellent job of developing and maintaining the MRTG code base, and his
decision will mean a sad loss for the progression of this marvelous
For the meat and potatoes of the technical discussion, Oetiker
concentrated on new improvements to the round-robin database (RRD)
management of the latest release of MRTG (called MRTG-3). RDD
enhancements include the ability to store multiple data sources in
parallel and a break between the database-storage and the
graph-generation interface, which is now called rrdtool. He
noted that these improvements alone have greatly improved the
performance of the MRTG data-logging capabilities.
There was some discussion about using other databases, such as Oracle,
to manage the databasing (RRD) and graphical functions
(rrdtool) of MRTG for large datasets and networks. Oetiker was
not sure if this would provide any major performance advances over the
current method. RRD uses a "lossy database" topology in which older
data is statistically averaged over spreading sample rates as data
ages. Using this method of data management, the RRD is able to maintain
a fixed-sized dataset.
While the latest performance enhancements are great news about
advancement in the MRTG package, it is sad news that Tobias is limiting
his future involvement with MRTG.
Wide Area Network Ecology
Jon T. Meek, Edwin S. Eichert, and Kim Takayama, American Home
This was a good nuts-and-bolts discussion of how to improve your WAN
performance. Jon Meek and his team concisely and effectively described
practical and innovative solutions for analyzing and enhancing network
performance at the American Home Products Corp.
Monitoring basics, such as tcpdump and Perl scripting, were some of the
methods used to measure WAN performance of the round-trip times (RTT),
committed information rates (CIR), and reliability of these parameters
across numerous WAN direct- and virtual-circuit connections. For more
unusual problems, Meek and his team looked directly at packets and
protocols on the WAN and performed system monitoring of process
utilization and file sizes to gather more process data. Often,
network-performance measurements and diagnostics were limited in their
detail by the complexity of the private section of the frame-relay
cloud of their WAN provider. Some solutions to this dilemma were to
obtain circuit and network maps from their frame-relay provider(s) and
insist on read-only access to the SNMP port of the frame-relay routers
to be able to quickly map and identify stops and bottlenecks in the WAN
Other core essentials to network performance were also checked.
Redundant successive database queries, large PostScript files, email
attachments, file sharing over the WAN, and SQL network applications
were major sources of bandwidth utilization. Hard disks were
implemented on many of the network printers to handle redundant headers
and footers, such as company logos and graphics. This was a
particularly novel idea since it enabled local caching of redundant
data at the destination point (the printer). It was found that Web
applications used far less bandwidth than the SQL applications, since
the Web applications needed to transmit only the interface instead of a
client/server database link.
Meek and his team took advantage of a packet prioritization determined
by protocol and "weighted fair queueing" to lessen the severity of
"bandwidth hogs." Although the use of access-control lists (ACLs) and
special packet handling introduces extra load on the network routers,
it enabled interactive network sessions to take precedence in the
network bandwidth, thereby giving the appearance of better response by
online applications such as Telnet and Web interfaces.
In one instance, a WAN circuit was upgraded from a CIR of 128KBps to
256KBps and the RTTs markedly increased, degrading network performance.
It was found that the frame-relay provider actually routed the new
connection upgrade over a more complex set of routes in the WAN just to
get the "faster" circuit connected to Meek's site. If Meek's team did
not statistically and periodically analyze network-performance links,
they would have had no idea where the trouble was, since their
frame-relay provider did not do any network performance monitoring and
analysis, only up/down-time status. Meek's analysis data was enough to
give his frame-relay provider the impetus to get working on the
solutions. Meek stated that many frame-relay providers have this same
Further analysis of WAN RTTs and RTTs of Internet-bound connections
showed a much lower RTT for Internet connections than for WAN
connections. This left a few questions about the possibility of using
Virtual Private Network (VPN) technology routed over the Internet to
handle some of the currently poor-performing WAN links.
Automatically Selecting a Close Mirror Based on Network Topology
Giray Pultar discussed issues involved in automatically redirecting
queries to an HTTP server on the basis of an HTTP client's proximity to
the nearest HTTP mirror server.
Automated mirror selection would be of great value to sites that have
multiple Internet presence points across large geographical areas. An
automated mirror-selection service or tool would provide a
single-presence appearance to the client host and greatly reduce the
need for a user to manually choose a mirror site from a lead page or
Giray made suggestions for implementing such a system and noted some
defects of both client- and server-side implementations of close-mirror
selection. Java (software overhead) and traceroute (routing
difficulties due to propagation delays and return-trip connections to
the client host through an origin firewall) are limited as client-side
solutions. In addition, the traceroute method could conceivably add
large delays for sites with numerous mirroring sites.
Giray's approach to solving the close-mirror problems is to build a
mirror table of known networks and relate them to the geographically
"closest" mirror. Such a table, if based on all IP network address
combinations, would be massive and difficult to construct. How can each
network be correctly identified and categorized? How can physical
"closeness" be determined when so many ISPs have multiple redundant
links and dynamic failover routes?
By identifying collections of networks as autonomous systems, the
definition of "which mirror to use" becomes less complicated. The
Internet routing registry (IRR) databases are then used to compile a
correlation table which a server can use to redirect a client to a
"closer" site based on the client's IP number. Two scripts were
developed for building the close-mirror tables. Script
closest.cgi is the CGI interface to be called by the server to
determine the "nearest" location. The script mkmirrortable
contacts the IRR databases and compiles the correlation table.
Some caveats still apply to the development of this technology. There
were questions about "what should be" and "what is" when using AS
paths, since there is no hard relation between an AS path and a real
physical network route. Also, expansion delays depend on the
geographical span and size of a particular AS, which can vary in real
physical size and distance within the definition of the AS itself.
Other issues, such as load balancing and mirror-site downtime, pose
several challenges to the development of this new technology which,
once solved, will prove to have highly useful applications not
primarily limited to HTTP redirection.
The close-mirror package can be found at
Summary by Carolyn M. Hennings
What to Do When the Lease Expires: A Moving Experience
Lloyd Cha, Chris Motta, Syed Babar, and Mukul Agarwal, Advanced
Micro Devices, Inc.; Jack Ma and Waseem Shaikh, Taos Mountain, Inc.;
Istvan Marko, Volt Services Group
Chris Motta related his experience in moving approximately 1,000
machines and 220 users from a single building into two different
buildings. He listed some of the things that helped the move go
smoothly. This list included a well-defined scope, using email to
communicate during the planning phase, organization and planning, a
central command center, allowing extra time for unforeseen problems,
blanket purchase orders with key vendors, new networks staged and
tested in advance, and insisting that managers and users were not
present during the move. Some things that hindered the operations were
insufficient checking of scripts and lists, inaccurate audit from a
vendor, poor estimate and execution from movers, poor
estimate/execution from a fileserver vendor, lack of working phone
lines, weekly bureaucratic meetings too far in advance of the move, and
not getting enough sleep during the move itself. Motta made the
following suggestions: Have independent verification of scripts and
audit; have a single person in charge of controlling the entire move
and making key decisions; have laptops with network cards available for
use as terminal emulators; have cellular phones and/or radios available
for everyone, and plenty of spare cables and adapters.
Anatomy of an Athena Workstation
Thomas Bushnell, BSG; Karl Ramm, MIT Information Systems
Thomas Bushnell described an academic-computing environment with
approximately 30,000 users and 1,000 workstations. The workstations are
located in public clusters -- libraries and hallways -- as well
as in faculty and staff offices and dorm rooms. They are all standard
UNIX workstations configured for a single user and serial reuse.
Bushnell described the concept of "lockers" -- storage areas
specified for a particular use such as a home directory, packages of
software, or common areas for collaborative efforts by groups of
people. The "lockers" support the release cycle for operating systems
and software updates. A group of "system packs" made up of "lockers"
comprise the operating system and other software layered on top of the
OS. Machines are identified as parts of clusters that determine at what
point in the release cycle new software will be loaded. An
"auto-update" facility allows for these lockers of software to be
loaded as the machines are booted. The presenters concluded with the
following observations: the security model gives clarity; the serial
reuse model presents problems with time sharing and long-running batch
jobs; and the hands-off auto-update and installation allows a team of
10 system administrators to support the 30,000 users and 1,000
Bootstrapping an Infrastructure
Steve Traugott, Sterling Software and NASA Ames Research Center;
Joel Huddleston, Level 3 Communications
Steve Traugott provided an insightful overview of the steps for
creating and managing a solid infrastructure. Some key steps are
determining how version control is going to be done and setting up a
"gold server." With this foundation, other infrastructure elements such
as installation tools, directory and authentication services, network
fileservers, client file access, and configuration management can be
implemented. These steps and others are detailed in the paper along
with a graphic describing the order in which they should be performed.
This architecture has advantages in disaster recovery, software
distribution, and lowering total cost of ownership. Traugott concluded
the presentation with the observation that when it comes to defining an
infrastructure, the role being filled is larger than "system
administrator" and might more accurately be called "system architect."
Session: Distributing Software Packages
Summary by Chastity D. Arthur
mkpkg: A Software Packaging Tool
Carl Staelin, Hewlett-Packard Laboratories
Carl Staelin has developed a remarkable tool to allow software
publishers to easily create installation packages. Staelin pointed out
that the industry has focused on the end users and systems
administrators, allowing them to easily install and uninstall software,
and has not focused on the first step, the software distributor who has
to create the binary installation package. With Staelin's tool,
mkpkg, the software distributor can add a description of the
package, develop manifests, include certain dependencies, create
install and uninstall scripts, and customize the post-installation.
mkpkg can take as little as three minutes to complete,
provided the software is ready for distribution.
Staelin addressed the portability of mkpkg. It was developed
on HP-UX and uses HP-UX-specific commands. He has successfully ported
mkpkg to ninstall, update, and SD-UX. His next conquest will be RPM,
but his work has slowed for lack of time. mkpkg is available
SEPP -- Software Installation and Sharing System
Tobias Oetiker, Swiss Federal Institute of Technology
Tobias Oetiker and his IT support group (ISG) saw the need to provide a
software-installation tool that would reduce the repetitive task of
installing software and configuring the systems throughout the various
departments in the institute. They were also looking to develop a tool
that would retain some independence in the installations. The ISG
tested software-distribution tools already on the market, comparing Red
Hat's Package Manager, GNU Stow, Depot-Lite, and LUDE, to name a few.
None of these tools met their requirements, nor did they use wrapper
scripts -- but the ISG did discover that in a mix of all these
tools and a few of their own ideas lay exactly the features they
needed. Thus SEPP came into existence. SEPP provides both a clean
system for system managers to use and a user-friendly environment. It
is currently supported only on Solaris and Irix.
SEPP includes a number of system-management features. The subdirectory
tree provides clean encapsulation to all files of the same
distribution; a special directory (called SEPP) in each software
subdirectory houses a description of the contents along with the
startup wrapper script, start.pl; the automounter tool, using
/usr/pack, helps to ensure paths during compiles; the
packages' binaries are actually symbolic links to
/usr/sepp/bin, which points to stub scripts; Perl scripts
start up the wrapper script; and a unique name field is generated for
each software-package distribution. Oetiker was not only very proud of
SEPP's system-management features, he also highlighted the user
features. One of SEPP's most convenient features is that the user only
needs to add /usr/sepp/bin to the PATH variable. The ISG also
developed both Web-based and manual-page-based SEPP documentation. SEPP
also allows for multiple versions of the same software distribution
using suffixes appended to the executable names.
With SEPP's reliance on the automounter, user applications that are
required during bootup will cause problems. The ISG is currently
addressing that issue by adding a feature that enforces the bootup
applications to be mirrored to the local machine.
SEPP is distributed under the GNU General Public License and can be
obtained from <http://www.ee.ethz.ch/sepp>, where there is also
information on the SEPP mailing list.
Synctree for Single Point Installation, Upgrades, and OS Patches
John Lockard, University of Michigan; Jason Larke, ANS
Synctree is a system-administration tool developed for a large network
requiring frequent OS or software updates and security patches. Lockard
and Larke had two goals in mind -- system security and uptime.
Synctree is capable of holding the network's complete configuration in
a secure, readable format. The idea behind Synctree is to bring a
machine up on the network and "sync" it to the templates for the
architecture so defined.
Comparing Synctree to cloning, the authors stated that although a clone
could be made that meets your requirements, each time an update is
added to that architecture a new clone would have to be established.
Another comparison was made to rdist in that rdist relies on each
machine being up and connected to the network when you run your update.
Under the direction of Paul Howell, the University of Michigan's
Computer Aided Engineering Network group wanted to create a utility
that provides verification of widely distributed patch installations
and ensures that files prone to hackers are in their expected state.
Synctree's template permissions are based at the client level, and only
the root user of that client can call a sync and order the classes the
client syncs to. Synctree relies on a server, and any other work is
copied in downloads to the clients. Synctree also allows images to
overlay each other, like GNU's cfengine. With
this feature, the client actually builds the final picture before
implementing any changes. Synctree has only one configuration file,
each class of machines is listed. Synctree goes down to the level of
In closing their presentation, Lockard and Larke talked of future
features they would like to incorporate into Synctree. One update for
the near future is allowing Synctree to install software packages
normally found on the network to the local hard drive. Currently
Synctree relies on AFS, which not everyone has or wants; a future goal
is to adapt Synctree to another secure copying system, such as krcp.
A Synctree sample is available for noncommercial use only at
Session: Mailing Lists
Summary by Brian Kirouac
Mailman: The GNU Mailing List Manager
John Viega, Reliable Software Technologies; Barry Warsaw and Ken
Manheimer, Corporation for National Research Initiatives
Have you ever subscribed to a list and later realized that you forgot
what type of mailing list it was and how to unsubscribe? As a list
owner it would be nice to add a footer to each message that describes
the process for unsubscribing. Viega, Warsaw, and Manheimer wanted to
add just such a footer to a majordomo mailing list. This worked fine
for individual messages. The problem was each message with its footer
was put in the digest, so there were multiple copies of the footer in
They started looking at different mailing-list-management software
packages for something that would allow the user to subscribe or
unsubscribe quickly and easily, and would allow the list owner to
manage the list. MajorCool was considered "cool" but limited.
Mailman came of this. Mailman offers a Web-based user interface that
allows list management on three levels: user, list, and site. It
includes email-based commands, but the Web based interface is the
driving force. A user or owner can subscribe or unsubscribe from a list
as well as choose between live and digest modes. A list owner can edit
the list's Web page and set various list options.
Drinking from the Fire(walls) Hose: Another Approach to Very Large
Strata Rose Chalup, Christine Hogan, Greg Kulosa, Bryan McDonald,
Bryan Stansell, Global Networking and Computing, Inc.
Strata Rose Chalup presented the authors' experience moving the
"Firewalls" mailing list. The original server used large ISPs to do
mail relaying instead of doing the delivery itself. When GNAC took over
the list, it did not have the same relationships with ISPs, and thus
the new server had to deliver the mail.
The typical two-queue system did not function well enough. The outbound
queue was growing faster than mail was getting delivered. The problem
was that majordomo was creating a single sendmail queue file generated
with 4000+ addresses in the RCPT line.
They created a Perl program run every five minutes out of cron, called
qsplit. This takes the original queue file and splits it up
into easier-to-spool chunks. Each chunk having a specified number of
recipients, they chose 25. To keep the uniqueness of queue file names,
each chunk has a sequence number appended to the original name. These
are then spread through 10 different queue directories.
Each queue directory runs a separate instance of sendmail to process
the queue. A process called spawn is responsible for keeping these
sendmail processes running. Spawn is smart enough to keep the system
busy but not have it swapping. This way as much mail is delivered as
fast as possible.
Request v3: A Modular, Extensible Task Tracking Tool
Joe Rhett, Navigist
This was a presentation on some of the modifications and extensions
that have been made to Request.
Some of the problems of not having a good tracking tool are: task
history is usually stored in human RAM, thus prone to loss; handoffs
are not always handled well; there's little or no information to
justify staff. Requirements of a good task tracking tool are: track
entire history of task; do not slow down admins who are using the tool;
support almost any operating system or platform; work well from remote;
be easy for untrained users to access; acquire statistics.
Commercial applications may fit some of these requirements, but they
are expensive, require a lot of training, and don't usually support all
platforms. Free applications are generally not updated often, require
UNIX-like skills, and don't always have Web and email interfaces.
The previous versions of Request had several problems. First, they were
not year-2000 compliant. Parts of the code aggregated Perl 5, and small
changes required many fixes. Most problems related to dispersion. The
design goals of the new request were to fix these problems and to allow
others to add code easily. It actually resulted in fewer lines of code.
INVITED TALKS TRACK
Zero to LISA in One Year
Brent Chapman, Covad Communications Company
Summary by Chastity D. Arthur
Brent Chapman explained the successful and unsuccessful decisions made
as the Silicon Valley startup, Covad, coped with its one-year growth
from one region with 50 people to six regions with 400 people. He was a
member of the IT department, faced with continually scaling and
supporting the network and responding to systems demands. He discussed
the ongoing process of planning every detail possible and attempting a
proactive approach to situations. "No plan survives reality, but it's a
start," he said.
A startup company must recognize the challenges, both obvious and
hidden. The obvious issues are: keeping up with the growth; getting
ahead of the growth; attracting and retaining top talent; developing
adequate and scalable systems; maintaining daily development support;
introducing new tools, services, and concepts to the users; and
developing a strong infrastructure not just for IT but for the entire
company. Some hidden challenges are: the linear rise in number of hours
required; growth of users' expectations; loss of volunteers in the IT
department; and old users being more self-sufficient than newer users.
In Covad's case, company culture was also a factor. When Covad was
smaller, communication was always at peak performance; as the company
grew, disseminating information became more of a challenge, and it
became harder for older employees to find time to work with new ones.
Chapman discussed the considerations that went into site selection
(Silicon Valley offered many advantages) and described the headaches
and heartaches of surveying site after site. He explained how IT
planned for the move once a location was chosen. What better way than
to set up a MOCR, a NASA-like Missions Operation Control Room. Chapman
described in depth how the success of the MOCR enhanced the success of
Covad's move. The first decision was to hire trusted contractors to
augment the staff. Chapman then armed everyone with a radio and
appointed flight directors in a rotation that meant the MOCR always had
a manager. The flight plan was simple -- make all critical
decisions in advance. A successful move was completed in one weekend,
and there was still time to leave welcome packages in the cubicles. IT
left a welcome note for the employee, a map of the new building,
including restrooms and printer locations, plus a little treat. The
MOCR remained open as a help desk through the first official busy day.
Chapman described the MOCR as a "great centralized success."
Chapman then discussed the one demand that kept arising -- users
wanting more bandwidth. Although bandwidth is the first to be blamed,
it is often not the problem. IT should help the executives understand
the cost of more bandwidth, help the users understand how to use
software or choose their software, and explain the differences between
latency and bandwidth.
Chapman ended his talk with why he chose to join Covad as a startup
company. He wanted to be a part of something that could be successful
and to have the opportunity to work with outstanding and experienced
people, interesting technology, and vendors. He closed with the
observation that maximum productivity is nowhere equal to the maximum
number of hours worked.
Got LDAP? Deploying and Using the Lightweight Directory Access
Leif Hedstrom, Netscape Communications Corporation
Summary by Brian Kirouac
Is your directory information starting to become overwhelming? Someone
suggests LDAP, the Lightweight Directory Access Protocol, so you start
looking at the documentation. One of the first things to hit you is
that deploying and managing a directory server is a complicated task.
Leif Hedstrom of Netscape gave a good talk dealing with some of the
issues and pitfalls associated with installing a new LDAP-based system,
based on some of the issues Netscape faced when installing its LDAP
Before designing anything, you need to establish your goals. The first
is easy dissemination of information. Two other concerns should be the
scalability and performance of your server. Scalability and performance
have a direct impact on how to design your database tree structure.
During the design and implementation, you need a manager who will back
you, and you need to make sure you have all departments involved. It
was amazing how much input the legal department had in Netscape's
implementation. Several legal concerns can influence what data you
might be able to include. Pictures, home phone numbers, and car license
plates, for example, might be problematic.
Useful guidelines for planning and implementing LDAP: spend time
planning, analyzing, and testing the design; select a
directory-information tree that is as simple as possible; elect the
proper software based on your needs.
Succumbing to the Dark Side of the Force: The Internet as Seen from
an Adult Website
Dan Klein, Cybertainment, Inc.
Summary by Josh Simon
Dan Klein gave effectively the same talk as he had at USENIX '98 in New
Orleans, without displaying any defensiveness about the fact that he is
the technical person for a dozen pornography Web sites. He went over
some of the technical issues for maintaining such a site, and noted
that porn sites tend to have better security and adult-verification
than some banks. The talk was very well attended. (No, he didn't show
pictures; the talk was PG-13.)
On the technical side, Klein talked about techniques to reduce the load
on a Web server: load sharing, load shedding, and load boosting. Load
sharing is basically using DNS entries in a round-robin fashion to
distribute the load. The main issue with this is making sure that all
of the servers have the same data. Load shedding requires a front-end
server that hands off initial requests to back-end servers that have
the real content. The problem here, again, is keeping everything in
sync. Load boosting is performed on the client side. A lot of sites
make their money based on the number of hits a given URL receives. Thus
windows that access the same URL. Load boosting consists of turning off
thus reducing the amount of time it takes to load a page.
A good practice he mentioned is keeping logs. Logs help plan for the
future, and they help determine possible security breaches. And, in the
case of legal action, they can help cover you if someone falsely
accuses you of something.
Branchstart -- A Generic, Multi-OS Installation Server
Rory Toma, WebTV Networks, Inc.
Summary by Chastity D. Arthur
Rory Toma described his successful implementation of a
single-architecture yet multi-OS network installation server on
Intel-based platforms. His project isn't actually named Branchstart; he
is playing on the name of Sun's product, Jumpstart. Toma calls his
implementation MOSIP, an image- and package-based OS installer,
successfully tested on Red Hat Linux 5.2, NT 4 Workstation, NT 4
Terminal Server, and Windows 95.
Toma's project goals included: minimal user interaction, 100 percent
predictability, easy scalability, and functionality at a junior level.
He commented that MOSIP is reproducible, flexible, and fast to install.
On the more technical side, MOSIP has a binary failure mode; operators
can use the same install server for multiple OSes or platforms; and a
serial console or GUI is optional. Not so inviting is the amount of
front-end work and the level of knowledge and experience needed to set
up advanced installations.
Toma described how he made MOSIP come together. For each OS base
needed, a template machine must be installed to acquire the OS image
with dd and to record software- or hardware-specific parameters. He
then described what he terms "laying down the bits" -- basically,
booting Linux with NFS root filesystem and having an installation
script run automatically. Toma chose to replace init with his own
script. To finish, he described "modifying the bits." This is the point
at which the administrator would modify the IP address, create
auto-login scripts, and install LILO.
In closing, Toma discussed his next proj-ect, Internet OS Installation
Server Project (IOSISP), which will take MOSIP to the next level:
installation of free OSes from the Internet. He plans to automate the
installation of MacOS and Windows 2000 to include Active Directory and
Exchange. He would like his project to function on nonIntel hardware,
specifically for NT and Linux. His greatest challenge, he said, is
creating a generic NT image that would allow modification to suit a
wider range of hardware.
To learn more about MOSIP or IOSISP, visit
Moderator: Rob Kolstad. Panelists: Phil Scarr, GNAC; Leeland G. Artra, University of Washington; Linda True, TRW Space and
Electronics; Bruce Alan Wynn, Collective Technologies
Summary by Carolyn M. Hennings
If there is a hot topic in the SAGE community, it is definitely
certification. The issue has been in the air for a number of years, and
the SAGE Executive Committee decided it was time to take action toward making a decision to
either pursue or drop it. A SAGE Certification Subcommittee was formed,
and subsequently a Certification Advisory Council was created.
The purpose of "The Great Certification Debate" was to have a serious
discussion about the certification issue. Rob Kolstad asked the
panelists to introduce themselves and speak to their positions on the
issue. On the pro side of the discussion were Bruce Alan Wynn and Lynda
True. The cons were Phil Scarr and Leeland Artra.
Bruce Alan Wynn expressed his opinion in light of the SAGE charter of
advancing systems administration as a profession. The certification
project will help with the definition of our system-management
standards and then define the requirements for certification. Bruce
reminded the audience that SAGE uses the term "guild" in its name,
referring to a structure in which more experienced people help the less
experienced. The certification process would provide guidance and
direction to individuals who don't know where to start. He concluded by
saying that there are a number of different ways to do certification,
that some are better than others, and that SAGE needs to do it right.
Phil Scarr reiterated concerns expressed in his ;login: article
("When Worlds Collide," August 1997). He suggested that the best
administrators come from university programs where there is an
education focus rather than a certification focus. Experience is a
better indicator of ability than certification. Certification is touted
as a way to hire but is not effective.
Lynda True explained that her organization has been certifying UNIX
administrators for approximately two years. Management recognized
inadequate system administration support and lack of training to be
potential threats to information security and Internet availability.
Although the process has been painful, some benefits have been that
hiring has become easier and salaries have risen. She suggested that
the peer-review portion of the certification process was an important
Leeland Artra commented that certification might be good if done
correctly. He commented that most vendor-sponsored certification
programs have little value. He expressed concerns regarding the
difficulty of managing a certification program. A focus on education
should be a priority over certification.
Questions and comments from the audience expressed concerns regarding
how the certification process might work. Numerous people brought up
the issue of education and how it relates to certification. Suggestions
included looking at the certification processes used by the medical,
project-management, and aviation professions. An important point
brought up by one participant was that if SAGE doesn't certify
system-management professionals, some other organization will. If that
happens, we run the risk of having to live with something that isn't
the best and doesn't quite work.
INVITED TALKS TRACK
Security as Infrastructure
Tom Perrine, San Diego Supercomputer Center
Summary by Kurt Dillard
Tom Perrine convincingly asserted that an effective system
administrator must address security at all seven network layers as well
as two others, the economic and political layers of your organization.
He summarizes effective approaches as "building fences" while everyone
else stays busy "shooting rabbits." A long-term solution to effective
security will take time to implement, and a few "rabbits" may get in
while you build it, but the final result will be much more satisfactory
and manageable. Perrine suggests that you undertake implementing a
secure infrastructure by first defining goals -- figure out what
needs to be secure and rank those items by importance. Then decide how
you will respond to different types of attacks, automate dealing with
unsophisticated attempts, and don't even waste time investigating them
personally. Automate dealing with other types of probes and analyze the
data personally. Figure out what attacks are so sophisticated that you
will have to deal with them personally from start to finish. Basically,
this is risk analysis -- figuring out what is important, what the
threats are, and how much time you need to spend protecting what's
important from the different types of hosts. By the way, automate or
die, because if you cannot scale your solution it will fail as your
Perrine then reviewed current attack methods and ways to deal with
them. What's being used? Every approach you have ever heard of. Even
the weaknesses with known fixes are effective for hacking because most
sites never bother to apply the correct fixes properly. The biggest
threats are automated exploits that allow attackers to probe numerous
hosts very quickly, and any type of authentication that requires a
password transmitted as plain text.
To deal with these attacks you need to automate as much as possible in
your environment. Implement CfEngine or SMS to allow for quick host
installs and simplified patches and upgrades. Set a security policy and
automate a method of auditing all hosts to ensure that they are meeting
your policy. Segregate the weak hosts from the rest of the network by
placing them behind filtering routers. Automate the detection of
anomalies and as much as possible automate resolutions for those
anomalies. Don't forget to educate everyone who uses your network;
"social engineering" is a very effective method for hacking sites.
Finally, remember that you have to win the support of your users and
your management chain. Build your secure infrastructure slowly so that
your users have a chance to get used to it and so that you have time to
verify that everything is working at each step.
David Kensiski, David Kuncicky, Daniel Klein, Lee Damon, & Matt
Teaching Systems Administration
Chair: Lee Damon, QUALCOMM, Inc. Panelists: David Kensiski, Digital
Island, Inc.; David Kuncicky, Florida State University; Daniel Klein,
USENIX Association; Matt Shibla, Montgomery Blair High School
Summary by Chastity D. Arthur
Consensus at this practicum was that no one has all the answers to
teaching system administration, and one particular method is not going
to solve this issue. Only a combination of school programs, extension
programs, in-house training,
on-the-job training, and vendor courses is the answer for today.
David Kensiski appeared to be in favor of a combination of vendor
courses and in-house training. His answers were clear and concise. One
audience member asked him what he did with a junior employee who just
wasn't grasping the concepts. He politely stated that there was really
only one option -- either find them something
they can do in another area or terminate them. (This really only brings
up another question, what if it's the teaching method?)
David Kuncicky thinks universities need a way of defining the
appropriate levels of system administration in order to advance a
student, and he compared the teaching methods and coursework with those
for programming languages. Kuncicky presented his department's current
course design. FSU's program is a mix of books, courses taught on
campus by FSU instructors, and third-party trainers. He defined the
program by step levels in Systems Administration Proper, SA Tools, SA
Networking, and SA Applications. Kuncicky also mentioned that what may
separate the different universities is state legislation that mandates
the number of hours, which led FSU to a system-administration master's
track. He was not necessarily an advocate of the master's track, but
FSU's goal was simply to get a program started.
Daniel Klein, tutorial coordinator for USENIX, brought up a major
concern of the system-administration industry today: How do you know
when or which training is appropriate? He discussed the assumptions
that must be made prior to proceeding with any type of training. Klein
mentioned that a certain amount of prior experience here and there is
needed and that the potential trainee must be self-driven, willing to
read, use references, and know when to ask for assistance. Klein was an
excellent communicator and he immediately grabbed the audience when he
said, "Training does not fix your problems." He followed that up with a
concise statement that system administrators can be taught with the
see-one, do-one, teach-one theory. He pointed out that indeed vendor
training feeds the curriculum, but it is also a general assumption that
the instructor knows the course materials but may not necessarily be an
expert on the subject. This is what makes USENIX's curriculum flexible;
the instructors are the gurus and as technology changes, advances, or
grows, so will the course.
The last panelist, Matt Shibla, was there to discuss the Maryland
Virtual High School established through Montgomery High School. This is
an online curriculum downloaded from Cisco Network Academy to the high
school's private network. Shibla stated he felt the program was 60
percent generic and 40 percent vendor-related. The main issue that
arose from the audience was that this type of course is limited and
teacher-deficient. Shibla pretty much agreed and did not offer any
insight into developing the program further other than trying to get
This practicum was not set up to provide the answers but to give the
panelists and audience ideas on different directions they could take in
designing their own training methods. The audience was primarily
focused on the motivation of trainees and teachers and the management
issue of "you train them, they leave."
Network Administration and Remote Computing
Moderator: Rob Kolstad. Panelists: Pete Lynch, Jyra Research; Shaula
Yemini; Oljad Singh, System Management Arts
Summary by Kurt Dillard
All three speakers presented information about the network-monitoring
tools that their companies have developed. Oljad Singh's approach is to
focus on critical services and warn of impending and existing problems.
His company has installed a server, running their monitoring software,
that executes ghost transactions every few minutes on each server being
monitored. The monitoring software times each transaction and over time
learns what the "normal" performance signature is of each application
on each server at different times of the day and different days of the
week. When several transactions in a short period of time fall too far
outside of the "normal" signature, the monitoring software
automatically notifies an operator of a possible problem.
Pete Lynch promoted a product from Jyra that proactively monitors
defined business and performance goals. He asserted that service-level
agreements don't adequately measure the user's experience, while Jyra's
monitoring solution does so by measuring the response time of
applications using a scheduled polling agent to see how each
application server is performing. Their scalable solution utilizes
distributed Java agents and provides automatic reporting, summaries,
and exception alarms.
Shaula Yemini has another proactive realtime management tool, SMARTS,
which diagnoses problems before they have had an impact on the network,
allowing them to be resolved before users realize that anything has
gone awry. SMARTS creates a "codebook" that combines generic models for
each network object with the customer's specific topology to create
"problem signatures." When the problem signatures are detected the
system raises an alert. Her firm already has models for a wide variety
of network hardware, and the codebook can be created by connecting
their system to some common network-management systems such as OpenView
and NetView. The codebook is automatically updated every time the
models or topology change.
Coordinator: Peg Schafer, Harvard University
Summary by John Talbot
The WIPs Practicum was more than the advertised "pithy" display of
current techniques and issues. It was a nonstop blitz, in concise
10-minute samples, of interesting and thought-provoking descriptions of
"real-world" problems and solutions. The WIPs were pitted against one
another with a real whip bestowed upon the winner, who was determined
by group applause.
Steven Nelson's "Multiplatform Storage Area Network" WIP
was a nuts-and-bolts discussion of managing a network of over 1.6 TB of
database and data sources in a 24x7 environment. Nike's original
fiber-channel network was initially spread out over all storage areas,
but problems persisted because of shared data paths for both the backup
and data-delivery services. They have used the EMC array and parallel
pipes to have transparent access to data dumps without interfering with
network and system resources. They still have large-size backups and
multiple filesystems for data storage. They are looking into ways of
using the Veritas volume manager filesystems on their EMC array to have
a commonly mountable local vxvm filesystem between heterogeneous
John Buckman<firstname.lastname@example.org> presented a unique
implementation of email as a mission-critical application. Instead of a
centralized MTA and MDA, Lyris opted for a thin multithreaded mail
daemon that uses SQL as the message store. This enables his site to
utilize the text searching and user access of the existing SQL
structures to manage email. Also, they are able to deliver an email
interface using HTML and Tcl more easily with the existing SQL
Lowell Snyder<email@example.com>presented another good email
implementation. (I might be a little biased since I was a cohort of
this WIP, but it did come in a close second.) Lowell presented some of
the work done at Parametric Technology to remove the standard UNIX
aliasing from the central MTA and institute LDAP hooks into the
internal mail exchangers to directly deliver mail based on the
company's managed LDAP database. A motive was that a majority of users
were consolidated on a central POP/IMAP MDA, and the management of
simple UNIX mail aliases became cumbersome as the user base grew to
over 4,000. Snyder described the code changes required on the sendmail
and LDAP daemons necessary to invoke the features and handle several
exceptions for whitespace and parsing symbols not generally understood
or interpreted by general sendmail parsing rules.
Tom Limoncelli<firstname.lastname@example.org> presented "Tricks you can
do when your firewall
is a bridge." This WIP stemmed from a project Limoncelli had at Bell
Labs/Lucent when cutting over backbone routers to new firewalls to the
Internet. The problem was originally complicated by the fact that he
didn't have access to all the routers in the schema and still needed to
implement the firewall changes transparently. Of note was the fact that
when the firewall acts as a bridge and doesn't have an IP number, it is
less likely to be subject to hacking attempts since it has no
TCP/IP-bound interface. See <http://www.bell-labs.com/usr/tal>.
Charles Tatum<email@example.com> of the U.S. Navy
presented "Computationally Expensive Intrusion Prevention." He focused
mainly on his modifications to the popular Crack program. He surmised
that a massive amount of time was being used by the Crack code to test
less probable circumstances of password guessing. His new approach was
to apply common rules first. For example, he found that most people
don't use more than one word for their password, so applying a number
of permutation rules to the password guessing before trying "obvious"
guesses makes running Crack computationally expensive. Also,
implementing a separate dictionary of names ahead of the standard
dictionary will weed out more obvious guesses. Reducing the salt
guesses also lowers process runtime. Tatum's approach manipulated the
original Crack encrypt and compare looping structure from
Dict*Rules*Users*Salts to Rules*Dict*Users*Salts and implemented his
common-guess strategies to greatly lower computing times.
Geoff Halprin<firstname.lastname@example.org>of SysAdmin
Group, winner (by a hair over Lowell Snyder) of the WIP prize, gave a
stunning talk about the "Taxonomy of Best Practice." This was one of
the clearest talks classifying and describing what sysadmins do that I
have heard in a long while. Halprin not only explored the need for
sysadmins to understand their
own personal technical capabilities, but also explained that the user
community needs to understand what sysadmins do and of what they are
capable so thatsysadmins can be properly matched to the environments
they support. He defined system-administration core competencies as
control, organization, protection, optimization, and planning, and he
included a diagrammatic breakdown of each of these characteristics. He
established models for organizing these responsibilities on a set of
five levels, with three to five of those levels being consistent across
all platforms. The five levels are determined by a Capability Maturity
Model by which certification is based upon core competencies. This WIP
would be a great basis for a future LISA invited talk.
Michael Ewan<email@example.com> of Tektronics described
using LDAP to create printer definitions and determine printers by
class (e.g., color, resolution, paper size) and location. He described
how the current workstation environment DISPLAY variable and user
profiles could be used to reference the "nearest" user-default printer.
He has also looked into ways to use the LDAP information base to manage
printers. His environment eased implementation by its homogeneous use
of Tektronics printers and the standardization of UNIX workstations.
Andrew Hume<firstname.lastname@example.org> and Tom Scola
from AT&T Labs presented "How to Handle Microsoft Attachments" in
UNIX email. They wanted to have a UNIX-based reader that would be able
to read email attachments in realtime. To reduce the need for
specialized windowing software, they devised a plan that sends the
Microsoft attachment to an NT system that runs a PostScript converter
and sends it back to the Xwindows-based MTA. Unfortuna-tely, time was
running short at this point of the WIP session and many of the details
were abbreviated before Hume could complete his WIP.
Summary by Douglas Stewart
Variable Length Subnet Masks on TCP/IP Networks
The basics of Class A/B/C and their corresponding netmasks, and
reserved classes for testing and internal use, were covered. Using all
of your addresses in a single network is wasteful, especially if your
company has a Class A address allocated to it and has its network
broken up into geographically separate chunks. By modifying your
netmask, you can break up your network into smaller, separate pieces
that can be routed separately. A useful example is an ISP that resells
large number of T1s. Typically you'll have a subnet composed of only
the router on each side. By setting your netmask to 255.255.255.252,
you break your network into 64 subnetworks with four addresses, which
include a network address, the broadcast address, and two usable host
addresses you can use for the routers. Routing protocols and commands
for troubleshooting routing problems were suggested. Some light reading
(RFCs 950, 1918, and 1878) was suggested.
Most of this presentation was on new features of BSDI 4.0. Filesystem
code updates: soft update (delays certain file operations for speedup,
very temporary files may never be created); even out update; 64-bit
file offsets; mount options to disable access time updates; mount
options for sync/asynch writes; and soft read-only. Networking
improvements: PCB lookup hashing; IP address hashing for fast virtual
hosting; per-address IP statistics; and kernel-level packet filter.
IPv4 enhancements: slow start; congestion avoidance; multi-cast; large
windows; MTU discovery; and IPSEC. IPv6 support! Includes SAMBA,
IP/IPX, Novell 3.x file and print services, VPN support. NFS: v2 and v3
support over UDP or TCP; NFS lock daemon. New network media: frame
relay. SMP: performance improvements (user-level processes show best
improvements); threads are all user-level for now but the kernel will
be threaded in the future. New hardware improvements include bootable
CD-ROMs and plug-and-play Ethernet, modem, and sound cards. Binaries
are now ELF, and the math libraries have been proved. There's a console
debugger and trace facility called Kdebug and KTR. Things to look for
in the future: Linux binary compatibility, Java application
environment, finer-grained SMP with kernel threads, SPARC port, ATM,
and channelized T1 and T3 support.
The people who attended this BOF were almost entirely AFS users from
university environments. The first topic was release dates for Linux
and NT-based AFS servers -- apparently in February 1999. Transarc,
which has been bought by IBM, was a topic for heavy discussion. It has
a new CEO, has changed its mind about dropping the development of AFS,
and has opened a London office. Something else people were looking for
was Kerberos 5 support. There were complaints about poor support from
Transarc, especially with the 5 pm EST closing time that is
inconvenient for West Coast customers. KNFS was discussed as something
that people were experimenting with and had had some success with. In
the end, people had little faith in most of the alternatives to AFS
(NFS, DFS, CacheFS) and felt that the advantages of AFS outweighed the
problems they had encountered.
ADVANCED TOPICS WORKSHOP
Adam Moskowitz, Facilitator
Rob Kolstad, Co-chair and Scribe
Summary by Josh Simon
We first went around the room introducing ourselves, the quantity of
users and the quantity and type of hosts we supported (whether
individually or as part of a team), and two to five topics we wanted to
discuss during the day. We came up with ranges of up to 10,000 users,
10,000 PCs, 2,000 Macs, and 3,000 UNIX hosts of various flavors. Other
notes were multiple terabytes of disk storage (with projected
short-term growth to exceed a petabyte), strange printer requirements,
and extremely high growth rates (up to 400% a year).
We determined that we wanted to talk, in general, about:
* Consistency/standardization in sysadmin practices as
* Cool system-administration tools and paradigms
* Specific hot technologies/paradigms to prepare for/crystal ball
First, we discussed the issue of internal consistency and
standardization in technical practices. We tried to look at the
"problem," but realized that we all had slightly different ways of
looking at it -- which was not surprising considering that we had 31
people in the room, all with different backgrounds and experiences. We
seemed to agree in general that creating standards is challenging,
enforcing them is a hard problem, and that there are many more
variables than may be obvious at first look. The concept of a
"taxonomy" or categorization of problems into areas seemed to make
sense to a lot of the folks present.
We next had a free-form discussion on cool system-administration tools
and paradigms. Some general comments were:
* DSL is great. (10 people have 56K or more to the home, all but
one of those has 56K or faster bidirectionally; six people have
>128K. Most of these are business-paid and not individual-paid.)
* MRTG (discussed in Tobias Oetiker's paper at the conference) was
hailed as a wonderful network-mapping tool. It uses SNMP polling on a
5-minute interval and creates Web pages with usage graphs. It ages data
appropriately and is freely available. Big Brother, a systems
monitoring package, integrates with MRTG and is also freely available.
* Intrusion-detection systems now are in the same sort of
not-yet-well-understood position as firewalls were a decade ago. While
then we had free firewalls which later became commercial, now we have
commercial intrusion-detection systems (IDS) even though the problem is
neither understood nor solved.
* Turnover can be interesting. 16 people changed jobs at least
once in the past year. Four of these were internal (same company) job
changes. Raises in the new job ranged from 0-90% and seemed to average
around 28%. And 14 people present have open requisitions they are
actively hiring for.
* Enforcing the use of a PDA like the PalmPilot has improved the
follow-through for members of the group. Many folks at one company have
bought one with their own money.
* 19 of those present carry a cell phone; 26 carry pagers; seven
carry authentication devices. A few have two-way pagers; 12
participants pay at least part
of their monthly fees for the portable communications devices.
* Five members use a Ricochet or similar device for wireless
digital communications. 14 more would use it if it were available in
* Seven people have an agreement to attend conferences annually.
In spite of that small number, 10 have some kind of permission to
attend more than two per year. A couple can go to even more if they
have papers presented at them. Everyone pretty much gets at least one
per year. About half can attend two or more per year, depending on
* Some cool utilities are ssh and Curl. LDAP or similar directory
services are on the rise; 11 attendees have this.
* Cordless phones in machine rooms are a major win.
* Tools sometimes die for lack of nurturing. It would be nice if
there were some way to solve that problem (like a MacArthur grant type
of thing). Even finding current versions is too hard. See
<ftp.sage-au.org.au> for lots of sysadmin tools.
Next we discussed hot technologies, rumors, and similar
prognostications. One hot technology we talked about is XML, the
Extensible Markup Language. It is self-verifying, easy to parse, easy
to search, and has a universal file format. It's different from SGML in
that it doesn't include the hard-to-implement features. XML supports
Unicode. Unicode is the next hot technology we see on the horizon. It
represents all characters (including nonRoman alphabets like Cyrillic,
Hebrew, and Farsi). Microsoft Office 2000 uses Unicode; rumor has it
that Word already supports it.
Other predictions are: Voice over IP will be a hot technology soon;
directory services are becoming more important; applications will
support more location independence; voice input and/or recognition will
grow in the next year; and digital camera use will continue to rise.
SAGE COMMUNITY MEETING AND CANDIDATES FORUM
Summary by Carolyn M. Hennings
The annual SAGE Community Meeting had a significantly larger attendance
this year than in the past two years that I've attended. Pat Wilson
kicked off the meeting with announcements, introduced the candidates
for the upcoming Executive Committee elections, and moderated a
Announcements included the status of publications in the "Short Topics"
series. Educating and Training Systems Administrators is in the
mail to SAGE members. Pat provided an update on the certification
debate, saying that an advisory committee has been formed and charged
with the investigation of whether or not SAGE should continue to pursue
the certification issue and how it should be done if SAGE chooses to
move forward. Pat mentioned the efforts of the "Day-in-the-Life" survey
and announced that Rob Ferrell and Brian Kirouac have been named SAGE
Historians. Items to watch for are "How-To Notes," revamped Rosetta
Stone, new booklets on site audits and on hiring system administrators.
Efforts are under way in reviewing the ethics policy and in developing
a mentoring program.
Moving on to the candidates forum, Pat asked the candidates to
introduce themselves, and the floor was opened up for questions.
Question topics included the certification issue, the ethics policy,
education and training, the inclusion of Windows NT topics in the LISA
program, and the market's view of SAGE.
In the BPF format, the SAGE Community Meeting continued under the
guidance of Hal Miller, SAGE Executive Committee President. The
open-forum discussion focused on the professional development of
systems administrators in the sense of gaining skills to communicate
and work with management. David Parter, chair for LISA '99, took many
suggestions for topics for the next conference. The smaller group
meeting provided an informal opportunity to meet other individuals who
are actively involved with SAGE at the national and local levels.
Summary by Carolyn M. Hennings
The 1998 Champion, Daniel Boyd, with Rob Kolstad
At a lot of conferences the attendee population markedly diminishes on
the last day of the conference. LISA is different for one reason. As
always, Rob Kolstad's LISA Quiz Show is a major attraction at the
conclusion of the conference.
This year was no exception. Maintaining the same format from previous
years and always making technical improvements,
the Quiz Show gets better and better. Conference attendees vied to
answer questions in areas such as UNIX administration, the WWW,
computer executives, physics, circus acts, electrical current, coins,
certification questions, and European dictators. In the category of
conductors questions about electrical conductors set the stage, but the
final question was "Who wrote West Side Story?" One never knows
what to expect.
A bonus this year was the "Tournament of Champions." Last year Snoopy
Beagle, who hails from Germany, lost to Hal Pomeranz. Snoopy then
challenged the fairness of the Quiz Show, saying that too many
questions were based on American pop culture. Apparently Rob Kolstad
heard enough feedback over the past year regarding this issue and
decided to rectify the situation.
The "Tournament of Champions" was a contest between this year's winner,
last year's winner, and Snoopy. Notable categories for the final round
included European history, television shows, security, and match the
dictator. In a surprising victory, Daniel Boyd made mincemeat of his
competitors. Who knows what will happen next year!
Tina Darmohray, Winner of the
1998 SAGE Outstanding Achievement Award
Dan Geer & Greg Rose comparing ribbons
Adam Moskowitz & friend, demonstrating the latest LISA dress