Twelfth Systems Administration Conference (LISA '98)

Boston, Massachusetts
December 6-11, 1998

KEYNOTE ADDRESS

The Evolution of Open Source Software

Eric Allman, Sendmail, Inc.

Summary by Carolyn M. Hennings

The original author of sendmail, Eric Allman, has been actively involved in all aspects of software development. He provided an authoritative view of the history, current state, and possible future of what we now call open-source software.

Allman reminded us that the first implementations of computer systems were mainframes that required special facilities and staff. These systems came with the source code for the operating system, and all configuration information was in the source code. The next systems to become available were minicomputers designed originally for lab environments. These systems were less expensive and required less support, but the users were more sophisticated. Some commercial software started coming on the scene, but most software was free, and the source code was available. With both the mainframe and minicomputer systems, the hardware blueprints were available.

The advent of microcomputers changed the characteristics of the computer-user community. Microcomputers were affordable and available to the hobbyist. As the hardware became more and more available and software became easier and easier to use, the demand for systems increased. With the availability of cheap, reliable hardware, the need for blueprints and specifications decreased. At the same time, the demand and need for knowledge about the internal workings of the computer software decreased. These factors led us to our current situation of cheap hardware and expensive software.

The recent movement toward open-source software has been built on the desire and contributions of avid technologists who were frustrated by the inability to access, understand, and optimize the software. The expansion of the Internet and the resulting ability quickly and easily to share code has provided a forum for this community to develop software on "Internet time."

Allman referred to "The Cathedral and The Bazaar" article <https://www.tuxedo.org/~esr/writings/cathedral-bazaar/> by Eric S. Raymond comparing the commercial centralized software-development process to a cathedral and the decentralized development of open-source software on the Internet to a bazaar. Allman suggested that a moderator is necessary for truly successful open-source software development. The "bazaar" analogy does not extend to this level. He described a number of new company models supporting the open source philosophy in different ways:

• Give away old source code, sell new.
• Develop and give away source code, sell service.
• Market and give away source code, sell service.
• Develop and give away base set of source code, sell ease-of-use extensions, other tools, and service.
• Develop and give away parts, sell most as closed source.
• Develop and give away source, place restrictions on the subsequent use of the code.

Allman's answer to the question, "Why give it away?" is that "religious" arguments don't make good business. Sendmail, Inc. continues giving away the source code in order to maintain market share, ensure universal interoperability, and improve the quality of the code. He commented that the volunteer individuals likely to support and contribute to open-source code development are naturally focused on the "interesting" and "cool" work. However, commercially viable software requires development that might be less exciting, such as GUIs for configuration and/or management tools. His proposal is to develop and sell these extensions as a commercial package, while continuing to release the base software in the open-source arena.

Allman predicted that features-based code will tend toward open source, while algorithm-based code will generally stay closed. He also suggested that open- protocol-based code will tend toward open source and that, subsequently, mission-critical applications will tend toward open protocols.

REFEREED PAPERS TRACK

Session: Security

Summary by Kurt Dillard

TITAN
Dan Farmer, Earthlink Network; Brad Powell, Sun Microsystems, Inc.; Matthew Archibald, KLA-Tencor

Titan is a freely available host-based tool that can be used to audit or improve the security of a UNIX system. It started as a Bourne Shell script to reconfigure various daemons. Checks for verifying configurations were added, and over time Titan became an effective tool for auditing computers. The authors made it clear that this is a powerful tool not designed for the weak or timid sysadmin. Using it incorrectly, you could easily render a system unusable or even unbootable. For the SA willing to put in the time to learn Titan thoroughly, it can save a great deal of time while helping to verify and maintain security across multiple hosts. The authors also made it clear that Titan is not the be-all and end-all of information-systems security; it is designed to be only part of the overall infrastructure. Titan now runs on most versions of Solaris, but it shouldn't be too difficult to port the scripts to other flavors of UNIX. By editing the scripts you can reconfigure Titan so that it performs auditing and configuration changes appropriate to the type of host you are running it on and the security policies that your network requires.

See <https://www.fish.com/security/titan.html>.

Infrastructure: A Prerequisite for Effective Security
Bill Fithen, Steve Kalinowski, Jeff Carpenter, and Jed Pickel, CERT Coordination Center

The authors started their presentation with some scary data compiled by CERT. A 1997 survey shows that 50% of systems were not kept up to date with security patches after they were compromised. One site appeared in 35 incidents between 1997 and 1998; the site was used for password sniffing and probing of other sites in many of those cases. Ten of the 35 incidents involved root compromise of the host. In another break-in, 20-25 hosts were compromised. All of these systems needed to be rebuilt, but the site's administrator said that they didn't have enough resources to do so. The authors set out to improve infrastructure manageability at CERT by creating an easily maintained system of distributing software packages. The result is SAFARI, a centralized repository of 900 collections of software for multiple versions of UNIX. Using SAFARI, a sysadmin can build new systems from scratch and update existing systems with patches and new packages. SAFARI includes flexible version controls so that developers and admins can easily post and retrieve software packages from the same central repository.

SSU: Extending SSH for Secure Root Administration
Christopher Thorpe, Yahoo!, Inc.

Christopher Thorpe needed to create a low-cost method of allowing distributed access to privileged operations. As a system administrator at Harvard University, he had 200 systems to manage, and many of the students, staff, and faculty required root access to one or more processes on those systems. A method was needed to allow these users to execute certain processes as root in a secure environment. SSU combines SSH and Perl to create a system that allows this by combining RSA key pairs with those commands. Unfortunately, users need a separate key for each command that they need access to, but the solution works at Harvard because most of the users need to execute only a few commands with root access. A side-benefit of SSU is that everyone is now using SSH for all of their console connections, making all network activity more secure.

Session: Pushing Users and Scripts Around

Summary by Allen Supynuk

System Management with NetScript
Apratim Purakayastha and Ajay Mohindra, IBM T.J. Watson Research Center

NetScript is a BASIC-like (could also be Perl- or Tcl-like) scripting language for remote administration of heterogeneous systems (UNIX, Wintel, and soon PDAs like the PalmPilot). It includes neat features like parallel scripts, isolation, and disconnected operation.

Single Sign-On and the System Administrator
Michael Fleming Grubb and Rob Carter, Duke University

This presentation did a nice job of covering the major interpretations and issues involved in single sign-on, but, not surprisingly, was a bit short on satisfying solutions. The paper is worth reading for anyone interested in this holiest of grails.

Session: Storage Performance

Summary by Carolyn M. Hennings

Using Gigabit Ethernet to Backup Six Terabytes
W. Curtis Preston, Collective Technologies

Curtis Preston presented his paper and some interesting additional information. He talked about two different backup systems he implemented and some of the things he learned along the way. One conclusion from these experiences was that the limitation in backing up this amount of data in a reasonable timeframe was really in the network. He suggested that private "storage-area networks" will be the future direction.

Configuring Database Systems
Christopher R. Page, Millennium Pharmaceuticals

Christopher Page described relational databases from a system-management viewpoint. Relational databases reside on computer systems that are managed by system administrators. The system administrator needs to be knowledgeable about how the database works with, and uses, the operating system. Page began with information about the relational-database architecture. From a database user's perspective, data is conceptualized in tables; it is accessed and controlled through the Structured Query Language (SQL), and data manipulation is transaction-oriented. From the relational-database server perspective, it is an operating system on top of the native operating system, it manages the steps in processing requests, and it maintains four different file types (data, log, temporary, and control files). Issues that system administrators need to be aware of and work with the database administrators on include: setting up and monitoring memory structures; optimizing "Intimate Shared Memory" (ISM); and configuring swap space, CPU, and network usage. Page noted that some key things to be aware of are data-block size, raw versus filesystem disk usage, and synchronous I/O.

Session: Distributed Computing

Summary by Carolyn M. Hennings

Design and Implementation of an Administration System for Distributed Web Server
C. S. Yang and M. Y. Luo, National Sun Yat-Sen University

This presentation, winner of the Best Student Paper award, described a system for managing distributed Web servers. The components of the system include a control interface, a controller, broker, agent, and remote console GUI. The control interface is used by system administrators for turning distribution on and off, adding and removing nodes, managing Web content, and reading statistical information. The controller is a Java application that runs in the background on the distributor and responds to system-administration requests. The broker is a standalone Java app on each server that consists of an agent and monitoring thread. The agent performs the delegated task on requested nodes such as adding and deleting files, searching for a file, and analyzing log files. The remote console is used for executing management operations and interacting with the controller. The presenters described the system as extensible and applicable to any Web site.

Session: Networking

Summary by John Talbot

MRTG -- The Multi Router Traffic Grapher
Tobias Oetiker, Swiss Federal Institute of Technology

MRTG is one of the more valuable free tools in use today in thousands (a very conservative guesstimate) of network and Internet sites around the globe. Tobias Oetiker, MRTG's creator, maintainer, and chief cook and bottlewasher, originally created MRTG in the summer of 1994 using his free time on a hobby project to analyze network traffic at the Montfort University. In the short time since then, MRTG has established its place as an essential tool for network monitoring.

Unfortunately Oetiker is ending his support of MRTG because of personal and professional time constraints. He delivered this news with a touch of sarcasm and modesty by stating that he "sometimes wonder[s] if no one can program" since it is "just a Perl script." Oetiker has done an excellent job of developing and maintaining the MRTG code base, and his decision will mean a sad loss for the progression of this marvelous tool.

For the meat and potatoes of the technical discussion, Oetiker concentrated on new improvements to the round-robin database (RRD) management of the latest release of MRTG (called MRTG-3). RDD enhancements include the ability to store multiple data sources in parallel and a break between the database-storage and the graph-generation interface, which is now called rrdtool. He noted that these improvements alone have greatly improved the performance of the MRTG data-logging capabilities.

There was some discussion about using other databases, such as Oracle, to manage the databasing (RRD) and graphical functions (rrdtool) of MRTG for large datasets and networks. Oetiker was not sure if this would provide any major performance advances over the current method. RRD uses a "lossy database" topology in which older data is statistically averaged over spreading sample rates as data ages. Using this method of data management, the RRD is able to maintain a fixed-sized dataset.

While the latest performance enhancements are great news about advancement in the MRTG package, it is sad news that Tobias is limiting his future involvement with MRTG.

Wide Area Network Ecology
Jon T. Meek, Edwin S. Eichert, and Kim Takayama, American Home Products Corporation

This was a good nuts-and-bolts discussion of how to improve your WAN performance. Jon Meek and his team concisely and effectively described practical and innovative solutions for analyzing and enhancing network performance at the American Home Products Corp.

Monitoring basics, such as tcpdump and Perl scripting, were some of the methods used to measure WAN performance of the round-trip times (RTT), committed information rates (CIR), and reliability of these parameters across numerous WAN direct- and virtual-circuit connections. For more unusual problems, Meek and his team looked directly at packets and protocols on the WAN and performed system monitoring of process utilization and file sizes to gather more process data. Often, network-performance measurements and diagnostics were limited in their detail by the complexity of the private section of the frame-relay cloud of their WAN provider. Some solutions to this dilemma were to obtain circuit and network maps from their frame-relay provider(s) and insist on read-only access to the SNMP port of the frame-relay routers to be able to quickly map and identify stops and bottlenecks in the WAN architecture.

Other core essentials to network performance were also checked. Redundant successive database queries, large PostScript files, email attachments, file sharing over the WAN, and SQL network applications were major sources of bandwidth utilization. Hard disks were implemented on many of the network printers to handle redundant headers and footers, such as company logos and graphics. This was a particularly novel idea since it enabled local caching of redundant data at the destination point (the printer). It was found that Web applications used far less bandwidth than the SQL applications, since the Web applications needed to transmit only the interface instead of a client/server database link.

Meek and his team took advantage of a packet prioritization determined by protocol and "weighted fair queueing" to lessen the severity of "bandwidth hogs." Although the use of access-control lists (ACLs) and special packet handling introduces extra load on the network routers, it enabled interactive network sessions to take precedence in the network bandwidth, thereby giving the appearance of better response by online applications such as Telnet and Web interfaces.

In one instance, a WAN circuit was upgraded from a CIR of 128KBps to 256KBps and the RTTs markedly increased, degrading network performance. It was found that the frame-relay provider actually routed the new connection upgrade over a more complex set of routes in the WAN just to get the "faster" circuit connected to Meek's site. If Meek's team did not statistically and periodically analyze network-performance links, they would have had no idea where the trouble was, since their frame-relay provider did not do any network performance monitoring and analysis, only up/down-time status. Meek's analysis data was enough to give his frame-relay provider the impetus to get working on the solutions. Meek stated that many frame-relay providers have this same problem.

Further analysis of WAN RTTs and RTTs of Internet-bound connections showed a much lower RTT for Internet connections than for WAN connections. This left a few questions about the possibility of using Virtual Private Network (VPN) technology routed over the Internet to handle some of the currently poor-performing WAN links.

Automatically Selecting a Close Mirror Based on Network Topology
Giray Pultar

Giray Pultar discussed issues involved in automatically redirecting queries to an HTTP server on the basis of an HTTP client's proximity to the nearest HTTP mirror server.

Automated mirror selection would be of great value to sites that have multiple Internet presence points across large geographical areas. An automated mirror-selection service or tool would provide a single-presence appearance to the client host and greatly reduce the need for a user to manually choose a mirror site from a lead page or site.

Giray made suggestions for implementing such a system and noted some defects of both client- and server-side implementations of close-mirror selection. Java (software overhead) and traceroute (routing difficulties due to propagation delays and return-trip connections to the client host through an origin firewall) are limited as client-side solutions. In addition, the traceroute method could conceivably add large delays for sites with numerous mirroring sites.

Giray's approach to solving the close-mirror problems is to build a mirror table of known networks and relate them to the geographically "closest" mirror. Such a table, if based on all IP network address combinations, would be massive and difficult to construct. How can each network be correctly identified and categorized? How can physical "closeness" be determined when so many ISPs have multiple redundant links and dynamic failover routes?

By identifying collections of networks as autonomous systems, the definition of "which mirror to use" becomes less complicated. The Internet routing registry (IRR) databases are then used to compile a correlation table which a server can use to redirect a client to a "closer" site based on the client's IP number. Two scripts were developed for building the close-mirror tables. Script closest.cgi is the CGI interface to be called by the server to determine the "nearest" location. The script mkmirrortable contacts the IRR databases and compiles the correlation table.

Some caveats still apply to the development of this technology. There were questions about "what should be" and "what is" when using AS paths, since there is no hard relation between an AS path and a real physical network route. Also, expansion delays depend on the geographical span and size of a particular AS, which can vary in real physical size and distance within the definition of the AS itself. Other issues, such as load balancing and mirror-site downtime, pose several challenges to the development of this new technology which, once solved, will prove to have highly useful applications not primarily limited to HTTP redirection.

The close-mirror package can be found at <ftp://ftp.coubros.com/pub/lisa98>.

Session: Infrastructure

Summary by Carolyn M. Hennings

What to Do When the Lease Expires: A Moving Experience
Lloyd Cha, Chris Motta, Syed Babar, and Mukul Agarwal, Advanced Micro Devices, Inc.; Jack Ma and Waseem Shaikh, Taos Mountain, Inc.; Istvan Marko, Volt Services Group

Chris Motta related his experience in moving approximately 1,000 machines and 220 users from a single building into two different buildings. He listed some of the things that helped the move go smoothly. This list included a well-defined scope, using email to communicate during the planning phase, organization and planning, a central command center, allowing extra time for unforeseen problems, blanket purchase orders with key vendors, new networks staged and tested in advance, and insisting that managers and users were not present during the move. Some things that hindered the operations were insufficient checking of scripts and lists, inaccurate audit from a vendor, poor estimate and execution from movers, poor estimate/execution from a fileserver vendor, lack of working phone lines, weekly bureaucratic meetings too far in advance of the move, and not getting enough sleep during the move itself. Motta made the following suggestions: Have independent verification of scripts and audit; have a single person in charge of controlling the entire move and making key decisions; have laptops with network cards available for use as terminal emulators; have cellular phones and/or radios available for everyone, and plenty of spare cables and adapters.

Anatomy of an Athena Workstation
Thomas Bushnell, BSG; Karl Ramm, MIT Information Systems

Thomas Bushnell described an academic-computing environment with approximately 30,000 users and 1,000 workstations. The workstations are located in public clusters -- libraries and hallways -- as well as in faculty and staff offices and dorm rooms. They are all standard UNIX workstations configured for a single user and serial reuse. Bushnell described the concept of "lockers" -- storage areas specified for a particular use such as a home directory, packages of software, or common areas for collaborative efforts by groups of people. The "lockers" support the release cycle for operating systems and software updates. A group of "system packs" made up of "lockers" comprise the operating system and other software layered on top of the OS. Machines are identified as parts of clusters that determine at what point in the release cycle new software will be loaded. An "auto-update" facility allows for these lockers of software to be loaded as the machines are booted. The presenters concluded with the following observations: the security model gives clarity; the serial reuse model presents problems with time sharing and long-running batch jobs; and the hands-off auto-update and installation allows a team of 10 system administrators to support the 30,000 users and 1,000 workstations.

Bootstrapping an Infrastructure
Steve Traugott, Sterling Software and NASA Ames Research Center; Joel Huddleston, Level 3 Communications

Steve Traugott provided an insightful overview of the steps for creating and managing a solid infrastructure. Some key steps are determining how version control is going to be done and setting up a "gold server." With this foundation, other infrastructure elements such as installation tools, directory and authentication services, network fileservers, client file access, and configuration management can be implemented. These steps and others are detailed in the paper along with a graphic describing the order in which they should be performed. This architecture has advantages in disaster recovery, software distribution, and lowering total cost of ownership. Traugott concluded the presentation with the observation that when it comes to defining an infrastructure, the role being filled is larger than "system administrator" and might more accurately be called "system architect."

Session: Distributing Software Packages

Summary by Chastity D. Arthur

mkpkg: A Software Packaging Tool
Carl Staelin, Hewlett-Packard Laboratories

Carl Staelin has developed a remarkable tool to allow software publishers to easily create installation packages. Staelin pointed out that the industry has focused on the end users and systems administrators, allowing them to easily install and uninstall software, and has not focused on the first step, the software distributor who has to create the binary installation package. With Staelin's tool, mkpkg, the software distributor can add a description of the package, develop manifests, include certain dependencies, create install and uninstall scripts, and customize the post-installation. mkpkg can take as little as three minutes to complete, provided the software is ready for distribution.

Staelin addressed the portability of mkpkg. It was developed on HP-UX and uses HP-UX-specific commands. He has successfully ported mkpkg to ninstall, update, and SD-UX. His next conquest will be RPM, but his work has slowed for lack of time. mkpkg is available at <https://www.hp1.hp.com/personal/Carl_Staelin/mkpkg>.

SEPP -- Software Installation and Sharing System
Tobias Oetiker, Swiss Federal Institute of Technology

Tobias Oetiker and his IT support group (ISG) saw the need to provide a software-installation tool that would reduce the repetitive task of installing software and configuring the systems throughout the various departments in the institute. They were also looking to develop a tool that would retain some independence in the installations. The ISG tested software-distribution tools already on the market, comparing Red Hat's Package Manager, GNU Stow, Depot-Lite, and LUDE, to name a few. None of these tools met their requirements, nor did they use wrapper scripts -- but the ISG did discover that in a mix of all these tools and a few of their own ideas lay exactly the features they needed. Thus SEPP came into existence. SEPP provides both a clean system for system managers to use and a user-friendly environment. It is currently supported only on Solaris and Irix.

SEPP includes a number of system-management features. The subdirectory tree provides clean encapsulation to all files of the same distribution; a special directory (called SEPP) in each software subdirectory houses a description of the contents along with the startup wrapper script, start.pl; the automounter tool, using /usr/pack, helps to ensure paths during compiles; the packages' binaries are actually symbolic links to /usr/sepp/bin, which points to stub scripts; Perl scripts start up the wrapper script; and a unique name field is generated for each software-package distribution. Oetiker was not only very proud of SEPP's system-management features, he also highlighted the user features. One of SEPP's most convenient features is that the user only needs to add /usr/sepp/bin to the PATH variable. The ISG also developed both Web-based and manual-page-based SEPP documentation. SEPP also allows for multiple versions of the same software distribution using suffixes appended to the executable names.

With SEPP's reliance on the automounter, user applications that are required during bootup will cause problems. The ISG is currently addressing that issue by adding a feature that enforces the bootup applications to be mirrored to the local machine.

SEPP is distributed under the GNU General Public License and can be obtained from <https://www.ee.ethz.ch/sepp>, where there is also information on the SEPP mailing list.

Synctree for Single Point Installation, Upgrades, and OS Patches
John Lockard, University of Michigan; Jason Larke, ANS Communications

Synctree is a system-administration tool developed for a large network requiring frequent OS or software updates and security patches. Lockard and Larke had two goals in mind -- system security and uptime. Synctree is capable of holding the network's complete configuration in a secure, readable format. The idea behind Synctree is to bring a machine up on the network and "sync" it to the templates for the architecture so defined.

Comparing Synctree to cloning, the authors stated that although a clone could be made that meets your requirements, each time an update is added to that architecture a new clone would have to be established. Another comparison was made to rdist in that rdist relies on each machine being up and connected to the network when you run your update. Under the direction of Paul Howell, the University of Michigan's Computer Aided Engineering Network group wanted to create a utility that provides verification of widely distributed patch installations and ensures that files prone to hackers are in their expected state.

Synctree's template permissions are based at the client level, and only the root user of that client can call a sync and order the classes the client syncs to. Synctree relies on a server, and any other work is copied in downloads to the clients. Synctree also allows images to overlay each other, like GNU's cfengine. With
this feature, the client actually builds the final picture before implementing any changes. Synctree has only one configuration file, /etc/hostconfig, where each class of machines is listed. Synctree goes down to the level of file-to-file comparison.

In closing their presentation, Lockard and Larke talked of future features they would like to incorporate into Synctree. One update for the near future is allowing Synctree to install software packages normally found on the network to the local hard drive. Currently Synctree relies on AFS, which not everyone has or wants; a future goal is to adapt Synctree to another secure copying system, such as krcp.

A Synctree sample is available for noncommercial use only at <ftp.math.lsa.umich.edu/pub/Synctree/>.

Session: Mailing Lists

Summary by Brian Kirouac

Mailman: The GNU Mailing List Manager
John Viega, Reliable Software Technologies; Barry Warsaw and Ken Manheimer, Corporation for National Research Initiatives

Have you ever subscribed to a list and later realized that you forgot what type of mailing list it was and how to unsubscribe? As a list owner it would be nice to add a footer to each message that describes the process for unsubscribing. Viega, Warsaw, and Manheimer wanted to add just such a footer to a majordomo mailing list. This worked fine for individual messages. The problem was each message with its footer was put in the digest, so there were multiple copies of the footer in the digest.

They started looking at different mailing-list-management software packages for something that would allow the user to subscribe or unsubscribe quickly and easily, and would allow the list owner to manage the list. MajorCool was considered "cool" but limited.

Mailman came of this. Mailman offers a Web-based user interface that allows list management on three levels: user, list, and site. It includes email-based commands, but the Web based interface is the driving force. A user or owner can subscribe or unsubscribe from a list as well as choose between live and digest modes. A list owner can edit the list's Web page and set various list options.

Drinking from the Fire(walls) Hose: Another Approach to Very Large Mailing Lists
Strata Rose Chalup, Christine Hogan, Greg Kulosa, Bryan McDonald, Bryan Stansell, Global Networking and Computing, Inc.

Strata Rose Chalup presented the authors' experience moving the "Firewalls" mailing list. The original server used large ISPs to do mail relaying instead of doing the delivery itself. When GNAC took over the list, it did not have the same relationships with ISPs, and thus the new server had to deliver the mail.

The typical two-queue system did not function well enough. The outbound queue was growing faster than mail was getting delivered. The problem was that majordomo was creating a single sendmail queue file generated with 4000+ addresses in the RCPT line.

They created a Perl program run every five minutes out of cron, called qsplit. This takes the original queue file and splits it up into easier-to-spool chunks. Each chunk having a specified number of recipients, they chose 25. To keep the uniqueness of queue file names, each chunk has a sequence number appended to the original name. These are then spread through 10 different queue directories.

Each queue directory runs a separate instance of sendmail to process the queue. A process called spawn is responsible for keeping these sendmail processes running. Spawn is smart enough to keep the system busy but not have it swapping. This way as much mail is delivered as fast as possible.

Request v3: A Modular, Extensible Task Tracking Tool
Joe Rhett, Navigist

This was a presentation on some of the modifications and extensions that have been made to Request.

Some of the problems of not having a good tracking tool are: task history is usually stored in human RAM, thus prone to loss; handoffs are not always handled well; there's little or no information to justify staff. Requirements of a good task tracking tool are: track entire history of task; do not slow down admins who are using the tool; support almost any operating system or platform; work well from remote; be easy for untrained users to access; acquire statistics.

Commercial applications may fit some of these requirements, but they are expensive, require a lot of training, and don't usually support all platforms. Free applications are generally not updated often, require UNIX-like skills, and don't always have Web and email interfaces.

The previous versions of Request had several problems. First, they were not year-2000 compliant. Parts of the code aggregated Perl 5, and small changes required many fixes. Most problems related to dispersion. The design goals of the new request were to fix these problems and to allow others to add code easily. It actually resulted in fewer lines of code.

INVITED TALKS TRACK

Zero to LISA in One Year

Brent Chapman, Covad Communications Company

Summary by Chastity D. Arthur

Brent Chapman explained the successful and unsuccessful decisions made as the Silicon Valley startup, Covad, coped with its one-year growth from one region with 50 people to six regions with 400 people. He was a member of the IT department, faced with continually scaling and supporting the network and responding to systems demands. He discussed the ongoing process of planning every detail possible and attempting a proactive approach to situations. "No plan survives reality, but it's a start," he said.

A startup company must recognize the challenges, both obvious and hidden. The obvious issues are: keeping up with the growth; getting ahead of the growth; attracting and retaining top talent; developing adequate and scalable systems; maintaining daily development support; introducing new tools, services, and concepts to the users; and developing a strong infrastructure not just for IT but for the entire company. Some hidden challenges are: the linear rise in number of hours required; growth of users' expectations; loss of volunteers in the IT department; and old users being more self-sufficient than newer users. In Covad's case, company culture was also a factor. When Covad was smaller, communication was always at peak performance; as the company grew, disseminating information became more of a challenge, and it became harder for older employees to find time to work with new ones.

Chapman discussed the considerations that went into site selection (Silicon Valley offered many advantages) and described the headaches and heartaches of surveying site after site. He explained how IT planned for the move once a location was chosen. What better way than to set up a MOCR, a NASA-like Missions Operation Control Room. Chapman described in depth how the success of the MOCR enhanced the success of Covad's move. The first decision was to hire trusted contractors to augment the staff. Chapman then armed everyone with a radio and appointed flight directors in a rotation that meant the MOCR always had a manager. The flight plan was simple -- make all critical decisions in advance. A successful move was completed in one weekend, and there was still time to leave welcome packages in the cubicles. IT left a welcome note for the employee, a map of the new building, including restrooms and printer locations, plus a little treat. The MOCR remained open as a help desk through the first official busy day. Chapman described the MOCR as a "great centralized success."

Chapman then discussed the one demand that kept arising -- users wanting more bandwidth. Although bandwidth is the first to be blamed, it is often not the problem. IT should help the executives understand the cost of more bandwidth, help the users understand how to use software or choose their software, and explain the differences between latency and bandwidth.

Chapman ended his talk with why he chose to join Covad as a startup company. He wanted to be a part of something that could be successful and to have the opportunity to work with outstanding and experienced people, interesting technology, and vendors. He closed with the observation that maximum productivity is nowhere equal to the maximum number of hours worked.

Got LDAP? Deploying and Using the Lightweight Directory Access Protocol

Leif Hedstrom, Netscape Communications Corporation

Summary by Brian Kirouac

Is your directory information starting to become overwhelming? Someone suggests LDAP, the Lightweight Directory Access Protocol, so you start looking at the documentation. One of the first things to hit you is that deploying and managing a directory server is a complicated task.

Leif Hedstrom of Netscape gave a good talk dealing with some of the issues and pitfalls associated with installing a new LDAP-based system, based on some of the issues Netscape faced when installing its LDAP servers.

Before designing anything, you need to establish your goals. The first is easy dissemination of information. Two other concerns should be the scalability and performance of your server. Scalability and performance have a direct impact on how to design your database tree structure.

During the design and implementation, you need a manager who will back you, and you need to make sure you have all departments involved. It was amazing how much input the legal department had in Netscape's implementation. Several legal concerns can influence what data you might be able to include. Pictures, home phone numbers, and car license plates, for example, might be problematic.

Useful guidelines for planning and implementing LDAP: spend time planning, analyzing, and testing the design; select a directory-information tree that is as simple as possible; elect the proper software based on your needs.

Succumbing to the Dark Side of the Force: The Internet as Seen from an Adult Website

Dan Klein, Cybertainment, Inc.

Summary by Josh Simon

Dan Klein gave effectively the same talk as he had at USENIX '98 in New Orleans, without displaying any defensiveness about the fact that he is the technical person for a dozen pornography Web sites. He went over some of the technical issues for maintaining such a site, and noted that porn sites tend to have better security and adult-verification than some banks. The talk was very well attended. (No, he didn't show pictures; the talk was PG-13.)

On the technical side, Klein talked about techniques to reduce the load on a Web server: load sharing, load shedding, and load boosting. Load sharing is basically using DNS entries in a round-robin fashion to distribute the load. The main issue with this is making sure that all of the servers have the same data. Load shedding requires a front-end server that hands off initial requests to back-end servers that have the real content. The problem here, again, is keeping everything in sync. Load boosting is performed on the client side. A lot of sites make their money based on the number of hits a given URL receives. Thus sites will have banners and Javascript programs that pop up other windows that access the same URL. Load boosting consists of turning off Javascript on the client to prevent these other windows from appearing, thus reducing the amount of time it takes to load a page.

A good practice he mentioned is keeping logs. Logs help plan for the future, and they help determine possible security breaches. And, in the case of legal action, they can help cover you if someone falsely accuses you of something.

Branchstart -- A Generic, Multi-OS Installation Server

Rory Toma, WebTV Networks, Inc.

Summary by Chastity D. Arthur

Rory Toma described his successful implementation of a single-architecture yet multi-OS network installation server on Intel-based platforms. His project isn't actually named Branchstart; he is playing on the name of Sun's product, Jumpstart. Toma calls his implementation MOSIP, an image- and package-based OS installer, successfully tested on Red Hat Linux 5.2, NT 4 Workstation, NT 4 Terminal Server, and Windows 95.

Toma's project goals included: minimal user interaction, 100 percent predictability, easy scalability, and functionality at a junior level. He commented that MOSIP is reproducible, flexible, and fast to install. On the more technical side, MOSIP has a binary failure mode; operators can use the same install server for multiple OSes or platforms; and a serial console or GUI is optional. Not so inviting is the amount of front-end work and the level of knowledge and experience needed to set up advanced installations.

Toma described how he made MOSIP come together. For each OS base needed, a template machine must be installed to acquire the OS image with dd and to record software- or hardware-specific parameters. He then described what he terms "laying down the bits" -- basically, booting Linux with NFS root filesystem and having an installation script run automatically. Toma chose to replace init with his own script. To finish, he described "modifying the bits." This is the point at which the administrator would modify the IP address, create auto-login scripts, and install LILO.

In closing, Toma discussed his next proj-ect, Internet OS Installation Server Project (IOSISP), which will take MOSIP to the next level: installation of free OSes from the Internet. He plans to automate the installation of MacOS and Windows 2000 to include Active Directory and Exchange. He would like his project to function on nonIntel hardware, specifically for NT and Linux. His greatest challenge, he said, is creating a generic NT image that would allow modification to suit a wider range of hardware.

To learn more about MOSIP or IOSISP, visit <https://www.munitions.com/rory/MOSIP/welcome.html>.

THE GREAT CERTIFICATION DEBATE

Moderator: Rob Kolstad. Panelists: Phil Scarr, GNAC; Leeland G. Artra, University of Washington; Linda True, TRW Space and Electronics; Bruce Alan Wynn, Collective Technologies

Summary by Carolyn M. Hennings

If there is a hot topic in the SAGE community, it is definitely certification. The issue has been in the air for a number of years, and the SAGE Executive Committee decided it was time to take action toward making a decision to either pursue or drop it. A SAGE Certification Subcommittee was formed, and subsequently a Certification Advisory Council was created.

The purpose of "The Great Certification Debate" was to have a serious discussion about the certification issue. Rob Kolstad asked the panelists to introduce themselves and speak to their positions on the issue. On the pro side of the discussion were Bruce Alan Wynn and Lynda True. The cons were Phil Scarr and Leeland Artra.

Bruce Alan Wynn expressed his opinion in light of the SAGE charter of advancing systems administration as a profession. The certification project will help with the definition of our system-management standards and then define the requirements for certification. Bruce reminded the audience that SAGE uses the term "guild" in its name, referring to a structure in which more experienced people help the less experienced. The certification process would provide guidance and direction to individuals who don't know where to start. He concluded by saying that there are a number of different ways to do certification, that some are better than others, and that SAGE needs to do it right.

Phil Scarr reiterated concerns expressed in his ;login: article ("When Worlds Collide," August 1997). He suggested that the best administrators come from university programs where there is an education focus rather than a certification focus. Experience is a better indicator of ability than certification. Certification is touted as a way to hire but is not effective.

Lynda True explained that her organization has been certifying UNIX administrators for approximately two years. Management recognized inadequate system administration support and lack of training to be potential threats to information security and Internet availability. Although the process has been painful, some benefits have been that hiring has become easier and salaries have risen. She suggested that the peer-review portion of the certification process was an important aspect.

Leeland Artra commented that certification might be good if done correctly. He commented that most vendor-sponsored certification programs have little value. He expressed concerns regarding the difficulty of managing a certification program. A focus on education should be a priority over certification.

Questions and comments from the audience expressed concerns regarding how the certification process might work. Numerous people brought up the issue of education and how it relates to certification. Suggestions included looking at the certification processes used by the medical, project-management, and aviation professions. An important point brought up by one participant was that if SAGE doesn't certify system-management professionals, some other organization will. If that happens, we run the risk of having to live with something that isn't the best and doesn't quite work.

INVITED TALKS TRACK

Security as Infrastructure

Tom Perrine, San Diego Supercomputer Center

Summary by Kurt Dillard

Tom Perrine convincingly asserted that an effective system administrator must address security at all seven network layers as well as two others, the economic and political layers of your organization. He summarizes effective approaches as "building fences" while everyone else stays busy "shooting rabbits." A long-term solution to effective security will take time to implement, and a few "rabbits" may get in while you build it, but the final result will be much more satisfactory and manageable. Perrine suggests that you undertake implementing a secure infrastructure by first defining goals -- figure out what needs to be secure and rank those items by importance. Then decide how you will respond to different types of attacks, automate dealing with unsophisticated attempts, and don't even waste time investigating them personally. Automate dealing with other types of probes and analyze the data personally. Figure out what attacks are so sophisticated that you will have to deal with them personally from start to finish. Basically, this is risk analysis -- figuring out what is important, what the threats are, and how much time you need to spend protecting what's important from the different types of hosts. By the way, automate or die, because if you cannot scale your solution it will fail as your network grows.

Perrine then reviewed current attack methods and ways to deal with them. What's being used? Every approach you have ever heard of. Even the weaknesses with known fixes are effective for hacking because most sites never bother to apply the correct fixes properly. The biggest threats are automated exploits that allow attackers to probe numerous hosts very quickly, and any type of authentication that requires a password transmitted as plain text.

To deal with these attacks you need to automate as much as possible in your environment. Implement CfEngine or SMS to allow for quick host installs and simplified patches and upgrades. Set a security policy and automate a method of auditing all hosts to ensure that they are meeting your policy. Segregate the weak hosts from the rest of the network by placing them behind filtering routers. Automate the detection of anomalies and as much as possible automate resolutions for those anomalies. Don't forget to educate everyone who uses your network; "social engineering" is a very effective method for hacking sites. Finally, remember that you have to win the support of your users and your management chain. Build your secure infrastructure slowly so that your users have a chance to get used to it and so that you have time to verify that everything is working at each step.

PRACTICUM TRACK

David Kensiski, David Kuncicky, Daniel Klein, Lee Damon, & Matt Shibla

Teaching Systems Administration

Chair: Lee Damon, QUALCOMM, Inc. Panelists: David Kensiski, Digital Island, Inc.; David Kuncicky, Florida State University; Daniel Klein, USENIX Association; Matt Shibla, Montgomery Blair High School

Summary by Chastity D. Arthur

Consensus at this practicum was that no one has all the answers to teaching system administration, and one particular method is not going to solve this issue. Only a combination of school programs, extension programs, in-house training, on-the-job training, and vendor courses is the answer for today.

David Kensiski appeared to be in favor of a combination of vendor courses and in-house training. His answers were clear and concise. One audience member asked him what he did with a junior employee who just wasn't grasping the concepts. He politely stated that there was really only one option -- either find them something they can do in another area or terminate them. (This really only brings up another question, what if it's the teaching method?)

David Kuncicky thinks universities need a way of defining the appropriate levels of system administration in order to advance a student, and he compared the teaching methods and coursework with those for programming languages. Kuncicky presented his department's current course design. FSU's program is a mix of books, courses taught on campus by FSU instructors, and third-party trainers. He defined the program by step levels in Systems Administration Proper, SA Tools, SA Networking, and SA Applications. Kuncicky also mentioned that what may separate the different universities is state legislation that mandates the number of hours, which led FSU to a system-administration master's track. He was not necessarily an advocate of the master's track, but FSU's goal was simply to get a program started.

Daniel Klein, tutorial coordinator for USENIX, brought up a major concern of the system-administration industry today: How do you know when or which training is appropriate? He discussed the assumptions that must be made prior to proceeding with any type of training. Klein mentioned that a certain amount of prior experience here and there is needed and that the potential trainee must be self-driven, willing to read, use references, and know when to ask for assistance. Klein was an excellent communicator and he immediately grabbed the audience when he said, "Training does not fix your problems." He followed that up with a concise statement that system administrators can be taught with the see-one, do-one, teach-one theory. He pointed out that indeed vendor training feeds the curriculum, but it is also a general assumption that the instructor knows the course materials but may not necessarily be an expert on the subject. This is what makes USENIX's curriculum flexible; the instructors are the gurus and as technology changes, advances, or grows, so will the course.

The last panelist, Matt Shibla, was there to discuss the Maryland Virtual High School established through Montgomery High School. This is an online curriculum downloaded from Cisco Network Academy to the high school's private network. Shibla stated he felt the program was 60 percent generic and 40 percent vendor-related. The main issue that arose from the audience was that this type of course is limited and teacher-deficient. Shibla pretty much agreed and did not offer any insight into developing the program further other than trying to get teachers certified.

This practicum was not set up to provide the answers but to give the panelists and audience ideas on different directions they could take in designing their own training methods. The audience was primarily focused on the motivation of trainees and teachers and the management issue of "you train them, they leave."

Network Administration and Remote Computing

Moderator: Rob Kolstad. Panelists: Pete Lynch, Jyra Research; Shaula Yemini; Oljad Singh, System Management Arts

Summary by Kurt Dillard

All three speakers presented information about the network-monitoring tools that their companies have developed. Oljad Singh's approach is to focus on critical services and warn of impending and existing problems. His company has installed a server, running their monitoring software, that executes ghost transactions every few minutes on each server being monitored. The monitoring software times each transaction and over time learns what the "normal" performance signature is of each application on each server at different times of the day and different days of the week. When several transactions in a short period of time fall too far outside of the "normal" signature, the monitoring software automatically notifies an operator of a possible problem.

Pete Lynch promoted a product from Jyra that proactively monitors defined business and performance goals. He asserted that service-level agreements don't adequately measure the user's experience, while Jyra's monitoring solution does so by measuring the response time of applications using a scheduled polling agent to see how each application server is performing. Their scalable solution utilizes distributed Java agents and provides automatic reporting, summaries, and exception alarms.

Shaula Yemini has another proactive realtime management tool, SMARTS, which diagnoses problems before they have had an impact on the network, allowing them to be resolved before users realize that anything has gone awry. SMARTS creates a "codebook" that combines generic models for each network object with the customer's specific topology to create "problem signatures." When the problem signatures are detected the system raises an alert. Her firm already has models for a wide variety of network hardware, and the codebook can be created by connecting their system to some common network-management systems such as OpenView and NetView. The codebook is automatically updated every time the models or topology change.

WORKS-IN-PROGRESS (WIPS)

Coordinator: Peg Schafer, Harvard University

Summary by John Talbot

The WIPs Practicum was more than the advertised "pithy" display of current techniques and issues. It was a nonstop blitz, in concise 10-minute samples, of interesting and thought-provoking descriptions of "real-world" problems and solutions. The WIPs were pitted against one another with a real whip bestowed upon the winner, who was determined by group applause.

Steven Nelson's "Multiplatform Storage Area Network" WIP was a nuts-and-bolts discussion of managing a network of over 1.6 TB of database and data sources in a 24x7 environment. Nike's original fiber-channel network was initially spread out over all storage areas, but problems persisted because of shared data paths for both the backup and data-delivery services. They have used the EMC array and parallel pipes to have transparent access to data dumps without interfering with network and system resources. They still have large-size backups and multiple filesystems for data storage. They are looking into ways of using the Veritas volume manager filesystems on their EMC array to have a commonly mountable local vxvm filesystem between heterogeneous platforms.

John Buckman<john@lyris.com> presented a unique implementation of email as a mission-critical application. Instead of a centralized MTA and MDA, Lyris opted for a thin multithreaded mail daemon that uses SQL as the message store. This enables his site to utilize the text searching and user access of the existing SQL structures to manage email. Also, they are able to deliver an email interface using HTML and Tcl more easily with the existing SQL structures.

Lowell Snyder<lsnyder@ptc.com>presented another good email implementation. (I might be a little biased since I was a cohort of this WIP, but it did come in a close second.) Lowell presented some of the work done at Parametric Technology to remove the standard UNIX aliasing from the central MTA and institute LDAP hooks into the internal mail exchangers to directly deliver mail based on the company's managed LDAP database. A motive was that a majority of users were consolidated on a central POP/IMAP MDA, and the management of simple UNIX mail aliases became cumbersome as the user base grew to over 4,000. Snyder described the code changes required on the sendmail and LDAP daemons necessary to invoke the features and handle several exceptions for whitespace and parsing symbols not generally understood or interpreted by general sendmail parsing rules.

Tom Limoncelli<tal@lucent.com> presented "Tricks you can do when your firewall
is a bridge." This WIP stemmed from a project Limoncelli had at Bell Labs/Lucent when cutting over backbone routers to new firewalls to the Internet. The problem was originally complicated by the fact that he didn't have access to all the routers in the schema and still needed to implement the firewall changes transparently. Of note was the fact that when the firewall acts as a bridge and doesn't have an IP number, it is less likely to be subject to hacking attempts since it has no TCP/IP-bound interface. See <https://www.bell-labs.com/usr/tal>.

Charles Tatum<tatum@nswc.navy.mil> of the U.S. Navy presented "Computationally Expensive Intrusion Prevention." He focused mainly on his modifications to the popular Crack program. He surmised that a massive amount of time was being used by the Crack code to test less probable circumstances of password guessing. His new approach was to apply common rules first. For example, he found that most people don't use more than one word for their password, so applying a number of permutation rules to the password guessing before trying "obvious" guesses makes running Crack computationally expensive. Also, implementing a separate dictionary of names ahead of the standard dictionary will weed out more obvious guesses. Reducing the salt guesses also lowers process runtime. Tatum's approach manipulated the original Crack encrypt and compare looping structure from Dict*Rules*Users*Salts to Rules*Dict*Users*Salts and implemented his common-guess strategies to greatly lower computing times.

Geoff Halprin<geoff.halprin@sysadmin.com.au>of SysAdmin Group, winner (by a hair over Lowell Snyder) of the WIP prize, gave a stunning talk about the "Taxonomy of Best Practice." This was one of the clearest talks classifying and describing what sysadmins do that I have heard in a long while. Halprin not only explored the need for sysadmins to understand their own personal technical capabilities, but also explained that the user community needs to understand what sysadmins do and of what they are capable so thatsysadmins can be properly matched to the environments they support. He defined system-administration core competencies as control, organization, protection, optimization, and planning, and he included a diagrammatic breakdown of each of these characteristics. He established models for organizing these responsibilities on a set of five levels, with three to five of those levels being consistent across all platforms. The five levels are determined by a Capability Maturity Model by which certification is based upon core competencies. This WIP would be a great basis for a future LISA invited talk.

Michael Ewan<michael.ewan@tek.com> of Tektronics described using LDAP to create printer definitions and determine printers by class (e.g., color, resolution, paper size) and location. He described how the current workstation environment DISPLAY variable and user profiles could be used to reference the "nearest" user-default printer. He has also looked into ways to use the LDAP information base to manage printers. His environment eased implementation by its homogeneous use of Tektronics printers and the standardization of UNIX workstations.

Andrew Hume<andrew@research.att.com> and Tom Scola from AT&T Labs presented "How to Handle Microsoft Attachments" in UNIX email. They wanted to have a UNIX-based reader that would be able to read email attachments in realtime. To reduce the need for specialized windowing software, they devised a plan that sends the Microsoft attachment to an NT system that runs a PostScript converter and sends it back to the Xwindows-based MTA. Unfortuna-tely, time was running short at this point of the WIP session and many of the details were abbreviated before Hume could complete his WIP.

BOFS

Summary by Douglas Stewart

Variable Length Subnet Masks on TCP/IP Networks

Mike Andrews

The basics of Class A/B/C and their corresponding netmasks, and reserved classes for testing and internal use, were covered. Using all of your addresses in a single network is wasteful, especially if your company has a Class A address allocated to it and has its network broken up into geographically separate chunks. By modifying your netmask, you can break up your network into smaller, separate pieces that can be routed separately. A useful example is an ISP that resells large number of T1s. Typically you'll have a subnet composed of only the router on each side. By setting your netmask to 255.255.255.252, you break your network into 64 subnetworks with four addresses, which include a network address, the broadcast address, and two usable host addresses you can use for the routers. Routing protocols and commands for troubleshooting routing problems were suggested. Some light reading (RFCs 950, 1918, and 1878) was suggested.

BSDI

Doug Urner

Most of this presentation was on new features of BSDI 4.0. Filesystem code updates: soft update (delays certain file operations for speedup, very temporary files may never be created); even out update; 64-bit file offsets; mount options to disable access time updates; mount options for sync/asynch writes; and soft read-only. Networking improvements: PCB lookup hashing; IP address hashing for fast virtual hosting; per-address IP statistics; and kernel-level packet filter. IPv4 enhancements: slow start; congestion avoidance; multi-cast; large windows; MTU discovery; and IPSEC. IPv6 support! Includes SAMBA, IP/IPX, Novell 3.x file and print services, VPN support. NFS: v2 and v3 support over UDP or TCP; NFS lock daemon. New network media: frame relay. SMP: performance improvements (user-level processes show best improvements); threads are all user-level for now but the kernel will be threaded in the future. New hardware improvements include bootable CD-ROMs and plug-and-play Ethernet, modem, and sound cards. Binaries are now ELF, and the math libraries have been proved. There's a console debugger and trace facility called Kdebug and KTR. Things to look for in the future: Linux binary compatibility, Java application environment, finer-grained SMP with kernel threads, SPARC port, ATM, and channelized T1 and T3 support.

AFS

Esther Filderman

The people who attended this BOF were almost entirely AFS users from university environments. The first topic was release dates for Linux and NT-based AFS servers -- apparently in February 1999. Transarc, which has been bought by IBM, was a topic for heavy discussion. It has a new CEO, has changed its mind about dropping the development of AFS, and has opened a London office. Something else people were looking for was Kerberos 5 support. There were complaints about poor support from Transarc, especially with the 5 pm EST closing time that is inconvenient for West Coast customers. KNFS was discussed as something that people were experimenting with and had had some success with. In the end, people had little faith in most of the alternatives to AFS (NFS, DFS, CacheFS) and felt that the advantages of AFS outweighed the problems they had encountered.

ADVANCED TOPICS WORKSHOP

Adam Moskowitz, Facilitator
Rob Kolstad, Co-chair and Scribe

Summary by Josh Simon

We first went around the room introducing ourselves, the quantity of users and the quantity and type of hosts we supported (whether individually or as part of a team), and two to five topics we wanted to discuss during the day. We came up with ranges of up to 10,000 users, 10,000 PCs, 2,000 Macs, and 3,000 UNIX hosts of various flavors. Other notes were multiple terabytes of disk storage (with projected short-term growth to exceed a petabyte), strange printer requirements, and extremely high growth rates (up to 400% a year).

We determined that we wanted to talk, in general, about:

* Consistency/standardization in sysadmin practices as organizations grow

* Cool system-administration tools and paradigms

* Specific hot technologies/paradigms to prepare for/crystal ball

First, we discussed the issue of internal consistency and standardization in technical practices. We tried to look at the "problem," but realized that we all had slightly different ways of looking at it -- which was not surprising considering that we had 31 people in the room, all with different backgrounds and experiences. We seemed to agree in general that creating standards is challenging, enforcing them is a hard problem, and that there are many more variables than may be obvious at first look. The concept of a "taxonomy" or categorization of problems into areas seemed to make sense to a lot of the folks present.

We next had a free-form discussion on cool system-administration tools and paradigms. Some general comments were:

* DSL is great. (10 people have 56K or more to the home, all but one of those has 56K or faster bidirectionally; six people have >128K. Most of these are business-paid and not individual-paid.)

* MRTG (discussed in Tobias Oetiker's paper at the conference) was hailed as a wonderful network-mapping tool. It uses SNMP polling on a 5-minute interval and creates Web pages with usage graphs. It ages data appropriately and is freely available. Big Brother, a systems monitoring package, integrates with MRTG and is also freely available.

* Intrusion-detection systems now are in the same sort of not-yet-well-understood position as firewalls were a decade ago. While then we had free firewalls which later became commercial, now we have commercial intrusion-detection systems (IDS) even though the problem is neither understood nor solved.

* Turnover can be interesting. 16 people changed jobs at least once in the past year. Four of these were internal (same company) job changes. Raises in the new job ranged from 0-90% and seemed to average around 28%. And 14 people present have open requisitions they are actively hiring for.

* Enforcing the use of a PDA like the PalmPilot has improved the follow-through for members of the group. Many folks at one company have bought one with their own money.

* 19 of those present carry a cell phone; 26 carry pagers; seven carry authentication devices. A few have two-way pagers; 12 participants pay at least part

of their monthly fees for the portable communications devices.

* Five members use a Ricochet or similar device for wireless digital communications. 14 more would use it if it were available in their area.

* Seven people have an agreement to attend conferences annually. In spite of that small number, 10 have some kind of permission to attend more than two per year. A couple can go to even more if they have papers presented at them. Everyone pretty much gets at least one per year. About half can attend two or more per year, depending on circumstances.

* Some cool utilities are ssh and Curl. LDAP or similar directory services are on the rise; 11 attendees have this.

* Cordless phones in machine rooms are a major win.

* Tools sometimes die for lack of nurturing. It would be nice if there were some way to solve that problem (like a MacArthur grant type of thing). Even finding current versions is too hard. See <ftp.sage-au.org.au> for lots of sysadmin tools.

Next we discussed hot technologies, rumors, and similar prognostications. One hot technology we talked about is XML, the Extensible Markup Language. It is self-verifying, easy to parse, easy to search, and has a universal file format. It's different from SGML in that it doesn't include the hard-to-implement features. XML supports Unicode. Unicode is the next hot technology we see on the horizon. It represents all characters (including nonRoman alphabets like Cyrillic, Hebrew, and Farsi). Microsoft Office 2000 uses Unicode; rumor has it that Word already supports it.

Other predictions are: Voice over IP will be a hot technology soon; directory services are becoming more important; applications will support more location independence; voice input and/or recognition will grow in the next year; and digital camera use will continue to rise.

SAGE COMMUNITY MEETING AND CANDIDATES FORUM

Summary by Carolyn M. Hennings

The annual SAGE Community Meeting had a significantly larger attendance this year than in the past two years that I've attended. Pat Wilson kicked off the meeting with announcements, introduced the candidates for the upcoming Executive Committee elections, and moderated a question-and-answer session.

Announcements included the status of publications in the "Short Topics" series. Educating and Training Systems Administrators is in the mail to SAGE members. Pat provided an update on the certification debate, saying that an advisory committee has been formed and charged with the investigation of whether or not SAGE should continue to pursue the certification issue and how it should be done if SAGE chooses to move forward. Pat mentioned the efforts of the "Day-in-the-Life" survey and announced that Rob Ferrell and Brian Kirouac have been named SAGE Historians. Items to watch for are "How-To Notes," revamped Rosetta Stone, new booklets on site audits and on hiring system administrators. Efforts are under way in reviewing the ethics policy and in developing a mentoring program.

Moving on to the candidates forum, Pat asked the candidates to introduce themselves, and the floor was opened up for questions. Question topics included the certification issue, the ethics policy, education and training, the inclusion of Windows NT topics in the LISA program, and the market's view of SAGE.

In the BPF format, the SAGE Community Meeting continued under the guidance of Hal Miller, SAGE Executive Committee President. The open-forum discussion focused on the professional development of systems administrators in the sense of gaining skills to communicate and work with management. David Parter, chair for LISA '99, took many suggestions for topics for the next conference. The smaller group meeting provided an informal opportunity to meet other individuals who are actively involved with SAGE at the national and local levels.

CLOSING SESSION

Summary by Carolyn M. Hennings

The 1998 Champion, Daniel Boyd, with Rob Kolstad

At a lot of conferences the attendee population markedly diminishes on the last day of the conference. LISA is different for one reason. As always, Rob Kolstad's LISA Quiz Show is a major attraction at the conclusion of the conference.

This year was no exception. Maintaining the same format from previous years and always making technical improvements, the Quiz Show gets better and better. Conference attendees vied to answer questions in areas such as UNIX administration, the WWW, computer executives, physics, circus acts, electrical current, coins, certification questions, and European dictators. In the category of conductors questions about electrical conductors set the stage, but the final question was "Who wrote West Side Story?" One never knows what to expect.

A bonus this year was the "Tournament of Champions." Last year Snoopy Beagle, who hails from Germany, lost to Hal Pomeranz. Snoopy then challenged the fairness of the Quiz Show, saying that too many questions were based on American pop culture. Apparently Rob Kolstad heard enough feedback over the past year regarding this issue and decided to rectify the situation.

The "Tournament of Champions" was a contest between this year's winner, last year's winner, and Snoopy. Notable categories for the final round included European history, television shows, security, and match the dictator. In a surprising victory, Daniel Boyd made mincemeat of his competitors. Who knows what will happen next year!

Tina Darmohray, Winner of the 1998 SAGE Outstanding Achievement Award	Dan Geer & Greg Rose comparing ribbons
Adam Moskowitz & friend, demonstrating the latest LISA dress code	Snoopy