Summary of the Advanced Topics Workshop

Held at the 12th Systems Administration Conference
(LISA XII), Boston, MA, December 8th, 1998.

Sponsored by USENIX and SAGE

Attendees

Adam Moskowitz (Chair)
Rob Kolstad (Co-chair and Scribe)

Paul Anderson
Christopher L. Barnard
Ann Benninger
David Blank-Edelman
Strata Rose Chalup
Brent Chapman
Albert Chin-A-Young
Richard Chycoski
Lee Damon
Tina Darmohray
Xev Gittler
Geoff Halprin
Helen Harrison
Christine Hogan
Brian Kantor
Tom Limoncelli
David Lindes
Bryan McDonald
Mark Mellis
Hal Miller
Ruth Milner
Paul Moriarty
Cat Okita
David Parter
Giray Pultar
Ian Reddy
Shawn Rutledge
Phil Scarr
John Sellens
Dimitri Shcherban
Josh Simon
David Williamson
Pat Wilson

Agenda

9:00 - 10:30		Introductions and Statements of Problems
11:00 - 12:30		Discussion of Selected Problem(s)
13:30 - 15:00		Presentations of Recent or Current Work Group Surveys
15:30 - 17:00		Presentations and Surveys (cont'd) Hot Technologies: Rumors and Refutations

Atendees were reminded that they were there to share their own experience and knowledge, and not as representatives of their company (insofar as the goal of the workshop was not to sell either companies or products, but rather, to share ideas and problems with our peers). Since many of the issues and problems that were likely to be discussed were difficult simple because of the (sometimes staggering) scale of the problem (e.g., backing up 10 or even 100 gigabytes is hard -- backing up tens of terabytes is nearly impossible!), attendees were asked not to play ``mine's bigger than yours.'' However, when Lee Damon (of Qualcomm) introduced himself as ``working for the people that make these'' (and holding up a Qualcomm Q800 -- a very small cellular telephone), Brent Chapman held up his Morotola StarTAC (another very small cellular telephone) and said ``mine's smaller!'' Lee then noted that it's only when discussing ``cell phones, cameras, and pagers will you hear a man say `mine's smaller than yours' with pride.'' :-)

The attendees, their environment, and their problems are presented in alphabetical order.

Name:		Paul Anderson
Affiliation:		University of Edinburgh
Scale:		1,000- machines
Comments:
» Formerly mostly-Sun shop, now diverging to NT and Linux (PDAs, too)
Problems/Issues:
» Providing core infrastructure for user self-support
» NT Integration
» User authentication and security
Name:		Christopher L. Barnard
Affiliation:		Chicago Board of Trade
Scale:		530 workstations
Problems/Issues:
» Remote administration without remote admins
» 99.98% uptime [1.75 hours downtime per year]
Name:		Ann Benninger
Affiliation:		Exelixis Pharmaceuticals
Scale:		100 people, 150 machines, 500 GB
Problems/Issues:
» Explosive data growth
» 7x24 expectations without appropriate budgets
Name:		David Blank-Edelman
Affiliation:		Northeastern University
Problems/Issues:
» Increased security requirements
» Growing demands on sysadmin staff by ever-maturing user population
» Work satisfaction for sysadmins
Name:		Strata Rose Chalup
Problems/Issues:
» 200K users messaging system for roaming palm computing
» Scalability of internet services for clients
» Distributed database management (accounts, auth)
» `Fighting the conduit war' [delivering services, not technologies]
» Showing that sysadmins affect the bottom line (and thus are professionals)
Name:		Brent Chapman
Affiliation:		Covad
Comments:
» Covad is a DSL startup
Problems/Issues:
» Growth -- doubling every 4 months (340 now) [moving frequently]
» VP's who think think understand IT
» 7x24 uptime w/limited staff
Name:		Albert Chin-A-Young
Affiliation:		The Written Word/Abbott Labs
Scale:		350 machines, 1,000 users
Problems/Issues:
» Packaging of software for distribution
» Managing and exploiting testing environments for multiple platforms
» Single signon [authentication]
» Global filesystem design and implementation [file sharing, UNIX/NT]
» NT/Unix integration; account name management, authentication
Name:		Richard Chycoski
Affiliation:		Cisco
Problems/Issues:
» User accounting admin on multiple platforms
» Downtime in 7x24 environment
Name:		Lee Damon
Affiliation:		Qualcomm
Scale:		10,000 users, 10,000 PCs, 2,000 Macs, 3,000 Unix
Problems/Issues:
» Y2K
» Politics
» Merging UNIX & NT environments
Name:		Xev Gittler
Affiliation:		Goldman, Sachs
Scale:		10,000+ users
Problems/Issues:
» Integration UNIX/NT/Mainframe/Pilot [no macs]
» Applications/developer management (1,000 developers)
» Large scaling issues (including remote offices w/small staff)
Name:		Geoff Halprin
Affiliation:		Consulting w/Telstra and IBM
Problems/Issues:
» Software currency in large (650+) mission-critical environments
» Standardization of sysadmin practices
Name:		Helen Harrison
Affiliation:		SAS Institute
Problems/Issues:
» Unix-NT Integration [accounts, filesharing, global sysadmin]
Name:		Christine Hogan
Affiliation:		Global Networking and Computing, Inc. (GNAC)
Problems/Issues:
» Consulting/Data center outsourcing
» Cost-cutting (sysadmin is cost-center, not profit center)
» Growing demands on sysadmin staff by ever-maturing user population
Name:		Brian Kantor
Affiliation:		UCSD
Scale:		25,000 hosts, 40,000 users, high turnover
Comments:
» [Most problems are political for Brian, who lives downhill from the outhouse] [Kolstad]
Problems/Issues:
» Finding competent people to work for university salaries
Name:		Tom Limoncelli
Affiliation:		Bell Labs (a division of Lucent Technologies)
Problems/Issues:
» Single signon
» Cramped machine room space resources
» Lack of theoretical basis for sysadmin
Name:		David Lindes
Affiliation:		Netscape Communications
Problems/Issues:
» Employee turnover (loss of project memory)
» Maintaining namespaces across worldwide org
» Package management on multiple platforms
Name:		Bryan McDonald
Affiliation:		Global Networking and Computing, Inc.(GNAC)
Scale:		10-10,000 users @ clients
Problems/Issues:
» Security in face of declining budgets
» Security while outsourcing
» Security with new un-secure applications
» Security in environment of non-technical management
Name:		Hal Miller
Affiliation:		University of Washington
Scale:		200 machines, 1,000 next year, 1PB in four years(!) 1TB, doubling every 5 months
Problems/Issues:
» Storage management (including backup)
» Budget allocation/chargeback schemes
» Account management systems (must work w/UNIX, NT, and Macs)
» Building services for space (moving people/networks around)
Name:		Ruth Milner
Affiliation:		National Radio Astronomy Observatory
Scale:		500 users, 600 PC/UNIX
Problems/Issues:
» Y2K testing of 3rd party s/w (250 apps)
» Single signon/single user account [NT/Unix]
» Consensus on technical decisions
Name:		Paul Moriarty
Affiliation:		U S Web/CKS
Problems/Issues:
» How does one make sysadmins better project managers?
Name:		Adam Moskowitz
Affiliation:		Genome Therapeutics
Scale:		250 users, 300 machines, 1TB store
Problems/Issues:
» Explosive data growth
Name:		Cat Okita
Affiliation:		Earthworks; Toronto Stock Exchange
Scale:		500 users, 500 machines, varied brokers
Problems/Issues:
» Soft skills [writing/presenting, managing, selling ideas/ourselves]
» Marketing/training/certification -- gaining recognition for profession
» Planning
Name:		David Parter
Affiliation:		University of Wisconsin, Computer Sciences Department
Scale:		6,000 users
Problems/Issues:
» Communication skills [w/staff and w/users]
» Users expectations too high
» `Doing things right' (avoiding ad hoc solutions)
Name:		Giray Pultar
Affiliation:		John Hancock Funds
Scale:		2 hosts, 12 users
Problems/Issues:
» Data growth
» Euro currency conversion
» Linux in workplace (users bringing in versions, flaky h/w, mgmt buy-in)
Name:		Ian Reddy
Affiliation:		Cisco
Scale:		15,000 users (half engr)
Problems/Issues:
» Maintaining consistency in sysadmin practices as orgs grow
» Growing into management
» Beyond Y2K
Name:		Shawn Rutledge
Affiliation:		Sequent Computer Systems (Consulting Division -- @ Boeing)
Problems/Issues:
» Scaling vendor apps to large environments
» Politics of consulting to huge company
» Scaling NT to large environments
» ``Can't do too much to solve problems, because it's NT''
Name:		Phil Scarr
Affiliation:		Global Network and Computing, Inc. (GNAC)
Problems/Issues:
» Managing (people/projects) and its transition from technical positions
» Working in NT-only environments at customer sites who have NO tech staff
» Managing politics at numerous client sites
Name:		John Sellens
Affiliation:		UUNet Canada (Engineering)
Problems/Issues:
» Large storage
» Upgrades without interrupting services
» 7x24 Unmanned remote servers
Name:		Dimitri Shcherban
Affiliation:		Contractor at NASA w/users also in Japan
Scale:		~60 users
Problems/Issues:
» Storage
» UNIX/NT integration (by elimination of UNIX)
» Support of tropical rainforest mission
Name:		Josh Simon
Affiliation:		Consultant, Collective Technology, on-site at Motorola
Scale:		team supports thousands of users
Problems/Issues:
» Time for projects (vs. process for operations)
» Writing skills for technical and management operations
» Upgrading hw/sw in 7x24 shop
Name:		David Williamson
Affiliation:		Global Networking and Computing, Inc. (GNAC)
Problems/Issues:
» Selling cost of VPN to management
» Juggling n business clients with n x m passwords
» Scaling
» Dealing with utilities (like telcos)
Name:		Pat Wilson
Affiliation:		Dartmouth College
Scale:		3,000 users, 35 unix servers
Problems/Issues:
» Retrofitting infrastructure (central backup, authentication, policies)
» Security in academic environment

An attempt was made to define the problem, but this proved rather more difficult than expected. What follows is Rob Kolstad's herculean attempt to summarize the discussion:

Topic: Internal consistency/standardization in technical practices

Problem statement(s):

System administration is a difficult job to describe

It is possible to construct any of a number of paradigms for system administration at a given site

It is desirable to reduce the number of various paradigms and thus reduce complexity and thus reduce costs

It is desirable to have shared goals among the sysadmin group and global organization

Standard procedures are challenging to create

Standard procedures are difficult to enforce

Motivation for decisions is often omitted from documentation

Turnover and individualism thwart standardization

Intergroup politics can thwart more general standardization

How are goals set for sysadmins?

Are procedures documented so that results are repeatable?

Desirable to create an environment where pre-defined responses exist for all conceivable situations (including routine situations and rare ones)

Management wants to be able to solve problems as they occur, with interchangeable people and standard procedures

Larger organizations demand control mechanisms that differ from the `benign dictatorship' that often works well in smaller shops

Professionalism demand that sysadmins address problems in a standard way that users/etc. can expect

Sysadmin paradigms and tools change very quickly

Sysadmins (or their management) require authority to implement certain paradigm mandates

Big iron data centers might provide a model for solving this problem

Predictability reduces chaos and increases RAS

Geoff Halprin is developing sysadmin taxonomy (!); maybe a best practices document can define a consistent attack

Comments on the problem:

Doing something `right' vs. doing it `fast' is a conflict

Setting standards is ongoing at several participants' organizations

Current problems similar to those cited by mainframe people when Unix came out

Technicians are not architects and architects are rare

Different people have different sets of strengths

Companies need to exert ownership over their computers -- they are not ``PERSONAL'' computers

Management perceptions of budgetary issues surrounding support organizations is complex, chaotic, and potentially bizarre

Responsibility and authority really must go hand-in-hand

What follows are the highlights of this "discussion"; no attempt was made to record who "presented" what task/tool/idea.

• I have DSL at home and it's very fast.

• Dartmouth has a new enterprise-wide backup system (very inexpensive for universities) from IBM. Lots of configuration capabilities. Runs with a tape library. Not perfect but solves a whole class of problems. It currently backs up about 500GB.

• Another firm spent $8M to design, architect and install a huge Legato-based backup solution.

• I have Kerberos V, AFS, no NT Domain. And it works!

• MRTG -- a miracle program. It polls SNMP variables or external programs every five minutes and creates attractive web pages for monitoring. It handles data-aging gracefully with fixed-size datasets. Freely available.

• 802.11 is the new IEEE standard for wireless ethernets. DHCP and wireless networks work in our building now -- up to ten people in a conference room. Two megabits per second.

• I built an IT department in a year and a half. It's been fun and very rewarding. We built a farm of Solaris-based PCs in the server room for compute cycles. We built a distributed processing tool kind of like LSF.

• We examined how to re-architect DNS. I built some perl scripts that acquired zone information for every zone at our company. I generated a postscript map of the entire tree, with color coding.

• I've been focusing on intrusion detection systems. It reminds me of where firewalls were about ten years. It does not appear that the problem is understood or solved. The difference now? In the past you could build firewalls cheaply using free tools and the vendors came into the market. This time, the cart is before the horse and vendors are selling IDS solutions that might not be as complete as I would like.

• I changed jobs to get a 30% raise. My replacement made 25% more than I made when I left. I am bored in my new job so far. I am looking for another job.

• I use Big Brother to monitor large numbers of systems. I like it. It connects with MRTG. BB is freely available at https://maclawran.ca/bb-dnld/. We are also installing a Lucent Managed Firewall.

• I am facing the challenge of managing applications in 7x24 environment, including monitoring and management. I am working with developers to improve our applications.

• I wrote a carefully composed e-mail message to our developers explaining about network latency and bandwidth and it sure helped our locally developed applications.

• We backup 6 TB with FiberChannel and a silo. We exploit multiporting.

• We use Cisco's Local Director for Oracle (!).

• We're working with data-over-cellphone and palm pilot communications. I'm also trying to raise awareness of the Y2K problem. We can sync Palm Pilot calendar to MeetingMaker and that's helped our meeting attendance.

• We gave everyone in our group a Palm Pilot and our group's follow through is much better now.

• At our company, 70% of employees have bought Palm Pilots with their own money. Higher level employees approach 100% penetration.

• Many group members like their Palm Pilots and other PDAs. Most of those who don't use a PDA use other aids, including Franklin Day Planners and just plain paper.

• We back all our data up at least once per day onto DLT. We made our own backup software and architecture. It requires a bit of hardware.

• We like MRTG, too. A new generation version is being worked on; it's called super-MRTG and solves some of the scaling problems of MRTG. Collection is much faster. Configuration language is now compressed and includes an object oriented hierarchy. The performance increases are radical. See a paper about it at the Network Admin conference in Santa Clara. Software will be freely available.

• I am learning that my skills are a bit rare and it is very fulfilling for me. By providing a small voice of sanity and a higher level viewpoint, I've been able to contribute far more than just by memorizing commands. My current project is scaling a lot and very unpredictably. I am now working with management in ways that are `cool' but I'm still avoiding the bad parts (e.g., PERT charts).

• We put a new NEWS network provider up on the net. It's interesting to build a site for 1,000,000 people from scratch.

• We're doing VPN technology, using commercial building blocks. There are lots of new RFC's for IPSEC. We're also working with NT Terminal Server (a Citrix-like product). We're pretty happy with using it in a security environment that resembles a central computer center. We're also working with I-Beam, a satellite multi-cast caching company.

• I like ssh. I use rsync to keep my laptop filesystem synchronized. It uses ssh and is available from the same people who make Samba. I really like email pagers and two-way pagers. VNC (https://www.orl.co.uk/vnc) puts another screen (from a Windows or UNIX machine) onto yours (also freely available). I like XML. I also like Trinux (https://www.trinux.org), a tiny linux networking kit on two floppies where it runs from RAM (it doesn't access hard drives). It's very useful for hacking, cracking, and even repairing your network.

• I like Curl, a new programming language that sports the best features of TCL and HTML. Runs on lots of platforms. There's a browser implemented in it.

• One of our customers is implementing technology for electronic books (the RocketBook). The display technology is a fabulous LCD screen. (Another attendee is a beta tester and has a special affinity for paper that the product did not satisfy.) These might be great for reference manuals.

• I enjoy working in a company that has its own firewall product. They didn't have a proxy for it so our developers listened to us and we had one! That's fun. It's also been fun to construct a data center. We're now being challenged by bandwidth problems with our CO.

• A year ago we examined replacement of a pure Sun shop with other things, including Linux. We're 60% Linux now.

• I wrote a tool called HCB (Host Configuration Baseline). It runs around gathering all the things that make a host unique (different from when it was unboxed.) It sucks them all into a spool area and then compares that to a baseline and sends mail to the appropriate admin telling of changes. This solves 90% of the mistyping problems. It is proactive about change alerts. It also encourages an approved rollback snapshot. It will be publicly available soon. This differs from tripwire in that it shows *how* things have changed. I also have tripwire-on-steroids. Tripwire has a single database per host. Adding tools on top of tripwire that enable it to address, e.g., only the OS part of a filetree, gives you a more useful system. Changes are aggregated into one e-mail per day per site! [Permissions and ownerships are not yet easily correct... that work is in progress.]

• I have a program called ypmake that builds NIS databases. Files are kept under RCS. The program checks out the latest version of the (configuration) file and syntax checks it. The program then does a yppush and potentially rdist to other YP masters. This is all controlled with a Berkeley amd-style configuration file. There is a notion of `map handlers' that can be specified and dynamically loaded. I also have SB/PB for software build/package build. I create an instruction file for each package I build (potentially for each architecture). Builds proceed accordingly.

• We have 500 machines remotely administered with NO LOCAL ADMINS! The WorldComm guys set up the net and plug our machine in and the machine comes up and `everything works'. We even capture consoles remotely. I also have perl combing my system to send anomalies to my pager.

• We implemented Microsoft Exchange and found benefits in it. Directory management, public folders, global namespace, global directory system. This enables you to know who is your organization. We share files with Europe through the folders. We have unique e-mail IDs. We like calendaring, too.

• I've noticed that tools sometimes die for lack of nurturing. I wish there was some way to solve that problem (like a MacArthur grant type thing). Even finding current versions is too hard. See ftp.sage-au.org.au for LOTS of sysadmin tools.

  I have 15 years of e-mail and I've extracted the addresses. So, I built a database to use as an incoming spam filter that dumps suspected spam into a special folder.

• I changed jobs and spent four months without root. It worked out OK. I liked it. We bought a NetApp server and are getting rid of the Auspex server. We're deploying Linux on our scientists' desktops. We don't use auto-client since it's hard to tell it which disks to use or not use.

• In my new job, after spending years eking bits out of network links, I'm enjoying my new job and its ample bandwidth. I have two orders of magnitude more bandwidth now. I'm having fun managing now.

• I moved from an academic environment in Canada to a commercial environment in USA. I've been working with NIS+. It's still got growing problems but is fun to poke at. I play with Netapp's. USA corporate environment is different from Canada. It's refreshing to have a bottom line that means something.

• I worked with lprng. We print a lot. So, we designed a system with no single point of failure. Everyone wants a personal printer (e.g., $200 inkjet). Every (PC) app wants a printer list. I invented an abstraction called `MYPRINTER' that solves lots of problems.

  I wish there was a metric that enabled me to choose ISPs from whom to purchase effective bandwidth.

• I've faced explosive data growth. I built a filesystem walker. I'm seeing 100MB/day growth in some people's directories! 10% of my disk is consumed by symbolic links. I'm learning why my network is slow. My systems have 40M tiny files. I'm working on a trend plotter (web page). I have directories with 100,000 files in them.

• Cordless phones in machine room are a big win.

• Truly archival media are difficult to find.

Of the 35 attendees:

• About 10 have network access faster than 56K at home; all but one faster in *both* directions; about six have faster than 128K; four have non-DSL, including T-1, wireless, and cable modems; in most cases, employers pay for this access.

• Sixteen (16) changed jobs in the last year; four stayed within their same company; raises ranged from 0% to 90% (yes, 90), as follows: 0%, 5.5%, 7.75%, 10%, 15%, 20%, 24%, 25%, 31%, 40%, 50%, 50%, and 90%.

• Nineteen (19) carry a cell phone; 26 carry pagers; seven have authentication devices; only a few have pagers capable of sending e-mail; twelve pay at least part of the monthly fees for their portable communications.

• Five use a Richochet (or similar) device for mobile wireless digital communications; 14 more would carry one if it were available in their area.

• Fourteen (14) attendees have jobs they are hiring for; almost all of those have more than one opening; five are looking for a job.

• Three are consultants (not contractors), the rest are employees.

• Eleven (11) of the attendees have some sort of LDAP directory service.

• Whether by formal or informal arrangement, all workshop participants get to go to at least one conference per year (circumstances permitting); about half can attend two or more per year (again, circumstances permitting); seven have formal agreements, and ten have some sort of permission to attend more than two per year (as opposed to on an ad-hoc basis); two or three can go to even more (conferences) if they are presenting a paper.

• XML. Self-verifying; easy to parse. Extensible. Universal file format. Server to server, it's like automatic-RPC. It's like the difference between MIME and X.400. XML is also easy to search. We maintain our FAQs using XML with a very uniform format and have derived good advantage. Office-2000 apps will use XML as its file format. Differs from SGML in that it does not include the features that were hard to implement. XML is all Unicode.

• Unicode. Represents all characters (including Farsi, etc.). Coding of the future, really. Office 2000 uses Unicode; rumor has it that MSWord already does.

• Voice over IP. One bit over a voice line from SFO to NYC is 9 nanocents. On IP, it's 3 nanocents. But this will end up being a commodity.

• Directory services will be more important over time. More machines -> more data about people. The closest thing to a network database is a directory service. Probably going to be LDAP based and also will be highly distributed.

• Applications will sport extreme location independence and will shed their idea of having a `home' and will just magically know your environment. A meta-paradigm of configuration will specify these connections. ``Directory services are not just about people.''

• San Jose Mercury News said: the network is a cloud, according to some techies. Does anyone really care about the network details? Yes, network engineers do. Does it matter that no one has the complete picture? No, says Rob. See https://www.cs.bell-labs.com/~ches/map. And posters WILL be available soon (probably). See book `Spoiled' by Nicols Fox (about decoupling checks and balances in food distribution chain). One popular way of dealing with ecological cleanup for industrial waste in USA is to sell it to fertilizer manufacturers who then put heavy metals into your tomatoes.

  Structure of internet is important to politicians because diversity of suppliers can enhance reliability.

• Voice input. Some think voice input will grow in the next year.

• Digital cameras. Cheap digital (full motion) cameras will cost only $50.

Acknowledgements

I'd also like to thank Rob Kolstad for his work as scribe and co-chair. His efforts allowed me to concentrate on interacting with the attendees without worrying that what was being said would be lost; he also provided valuable assistance that helped keep the workshop running smoothly. It is also thanks to Rob efforts that this summary exists at all; the only thing I did was sprinkle in some HTML.

Finally, I'd like to thank Judy DesHarnais, the rest of the USENIX conference staff, the USENIX office staff, the USENIX and SAGE boards, and all the rest of the volunteer that helped organize and run the conference. Without them, there would have been no conference.

Adam S. Moskowitz, Workshop chair