• USENIX Home
• Conferences
• Proceedings
• Membership

• LISA '05 Home
• Registration
• Organizers
• Invitation
• 
• At a Glance
• Training
• Tech Sessions
• Hit the Ground...
• Workshops
• WiPs
• BoFs
• 
• Exhibition
• Sponsors
• Activities
• 
• Hotel/Travel
• Services
• 
• Students
• Questions?
• Help Promote!
• Brochure PDF
• 
• Call for Papers
• Past Proceedings
• 
• Authors
• Speakers
• 

Monday, December 5, 2005

Full-Day Tutorials

M1 Hands-On Linux Security Class: Learn How to Defend Linux/UNIX Systems by Learning to Think Like a Hacker (Day 2 of 2) Rik Farrow, Security Consultant 9:00 a.m.–5:00 p.m. See Part 1, S1, for the description of the first day of this tutorial. Day two of this class focuses on practical forensics, that is, how to analyze a possibly hacked Linux or UNIX system from a system administrator's perspective. As a system administrator, you will not be acting as law enforcement, trying to find the perpetrator, but instead will be working as quickly as possible with the goal of uncovering what went wrong. Finding rootkits and backdoors on a sample hacked system gives you an idea of what you might find on other similar systems. You can also get clues about the nature of the attack by discovering the tools left behind on a system by an attacker. The final portion of this class focuses on patching, with a discussion of cfengine. As this is the second day of a two-day, hands-on course, we will not repeat material covered on the first day, including getting the CD working with your laptop. If you plan on attending the course only the second day, you might want to contact the instructor before the class and get a test CD to ensure that your laptop will work in the classroom environment. Exercises include: John the Ripper, password cracking Using and modifying KNOPPIX Linux boot CD Elevation of privilege and suid shells Rootkits, and finding rootkits (chkrootkit) Sleuth Kit (looking at intrusion timelines) iptables and netfilter cfengine configuration Rik Farrow (S1, M1) provides UNIX and Internet security consulting and training. He has been working with UNIX system security since 1984 and with TCP/IP networks since 1988. He has taught at the IRS, Department of Justice, NSA, NASA, US West, Canadian RCMP, Swedish Navy, and for many U.S. and European user groups. He is the author of UNIX System Security, published by Addison-Wesley in 1991, and System Administrator's Guide to System V (Prentice Hall, 1989). Farrow is the editor of ;login: and a network security columnist for Network magazine. Rik lives with his family in the high desert of northern Arizona and enjoys hiking and mountain biking when time permits. M2 Advanced Solaris System Administration Topics Peter Baer Galvin, Corporate Technologies, Inc. 9:00 a.m.–5:00 p.m. Who should attend: UNIX administrators who need more knowledge of Solaris administration, especially the next-generation features of Solaris 10. We will discuss the major new features of recent Solaris releases, including which to use (and how) and which to avoid. This in-depth course will provide the information you need to run a Solaris installation effectively. This tutorial has been updated to include Solaris 10 and several other new topics. Topics include: Installing and upgrading Planning your installation, filesystem layout, post-installation steps Installing (and removing) patches and packages Advanced features of Solaris Filesystems and their uses The /proc filesystem and commands ZFS The kernel Kernel and performance tuning: new features, adding devices, tuning, debugging commands DTrace Enhancing Solaris Virtual IP: configuration and uses Performance: how to track down and resolve bottlenecks Tools: useful free tools, tool use strategies Security: locking down Solaris, system modifications, tools, zones, privileges Resource management: fair share scheduler Resources and references Peter Baer Galvin (M2, T12) is the Chief Technologist for Corporate Technologies, Inc., a systems integrator and VAR, and was the Systems Manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines. He wrote the "Pete's Wicked World" and "Pete's Super Systems" columns at SunWorld. He is currently contributing editor for Sys Admin, where he manages the Solaris Corner. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials on security and system administration and has given talks at many conferences and institutions on such topics as Web services, performance tuning, and high availability. M3 Administering Linux in Production Environments Æleen Frisch, Exponential Consulting 9:00 a.m.–5:00 p.m. Who should attend: Both current Linux system administrators and administrators from sites considering converting to Linux or adding Linux systems to their current computing resources. We will be focusing on the administrative issues that arise when Linux systems are deployed to address a variety of real-world tasks and problems arising from both commercial and research-and-development contexts. Topics include: Recent kernel developments High-performance I/O Advanced filesystems and logical volumes Disk striping Optimizing I/O performance Advanced compute-server environments Beowulf Clustering Parallelization environments/facilities CPU performance optimization High availability Linux: fault-tolerance options Enterprise-wide authentication Fixing the security problems you didn't know you had (or, what's good enough for the researcher/hobbyist won't do for you) Automating installations and other mass operations Linux in the office environment Æleen Frisch (M3) has been a system administrator for over 20 years. She currently looks after a pathologically heterogeneous network of UNIX and Windows systems. She is the author of several books, including Essential System Administration (now in its 3rd edition).   M4 Introduction to VMware ESX Server John Gannon and John Arrasjid, VMware 9:00 a.m.–5:00 p.m. Who should attend: x86 sysadmins who want to dramatically improve the way they manage systems. Do any of these complaints sound familiar? Our datacenter is out of power/space/network infrastructure and adding new servers is a struggle. Our developers ask us for new servers constantly and we can't keep up with the demand. It takes us days or weeks to procure, rack, stack, and configure a new box. Our yearly disaster recovery simulations are hardly ever successful because the DR site has a different hardware configuration than the production site. Our DR site is too expensive to operate because it is an exact replica of our production environment. We can only do hardware upgrades late at night and on the weekends. If yes, VMware ESX Server can help by: reducing your x86 server count by up to 90% supporting up to 80 x86-based OS instances running simultaneously (Linux, FreeBSD, Netware, and Windows) on a single physical machine freeing up valuable rack space, SAN, and networking ports providing instantaneous rollback to a "known good configuration" to assist in software development and testing allowing you to provision a new x86 server in minutes instead of weeks enabling Disaster Recovery despite having different hardware (and less of it) at your DR site eliminating downtime traditionally associated with hardware maintenance In this tutorial, we will provide an overview of virtual machine technology as well as the features and functionality of ESX Server. Installation, configuration, and best practices will be the focus of the session. Topics include: Virtual infrastructure and ESX Server overview ESX Server installation and configuration Virtual Machine (VM) creation and operation Installing VMs from scratch Using templates and cloning to provision VMs in minutes Operations and administration Sizing the environment Automating tasks via scripting Operations best practices Enabling disaster recovery and business continuity with ESX Server Migration strategies and the P2V process (Physical-to-Virtual) Advanced configuration SAN Networking Performance Tuning Security John Gannon (M4) has over ten years of experience architecting and implementing UNIX, Linux, and Windows infrastructures. John has worked in network engineering, operations, and professional services roles with various companies including Sun Microsystems, University of Pennsylvania, Scient Corporation, and FOX Sports. John's current work at VMware involves delivering server consolidation, disaster recovery, and virtual infrastructure solutions to FORTUNE 500 clients. John Arrasjid (M4) has 20 years experience in the Computer Science field. His experience includes work with companies such as AT&T, Amdahl, 3Dfx Interactive, Kubota Graphics, Roxio, and his own company, WebNexus Communications, where he developed consulting practices and built a cross-platform IT team. John is currently a senior member of the VMware Professional Services Organization as a Consulting Architect. John has developed a number of PSO engagements including Performance, Security, and Disaster Recovery and Backup. M5 System and Network Monitoring: Tools in Depth John Sellens, SYONEX 9:00 a.m.–5:00 p.m. Who should attend: Network and system administrators ready to implement comprehensive monitoring of their systems and networks using the best of the freely available tools. Participants should have an understanding of the fundamentals of networking, familiarity with computing and network components, UNIX system administration experience, and some understanding of UNIX programming and scripting languages. This tutorial will provide in-depth instruction in the installation and configuration of some of the most popular and effective system and network monitoring tools, including Nagios, Cricket, MRTG, and Orca. Participants should expect to leave the tutorial with the information needed to immediately implement, extend, and manage popular monitoring tools on their systems and networks. Topics include, for each of Nagios, Cricket, MRTG, and Orca: Installation—Basic steps, prerequisites, common problems, and solutions Configuration, setup options, and how to manage larger and non-trivial configurations Reporting and notifications—proactive and reactive Special cases—how to deal with interesting problems Extending the tools—how to write scripts or programs to extend the functionality of the basic package Dealing effectively with network boundaries and remote sites Security concerns and access control Ongoing operation John Sellens (S5, M5, T11) has been involved in system and network administration since 1986 and is the author of several related USENIX papers, a number of ;login: articles, and the SAGE Short Topics in System Administration booklet #7, System and Network Administration for Higher Reliability. He holds an M.Math. in computer science from the University of Waterloo and is a chartered accountant. He is the proprietor of SYONEX, a systems and networks consultancy. From 1999 to 2004, he was the General Manager for Certainty Solutions in Toronto. Prior to joining Certainty, John was the Director of Network Engineering at UUNET Canada and was a staff member in computing and information technology at the University of Waterloo for 11 years. Monday Morning Half-Day Tutorials M7 Security Without Firewalls Abe Singer, San Diego Supercomputer Center 9:00 a.m.–12:30 p.m. Who should attend: Administrators who want or need to explore strong, low-cost, scalable security without firewalls. Good, possibly better, network security can be achieved without relying on firewalls. The San Diego Supercomputer Center does not use firewalls, yet managed to go almost 4 years without an intrusion. Our approach defies some common beliefs, but it seems to work, and it scales well. "Use a firewall" is the common mantra of much security documentation, and are the primary security "solution" in most networks. However, firewalls don't protect against activity by insiders, nor do firewalls provide protection against any activity that is allowed through the firewall. And, as is true for many academic institutions, firewalls just don't make sense in our environment. Weighting internal threats equally with external threats, SDSC has built an effective, scalable, host-based security model. The keys parts to our model are: centralized configuration management; regular and frequent patching; and strong authentication (no plaintext passwords). This model extends well to many environments beyond the academic. Of course, we're not perfect, and last year we had a compromise as part of a security incident that spanned numerous institutions. However, firewalls would have done little if anything to have mitigated that attack, and we believe our approach to security reduced the scope of compromise and helped us to recover faster than some of our peers. In addition to a good security model and faster recovery, our system administration costs scale well. The incremental cost of adding a host to our network (beyond the cost of the hardware) is negligible, as is the cost of reinstalling a host. Topics include: The threat perspective from a data-centric point of view How to implement and maintain centralized configuration management using cfengine, and how to build reference systems for fast and consistent (re)installation of hosts Secure configuration and management of core network services such as NFS, DNS, and SSH Good system administration practices Implementing strong authentication and eliminating use of plaintext passwords for services such as POP/IMAP A sound patching strategy An overview of last year's compromise, how we recovered, and what we learned Abe Singer (S4, M7) is a Computer Security Researcher in the Security Technologies Group at the San Diego Supercomputer Center. In his operational security responsibilities, he participates in incident response and forensics and in improving the SDSC logging infrastructure. His research is in pattern analysis of syslog data for data mining. He is co-author of of the SAGE booklet Building a Logging Infrastructure and author of a forthcoming O'Reilly book on log analysis. M8 Intellectual Property Protection and the System Administrator Daniel L. Appelman, Heller Ehrman LLP 9:00 a.m.–12:30 p.m. Who should attend: System administrators of every level of experience and seniority, as well as their employers. Infringement of intellectual property rights through use of computer systems and networks is an increasingly visible issue. The proliferation of peer-to-peer networks, the ubiquity of copyrighted material available on the Internet, and the expanding bandwidth available to many users make it trivial to locate and obtain music and video files and other protected content of all kinds. System administrators are being called upon to recognize infringing behavior of their users and to prevent it from happening. They recognize the profound tension between facilitating wide-open access to the information society and the need to comply with laws that protect intellectual property rights. This tutorial will survey the fundamentals of intellectual property protection in the context of the system administrator's responsibilities. It will then discuss in some detail new laws and court cases that have addressed the scope of intellectual property protection in the context of electronic access and distribution. Attendees will gain an increased appreciation for the complexity of the issues, the pace at which the law is addressing them, and the parameters of the system administrator's responsibilities in the face of legal uncertainties. Topics include: Fundamentals of intellectual property law for the system administrator Copyright Trade secrecy Patent law and trademarks Copyright term extension and the expanding rights of copyright owners The DMCA: How does it affect system administrators? The Grokster case: What's new from the Supreme Court? File sharing after Grokster Trends in intellectual property protection abroad What should a sysadmin do or refrain from doing? Daniel L. Appelman (M8) is a lawyer in the Silicon Valley office of a major international law firm. He has been practicing in the areas of cyberspace and software law for many years. He was the lawyer for Berkeley Software Design in the BSDi/UNIX System Laboratories (AT&T) case. Dan is the attorney for the USENIX Association and for many tech companies. He is also founding chair of his firm's Information Technology practice group, is the former chair of the California Bar's Standing Committee on Cyberspace Law, and is a current member of the California Bar Business Law Section's Executive Committee, the Computer Law Association, and the American Bar Association's Cyberspace Committee. M9 Regular Expression Mastery Chip Salzenberg, Cloudmark, Inc. 9:00 a.m.–12:30 p.m. Who should attend: System administrators and users who use Perl, grep, sed, awk, procmail, vi, or emacs. Almost everyone has written a regex that produced unexpected results. Sometimes regexes appear to hang forever, and it's not clear what has gone wrong. Sometimes they behave differently in different utilities, and you can't tell why. This class will fix all these problems. The first section of the class will explore the matching algorithms used internally by common utilities such as grep and Perl. Understanding these algorithms will allow us to predict whether a regex will match, which of several matches will be found, and which regexes are likely to be faster than others, and to understand why all of these behaviors occur. We'll learn why commonly used regex symbols such as ".," "$." and "\1" may not mean what you thought they did. In the second section, we'll look at common matching disasters, a few practical parsing applications, and some advanced Perl features. We'll finish with a discussion of optimizations that were added to Perl 5.6, and why you should avoid using "/i." Topics include: Inside the regex engine Regular expressions are programs Backtracking NFA vs. DFA POSIX and Perl Quantifiers Greed and anti-greed Anchors and assertions Backreferences Disasters and optimizations Where machines come from Disaster examples Tokenizing New optimizations Matching strings with balanced parentheses Chip Salzenberg (M9, M13) is a well-known figure in the Perl and free/open source communities. Chip's been working with, and on, free and open source software for 20 years, and specifically Perl for over 15 years. In 1996 and 1997, Chip was project manager ("pumpking") for Perl 5.4, a release widely praised for its high quality. Chip teaches and has been published on Perl and other subjects. During the day he masquerades as a mild-mannered, spam-fighting programmer at Cloudmark, Inc.; but his secret identity is Architect of the Parrot virtual machine. M10 Backup on a Budget W. Curtis Preston, Glasshouse 9:00 a.m.–12:30 p.m. Who should attend: Administrators who need to back up their systems reliably but are on a limited budget. Many computing environments cannot afford a commercial backup and recovery package. Many more can afford the basic package but cannot afford add-ons to handle databases and bare-metal recovery. What can they do? There are more good answers than ever before to that question. Save your precious dollars for hardware and learn about the really good free software and techniques that can bring enterprise-level backup to even the smallest shop. His O'Reilly book UNIX Backup and Recovery is about to enter a second edition under the title Free Backup and Recovery: Basic Data Protection. Topics include: The use of disk in a low-budget backup system Open source backup packages AMANDA Bakula rsync snapshots Backing up Windows, NetWare, and Macintosh systems with open source tools Bare metal recovery techniques Solaris, including te new Flash Archive AIX HP-UX, including an updated make_recovery Windows, using a technique introduced six years ago at LISA Linux Macintosh Database backup and recovery basics Oracle Exchange SQL Server Sybase DB2 W. Curtis Preston (M10) is VP Data Protection for Glasshouse, a storage consulting firm focused on bridging the gap between the business and storage products. Curtis has twelve years of experience in designing storage systems for many environments, both large and small. As a recognized expert in the field, Curtis has advised the major product vendors regarding product features and implementation methods. Curtis is the administrator of the NetBackup and NetWorker FAQs and answers the "Ask The Experts" backup forum on SearchStorage.com. He is also the author of O'Reilly's UNIX Backup & Recovery and Using SANs & NAS, the co-author of the SAGE Short Topics booklet Backups and Recovery, and a contributing editor to Storage Magazine. Monday Afternoon Half-Day Tutorials M11 Google-Driven Web Development Deryck Hodge, Samba Team/Auburn University 1:30 p.m.–5:00 p.m. Who should attend: System administrators and Webmasters who are called upon to build and manage Web applications. Code examples for browser-based tools will use Javascript, and scripting examples will use Python. We will, however, discuss how to apply these examples in the programming language of your choice. For inspiration in building our Web apps, we'll look to Google, the most-used Web app today. We'll examine everything from Google maps and Gmail to Google's available APIs. The goal of this tutorial is to enable you to use the same tools as Google—Javascript, CSS/DOM, and Google search data—to build anything from a simple script to gather Web data to a sophisticated Web-based application. We'll look at building: A module for searching your own site without using server-side scripting A command-line program to mine Google's vast store of Web data A system to monitor your site's presence in Google's index and related-keyword searches A script to gather data from Google and build a Google map to display relevant locations Topics include: Google search syntax and services XHTML, Javascript, CSS, and DOM use in Google Gmail's Javascript UI Engine XMLHttpRequest/XSLTProcessor use in Google Maps Google APIs, SOAP, and WSDL Deryck Hodge (M11) is the current Webmaster for https://www.samba.org/ and https://news.samba.org. He has been instrumental is redesigning Samba's Web site to adhere to Web standards and follows similar pursuits while working for the Auburn University Libraries IT department.     M12 Introduction to Host Configuration and Maintenance with Cfengine Mark Burgess, Oslo University College 1:30 p.m.–5:00 p.m. Who should attend: System administrators with a minimal knowledge of a scripting language who wish to start using cfengine to automate the maintenance and security of their systems. UNIX administrators will be most at home in this tutorial, but cfengine can also be used on Windows 2000 and above. Cfengine is a tool for setting up and maintaining a configuration across a network of hosts. It is sometimes called a tool for "Computer Immunology"—your computer's own immune system. You can think of cfengine as a very high level language, much higher-level than Perl or shell, together with a smart agent. The idea behind cfengine is to create a single "policy" or set of configuration files that describes the setup of every host on your network, without sacrificing their autonomy. Cfengine runs on every host and makes sure that it is in a policy-conformant state; if necessary, any deviations from policy rules are fixed automatically. Unlike tools such as rdist, cfengine does not require hosts to open themselves to any central authority, nor to subscribe to a fixed image of files. It is a modern tool, supporting state-of-the-art encryption and IPv6 transport, that can handle distribution and customization of system resources in huge networks (tens of thousands of hosts). Cfengine runs on hundreds of thousands of computers all over the world. Topics include: The components of cfengine and how they are used How to get the system running How to develop a suitable policy, step by step Security Examples How to customize cfengine for special tasks Mark Burgess (M12, T8, R8) is Professor of Network and System Administration at Oslo University College, Norway. He is the author of the configuration management system cfengine and of several books and many papers on the topic.     M13 Welcome to My ~/bin Chip Salzenberg, Cloudmark, Inc. 1:30 p.m.–5:00 p.m. Who should attend: System administrators and Perl jockeys who want to learn by example and think in Perl. Over the years I've built up a large collection of handy utilities in Perl. I'll take you on a tour of these utilities and show you what they do and how they work. Topics include (depending on student requests and instructor whim): attach: send email with MIME attachments f: replacement for awk forge: forge email messages googles: what google searches are leading people to my Web site? localtime: process log file timestamps locate: locate any file on the system mailhold: challenge-response for incoming email makethumbnails: build image thumbnail pages mark: manage collections of email messages mypsmerge, mypstrim, mypsup2: PostScript formatting and conversion pgrep: replacement for grep printd: replacement printer daemon psgrep: ps | grep sortby: sort the contents of a mail folder sw: 21st-century way to run a job in the background tail and ftail: replacements for the standard tail utility ticker: watch files grow unrecv: make email archive directories smaller watcher: watch a collection of Web pages and report whenever one changes Chip Salzenberg (M9, M13) is a well-known figure in the Perl and free/open source communities. Chip's been working with, and on, free and open source software for 20 years, and specifically Perl for over 15 years. In 1996 and 1997, Chip was project manager ("pumpking") for Perl 5.4, a release widely praised for its high quality. Chip teaches and has been published on Perl and other subjects. During the day he masquerades as a mild-mannered, spam-fighting programmer at Cloudmark, Inc.; but his secret identity is Architect of the Parrot virtual machine. M14 The Latest Hacking Tools and Defenses David Rhoades, Maven Security Consulting, Inc. 1:30 p.m.–5:00 p.m. Who should attend: Anyone who's interested in how hackers work these days, and what system and network administrators can do to defend themselves. We'll examine the latest developments in hacker tools and techniques. Live demos of tools will be given as time permits, and defenses against the tools will be discussed. Bonus: A look at some recently headlined cybercrimes, with an emphasis on the techniques used. Topics may include: VoIP security Phishing Reverse engineering Anti-forensics Wi-Fi and Bluetooth Web application attacks Spyware and malware Network tools Denial of service attacks David Rhoades (S9, M14, T13) is a principal consultant with Maven Security Consulting, Inc. Since 1996, David has provided information protection services for various FORTUNE 500 customers. His work has taken him across the U.S. and abroad to Europe and Asia, where he has lectured and consulted in various areas of information security. David has a B.S. in computer engineering from the Pennsylvania State University and has taught for the SANS Institute, the MIS Training Institute, and ISACA.

Tuesday, December 6, 2005

Full-Day Tutorials

T1 Network Security Monitoring with Open Source Tools Richard Bejtlich, TaoSecurity.com 9:00 a.m.–5:00 p.m. Who should attend: Engineers and analysts who detect and respond to security incidents. Participants should be familiar with TCP/IP. Command-line knowledge of BSD, Linux, or another UNIX-like operating system is a plus. A general knowledge of offensive and defensive security principles is helpful. This tutorial will equip participants with the theory, tools, and techniques to detect and respond to security incidents. Network Security Monitoring (NSM) is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. NSM relies upon alert data, session data, full content data, and statistical data to provide analysts with the information needed to achieve network awareness. Whereas intrusion detection cares more about identifying successful and usually known attack methods, NSM is more concerned with providing evidence to scope the extent of an intrusion, assess its impact, and propose efficient, effective remediation steps. NSM theory will help participants understand the various sorts of data that must be collected. This tutorial will bring theory to life by introducing numerous open source tools for each category of NSM data. Attendees will be able to deploy these tools alongside existing commercial or open source systems to augment their network awareness and defensive posture. Topics include: NSM theory Building and deploying NSM sensors Accessing wired and wireless traffic Full content tools: Tcpdump, Ethereal/Tethereal, Snort as packet logger Additional data analysis tools: Tcpreplay, Tcpflow, Ngrep, Netdude Session data tools: Cisco NetFlow, Fprobe, Flow-tools, Argus, SANCP Statistical data tools: Ipcad, Trafshow, Tcpdstat, Cisco accounting records Sguil (sguil.sf.net) Case studies, personal war stories, and attendee participation Material in the class is supported by the author's book The Tao of Network Security Monitoring: Beyond Intrusion Detection (Addison-Wesley, 2005; https://www.taosecurity.com/books.html). Richard Bejtlich (T1, W1, R1) is founder of TaoSecurity (https://www.taosecurity.com), a company that helps clients detect, contain, and remediate intrusions using network security monitoring (NSM) principles. Richard was previously a principal consultant at Foundstone, performing incident response, emergency NSM, and security research and training. He has created NSM operations for ManTech International Corporation and Ball Aerospace & Technologies Corporation. From 1998 to 2001 then-Captain Bejtlich defended global American information assets in the Air Force Computer Emergency Response Team (AFCERT), performing and supervising the real-time intrusion detection mission. T2 Linux Network Service Administration Joshua Jensen, Cisco Systems Inc. 9:00 a.m.–5:00 p.m. Who should attend: System administrators who are implementing network services and are looking for a background in the configuration of those services, as well as basics of the protocols. Attendees should have some network client/server experience and have a basic knowledge of UNIX administration, but do not need to be experienced network administrators. Both new and intermediate network administrators will leave the tutorial having learned something. From a stand-alone client attached to the Internet to a distributed network of Web servers, systems administrators are being tasked with bringing their office environments online. The network services that need to be configured in order to do this can be daunting to administrators who aren't familiar with the required applications. Configuration examples as well as overviews of the underlying protocols will give attendees the tools to implement services on their own systems. Topics include (with a special emphasis on security): Overview Network services SSH—Secure Shell with OpenSSH FTP—Explore vsftpd HTTP—Apache and Tux and Squid SMTP—Postfix MTA NFS—Network File Systems LDAP—Global authentication with OpenLDAP DHCP—DHCPD and PXE DNS—ISC's BIND NTP—Network Time LPD—Printing with cups Host-based security with TCP wrappers and xinetd Linux packet filtering Network monitoring and logging Network utilities you should be using At the completion of the course, attendees should feel confident in their ability to set up and maintain secure network services. The tutorial will be conducted in an open manner that encourages question-and-answer interruption. Joshua Jensen (S7, T2) has worked for IBM and Cisco Systems, and was Red Hat's first instructor, examiner, and RHCE. He worked with Red Hat for four and a half years, during which he wrote and maintained large parts of the Red Hat curriculum: Networking Services and Security, System Administration, Apache and Secure Web Server Administration, and the Red Hat Certified Engineer course and exam. Joshua has been working with Linux since 1996 and finds himself having come full circle: he recently left IBM to work with Red Hat Linux for Cisco Systems. In his spare time he dabbles in cats, fish, boats, and frequent flyer miles. T3 Advanced Topics in System Administration Trent R. Hein and Ned McClain, Applied Trust Engineering 9:00 a.m.–5:00 p.m. Who should attend: System and network administrators who are interested in picking up several new technologies in an accelerated manner. The six main topics are all focused on performance and availability. Topics include: Web Server Performance Management If you can't measure it, you can't fix it! Using examples from Apache and Linux, we'll investigate tools and specific metrics that can be used to measure performance and identify bottlenecks. Later, we will discuss strategies for addressing various bottlenecks, from the network and storage infrastructure to CGI applications and static content. Security Crisis Case Studies #3 Before your very eyes, we'll dissect a set of real-life security incident case studies, using many tools available on your system or from the Net. We'll specifically describe how to avoid common security-incident pitfalls, and we'll cover the basics of incident investigation. Revision Control for SysAdmins Every good programmer uses revision control on a day-to-day basis, but many sysadmins are unaware of its many benefits. We'll look at practical ways to use CVS to track changes to important system and application files; add-on tools and configuration features; and realistic revision control processes focused on system administration. Linux box == VOIP Phone Switch Learn the basics of SIP VOIP communication, using your Linux box as a fully-featured phone switch/PBX for your small or medium-sized business. We'll also examine basic infrastructure accommodations to handle VOIP within your network. Crash Course in Database Administration As sysadmins, we're often forced to deal with database issues. This session provides an overview of key database administration tasks, including backups, monitoring, performance tuning, and general database management. Although most of our examples will be taken from MySQL and Oracle, the concepts should apply to managing any relational database. Packet Trace Analysis This in-depth look at network packet trace analysis will give you the skills you need to investigate, isolate, and resolve tricky problems in your environment. Using freely available tools, we'll show you how to shine a bright light on those troublesome network performance mysteries. Trent R. Hein (T3) is co-founder of Applied Trust Engineering, a leader in holistic infrastructure and security. Trent worked on the 4.4 BSD port to the MIPS architecture at Berkeley, is co-author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook, and holds a B.S. in Computer Science from the University of Colorado. Ned McClain (T3), co-founder and CTO of Applied Trust Engineering, lectures around the globe on applying cutting-edge technology in production computing environments. Ned holds a B.S. in Computer Science from Cornell University and is a contributing author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook.   T5 Help! Everyone Hates Our IT Department! Tom Limoncelli, Cibernet Corp. 9:00 a.m.–5:00 p.m. Who should attend: Managers and system administrators who feel "the users hate us" and want to improve the situation quickly; sysadmins with large user populations, especially those with large desktop user communities; anyone who wants to manage the help desk, desktop deployment, and PC refresh cycles better. With a bow to the popularity of TV makeover shows, we're proud to present Limoncelli Eye for the IT Guy/Gal! Based on the top tips from the The Practice of System and Network Administration, this day-long tutorial teaches how to "make over" your IT department. Topics include: Looking good: improving your IT department's visibility Getting love: the secret to making users feel they are the center of the universe Giving love: communicating to users effectively Making that great first impression on your users Help desks (both real and virtual) Pros and cons of formal help desks How to create and manage a help desk Survey of request and ticket systems Customer care: a 9-step troubleshooting process Knowing what's wrong before they do Monitoring services Historical trend analysis Should you have a NOC (Network Operations Center)? Tom Limoncelli (T5, R5), author of O'Reilly's Time Management for System Administrators and co-author of The Practice of System and Network Administration from Addison-Wesley, is Director of IT Services at Cibernet Corp. A sysadmin and network wonk since 1987, he has worked at Dean for America, Lumeta, Bell Labs/Lucent, Mentor Graphics, and Drew University. He is a frequent presenter at LISA conferences.   T6 Implementing LDAP Directories Gerald Carter, Samba Team/Hewlett-Packard 9:00 a.m.–5:00 p.m. Who should attend: Both LDAP directory administrators and architects. The focus is on integrating standard network services with LDAP directories. The examples are based on UNIX hosts and the OpenLDAP directory server and will include actual working demonstrations throughout the course. System administrators today run a variety of directory services, although these are referred to by names such as DNS and NIS. The Lightweight Directory Access Protocol (LDAP) is the successor to the X.500 directory and has the promise of allowing administrators to consolidate multiple existing directories into one. Topics include: Replacing NIS domains Integration with Samba file and print servers Integrating MTAs such as Sendmail and Postfix Creating address books for mail clients Managing user access to HTTP and FTP services Integrating with DHCP and DNS servers Scripting with the Net::LDAP Perl module Defining custom attributes and object classes Gerald Carter (T6, W3, W8, F1) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing. Tuesday Morning Half-Day Tutorials T7 Over the Edge System Administration, Vol. 1 David N. Blank-Edelman, Northeastern University 9:00 a.m.–12:30 p.m. Who should attend: Old-timers who think they've already seen it all, and those who want to develop inventive thinking early in their career. Join us and be prepared to be delighted, disgusted, and amazed. Most of all, be ready to enrich your network and system adminstration by learning to be different. It's time to learn how to break the rules, abuse the tools, and generally turn your system administration knowledge inside out. This class is a cornucopia of ideas for creative ways to take the standard (and sometimes not-so-standard) system administration tools and techniques and use them in ways no one would expect. We'll also cover some tools you may have missed. Topics include: How to (ab)use perfectly good network transports by using them for purposes never dreamed of by their authors How to increase user satisfaction during downtimes with 6 lines of Perl How to improve your network services by intentionally throwing away data How to drive annoying Web-only applications that don't have a command line interface—without lifting a finger How to use ordinary objects you have lying around the house, such as Silly Putty, to make your life easier (seriously!) Note: The teacher takes no responsibility should your head explode during this class. David N. Blank-Edelman (T7) is the Director of Technology at the Northeastern University College of Computer and Information Science and the author of the O'Reilly book Perl for System Administration. He has spent the last 19 years as a system/network administrator in large multi-platform environments, including Brandeis University, Cambridge Technology Group, and the MIT Media Laboratory. He has given several successful invited talks off the beaten path at LISA and is the chair of this year's conference. T8 Advanced Topics in Host Configuration and Maintenance with Cfengine Mark Burgess, Oslo University College 9:00 a.m.–12:30 p.m. Who should attend: System administrators with a working knowledge of cfengine (or who have attended the introductory course) and who wish to extend their understanding of cfengine with examples and usage patterns. UNIX and Mac OS X administrators will be most at home in this tutorial, but cfengine can also be used on Windows 2000 and above. Cfengine contains many features and facilities that make it a powerful tool for system administration, but it has a large manual that is difficult to absorb without training. In this tutorial we assume that attendees have a basic understanding of how cfengine works and would like to develop a number of "best practices" and examples to maximize their returns. Topics include: Review of some basics Automating deployment of software throughout your infrastructure UNIX/Mac/Windows update.conf cron and cfexecd When to run Integrating data from information sources Structure and organization of config The overlapping-set model Import Modules Methods When to use these tools Special functions and variables Variables, scalars, arrays Associative arrays and their limitations ExecResult, ReturnsZero, etc. ReadArray, ReadList, etc. IsNewerThan, IsDir, etc. Searching, matching, and wildcards Search filters Regular expressions Wildcard expansions How does cfagent evaluate things? Thinking declaratively Ordering: When does it matter? Locks; What are they, and why are they there? Iteration over lists Control, actionsequence, alerts Services and security PP keys and exchange (trust model) Authentication stages Rule orderings IPv6 issues Peer-to-peer services Example: Backing up laptops Host monitoring cfenvd Interfacing to tcpdump Understanding cfenvgraph output PeerCheck neighborhood watch FriendStatus function Future developments and discussion Mark Burgess (M12, T8, R8) is Professor of Network and System Administration at Oslo University College, Norway. He is the author of the configuration management system cfengine and of several books and many papers on the topic.   T9 Disk-to-Disk Backup and Eliminating Backup System Bottlenecks Jacob Farmer, Cambridge Computer Corp. 9:00 a.m.–12:30 p.m. Who should attend: System administrators involved in the design and management of backup systems and policymakers responsible for protecting their organization's data. A general familiarity with server and storage hardware is assumed. The class focuses on architectures and core technologies and is relevant regardless of what backup hardware and software you currently use. Students will leave this lecture with immediate ideas for effective, inexpensive improvements to their backup systems. The end may finally be in sight for the pains of backup and restore. The cost of disk storage has crossed the line to where it is finally practical to use disk to enhance or replace tape-based backup systems. In turn, software applications have come to market to facilitate the use of disk in backup systems. Now the problem is sorting out all of the options and reconciling them with your existing infrastructure. This tutorial identifies the major bottlenecks in conventional backup systems and explains how to address them. The emphasis is placed on the various roles for inexpensive disk in your data protection strategy; however, attention is given to SAN-enabled backup, the current state and future of tape drives, iSCSI, and virtual tape. Topics include: Identifying and eliminating backup system bottlenecks Conventional disk staging Virtual tape libraries Incremental forever and synthetic full backup strategies Information lifecycle management and nearline archiving Data replication Continuous backup Snapshots Current and future tape drives Zero duplication file systems iSCSI Jacob Farmer (T9) is a well-known figure in the data storage industry. He has authored numerous papers and articles and is a regular speaker at trade shows and conferences. In addition to his regular expert advice column in the "Reader I/O" section of InfoStor Magazine, the leading trade magazine of the data storage industry, Jacob also serves as the publication's senior technical advisor. Jacob has over 18 years of experience with storage technologies and is the CTO of Cambridge Computer Services, a national integrator of data storage and data protection solutions.   T10 Taming the Wild Project Strata Rose Chalup, Project Management Consultant 9:00 a.m.–12:30 p.m. Who should attend: Anyone with an existing project that isn't going well, and they're not sure why, or with a big initiative at work that they'd like to turn into a project but can't seem to get beyond a certain point with it; anyone who's been getting involved with open source software development, and things have gotten complex now that more folks are on board. If you've been thinking, "Hey, if we had a little more structure, we could get a lot more accomplished," this tutorial is for you. It's likely, but not strictly required, that you've taken some kind of project management training or done some reading on your own. As for me: I've been pulling clients' projects out of the fire for years. As a career consultant, I'm constantly running into the "When all else fails, hire a consultant" syndrome. I've seen projects without a plan, plans without a project, and just about everything in between—including a lot of busy people who don't seem to know what the common goal is, or even whether there is one! So come on down, bring your laptop, your notes, and your questions, and get your project back on track. Strata Rose Chalup (T10, T14, W5) began as a fledgling sysadmin in 1983 and has been leading and managing complex IT projects for many years, serving in roles ranging from Project Manager to Director of Network Operations. She has written a number of articles on management and working with teams and has applied her management skills on various volunteer boards, including BayLISA and SAGE. Strata has a keen interest in network information systems and new publishing technologies and built a successful consulting practice around being an avid early adopter of new tools, starting with ncsa_httpd and C-based CGI libraries in 1993 and moving on to wikis, RSS readers, and blogging. Another MIT dropout, Strata founded VirtualNet Consulting in 1993. Tuesday Afternoon Half-Day Tutorials T11 Databases: What You Need to Know John Sellens, SYONEX 1:30 p.m.–5:00 p.m. Who should attend: System and application administrators who need to support databases and database-backed applications. Databases used to run almost exclusively on dedicated database servers, with one or more database administrators (DBAs) dedicated to their care. These days, with the easy availability of database software such as MySQL and PostgreSQL, databases are popping up in many more places, and are used by many more applications. As a system administrator you need to understand databases, their care and feeding. Attendees will leave the tutorial with a better understanding of databases and their use and will be ready to deploy and support common database software and database-backed applications. Topics include: An introduction to database concepts The basics of SQL (Structured Query Language) Common applications of databases Berkeley DB and its applications MySQL installation, configuration, and management PostgreSQL installation, configuration, and management Security, user management, and access controls Ad-hoc queries with standard interfaces ODBC and other access methods Database access from other tools (Perl, PHP, sqsh, etc.) John Sellens (S5, M5, T11) has been involved in system and network administration since 1986 and is the author of several related USENIX papers, a number of ;login: articles, and the SAGE Short Topics in System Administration booklet #7, System and Network Administration for Higher Reliability. He holds an M.Math. in computer science from the University of Waterloo and is a chartered accountant. He is the proprietor of SYONEX, a systems and networks consultancy. From 1999 to 2004, he was the General Manager for Certainty Solutions in Toronto. Prior to joining Certainty, John was the Director of Network Engineering at UUNET Canada and was a staff member in computing and information technology at the University of Waterloo for 11 years. T12 Solaris 10 Security Features Workshop Peter Baer Galvin, Corporate Technologies, Inc. 1:30 p.m.–5:00 p.m. Who should attend: Solaris systems managers and administrators interested in the new security features in Solaris 10 (and features in previous Solaris releases that they may not be using). This course covers a variety of topics surrounding Solaris 10 and security. Solaris 10 includes many new features, and there are new issues to consider when deploying, implementing, and managing Solaris 10. This will be a workshop featuring instruction and practice/exploration. Each student should have a laptop with wireless access for remote access into a Solaris 10 machine. Topics include: Solaris cryptographic framework NFSv4 Solaris privileges Solaris Flash archives and live upgrade Moving from NIS to LDAP Dtrace WBEM Smartcard interfaces and APIs Kerberos enhancements Zones FTP client and server enhancements PAM enhancements Auditing enhancements Password history checking ipfilters Peter Baer Galvin (M2, T12) is the Chief Technologist for Corporate Technologies, Inc., a systems integrator and VAR, and was the Systems Manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines. He wrote the "Pete's Wicked World" and "Pete's Super Systems" columns at SunWorld. He is currently contributing editor for Sys Admin, where he manages the Solaris Corner. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials on security and system administration and has given talks at many conferences and institutions on such topics as Web services, performance tuning, and high availability. T13 In-depth Topics for Web Application Security David Rhoades, Maven Security Consulting, Inc. 1:30 p.m.–5:00 p.m. Who should attend: People who are designing and/or developing Web applications, or managing the deployment of a Web application. Participants should have working knowledge of HTTP v1.1. Experience administering or configuring Apache is a plus. This course will cover in depth a variety of topics for enhancing the overall security of the Web application infrastructure. Practical steps for implementation will be the focus. Topics include: Securing database access Identifying attacks by analyzing web logs Implementing open source application firewalls, including Apache's mod_security David Rhoades (S9, M14, T13) is a principal consultant with Maven Security Consulting, Inc. Since 1996, David has provided information protection services for various FORTUNE 500 customers. His work has taken him across the U.S. and abroad to Europe and Asia, where he has lectured and consulted in various areas of information security. David has a B.S. in computer engineering from the Pennsylvania State University and has taught for the SANS Institute, the MIS Training Institute, and ISACA. T14 RSS vs. Information Overload Strata Rose Chalup, Project Management Consultant 1:30 p.m.–5:00 p.m. Who should attend: People who want to manage incoming information streams and go "on beyond Slashdot"; people who never heard of RSS before Microsoft announced it was going to do an embrace/extend/exterminate on it. There are so many sources of information out there that keeping up can be a big challenge. Wading through folders of postings to various lists, even quickly scanning the digest version, is fundamentally not scalable. What if I told you there's a tool out there designed for such things, which can publish headlines of articles, aggregate them into a reading interface, and even be used to fetch (or pre-fetch) the content? Better yet, these tools are a natural fit for managing some kinds of system information. And, like any hammer, RSS and its cohorts will undoubtedly be used to pound on things that were never nails. Look at the uses the Web is put to nowadays simply because it is a robust, simple, well-defined protocol, although it was never intended or designed for them. RSS is in the same boat. After completing this tutorial, participants will have an understanding of how to harness RSS feeds for information management, the tradeoffs among various publishing methods, and the toolkits available for working with RSS. We'll discuss methods whereby RSS can augment traditional system logging tools such as syslog and swatch, as well as hook into conventional distribution tools such as mailman and majordomo. Class materials will include pointers to RSS clients for a wide range of platforms. Topics include: RSS basics Origins and standards Growing pains: Tim, Dave, and a cast of hundreds RSS 2.0: a new beginning? RSS in context XML, DHTML, and RSS Where does Tibco fit in? The mod_pubsub model Weed 'n' feed Publishing basics Reputation communities (Syndic8 et al.) Atom: RSS on steroids, or annoyance? Bonus: what's this "tagging" stuff, and do I give a damn? Getting the goodies Aggregation clients Pre-fetch or post-fetch? Archiving feeds If I had a hammer . . . Toolkits and libraries Server-side fun for everyone Client building blocks Applied RSS syslog and MRTG: the low-hanging fruit Filtering and tagging Bugzilla and Wiki hooks Augmenting ticket systems Next generation Proposed RSS extensions Microsoft gets on the bandwagon Malice aforethought Scaling aspects to consider The coming deluge: spamvertising via RSS Security caveats Strata Rose Chalup (T10, T14, W5) began as a fledgling sysadmin in 1983 and has been leading and managing complex IT projects for many years, serving in roles ranging from Project Manager to Director of Network Operations. She has written a number of articles on management and working with teams and has applied her management skills on various volunteer boards, including BayLISA and SAGE. Strata has a keen interest in network information systems and new publishing technologies and built a successful consulting practice around being an avid early adopter of new tools, starting with ncsa_httpd and C-based CGI libraries in 1993 and moving on to wikis, RSS readers, and blogging. Another MIT dropout, Strata founded VirtualNet Consulting in 1993.

Wednesday, December 7, 2005

Full-Day Tutorials

W1 Network Incident Response Richard Bejtlich, TaoSecurity.com 9:00 a.m.–5:00 p.m. Who should attend: Security staff and sysadmins who detect and respond to intrusions. Participants should be familiar with TCP/IP. Command-line knowledge of BSD, Linux, or a UNIX-like operating system is a plus. A general knowledge of offensive and defensive security principles is helpful. The author's USENIX course "Network Security Monitoring with Open Source Tools" (T1) and his book The Tao of Network Security Monitoring: Beyond Intrusion Detection are very helpful prerequisites, but they are not mandatory. You've just discovered that one or more of your systems has been compromised. Now what? This tutorial will answer that question from a network-centric approach. It is based on the author's experience handling multiple systematic, long-term compromises at a variety of enterprises. The majority of the course will approach the incident response (IR) problem from the network perspective; host-based forensics will not be a priority. Attendees will first learn the basic steps needed to facilitiate incident response prior to any compromise. Thoughts on the sorts of threats likely to be faced, common intrusion scenarios, and ways to be aware of intruder activities will be discussed. Next, attendees will hear about various means by which incidents are discovered, all based on real life intrusions. The course will cover how to perform first response actions from the network perspective, and how to make the "pursue and prosecute" or "recover and remediate" decision. Attendees will learn how to eject determined, patient, and stealthy intruders from the enterprise, and how to verify the effectiveness of ongoing defensive measures. Topics include: Simple steps to take now that make incident response easier later Characteristics of intruders, such as their motivation, skill levels, and techniques Common ways intruders are detected, and reasons they are often initially missed Improved ways to detect intruders based on network security monitoring principles First response actions and related best practices Secure communications among IR team members, and consequences of negligence Approaches to remediation when facing a high-end attacker Short, medium, and long-term verification of the remediation plan to keep the intruder out Richard Bejtlich (T1, W1, R1) is founder of TaoSecurity (https://www.taosecurity.com), a company that helps clients detect, contain, and remediate intrusions using network security monitoring (NSM) principles. Richard was previously a principal consultant at Foundstone, performing incident response, emergency NSM, and security research and training. He has created NSM operations for ManTech International Corporation and Ball Aerospace & Technologies Corporation. From 1998 to 2001 then-Captain Bejtlich defended global American information assets in the Air Force Computer Emergency Response Team (AFCERT), performing and supervising the real-time intrusion detection mission. W2 System and Network Performance Tuning Marc Staveley, Soma Networks 9:00 a.m.–5:00 p.m. Who should attend: Novice and advanced UNIX system and network administrators, and UNIX developers concerned about network performance impacts. A basic understanding of UNIX system facilities and network environments is assumed. We'll examine the virtual memory system, the I/O system and the file system, NFS tuning and performance strategies, common network performance problems, examples of network capacity planning, and application issues. We'll also cover guidelines for capacity planning and customized monitoring based on your workloads and traffic patterns. Analysis periods for particular situations will be provided. Topics include: Performance tuning strategies Server tuning Filesystem and disk tuning Memory consumption and swap space System resource monitoring NFS issues Automounter and other tricks Network performance, design, and capacity planning Application tuning System resource usage Memory allocation Code profiling Job scheduling and queuing Real-time issues Managing response time Marc Staveley (W2) works with Soma Networks, where he is applying his many years of experience with UNIX development and administration in leading their IT group. Previously Marc had been an independent consultant and also held positions at Sun Microsystems, NCR, Princeton University, and the University of Waterloo. He is a frequent speaker on the topics of standards-based development, multi-threaded programming, system administration, and performance tuning. Wednesday Morning Half-Day Tutorials W3 Kerberos 5: Revenge of the Three-Headed Dog Gerald Carter, Samba Team/Hewlett-Packard 9:00 a.m.–12:30 p.m. Who should attend: Administrators who want to understand Kerberos 5 implementations on both UNIX/Linux and Windows clients and servers. For many organizations, Kerberos is an an old technology that has been driven to the forefront by deployments of Microsoft Active Directory domains. The introduction of a standard authentication protocol into Windows domains has caused many network administrators to reexamine ways to integrate UNIX/Linux and Windows clients in a single authentication model. Topics include: Key concepts of the Kerberos 5 protocol Related authentication interfaces such as SASL and GSSAPI The specifics of implementing Krb5 realms Implementations of Krb5 cross-realm trusts Integration of Windows and UNIX/Linux clients into Krb5 realms Possible pitfalls of using popular Krb5 implementations such as those of MIT and Windows 200x Gerald Carter (T6, W3, W8, F1) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing. W4 Sure, You Can Archive Data, But Will You Be Able to Retrieve It in Ten Years? Evan Marcus, Archivas Software 9:00 a.m.–12:30 p.m. Who should attend: Anyone who will be held responsible for recovering files from backup. Every business has fixed content data that must be safely stored for the long term. Whether it's medical records, corporate financial data, security data, old photographs, or an MP3 collection, the data must be preserved. The important question is, will you be able to get it back when you need it? In this tutorial we'll look at the three key functions that any data archive must perform: ingestion, preservation, and retrieval. Topics include: Media for storage (tape, optical disks, NAS, SAN, DAS, CAS, etc.) Advantages and disadvantages Expected lifespans How each performs the three key functions The emerging technology of fixed-content archiving Media Hardware and software technologies Security Performance Availability Compliance issues Sarbanes-Oxley HIPAA General concerns for longterm retrieval Evan Marcus (W4, W7) joined Archivas, Inc., in 2005 as a Senior Systems Engineer in the Office of the CTO. He has more than 15 years of experience in UNIX systems. Before joining Archivas, he spent 8 years at VERITAS Software, as a systems engineer, speaker, and author. He also spent 5 years at Sun Microsystems, and 2+ years at Fusion Systems, where he worked to bring the first high availability software applications for SunOS and Solaris to market. He also spent 2 years as a system administrator on the equities trading floor of a multinational trading institution. He is the co-author of Blueprints for High Availability from John Wiley & Sons and co-author and co-editor of The Resilient Enterprise from VERITAS Publications. He is a well-regarded and popular speaker on the design of highly available and disaster resilient systems, and on fixed-content storage archives. W5 Practical Project Management for Sysadmins and IT Professionals Strata Rose Chalup, Project Management Consultant 9:00 a.m.–12:30 p.m. Who should attend: System administrators who want to stay hands-on as team leads or system architects and need a new set of skills with which to tackle bigger, more complex challenges. No previous experience with project management is required. Participants will get a no-nonsense grounding in methods that work without adding significantly to one's workload. After completing this tutorial, participants will be able to take an arbitrarily daunting task and reduce it to a plan of attack that will be realistic, lend itself to tracking, and have functional, documented goals. They will be able to give succinct and useful feedback to management on overall project viability and timelines and easily deliver regular progress reports. People who have been through traditional multi-day project management courses will be shocked, yet refreshed, by the practicality of our approach. To get the most out of this tutorial, participants should have some real-world project or complex task in mind for the lab sections. This tutorial focuses on complementing your own organizational style (or lack thereof) with a toolbox of ways to organize and manage complex tasks without drowning in paperwork or clumsy, meeting-intensive methodologies. Also emphasized is how to bridge the gap between ad-hoc methods and the kinds of tracking and reporting traditionally trained managers will understand. Topics include: Quick basics of project management The essentials you need to know How to map the essentials onto real-world projects Skill sets Defining success Chunking and milestoning Delegating Tracking Reporting Problem areas Teams, interactions among people The albatross project When to go deep and when to get "pointy-haired" When disaster strikes, should you scrap, or salvage? Project management tools What tools should do for you Leveraging the command line: UNIX PM Freeware PM tool options The only 15 minutes of MS Project you'll ever need Strata Rose Chalup (T10, T14, W5) began as a fledgling sysadmin in 1983 and has been leading and managing complex IT projects for many years, serving in roles ranging from Project Manager to Director of Network Operations. She has written a number of articles on management and working with teams and has applied her management skills on various volunteer boards, including BayLISA and SAGE. Strata has a keen interest in network information systems and new publishing technologies and built a successful consulting practice around being an avid early adopter of new tools, starting with ncsa_httpd and C-based CGI libraries in 1993 and moving on to wikis, RSS readers, and blogging. Another MIT dropout, Strata founded VirtualNet Consulting in 1993. Wednesday Afternoon Half-Day Tutorials W6 Advanced Shell Programming Mike Ciavarella, University of Melbourne 1:30 p.m.–5:00 p.m. Who should attend: Junior or intermediate system administrators or anyone with a basic knowledge of programming, preferably with some experience in Bourne/Korn shells (or their derivatives). The humble shell script is still a mainstay of UNIX/Linux system administration, despite the wide availability of other scripting languages. This tutorial details techniques that move beyond the quick-and-dirty shell script. Topics include: Common mistakes and unsafe practices Modular shell script programming Building blocks: awk, sed, etc. Writing secure shell scripts Performance tuning Choosing the right utilities for the job Addressing portability at the design stage When not to use shell scripts Mike Ciavarella (W6, R3, R6, F3) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer. W7 Disaster Planning (and Recovery): How to Keep Your Company (and Your Job) Alive Evan Marcus, Archivas Software 1:30 p.m.–5:00 p.m. Who should attend: System administrators and managers who want to know what they need to think about, what they need to plan for (and what they can safely avoid considering), and how to carry out the plan if (God forbid!) disaster ever strikes. Disaster planning is like insurance: nobody wants to talk about it and everyone runs from the salesmen. But when you need it, you are very glad to have it! And if you don't have it when you need it, it is too late to do anything about it. Have you ever been robbed or had an accident or a medical emergency? If you had insurance, you had done personal disaster planning. We will explore the key aspects of developing a disaster recovery plan, including identifying the key components, testing the plan, and some of the technology that can speed recovery, with an eye toward balancing costs and benefits. We will also take a close look at one organization that completely recovered very quickly after 9/11. Topics include: What a DR plan should contain, with real-world examples The costs of developing a plan Why do you need a plan? Legal and civil liabilities of not having a plan Four methods for testing your plan Downtime and data loss: two sides of the same coin DR as a subset of high availability Methods and technologies for protecting data through a disaster How a disaster may affect the people responsible for recovery Building and staff a DR team The role of senior management in DR Convincing management that a DR plan is necessary Case study of a company that survived 9/11 Evan Marcus (W4, W7) joined Archivas, Inc., in 2005 as a Senior Systems Engineer in the Office of the CTO. He has more than 15 years of experience in UNIX systems. Before joining Archivas, he spent 8 years at VERITAS Software, as a systems engineer, speaker, and author. He also spent 5 years at Sun Microsystems, and 2+ years at Fusion Systems, where he worked to bring the first high availability software applications for SunOS and Solaris to market. He also spent 2 years as a system administrator on the equities trading floor of a multinational trading institution. He is the co-author of Blueprints for High Availability from John Wiley & Sons and co-author and co-editor of The Resilient Enterprise from VERITAS Publications. He is a well-regarded and popular speaker on the design of highly available and disaster resilient systems, and on fixed-content storage archives. W8 Ethereal and the Art of Debugging Networks Gerald Carter, Samba Team/Hewlett-Packard 1:30 p.m.–5:00 p.m. Who should attend: System and network administrators who are interested in learning more about the TCP/IP protocol and how network traffic monitoring and analysis can be used as a debugging, auditing, and security tool. The focus of this course is using the Ethereal protocol analyzer as a debugging and auditing tool for TCP/IP networks. System logs can turn out to be incomplete or incorrect when you're trying to track down network application failures. Sometimes the quickest, or the only, way to find the cause is to look at the raw data on the wire. This course is designed to help you make sense of that data. Topics include: Introduction to Ethereal for local and remote network tracing TCP/IP protocol basics Analysis of popular application protocols such as DNS, DHCP, HTTP, NFS, CIFS, and LDAP Security How some kinds of network attacks can be recognized Gerald Carter (T6, W3, W8, F1) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing.

Thursday, December 8, 2005

Full-Day Tutorials

R1 Network Forensics Richard Bejtlich, TaoSecurity.com 9:00 a.m.–5:00 p.m. Who should attend: Security staff and system administrators who detect and respond to intrusions. Participants should be familiar with TCP/IP. Command-line knowledge of BSD, Linux, or a UNIX-like operating system is a plus. A general knowledge of offensive and defensive security principles is helpful. The author's USENIX course "Network Security Monitoring with Open Source Tools" (T1) and his book The Tao of Network Security Monitoring: Beyond Intrusion Detection are very helpful prerequisites, but they are not mandatory. You've just discovered that one or more of your systems have been compromised. You have instituted incident response procedures to contain the intrusion and are planning remediation steps. You want to ensure that you're capturing the proper network-based evidence in a forensically sound manner. You want to handle that evidence such that it can be used to prosecute an offender, and you want to understand exactly what it means. You are also concerned about your ability to explain that evidence to a jury or even to your human resources representative, or to survive questions from adversarial legal counsel. Do you need help? If your answer is yes, this tutorial is for you. Attendees will learn how to address these and related issues. Best practices will be demonstrated, and the course itself will provide an outline for security practitioners who find themselves in the challenging but important role of digital detective. Note that this tutorial will supplement the more prevalent host-based forensic evidence classes found in the security industry. The focus of this class is network-based evidence, which the instructor has found to be as reliable as, and sometimes more reliable than, host-based evidence. A record of this training may also provide additional legitimacy to investigators seeking expert witness status. Topics include: Collecting network traffic as evidence on wired and wireless networks Essential preparation Accessing traffic for collection Protecting and preserving traffic from tampering, either by careless helpers or the intruder himself Analyzing network evidence Open source tools Network security monitoring (NSM) principles Case studies Presenting findings to laypeople, such as management, juries, judges Defending your conclusions in the face of adversarial defense attorneys or skeptical business leaders Richard Bejtlich (T1, W1, R1) is founder of TaoSecurity (https://www.taosecurity.com), a company that helps clients detect, contain, and remediate intrusions using network security monitoring (NSM) principles. Richard was previously a principal consultant at Foundstone, performing incident response, emergency NSM, and security research and training. He has created NSM operations for ManTech International Corporation and Ball Aerospace & Technologies Corporation. From 1998 to 2001 then-Captain Bejtlich defended global American information assets in the Air Force Computer Emergency Response Team (AFCERT), performing and supervising the real-time intrusion detection mission. R2 Advanced Perl Programming Tom Christiansen, Consultant 9:00 a.m.–5:00 p.m. Who should attend: Anyone with a journeyman-level knowledge of Perl programming who wants to hone Perl skills. This class will cover a wide variety of advanced topics in Perl, including many insights and tricks for using these features effectively. After completing this class, attendees will have a much richer understanding of Perl and will be better able to make it part of their daily routine. Topics include: Symbol tables and typeglobs Symbolic references Useful typeglob tricks (aliasing) Modules Autoloading Overriding built-ins Mechanics of exporting Function prototypes References Implications of reference counting Using weak references for self-referential data structures Autovivification Data structure management, including serialization and persistence Closures Fancy object-oriented programming Using closures and other peculiar referents as objects Overloading of operators, literals, and more Tied objects Managing exceptions and warnings When die and eval are too primitive for your taste The use warnings pragma Creating your own warnings classes for modules and objects Regular expressions Debugging regexes qr// operator Backtracking avoidance Interpolation subtleties Embedding code in regexes Programming with multiple processes or threads The thread model The fork model Shared memory controls Unicode and I/O layers Named Unicode characters Accessing Unicode properties Unicode combined characters I/O layers for encoding translation Upgrading legacy text files to Unicode Unicode display tips Tom Christiansen (R2) has been involved with Perl since day zero of its initial public release in 1987. Author of several books on Perl, including The Perl Cookbook and Programming Perl from O'Reilly, Tom is also a major contributor to Perl's online documentation. He holds undergraduate degrees in computer science and Spanish and a Master's in computer science. He now lives in Boulder, Colorado. Thursday Morning Half-Day Tutorials R3 Pretty and Effective: Fast Wins with Graphical Monitoring Mike Ciavarella, University of Melbourne 9:00 a.m.–12:30 p.m. Who should attend: Novice and intermediate system administrators who want to make effective use of graphical monitoring with a minimum of effort. Advanced system administrators may also find the visualisation and planning aspects of this course useful. Some experience with Python, Perl, Bourne shell, or similar is assumed, as well as a cursory knowledge of SNMP and networking. This course examines graphical monitoring with an emphasis on getting effective visual results with a minimum of system administration effort. Examples of effective and not so effective applications of graphical monitoring are drawn from everyday system administration tasks. The tools used are all freely available, and although most are typically run on UNIX hosts, many of the techniques described in class can be applied directly to Windows hosts; examples of this are included. Topics include: Introduction to visualisation and data interpretation Planning your monitoring When to use graphical monitoring tools and when to avoid them MRTG, RRDTOOL, and friends Working with SNMP and other common data sources Internode nodemap Mike Ciavarella (W6, R3, R6, F3) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer. R4 Recovering from Linux Hard Drive Disasters Theodore Ts'o, IBM Linux Technology Center 9:00 a.m.–12:30 p.m. Who should attend: Linux system administrators and users. Ever had a hard drive fail? Ever kick yourself because you didn't keep backups of critical files, or you discovered that your regularly nightly backup didn't succeed? Of course not: you keep regular backups and verify them frequently to make sure they are successful, right? But for those of you who think you might nevertheless someday need this information, this tutorial will discuss ways of recovering from hardware or software disasters. Topics include: Low-level techniques to recover data from a corrupted ext2/ext3 filesystem when backups aren't available Recovering from a corrupted partition table Using e2image to back up critical ext2/3 filesystem metadata Using e2fsck and debugfs to sift through a corrupted filesystem Some measures to avoid needing to use heroic measures Theodore Ts'o (R4) has been a Linux kernel developer since almost the very beginnings of Linux: he implemented POSIX job control in the 0.10 Linux kernel. He is the maintainer and author of the Linux COM serial port driver and the Comtrol Rocketport driver, and he architected and implemented Linux's tty layer. Outside of the kernel, he is the maintainer of the e2fsck filesystem consistency checker. Ted is currently employed by IBM Linux Technology Center. R5 Time Management for System Administrators: Getting It All Done and Not Going (More) Crazy! Tom Limoncelli, Cibernet Corp. 9:00 a.m.–12:30 p.m. Who should attend: Sysadmins who want to improve their time-management skills, who want to have more control over their time and better follow-through on assignments. If you feel overloaded, miss appointments, and forget deadlines and tasks, this class is for you. Do any of these statements sound like you? I don't have enough time to get all my work done. I don't have control over my schedule I'm spending all my time mopping the floor; I don't have time to fix the leaking pipe. My boss says I don't work hard enough, but I'm always working my —— off! Based on a new book from O'Reilly, this tutorial will help you get more done in less time. You'll miss fewer deadlines, be more relaxed at work, and have more fun in your social life. If you think you don't have time to take this tutorial, you really need to take this tutorial! Topics include: Why typical "time management" books don't work for sysadmins How to delegate tasks effectively A way to keep from ever forgetting a user's request Why "to do" lists fail and how to make them work Prioritizing tasks so that users think you're a genius Getting more out of your Palm Pilot Having more time for fun (for people with a social life) How to leave the office every day with a smile on your face Tom Limoncelli (T5, R5), author of O'Reilly's Time Management for System Administrators and co-author of The Practice of System and Network Administration from Addison-Wesley, is Director of IT Services at Cibernet Corp. A sysadmin and network wonk since 1987, he has worked at Dean for America, Lumeta, Bell Labs/Lucent, Mentor Graphics, and Drew University. He is a frequent presenter at LISA conferences. Thursday Afternoon Half-Day Tutorials R6 Documentation Techniques for SysAdmins Mike Ciavarella, University of Melbourne 1:30 p.m.–5:00 p.m. Who should attend: System administrators who need to produce documention for the systems they manage or who want to improve their documentation skills. Attendees should be able to make immediate, practical use of the techniques presented in this tutorial in their day-to-day tasks. Particular emphasis is placed on documentation as a time-saving tool rather than a workload imposition. Topics include: Why system administrators need to document The document life cycle Targeting your audience An adaptable document framework Common mistakes Tools to assist the documentation process Mike Ciavarella (W6, R3, R6, F3) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer. R7 Mastering Massive Changes and Upgrades to Mission-Critical Systems Andrew Cowie, Operational Dynamics 1:30 p.m.–5:00 p.m. Who should attend: Anyone involved in operations and keeping production systems running: system administrators, database people, application developers, and IT management. How do you ensure that you don't make mistakes when carrying out upgrades to mission-critical systems? Massive changes and upgrades are a significant part of the life-cycle of any large site. These types of events are often complex, involving numerous interdependent systems and people both internal and external to the team carrying out the procedure. They can only be allowed to disrupt services minimally, if at all. Numerous people need to be coordinated. And you need to get it right the first time. Databases are often the cornerstone of such high-load mission-critical systems, and they offer unique challenges; for example, an update to the application code often results in schema changes. Similarly, ongoing systems administration work such as patching and reconfiguring cluster configurations requires direct action on production systems that mustn't go down. This tutorial will teach you proven methods for planning, rehearsing, and safely executing such events. Topics include: Know your enemy: learn what can go wrong in a mission-critical event and why preparation needs to be done with precision The best defense is a good offense: methodologies for preparing a sound procedure that will also help you get buy-in from management Beta tests for people: how to conduct effective rehearsals that will accustom people to working together and catch problems at the outset Make it happen: how to execute the procedure, keep people on track, and deal with the unexpected Afterglow: Only by effectively and honestly reviewing what happened can you avoid making the same mistakes in the future Andrew Cowie (R7) is a management consultant working in the operations and infrastructure space. Andrew is a longtime UNIX and Linux user, and, somewhat unusually, was an infantry officer in the Canadian army, having graduated from Royal Military College with a degree in engineering physics. He saw service across North America and a peacekeeping tour in Bosnia. He later ran operations for an Internet startup in Manhattan building communities via SMS and was a part of recovering the company after the September 11 attacks. Andrew is now based in Sydney, Australia, and works with clients worldwide. R8 Understanding Configuration Management Mark Burgess, Oslo University College 1:30 p.m.–5:00 p.m. Who should attend: Anyone with a basic knowledge of computing, whether you are interested in understanding the different tools or perhaps looking to design your own tool. This tutorial explains the basic issues and approaches. This is a new kind of tutorial for LISA, aimed at those wanting an overview of the theory and concepts surrounding configuration management. This is not a tutorial about a software package or a network protocol; rather, it is a semi-popular review of ideas from computer science. You will learn how to evaluate the principles used in configuring hosts and devices, relate them to standards, and apply them to your own environments. Topics include: What is a configuration? States, sequences, metrics, databases Case study: network change management (avoiding outages, managing risk) The meaning of policy Data types Languages and the Chomsky hierarchy Regular expressions Syntax versus semantics Examples XML SNMP Netconf Cfengine Computation versus constraint Declarative and imperative languages Constraints and promise Scalability, workflow, and efficiency considerations Event-Condition-Action systems Scheduled maintenance Optimization and control theory CIM and DEN-ng information models Standards and de facto standards BS/ISO 17799 BS 15000 ITIL eTOM Mark Burgess (M12, T8, R8) is Professor of Network and System Administration at Oslo University College, Norway. He is the author of the configuration management system cfengine and of several books and many papers on the topic.

Friday, December 9, 2005

Full-Day Tutorials

F1 Managing Samba 3.0 Gerald Carter, Samba Team/Hewlett-Packard 9:00 a.m.–5:00 p.m. Who should attend: System administrators who are currently managing Samba servers or are planning to deploy new servers this year. This course will outline the new features of Samba 3.0, including working demonstrations throughout the course session. Topics include: Providing basic file and print services Centrally managing printer drivers for Windows clients Cofiguring Samba's support for Access Control Lists and the Microsoft Distributed File System Making use of Samba VFS modules for features such as virus scanning and a network recycle bin Integrating with Windows NT 4.0 and Active Directory authentication services Implementing a Samba primary domain controller along with Samba backup domain controllers Migrating from a Windows NT 4.0 domain to a Samba domain Utilizing account storage alternatives to smbpasswd such as LDAP Gerald Carter (T6, W3, W8, F1) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing. F2 Advanced Technology in Sendmail Eric Allman, Sendmail, Inc. 9:00 a.m.–5:00 p.m. Who should attend: System administrators who want to learn more about the Sendmail program, particularly details of configuration and operational issues. This tutorial assumes that you are already familiar with Sendmail, including installation, configuration, and operation. This will be an intense, fast-paced tutorial. It is strongly recommended that you have read or are familiar with the materials in the Sendmail book published by O'Reilly and Associates, preferably the 3rd edition (but at least the 2nd edition). In the past few years the face of email has changed dramatically. No longer is it sufficient to use the default configurations, even in single-user systems. Spam, regulation, high loads, and increased concerns about privacy and authentication have caused major changes in sendmail and in the options available to you. After a very brief review of Sendmail functionality and terminology, we will explore some of the newer important features. Topics include: SMTP authentication TLS encryption The Milter (mail filter interface) Many of the newer policy control interfaces Eric Allman (F2) is the original author of Sendmail, co-founder and CTO of Sendmail, Inc., and co-author of Sendmail, published by O'Reilly. At U.C. Berkeley, he was the chief programmer on the INGRES database management project, leader of the Mammoth project, and an early contributer to BSD, authoring syslog, tset, the -me troff macros, and trek. Eric designed database user and application interfaces at Britton Lee (later Sharebase) and contributed to the Ring Array Processor project for neural-network-based speech recognition at the International Computer Science Institute. Eric is on the Editorial Review Board of ACM Queue magazine and is a former member of the Board of Directors of the USENIX Association. F3 Seven Habits of the Highly Effective System Administrator Mike Ciavarella, University of Melbourne, and Lee Damon, University of Washington 9:00 a.m.–5:00 p.m. Who should attend: Junior system administrators with anywhere from little to 3+ years of experience in computer system administration. We will focus on enabling the junior system administrator to "do it right the first time." Some topics will use UNIX-specific tools as examples, but the class is applicable to any sysadmin and any OS. Most of the material covered is "the other 90%" of system administration—things every sysadmin needs to do and to know, but which aren't details of specific technical implementation. We aim to accelerate the experience curve for junior system administrators by teaching them the time honored tricks (and effective coping strategies) that experienced administrators take for granted and which are necessary for successful growth of both the administrator and the site. The class covers many of the best practices that senior administrators have long incorporated in their work. We will touch on tools you should use, as well as tools you should try to avoid. We will touch on things that come up frequently, as well as those which happen only once or twice a year. We will look at a basic security approach. We will talk about issues such as why your computers should all agree on what time it is, why root passwords should not be the same on every computer, why backing up every filesystem on every computer is not always a good idea, policies—where you want them and where you might want to avoid them—ethical issues, and growth and success as a solo-sysadmin as well as in small, medium, and large teams. We will discuss training, mentoring, and personal growth planning, as well as site planning, budgeting, and logistics. We will discuss books that can help you and your users. Mike Ciavarella (W6, R3, R6, F3) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer. Lee Damon (S8, F3) has a B.S. in Speech Communication from Oregon State University. He has been a UNIX system administrator since 1985 and has been active in SAGE since its inception. He assisted in developing a mixed AIX/SunOS environment at IBM Watson Research and has developed mixed environments for Gulfstream Aerospace and QUALCOMM. He is currently leading the development effort for the Nikola project at the University of Washington Electrical Engineering department. He is past chair of the SAGE Ethics and Policies working groups and he chaired LISA '04. F4 Production Change Management: To Each, His or Her Own Geoff Halprin, The Sysadmin Group 9:00 a.m.–5:00 p.m. Who should attend: System administrators who wish to learn how to manage change and risk better and to become more professional in their system management practices, and system administrators who are responsible for developing or managing their organization's change management process or who are hoping to influence and improve their organization's process. The only way to ensure the integrity of a production computing environment is through a formal change management process. But anyone who's worked at a large facility can tell you horror stories about having to wait 60 days to reboot a machine, and other complete failures of change management processes. Get it wrong and the results are, well, bad: Slow-moving systems that can't keep pace with the business Systems with uptime figures below 97% Unhappy Web customers moving to a competitor's site Unhappy internal customers looking to outsource IT Lost productivity costing tens of thousands of dollars an hour And the obvious cost: Lost sysadmin productivity while they cool their heels in meetings, writing up forms, and waiting just to do their jobs This tutorial looks at change management from principles to implementation. We look at what should be in a CM process, and how to tune the process to meet your business's requirements. Geoff Halprin (F4) has spent over 25 years as a software developer, system administrator, consultant, and troubleshooter. He has written software from system management tools to mission-critical billing systems, has built and run networks for enterprises of all sizes, and has been called upon to diagnose problems in every aspect of computing infrastructure and software. He has spent more years troubleshooting other people's systems and programs than he cares to remember. Geoff is a member of the USENIX board of directors.