USENIX '05 Training Track

TRAINING TRACK
Overview | By Day (Sunday, Monday, Tuesday, Wednesday, Thursday) |
All in One File | By Instructor

Sunday, April 10, 2005

S1 Hands-on Linux Security Class: Learn How to Defend Linux/UNIX Systems by Learning to Think Like a Hacker (Day 1 of 2)

Rik Farrow, Security Consultant
9:00 a.m.–5:00 p.m.

Who should attend: System administrators of Linux and other UNIX systems; anyone who runs a public UNIX server.

Few people enjoy learning how to swim by being tossed into the ocean, but that's what happens if a system you manage gets hacked. You often have little choice other than to reload that system, patch it, and get it running again. This two-day class gives you a chance to work with systems that have been "hacked," letting you search for hidden files or services or other evidence of the intrusion. Examples are taken from real, recent attacks on Linux systems. You will perform hands-on exercises with dual-use tools to replicate what intruders do as well as with tools dedicated to security. The tools vary from the ordinary, such as find and strings, to less familiar but very important ones, such as lsof, scanners, sniffers, and the Sleuth Kit.

The lecture portion of this class covers the background you need to understand UNIX security principles, TCP/IP, scanning, and popular attack strategies.

Day Two will explore the defenses for networks and individual systems. The class will end with a discussion of the use of patching tools for Linux, including cfengine.

Class exercises will require that you have an x86-based laptop computer that can be booted from a KNOPPIX CD. Students will receive a version of Linux on CD that includes the tools, files, and exercises used in the course. If you have a laptop but don't know whether it can run a bootable Linux CD (that will not have an impact on your installed hard drive or operating systems), please download a copy of KNOPPIX (https://www.knoppix.org), burn it, and try it out. KNOPPIX support for wireless is the same as common Linux kernels (not exciting), but KNOPPIX does a superb job of handling most other hardware found in laptops.

Exercises include:

DAY ONE:

Finding hidden files and evidence of intrusion
TCP/IP and its abuses
hping2 probes, or xprobe with ethereal again
nmap while watching with ethereal or tcpdump (connect and SYN scans)
Working with buffer-overflow exploit examples
Apache servers and finding bugs in scripts
John the Ripper, password cracking

DAY TWO:

Elevation of privilege and suid shells
Rootkits, and finding rootkits (chkrootkit)
Sleuth Kit (looking at intrusion timelines)
iptables and netfilter
Tracking down DoS floods
cfengine configuration
Vulnerability scanning with nessus

Rik Farrow (S1, M1) provides UNIX and Internet security consulting and training.

He has been working with UNIX system security since 1984 and with TCP/IP networks since 1988. He has taught at the IRS, Department of Justice, NSA, NASA, US West, Canadian RCMP, Swedish Navy, and for many US and European user groups. He is the author of UNIX System Security, published by Addison-Wesley in 1991, and System Administrator's Guide to System V (Prentice Hall, 1989). Farrow writes a column for ;login: and a network security column for Network magazine. Rik lives with his family in the high desert of northern Arizona and enjoys hiking and mountain biking when time permits.

S3 Hot Topics in System Administration

Ned McClain, Applied Trust Engineering
9:00 a.m.–5:00 p.m.

Who should attend: System and network administrators who are interested in picking up several new technologies quickly.

Topics include:

BIND9 Tips and Tricks: A Better DNS
Most sites have migrated to BIND9, but are you really getting the most out of this major rewrite of the Internet's most popular nameserver? Learn about powerful new functionality such as split views, remote management, and even DNSSEC. This topic is a must for every modern administrator.

Rapid Linux Disaster Recovery
Tape backups are essential, but they are not an efficient way to restore a server in an emergency. We evaluate the ins and outs of Mondo, an open source disaster recovery tool that can create bootable recovery CDs from any Linux server. When used in tandem with a solid tape backup system, Mondo recovery CDs can reduce "bare metal" recovery time from hours to minutes.

Linux Kernel Tuning
As Linux's popularity in production environments increases, the need for Linux kernel tuning knowledge is more important than ever! Whether it's performance, security, or functionality you're looking to cajole your system into, we'll give you the "what to's" and the "how to's," and even the "what you can'ts" of this rare art.

Practical Integration of UNIX and Active Directory
With Active Directory, Microsoft introduced an open LDAP directory that has become the de facto authentication store at many organizations. UNIX/Linux administrators are often tasked with the unthinkable: to integrate UNIX authentication with Active Directory. We'll not only explore the standard integration tools, such as OpenLDAP, PAM, and NSS, but will show you how to create custom scripts to manage Active Directory from UNIX.

Performance Crises Case Studies #4
Don't miss the latest episode of this incredibly popular segment! We've taken a new set of real-life system administration performance crises and dissected them, providing insight on how to diagnose and remedy situations that you may someday face. This is a great way to gain practical knowledge in the performance arena.

Custom Open Source Performance Monitoring
Most organizations have monitoring systems that provide real-time problem alerts, but few can produce graphs of resource utilization over time. We provide practical examples of extending a monitoring system to collect historical performance trends. We'll use examples specific to Nagios and RRDtool, but the lessons and gotchas discussed here will prove useful to anyone looking to implement any new monitoring system.

Ned McClain (S3), co-founder and CTO of Applied Trust Engineering, lectures around the globe

on applying cutting-edge technology in production computing environments. Ned holds a B.S. in Computer Science from Cornell University and is a contributing author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook.

S4 Regular Expression Mastery
Mark-Jason Dominus, Consultant and Author
9:00 a.m.–12:30 p.m.

Who should attend: System administrators and users who use Perl, grep, sed, awk, procmail, vi, or emacs. Attendees should have prior experience using regexes in UNIX utilities such as grep, sed, Perl, Python, vi, or emacs.

Almost everyone has written a regex that produced unexpected results. Sometimes regexes appear to hang forever, and it's not clear what has gone wrong. Sometimes they behave differently in different utilities, and you can't tell why. This class will fix all these problems.

The first section of the class will explore the matching algorithms used internally by common utilities such as grep and Perl. Understanding these algorithms will allow us to predict whether a regex will match, which of several matches will be found, and which regexes are likely to be faster than others, and to understand why all of these behaviors occur. We'll learn why commonly used regex symbols such as ".," "$." and "\1" may not mean what you thought they did.

In the second section, we'll look at common matching disasters, a few practical parsing applications, and some advanced Perl features. We'll finish with a discussion of optimizations that were added to Perl 5.6, and why you should avoid using "/i."

Topics include:

Inside the regex engine
- Regular expressions are programs
- Backtracking
- NFA vs. DFA
- POSIX and Perl
- Quantifiers
- Greed and anti-greed
- Anchors and assertions
- Backreferences
Disasters and optimizations
- Where machines come from
- Disaster examples
- Tokenizing
- New optimizations
- Matching strings with balanced parentheses

Mark-Jason Dominus (S4, S9) has been programming in Perl since 1992. He is a moderator of the comp.lang.perl.moderated newsgroup, the author of the Text::Template, Tie::File, and Memoize modules, a contributor to the Perl core, and author of the perlreftut man page. His work on the Rx regular expression debugger won the 2001 Larry Wall Award for Practical Utility. He lives in Philadelphia with his wife, daughter, and several plush octopuses.

S5 Eliminating Backup System Bottlenecks Using Disk-to-Disk and Other Methods
Jacob Farmer, Cambridge Computer Corp.
9:00 a.m.–12:30 p.m.

Who should attend: System administrators involved in the design and management of backup systems and policymakers responsible for protecting their organization's data. A general familiarity with server and storage hardware is assumed. The class focuses on architectures and core technologies and is relevant regardless of what backup hardware and software you currently use. Students will leave this lecture with immediate ideas for effective, inexpensive improvements to their backup systems.

The end may finally be in sight for the pains of backup and restore. The cost of disk storage has crossed the line: it has finally become practical to use disk to enhance or replace tape-based backup systems. In turn, software applications have come to market to facilitate the use of disk in backup systems. Now the problem is sorting out all of the options and fitting them into your existing infrastructure. This lecture identifies the major bottlenecks in conventional backup systems and explains how to address them. The emphasis is placed on the various roles inexpensive disk can play in your data protection strategy; however, attention is given to SAN-enabled backup, the current state and future of tape drives, iSCSI, and virtual tape.

Topics include:

Identifying and eliminating backup system bottlenecks

Conventional disk staging

Virtual tape libraries

Incremental forever and synthetic full backup strategies

Information life cycle management and nearline archiving

Data replication

Continuous backup

Snapshots

The current and future tape drives

Zero duplication file systems

iSCSI

Jacob Farmer (S5, S10) is the CTO of Cambridge Computer Services, a specialized integrator of backup systems and storage networks. He has over 15 years of experience with storage technologies and writes an expert advice column for InfoStor magazine. He is currently writing a book on storage networking.

S6 Kerberos 5—Revenge of the Three-Headed Dog

Gerald Carter, Samba Team/Hewlett-Packard
9:00 a.m.–12:30 p.m.

Who should attend: Administrators who want to understand Kerberos 5 implementations on both UNIX/Linux and Windows clients and servers.

For many organizations, Kerberos is an an old technology that has been driven to the forefront by deployments of Microsoft Active Directory domains. The introduction of a standard authentication protocol into Windows domains has caused many network administrators to reexamine ways to integrate UNIX/Linux and Windows clients in a single authentication model.

Topics include:

Key concepts of the Kerberos 5 protocol

Specific related authentication interfaces such as SASL and GSSAPI

The specifics of implementing of Krb5 realms

Implementations of Krb5 cross-realm trusts

Integration of Windows and UNIX/Linux clients into Krb5 realms

Possible pitfalls of using popular Krb5 implementations such as MIT, Heimdal, and Windows 200x

Gerald Carter (S6, T6, W3) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing.

S7 Advanced Shell Programming

Mike Ciavarella, University of Melbourne
9:00 a.m.–12:30 p.m.

Who should attend: Junior or intermediate system administrators or anyone with a basic knowledge of programming, preferably with some experience in Bourne/Korn shells (or their derivatives).

The humble shell script is still a mainstay of UNIX/Linux system administration, despite the wide availability of other scripting languages. This tutorial details techniques that move beyond the quick-and-dirty shell script.

Topics include:

Common mistakes and unsafe practices

Modular shell script programming

Building blocks: awk, sed, etc.

Writing secure shell scripts

Performance tuning

Choosing the right utilities for the job

Addressing portability at the design stage

When not to use shell scripts

Mike Ciavarella (S7, S12, M7) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching Software Engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer.

S8 Next-Generation Security Tools

Peter Baer Galvin, Corporate Technologies
9:00 a.m.–12:30 p.m.

Who should attend: Systems managers and security managers interested in current security problems and the new generation of tools designed to solve those problems.

This course covers a variety of topics of importance to those designing or implementing security solutions for their installations. It starts with the nasty world of current security threats and the problems sites have to solve. It then talks about what is solvable and what still has no solution. Finally, it covers each of the possible solutions in detail. (Note: Most of these solutions are commercial products.)

Topics include:

A security methodology

Determining the state of your world
Determining the problems to solve
Policy and procedure
Risk assessment, security audit, and penetration testing

Firewalls: Why don't they work?

Protecting Web servers

Reducing spam

Patch management and avoiding patching

Network snooping

Gaining status knowledge of your facility

Content filtering and antivirus software

Weak and strong authentication

Spyware and peer-to-peer networks

Peter Baer Galvin (S8, M3, T3) is the Chief Technologist for Corporate Technologies, Inc., a systems integrator and VAR,

and was the Systems Manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines. He wrote the "Pete's Wicked World" and "Pete's Super Systems" columns at SunWorld. He is currently contributing editor for Sys Admin, where he manages the Solaris Corner. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials on security and system administration and has given talks at many conferences and institutions on such topics as Web services, performance tuning, and high availability.

S9 Perl Program Repair Shop and Red Flags

Mark-Jason Dominus, Consultant and Author
1:30 p.m.–5:00 p.m.

Who should attend: Anyone who writes Perl programs regularly. Participants should have at least three months' experience programming in Perl.

You've probably been working too hard when you program, writing twenty lines of code when you only needed ten. But there is a better way, and I will show it to you. You'll learn how to improve your own code and the code of others, making it cleaner, more readable, more reusable, and more efficient, while at the same time making it 30-50% smaller. Smaller code contains fewer bugs and takes less time to maintain.

We will examine several real code examples in detail and see how to improve them. We'll focus on red flags—warning signs in your code that are plainly visible once you know what to look for—and on techniques that require little complex thought or ingenuity. All the bad code in this class is guaranteed 100% genuine and typical.

Participants are encouraged to submit their own code for anonymous review in the class. (Send it to mjd-usenix-2005@plover.com by March 1.) Class content varies depending on submissions, but is sure to include some of the topics listed below.

Topics include:

Families of variables
Making relationships explicit
Refactoring
Programming by convention
The Flesh Blanket
Conciseness
Why you should avoid the "." operator
Elimination of global variables
Superstition
The "use strict" zombies
Repressed subconscious urges
The cardinal rule of computer programming
The psychology of repeated code
Techniques for eliminating repeated code
What can go wrong with "if" and "else"
The Condition That Ate Michigan
Resisting "Holy Doctrine"
Trying it both ways
Structural vs. functional code
Elimination of structure
Boolean values
Programs that take two steps forward and one step back
Programs that are 10% backslashes
'print print print print print '
C-style "for" loops
Loop counter variables
Array length variables
Unnecessary shell calls
How (and why) to let "undef" be the special value
Confusion of internal and external representations of data
Tool use
Elimination of repeated code with higher-order functions
Learning to use a hammer
The "swswsw" problem
Avoiding special cases
Using uniform data representations

S10 Next-Generation Storage Networking and Data Protection

Jacob Farmer, Cambridge Computer Services
1:30 p.m.–5:00 p.m.

Who should attend: Sysadmins running day-to-day operations and those who set or enforce budgets. This lecture is technical in nature, but it does not address command-line syntax or the operation of specific products or technologies. Rather, the focus is on general architectures and various approaches to scaling in both performance and capacity. Since storage technologies tend to be expensive, there is some discussion of the relative cost of different technologies and of strategies for managing cost and achieving results on a limited budget.

There has been tremendous innovation in the data storage industry in the past few years, and this year the pace has quickened. Proprietary monolithic SAN and NAS subsystems are giving way to open-system and distributed architectures. Data-transfer protocols such as SCSI, NFS, and CIFS are facing competition from VI and DAFS. Fibre-channel and parallel SCSI interfaces are challenged by Gigabit Ethernet, iSCSI, and serial ATA. Bottlenecks imposed by I/O buses and stacks stand to be eliminated by Infiniband and RDMA.

This tutorial describes the latest technologies to hit the market for storage networking: SAN and NAS architectures, virtual storage, parallel file systems, storage interfaces, etc.

Topics include:

Fundamentals of storage networking

Shortcomings of conventional SAN and NAS architectures

Comparison of storage interfaces: fibre channel, SCSI, serial ATA, Infiniband, Ethernet

Comparison of storage protocols: CIFS, NFS, SCSI, VI, DAFS

Open systems storage virtualization

The convergence of SAN and NAS

High-performance file sharing (NAS on steroids)

SAN-enabled file systems

Wide-area file systems

Parallel file systems

Content-addressable storage

S11 Over the Edge System Administration, Volume 1

David N. Blank-Edelman, Northeastern University
1:30 p.m.–5:00 p.m.

Who should attend: Old-timers who think they've already seen it all, and those who want to develop inventive thinking early in their career. Join us and be prepared to be delighted, disgusted, and amazed. Most of all, be ready to enrich your network and system adminstration by learning to be different.

Can you think "out of the box" about system administration? One of the things that distinguishes the really great sysadmins from the good ones is their ability to be creative in their approach to problems and solutions. It's time to learn how to break the rules, abuse the tools, and generally turn your system administration knowledge inside out. This class is a cornucopia of ideas for creative ways to take the standard (and sometimes not-so-standard) system administration tools and techniques and use them in ways no one would expect. We'll also cover some tools you may have missed.

Topics include:

How to (ab)use perfectly good network transports by using them for purposes never dreamed of by their authors

How to increase user satisfaction during downtimes with 6 lines of Perl

How to improve your network services by intentionally throwing away data

How to drive annoying Web-only applications that don't have a command line interface—without lifting a finger

How to use ordinary objects you have lying around the house, such as Silly Putty, to make your life easier (seriously!)

David N. Blank-Edelman (S11, M4) is the Director of Technology at the Northeastern University College of Computer and Information Science and the author of the O'Reilly book Perl for System Administration. He has spent the last 19 years as a system/network administrator in large multi-platform environments, including Brandeis University, Cambridge Technology Group, and the MIT Media Laboratory. He has given several successful invited talks off the beaten path at LISA and is the LISA '05 Program Chair.

S12 Documentation Techniques for SysAdmins

Mike Ciavarella, University of Melbourne
1:30 p.m.–5:00 p.m.

Who should attend: System administrators who need to produce documention for the systems they manage or who want to improve their documentation skills.

Attendees should be able to make immediate, practical use of the techniques presented in this tutorial in their day-to-day tasks. Particular emphasis is placed on documentation as a time-saving tool rather than a workload imposition.

Topics include:

Why system administrators need to document

The document life cycle

Targeting your audience

An adaptable document framework

Common mistakes

Tools to assist the documentation process

Mike Ciavarella (S7, S12, M7) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer.

S13 Troubleshooting: A Basic Skill

Geoff Halprin, The SysAdmin Group
1:30 p.m.–5:00 p.m.

Who should attend: System administrators wishing to hone their ability to troubleshoot a problem under pressure, on a system of which their knowledge may be limited.

One of the most basic skills a system administrator must be able to call upon is that of problem diagnosis and resolution, that is, troubleshooting. It doesn't matter what else you do; if the system is broken, your priority is to fix it.

Topics include:

A general process for troubleshooting

Specific techniques that will help you get to the root of the problem

Ways to identify candidate solutions with confidence

Geoff Halprin (S13, T8) has spent over 25 years as a software developer, system administrator, consultant, and troubleshooter. He has written software from system management tools to mission-critical billing systems, has built and run networks for enterprises of all sizes, and has been called upon to diagnose problems in every aspect of computing infrastructure and software. He has spent more years troubleshooting other people's systems and programs than he cares to remember. Geoff was on the board of the System Administrators Guild (SAGE) and is now a member of the USENIX board of directors.

Need help? Use our Contacts page.

Last changed: 31 March 2005 ch