LISA 2002 Paper
Stem: The System Administration Enabler
(Berkeley, CA: USENIX Association, 2002).
Stem is a system administration ``enabler.'' It is not an administrative tool, but rather a general-purpose development framework that allows an administrator to craft tools to perform a wide variety of tasks in a distributed environment easily and quickly. Many common tasks can be performed with Stem scripts involving a few lines of declarations. Current example applications include log file collating and service load balancing. Using Stem, a non-programmer can craft reliable network software in a few lines of declarations that would require hundreds or thousands of lines of code in a traditional programming language such as C.
It is a bit difficult to begin describing something that is rather unique. Stem is not an administrative tool, nor does it support a particular kind of administrative practice. It is instead a ``framework'' for declarative construction of networked software for a variety of purposes. Its intent is to make network software that is as easy to create as it is to describe. In many cases, Stem allows a system administrator to create such tools without learning to program by basing them upon a few templates that describe common network applications.
Stem's components include:
While it remains a general-purpose programming framework, Stem's primary goal is to help system and network administrators solve their problems with less work. The is the concept of ``enabling'' which will be a theme throughout this paper. Stem enables system administrators to easily glue existing systems together with network connections, create new networked applications with much less coding than before, and configure solutions to many common problems without any coding at all.
Stem is a general purpose system that can be used in many situations and problem spaces. By using the existing Stem modules and example configurations you can focus on your own problem space and issues and ignore many common network programming problems such as event handling and client-server and other interprocess communication. If your problem space is large, Stem enables you to cover that entire space under one architecture, thereby simplifying your design, coding and maintenance.
To Code or Not to Code
While most administrators write at least the occasional script, many (perhaps even most) do not have the time or skills to develop full-blown network applications from scratch. Stem enables both groups of administrators equally. Non-programmers can create Stem configurations or modify the supplied examples without any coding needed to solve their specific problems, simple or complex. Administrators who know Perl can create modules for functions not provided by existing Stem modules and still take advantage of the infrastructure and network services that Stem offers, allowing them to limit their programming to just their specific task. The advantages of this approach, with respect to simplifying and speeding up application creation, are obvious. A site can even split up the work, with a developer creating new Stem modules, and an administrator creating the configuration to drive and deploy them.
Gluing together disparate existing applications is a common and difficult problem when attempting to automate system administration tasks. Applications can have command line interfaces (CLI), be client/server based, or use common protocols (HTTP, SMTP, etc.). Stem enables an elegant solution to this task. Stem can be used to wrap each existing application in a module along with a new, message-based interface (essentially, an API). However, there is no need to predefine message queues or compile parameters as you do in many other message-passing or RPC type systems. Once this is done, Stem declarations invoke the module under conditions you specify.
This approach allows the task of gluing together applications to be divided into two phases: first, creating a wrapper with a message-passing interface and second, using the new module within Stem, allowing the cells to communicate automatically with one another. The first task is performed only once, and the resulting module is reusable, and generic, while the second task allows the module to be quickly put to work on a specific task.
Stem is a unique integration of several kinds of technology, but has its roots in several other tools that contain part, but not all, of its capabilities.
First, Stem is a ``message-passing system,'' but applying that name implicitly limits it more than is appropriate. For a programmer, the term ``message-passing'' generally refers to parallel programming libraries such as PVM and MPI [mpi]. Stem differs from such facilities in that it allows network applications to be created at a conceptually ``higher'' level and handles all of the lowest level message-passing functionality transparently to the application creator. Also Stem's messages can be sent to any cell in the current application, whether in the same hub (process), system or to a remote site. In fact, changing where a message is sent usually amounts to simply editing an address in a configuration file with no coding involved.
Another common use of the term ``message-passing'' is to refer to a class of commercial products commonly called Message Oriented Middleware (MOM). These products include guaranteed message delivery among their capabilities, which also typically include access to databases and transaction systems and other related services. Such products are usually targeted to the financial and business community and are rarely used by administrators and network developers. A few of the more well known MOM products include IBM's WebSphere MQ Family and Microsoft's MSMQ [mom].
Stem differs from MOM applications in several ways. First of all, MOM systems are designed for use by professional application programmers and usually require substantial programming expertise to use. In contrast, Stem is designed for use by administrators for administrative tasks while minimizing required programming. Secondly, traditional MOM systems are extremely large and entail considerable overhead, from both a computing and a staffing point of view. MOM systems typically need a database system to function, and some MOM vendors (e.g., IBM) recommend at least one full time staff person dedicated to running them. Stem is at the other extreme in that it is extremely lightweight. Finally, while Stem is Open Source, existing MOM applications are commercial products which are both expensive and proprietary.
Second, Stem is intended as an ``administrative tool'' but has almost nothing in common with existing administrative tools such as Cfengine [cfengine]. These tools are intended primarily to control the configuration of hosts within a network. Stem is instead intended to allow flexible interoperation between tools within a network. While Cfengine provides network communications layers so that hosts can exchange information, this information is limited to facts relative to host configuration. It cannot, for example, hand off a service request from one host to another, an operation that is trivial in Stem.
The Swatch package [swatch] overlaps to some extent with one of Stem's modules. This package monitors log file contents and searches for specified patterns set in its configuration file. The Stem::LogTail module performs a very similar function.
Stem is also not a monitoring tool. It is better to say that Stem is a framework that makes it easier than ever to create applications which collect any desired system and network data. Thus, it can subsume many of the data collection capabilities of these tools. Stem could also be used to feed data to existing monitoring tools like RRDTool [rrdtool] or Cricket [cricket], enabling the administrator to extend their capabilities while continuing to take advantages of these tools' mature visualization capabilities.
Stem has something in common with many other tools and approaches. Stem's ability to function as an application wrapper has its roots in many other message-passing and ``screen-scraping'' systems. Its basic philosophy of creating a distributed configuration engine that responds to flexible events was first documented in Distr [distr], though this mechanism was intended solely for file distribution.
To encompass so many facets of other work, Stem has evolved a unique architecture based upon a biological metaphor of ``Stem Cells.'' A Cell is the fundamental building block for a network application. In a running Stem application, one or more cells exist as objects in a Stem ``hub''; a hub corresponds to a single daemon process. Multiple hubs can run simultaneously, and they communicate with one another via constructs called ``portals'' that use TCP/IP sockets. Hubs can communicate with other hubs running on the same system or any network-accessible host, with Stem handling all of the interprocess communication.
Stem is a fully event driven system. Events can be any of the common network operations such as socket connections, I/O on character devices (terminals, sockets, pipes, etc.), and timers. Stem uses a consistent technique for the delivery of messages based on all kinds of events.
Stem cells are addressable objects which can send and receive messages. Cells are registered at creation time by name. There are three kinds of Stem Cells:
The heart of Stem is the messaging subsystem and the heart of that is the registry. This is where all knowledge of how to address cells is located. Each cell is registered by its name and if it is a cloned cell, also by its target name, and messages are directed to it via these names.
Stem Messages are how Cells communicate with each other. Messages are simple data structures with two major sections, the address and the content. The address contains the Cell name that the message is directed to and which Cell sent it. The content has the message type, command and data.
Message addresses are name triplets of Hub/ Cell/Target. The Cell name is required and the Hub and Target are optional. These triplets form globally unique addresses with the overall Stem system.
The Message address section has multiple address fields. The two primary fields correspond to the common email headers and are called `to' and `from'. The `to' address designates which Cell will get this message, and the `from' address says which Cell sent this message.
The Message content has information about this message and any data being sent to the destination Cell. The primary attribute is `type' which can be set to any string, but common types are data, cmd (command), response and status. Stem modules and Cells can create any Message types they want. The other major attribute of the content is the data, which holds a reference to the message data.
The various Cells classes are implemented as Perl modules, and many useful Cell types are included with the Stem package. These are among the most important:
In addition, Stem includes other utility modules which provide services to active Cells. These services include asynchronous I/O, cloning of cells, flow control or local and remote method calls and logical pipes.
Stem differs from all other networking toolkits by being architected around configuration rather than software. A configuration file instructs the Stem engine which Stem cells to construct and register. When you invoke Stem, it interprets this configuration and creates cells as needed without any need to compile networking code.
Configuration files are structured using Perl objects. Each desired cell is specified as a Perl object with a list of attributes. Several examples are discussed in the next section.
Example Application: An inetd-like Server
We will consider a few versions of a small Stem application, chosen for its use of typical Stem components as well as in response to the space limitations imposed on this paper. None of them require any programming on the part of the system administrator.
A Simple First Version
This version of the application serves as a good starting point for understanding Stem. It uses only existing Stem modules to create an application which can execute a process on a local or remote system. As such, the only task required to create the application is to set up a file containing the Stem configuration. The simplest version is given in Listing 1.
# uptime.stem [ class => 'Stem::TtyMsg', args => , ], [ class => 'Stem::Proc', name => 'mon', args => [ path => '/usr/bin/uptime', cell_attr => [ 'data_addr' => 'A', 'send_data_on_close' => 1, ], ], ], [ class => 'Stem::SockMsg', name => 'A', args => [ port => 6666, server => 1, cell_attr => [ 'data_addr' => 'A', ], ], ],
This configuration uses three cells:
In this example when a socket connection is made, the logical pipe to `mon' is created, which causes the `mon' cell to clone and fork the uptime program. Its output is collected and then sent back to the `A' send and then on to the socket. Stem will take care of creating and sending all of those messages automatically.
Running this configuration requires the following commands:
$ xterm -T Stem -n Stem \ -e run_stem uptime $ xterm -T Monitor -n Monitor \ -e telnet localhost 6666
We use two xterm commands to make Stem's operations visible. The first command starts the Stem hub daemon process and attaches the Stem::TtyMsg Cell to it so commands can be entered. The second command attaches a second window to port 6666, the Stem::SockMsg Cell, using a telnet command. Figure 1 illustrates the resulting windows.
In the ``Stem'' window, we send the cell_trigger command to the mon Cell. This causes the Stem::Proc Cell to execute its command. Note that the output appears in the ``Monitor'' window attached to port 6666 each time the cell is triggered.
A Piped Version
The problem with the example above is that you must enter a `cell_trigger' command to make the process execute and only one telnet session can be used. This new version (see Listing 2) will make the process execute when the telnet connects to the socket. Note it uses the `pipe_addr' attribute which will create a logical pipe between the cell `A' and the `mon' cell. Actually the `mon' cell will be cloned when a pipe to it is created and that cloned cell will run the process.
#uptime2.stem [ class => 'Stem::TtyMsg', args => , ], [ class => 'Stem::Proc', name => 'mon', args => [ path => '/usr/bin/uptime', cell_attr => [ 'cloneable' => 1, 'send_data_on_close' => 1, ], ], ], [ class => 'Stem::SockMsg', name => 'A', args => [ port => 6666, server => 1, cell_attr => [ pipe_addr => 'mon', ], ], ],
To run this example, name the configuration file up3.stem and run this command:
$ xterm -T Stem -n Stem \ -e run_stem uptime2Then, in another window run the telnet command:
$ telnet localhost 6666Each time you run telnet you will invoke the uptime command and see its output from telnet which will then exit.
A Multi-Hub Version
The previous two Stem applications were extremely simple, but their potential functionality is very powerful. They can be extended in several ways:
Listings 3 and 4 illustrate a multi-hub version of this application. Notice how easy it is to split the simple version into an implementation which can be run across the network.
#uptime_server.stem # Name the hub so the client can # refer to it. [ class => 'Stem::Hub', name => 'uptime_server', args => , ], [ class => 'Stem::TtyMsg', args => , ], # Set the portal to be a server: # listen for portal connections from # any host the port defaults to 10000 # but can be set here [ class => 'Stem::Portal', name => 'listener', args => [ 'server' => 1, 'host' => '', ], ], [ class => 'Stem::Proc', name => 'mon', args => [ path => '/usr/bin/uptime', cell_attr => [ 'cloneable' => 1, 'send_data_on_close' => 1, ], ], ],
#uptime_client.stem # Name the hub so server can refer to it. [ class => 'Stem::Hub', name => 'uptime_client', args => , ], [ class => 'Stem::TtyMsg', args => , ], # Create a client portal. # this will connect to the portal # in the 'monitoring' hub the default # host is 'localhost' but it can be # set here the port defaults to # 10000 but can be set here [ class => 'Stem::Portal', name => 'server', args => , ], [ class => 'Stem::SockMsg', name => 'A', args => [ port => 6666, server => 1, cell_attr => [ cloneable => 1, pipe_addr => 'uptime_server:mon', ], ], ],
To run this application, start processes like these (as before):
$ xterm -T Server -n Server \ -e run_stem uptime_server $ xterm -T Client -n Client \ -e run_stem uptime_clientThen, in another window run this command:
$ telnet localhost 6666This will behave the same as the uptime2 example but it is split over two hubs. Notice that other than adding the Hub and Portal cells, the only change to the configuration was adding a hub name to the `pipe_addr' attribute in the uptime_client configuration. This illustrates how easy it is to distribute applications written in Stem across a network. Sending messages to local or remote cells is done the same way - typically only the address will need to be changed.
This example application and its variations provide some insight into Stem's flexibility and capabilities. The Stem distribution comes with several other example applications and a cookbook that shows you how to create your own cells.
Critique and Analysis
The major infrastructure work of creating Stem has been completed, and the package is working well where it has been deployed. The design has proven to be as flexible as was intended, and Stem applications have been created by administrators unfamiliar with the package after just a few hours.
Stem's planned extensibility has also been verified in that administrators have successfully written additional Stem modules in Perl and integrated them with those that the package provides.
Stem's highly modular design has been proven to be instrumental in extending it. New modules can easily be created and integrated. Internal services have been developed and quickly used by other modules and cells. Its simple message-passing API allows almost any external application, service or protocol to interact with any other.
Stem's configuration file format currently takes the form of Perl data structures. This has performance implications and security issues as well as presenting a somewhat eccentric interface to non-Perl literate system administrators. In the future, we plan to support multiple configurations formats including XML.
Similarly the format of Stem's message (when serialized over a pipe) also needs to support other formats. But as with the configuration formats, it is just a matter of having modules that can convert the internal message structure to/from an external format. This will allow other systems to be more easily integrate as they can then send/receive Stem messages.
Stem's security support currently is weak. We have demonstrated to ourselves that we can use ssh for message-passing between Stem processes but the design was not good enough for production. We have plans to redesign it to be integrated with Stem's socket module so that any IPC (not just message-passing) can use it. Also the design would allow a choice of secure transport (ssh, SSL etc.) by using the same modular plug-in design as mentioned above.
Stem is extremely modular in design and can be extended easily in many directions. Here is a short list of some items that are in our development queue now:
Stem is Open Source software, it is licensed under the GPL and is available without charge from Stem Systems. Our website is https://www.stemsystems.com.
I wish to thank Aeleen Frisch, Alva Couch, and Will Partain for their help with this paper.
Uri Guttman graduated from MIT in 1983 with a B.S. CSE. He has been developing software for 25 years. Stem is the result of his extensive experience in systems architecture, networking, communications, API design and Perl. He is currently president of Stem Systems and can be reached at firstname.lastname@example.org.
References[cfengine] Burgess, Mark, ``Cfengine: A Site Configuration Engine,'' USENIX Computing Systems, USENIX, 1995.
[cricket] Allen, Jeff R., ``Driving by the Rear-View Mirror: Managing a Network with Cricket,'' Proceedings First Conference on Network Administration, USENIX 1999.
[distr] Couch, Alva L., ``Chaos Out of Order: A File Distribution Facility For `Intentionally Heterogeneous' Networks,'' Proceedings LISA 1997, USENIX, 1997.
[mom] https://www-3.ibm.com/software/ts/mqseries/ and https://www.microsoft.com/msmq/default.htm.
[mpi] The Message Passing Interface Forum, ``MPI: A Message-Passing Interface Standard'' and ``MPI-2: Extensions to the Message-Passing Interface,'' https://www.mpi-forum.org.
[rrdtool] Oetiker, Tobias, ``RRDTool,'' https://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html.
[swatch] Hansen, Stephen E., and E. Todd Atkins, ``Centralized System Monitoring With Swatch,'' Proceedings LISA 1993, USENIX, 1993.
This paper was originally published in the
Proceedings of the LISA 2002 16th System Administration Conference, November 3-8, 2002, Philadelphia, Pennsylvania, USA.
Last changed: 24 Oct. 2002 aw