Pp. 97–108 of the Proceedings

Abstract

The ability to embed NeoWebScript code into existing webpages, without requiring a URL name change, leverages work done with webpage creation tools such as Netscape Communicator and Net Objects Fusion, while not disturbing links from remote sites and search engines.

A mature application, NeoWebScript-equipped webservers are currently in production on the Internet, serving real-world loads of millions of webpage "hits" per day.

This paper describes the driving forces behind the creation of NeoWebScript, and how those forces shaped its design and evolution into its current-day form. NeoWebScript's software architecture is described, and its capabilities are demonstrated using numerous examples, including webpage "visitor counters", rotating banner ads, queuing email, posting news, storing form submissions into Berkeley-style dbopen databases, and creating graphical images on the fly.

Finally, NeoWebScript's current status is summarized, our near-term plans are detailed, and conclusions are drawn.

Why Create NeoWebScript?

As our customers became more sophisticated and creative, they began asking for active content features such as access counters and rotating banner ads. Many had CGI scripts that they had written, or obtained, and wanted to run them on our webserver. Others wanted us to write them. We found that CGI programs had a number of problems:

• The overhead of the CGI approach is fairly high. For each CGI executed, the webserver must set up, start, manage, and communicate with a child process that executes the CGI.
• Redirecting existing URLs to scripts required reconfiguring the webserver. Redirected URL names tended to be unwieldy, and the rewritten web page addresses were confusing to users.
• Letting untrusted users run CGIs with the same user ID as the webserver creates a security problem; likewise there are security problems with running the webserver as superuser so that it can obtain permissions of the user when running a script as the user. A user could, in either case, accidentally or intentionally compromise their own files; in the latter case, an intruder may be able to directly gain superuser privileges.
• CGI programs typically emit the entire webpage programmatically, either rendering HTML authoring tools unusable or requiring "hand-stitching" to weave the tool-created HTML into the program, a fairly expert task that must be performed every time the tool-authored HTML is altered.

To address these needs, we decided to integrate our own solution by mating an existing webserver with an embeddable scripting language.

We chose the Apache webserver because we were already using it successfully in production to provide virtual web services for hundreds of domains. We had some skill with it, and we knew it to be capable, under active development, and in widespread use -- indeed, Apache is the world's most popular webserver and comes with full source code. Since Apache is freely redistributable, including for commercial resale, it kept our options open with regard to creating a commercial version if we chose to. Derived from the widely used NCSA http server, among Apache's enhancements were a modular architecture, designed and documented for the purpose of accepting third-party "plug ins" such as the one we hoped to create.

For the scripting language we chose the Tool command language (Tcl). An explicit design goal of Tcl was that it be easy to integrate into other applications. Tcl is known for being easy to learn, and has strong text processing capabilities. We already had substantial experience with Tcl, which lowered the technical risk. Tcl's developer community had produced many excellent tools and packages, usually distributing them under the permissive Berkeley copyright, which we could utilize to add capabilities to the server that were far beyond our own resources to create and maintain. Finally, Safe Tcl, by then supported as part of the Tcl core, looked very promising for providing users with a way to program the webserver with far less risk than the traditional server-side programming technologies.

Requirements

Another requirement was that, as our code progressed from an experiment to a production web scripting system, we embrace Apache's configuration files and configuration technology, which we did. All NeoWebScript-specific capabilities are configured and controlled through compliant configuration file extensions, using Apache's configurable module architecture.

One requirement greatly influenced the evolution of NeoWebScript. As an ISP, we have thousands of users who are not employees, all of whom are potential NeoWebScript developers, where their scripts are running on our servers. Compare this to a more traditional organization where, for example, a handful of employee-developers produce the organization's Internet and/or Intranet content. This is a critical distinction. As a result, NeoWebScript was designed from its inception to maintain the webserver's security while operating with an untrusted user base. This led to a design choice between providing webpage developers with the full power of a normal Tcl interpreter and our need to protect the server from those same developers, most of whom we have never met face-to-face.

Therefore, at every decision point, we opted for security over unencumbered power. By default, NeoWebScript's user files are kept out-of-band from the users' webpage files, protecting those files from being overwritten by a script, at the cost of not being able to script to create or manage files in the user's home directory. External programs cannot be executed except for certain specific applications (posting news and sending electronic mail, for instance) which are handled through tightly controlled interfaces that must be custom-developed for each supported program.

Design Philosophy

While our plan was to build in lots of capability, we wanted something that a fledgling webpage developer, with decent HTML skills but little or no prior programming experience, would be able to use to create simple active content features. A number of demos would be included, enabling web developers to cut and paste a number of interesting active-content elements into their webpages, all the while leaving their choice of HTML development tools completely open. An example of NeoWebScript usage appears in Figure 1.
 

<title>Simple Form Result</title>
<h1>Simple Form Result</h1>
<nws>
    load_response response
    dbstore simple $response(name) response
</nws> 

Response stored. Thanks!

Finally, for experienced developers, we planned to provide interfaces to database back-ends, a way to make and/or modify graphical images from data, provide access to the environment variables normally available to CGIs, and do whatever else users could do with the general-purpose programmability of a Safe Tcl interpreter. This would allow developers to customize content based on browser type and version, date, address, host name of the client, etc.

Commerce Servers

Another commerce server option is the freely redistributable Apache-SSL. Although at one point no Netscape or Explorer-recognized certificating authorities would sign digital certificates for Apache-SSL, Thawte (https://www.thawte.com/) has been doing so for some time. With Verisign's (https://www.verisign.com/) recent decision to sign certificates for Apache-SSL, Apache-SSL represents another viable commerce server platform that is NeoWebScript-compatible.
 

How It Works

The embedded code is evaluated within the interpreter. A number of services are provided:

• A Safe Tcl interpreter
• Form elements included with pages sent as GET and POST requests can be imported into an array of key-value pairs. (An example is shown in Figure 2.)
• Arrays of key-value pairs can be written to and read from btree-indexed disk files with a single statement.
• Arrays of key-value pairs can be translated to SQL statements and written to Oracle (using Tom Poindexter's Oratcl package) and Postgres databases, on the local machine or across the network. SQL queries and cursor walks produce data as arrays of key-value pairs.
• Creation and/or modification of GIF-format graphic images from a Tcl script, using Thomas Boutelle's graphics draw (gd) package. Examples of on-the-fly graphic image generation can be seen in Figure 9 and Figure 11.
• Access to the environment variables normally available to CGI programs through the webenv array.

Apache Module Architecture

Apache provides a number of services to modules, controlled by a "switch" structure. Modules have the option of having initialization code executed at webserver startup time, of receiving configuration information when retrieving a page from config files in that page's directory, or merged among zero or more higher-level directories. Modules can opt to receive configuration information from the server config files, and can define handlers for one or more MIME data types.

Apache modules can install their code into a "chain" of handlers that can accept, pass on, or reject the aforementioned filename translations, user authentication and authorization checks, etc.

Processing Webpages

Upon the first occurrence of a NeoWebScript tag, a safe interpreter is created and populated with variables, procedures and exported commands for use by the Tcl code contained in the webpage.

The embedded Tcl code is then evaluated, and its output is merged with any static content (standard HTML) that may be present in the page.

The same interpreter that executes the first chunk of NeoWebScript code executes any subsequent NeoWebScript code contained within that page.

After the page has been completely emitted, the interpreter is destroyed, as it cannot safely be reused. Note that the cost of creating, configuring and destroying a Tcl interpreter is far less than starting up and running a separate CGI program.

If no NeoWebScript tag is found in a page, the page is emitted without creating an interpreter or executing any Tcl code. This makes a NeoWebScript-enhanced Apache server perform at just a tick under the performance of an unenhanced server. When the NeoWebScript module is added to Apache, the server's memory footprint grows by the size of the NeoWebScript module, plus a Tcl interpreter and associated code. The processing overhead increases slightly due to the small overhead of looking for NeoWebScript tags.

As NeoWebScript's capabilities have evolved, our use of Apache's services has increased. Currently we support per-directory and merged-directory configuration, general configuration through the server config files, and, of course, handlers to parse and send pages by executing the NeoWebScript code embedded in the page and merging it with any static text present in the HTML file.

Supervisor Mode Pages

We wanted to provide a way to allow trusted developers to have the full capability of the Tcl shell, even on a server where most developers were untrusted and hence would only have the safe functionality. Also, we wanted to be able to create special pages that could be accessed by many different users, where each user would use the same page to look at their files, but not at one another's.

Supervisor Mode gives pages in specified directories full Tcl capabilities (not just safe ones) and allows those pages to assume the identity of other users (with respect to the server-maintained user identities - to the operating system, all the NeoWebScript files are owned by the webserver). This allows construction of site-wide control pages, multi-user data upload and download pages, etc.

We added the ability for supervisor mode pages to do password authentications against the UNIX password file, allowing users to login to services provided by supervisor mode pages and using those pages to access and manage their NeoWebScript files.

Tcl-oriented logging module

901018069 203.162.3.234 {} {} 304 0 
vnmusicawards.com {GET /summit2.jpg HTTP/1.0} 
{Mozilla/4.05 [en] (Win95; I)}

The NeoWebStats application creates summary data from the logfiles, which average more than 100 megabytes per day for a site serving a million hits per day. Summaries can be combined to produce summaries spanning multiple days. NeoWebScript's on-the-fly graphics generation capability is used to produce pie charts showing the proportion of hits at various "depths" within the webserver. An example of NeoWebStats output is shown in Figure 11.

Examples

Retrieving Data from a Client Request

In either case, NeoWebScript parses the data in the request and stores it into a global Tcl array using the NeoWebScript command load_response. An example using load_response is shown in Figure 2.

Sending Email from a Webpage

set fp [open_outbound_mail \
        "Moving to Montana?" $toWhom]
puts $fp "Do you think I could interest you in"
puts $fp "a pair of zircon-encrusted tweezers?"
close $fp

Open_outbound_mail returns a Tcl filehandle. The embedded code uses puts and any other relevant file-oriented Tcl commands to create the message body. The email sender's address is automatically constructed from the username of the owner of the webpage that's being interpreted and the name of the server that did the serving. If to is not specified, the recipient is also set to be the user name of the owner of the webpage.

Posting News

After writing out the body of the news article, a line is written to the filehandle consisting of a single period, then the file is closed. (This is a requirement of NNTP. It could, of course, be hidden by a proc.)
 

set fp [open_post_news -subject $subject \
                       -newsgroups neosoft.announce \
                       -distribution neosoft]
puts $fp "This is the body of the message."
puts $fp "."
close $fp

Note that for this to work you must have a news server supporting Network News Transfer Protocol (NNTP) in the same domain as your webserver. For example, within neosoft.com, open_post_news will contact the news server news.neosoft.com.

Graphical Access Counters

The code that created this display is shown in Figure 7. Split is used with an empty string for the characters to split on, causing the number of visits to be exploded into a list, where each digit in that number is one element of the list. We walk the list using foreach, emitting image references to a GIF image file for each digit to be displayed.
 

<center>
<h1>Welcome!</h1>
This page bas been visited
<nws>
foreach num [split [incr_page_counter] {}] {
html "<img src=/images/counters/set22/$num.gif>"
}
</nws>
 times since July 1, 1998
</center>

Rotating banner ads

random_pick_html {
  {<a href=/neosoft/><img src=images/sroom2.gif></a>}
  {<a href=/neosoft/neorules.html><img src=images/resources.gif></a>}
  {<a href=/neopolis/><img src=images/neopolis.gif></a>}


  {<a href=/neowebscript/><img src=images/logos/nws7.gif></a>}
}

Issuing and retrieving cookies

    neo_make_cookie -user ellyn -days 30 \
        -path /myApp

    load_cookies cookies
    html $cookies(email)

Creating Graphic Images from NeoWebScript

We create live GIF files by creating Tcl scripts that are end in a .gd extension. When the file is referenced from an <img src=...> tag in a webpage, our code is executed and is expected to produce a GIF file when complete by executing gd writeGIF. (Note that the image filehandle is passed in as the hard-coded global variable imageFile.)
 

proc draw_analog_time {im color} {
    scan [clock format [clock seconds] -format "%I %M"] "%d %d" hour minute
    set hourStart [expr 90 - ($hour * 30 + $minute / 2)]
    set minuteStart [expr 90 - ($minute * 6)]

    set minuteX [expr int(40 + cos($minuteStart * 3.14159 / 180) * 30)] 
    set minuteY [expr int(40 - sin($minuteStart * 3.14159 / 180) * 30)]

    gd line $im $color 40 40 $minuteX $minuteY; # draw three times
    gd line $im $color 40 43 $minuteX $minuteY; # to make it pointy
    gd line $im $color 43 40 $minuteX $minuteY

    set hourX [expr int(40 + cos($hourStart * 3.14159 / 180) * 20)] 
    set hourY [expr int(40 - sin($hourStart * 3.14159 / 180) * 20)]

    gd line $im $color 40 40 $hourX $hourY
    gd line $im $color 40 43 $hourX $hourY
    gd line $im $color 43 40 $hourX $hourY

     gd text $im $color large 25 80 "[format %d:%02d $hour $minute]"
}

set im_out [gd create 82 110]; # create the image
set white [gd color new $im_out 255 255 255]; # background
set black [gd color new $im_out 0 0 0]

gd arc $im_out $black 40 40 70 70 0 360; # draw clock face
gd text $im_out $black small 8 95 "Server Time"
draw_analog_time $im_out $black
gd writeGIF $im_out $imageFile 

The code that produced the analog clock is shown in Figure 10. In this code, the dimensions of the GIF file are specified using gd create. Colors are allocated, then we draw a circle to represent the clock face. We use gd text to write the text "Server Time" to the image, then call the proc draw_analog_time to create the clock hands and write the numeric time into the image.

Draw_analog_time fetches the current time in hours and minutes into variables called hour and minute. A bit of trigonometry serves to calculate the endpoints of the clock hands. By drawing the hands three times, each with slightly different locations for the "center" of the clock face, the clock come out "thicker" in the center of the clock face, drawing to a point at the ends.

A more sophisticated instance of graphics generation from NeoWebScript can be found in NeoWebStats (see Figure 11), an on-demand graphic image generator that creates pie charts to show what areas of a website are getting what proportion of hits.

Accessing Web Environment Variables

The HTML code that created the display is shown in Figure 13.  Remote_hostname returns the name of the host requesting the page, if it can be determined. (Otherwise the IP address is returned.)  Estimate_hits_per_hour returns an estimate of the number of hits being served per hour.  It does this by examining the server's access_log file.  It works by seeking approximately 1,000 hits backwards into the access log, comparing the time that entry was made to the current time, and extrapolating the estimated number of hits per hour that the webserver is serving.  We then format the current time and the date when the webpage was last modified using the standard Tcl clock command.  Finally we emit an HTML mailto: link to the web administrator, as read from the SERVER_ADMIN variable.

<img src=gdclock.gd align=left>
Hello, visitor from <b>
<nws>html [remote_hostname]</nws>
</b> and welcome!

<p>This server is currently serving approximately <b>
<nws>html [estimate_hits_per_hour]</nws>
</b>hits per hour as of <b>
<nws>
    html [clock format [clock seconds] \
        -format "%d-%b-%y %I:%M %p %Z"]
</nws>
</b><br>This page was last modified <b>
<nws>html "[clock format $webenv(NEO_LAST_MODIFIED)]"
</nws></b>, <b>
<nws>html "[expr ([clock seconds] - \ 
  $webenv(NEO_LAST_MODIFIED)) / 86400]"</nws>
</b> days ago.<br>Your server administrator is 
<nws>
html "<a href=mailto:$webenv(SERVER_ADMIN)> \
    $webenv(SERVER_ADMIN)</a>"
</nws>

Subst-style NeoWebScript Pages

The webserver's ability to evaluate subst-style webpages is configured as a handler via Apache's srm.conf file. The AddHandler directive allows you to map certain file extensions to "handlers", which causes Apache to take actions based on file extension. For example, to cause files ending in .shtml to be Tcl-substituted and emitted as a text/html MIME type, the following lines must be enabled in the srm.conf file:

AddType text/html .shtml
AddHandler server-subst .shtml

<img src=gdclock.gd align=left>
Hello, visitor from <b>[remote_hostname]</b> and welcome!

<p>This server is currently serving approximately <b>[estimate_hits_per_hour]</b> hits per hour as of <b>
[clock format [clock seconds] -format "%d-%b-%y %I:%M %p %Z"]

</b><br>This page was last modified <b>[clock format $webenv(NEO_LAST_MODIFIED)]</b>, <b>
[expr ([clock seconds] - $webenv(NEO_LAST_MODIFIED)) / 86400]</b> days ago.
<br>
Your server administrator is  <a href=mailto:$webenv(SERVER_ADMIN)>$webenv(SERVER_ADMIN)</a>

Current Status

Availability

Version 3.0

The 1.3.1 version of Apache is much easier to build, using a standard GNU (ftp://prep.ai.mit.edu/pub/gnu) configure script to automatically configure the package for the capabilities of the particular UNIX system for which it's being compiled. Apache 1.3.1 also includes a compiler "driver" that greatly simplifies building third-party modules. Also, it allows those modules to be kept separately from the Apache core, and NeoWebScript is currently being built and tested using this technology.

Apache 1.3 includes support for Windows 95 and NT for the first time, and work is underway to make NeoWebScript run in those environments as well.

Making It Easier to Build

Cgi.tcl

Year 2000 Issues

Future Development

Conclusions

Web content developers, including those without significant prior programming experience, have enthusiastically received NeoWebScript.

Sizable applications have been written in NeoWebScript, including two shopping carts, an editable event calendar and to-do list, in-out board, a web-based chat system, and a Yahoo-like hierarchical organizer (https://www.ghofn.org), among many others, all using our standard safe-interpreter interfaces. In fact, NeoWebScript has all but eliminated the need for other server-side tools, by allowing us to quickly and easily develop NeoWebScript equivalents for virtually every other commercial web-related application we have seen.

Supervisor mode provides a way to bypass the limits enforced by the architecture of the safe-interpreter/Apache interface in environments populated with a trusted user base, providing trusted developers with significantly more power than they would otherwise have had. Supervisor mode offers many as-yet-unexplored possibilities, and does not yet provide as tight of an integration of capabilities as is seen on the safe side.

A sizable NeoWebScript code base now exists, and we must balance our desire to innovate with the need to maintain compatibility with the work that has already been done.

NeoSoft is committed to continuing to develop and maintain NeoWebScript. We have been able to leverage this work in our own internal intranet applications, use it to lure, win, and keep web developers and the customers they bring with them, and to successfully win and deliver on a high-profile extranet site for a Fortune 100 company.

Hooking Tcl up with Apache is a simple idea that was straightforward to implement. So far, it has been one of the differentiators that have given us an edge in a highly competitive business.