The following paper was originally published in the
Proceedings of the
USENIX
Fourth Annual
Tcl/Tk Workshop
Monterey, California, July 1996.
For more information about
USENIX Association
contact:
| 1. Phone: | (510) 528-8649 |
| 2. FAX: | (510) 548-5738 |
| 3. Email: | office@usenix.org |
| 4. WWW URL: | https://www.usenix.org |
Visual Tcl
Mike Hopkirk
Santa Cruz Operation Inc
hops@sco.com
https://www.sco.com/Products/vtcl
This paper details some of the Requirements, Design, Architecture, History and Lessons learnt in implementing Visual Tcl (Vtcl) - A distributed multiple GUI toolkit extension to Tcl oriented initially and mainly at generating User Interfaces for System Administration use.
Possibly the first question asked is why would we would even consider another tcl GUI extension given that Tk exists. These were some of our requirements:
Vtcl exposes (in some form) probably about 70% of the Motif Widget set to the scripting interface (exceptions are the Drawing Area Widgets, MainWindow and some of the separate Managers e.g Paned Window). This exposure is generally at a level higher than that of the Motif API and we don't give direct access to any underlying Support APIs like X or Xt (as opposed to interfaces like tclMotif, Wafe [4] and DtKsh) favouring instead more a component only model. For example we have a formDialog that encapsulates the notions of a TopLevel Shell, a Form (with associated geometry on its children) and capability for providing a set of default buttons. These conform to the style and policies defined by their style Guides (Motif [2]) or where these didn't exist from Human Factors Feedback.
Geometry management is presented in a manner similar to X/Motif with parent widgets giving behavioral constraints on their children. We give a very much simplified interface to the geometry control of the Motif Manager widgets (currently only Form and RowColumn). Most of the apps use Form exclusively. The form layout simplification is setup such that with no explicit layout directions a 'simple and reasonable' layout is generated, explicit geometry options provide both generic and direct script control, overriding or augmenting the default layout handling.
In addition to the Motif Primitive and Manager Widgets we currently add three new ones, A ComboBox, A SpinButton (implemented in Vtcl) and a DrawnList. The DrawnList is heavily used, it presents a multicolumn list interface which can contain embedded pictures - this is convenient for displaying pictorial hierarchical trees and visually tagging related items.
We abstract the notion of colors and fonts by a single level of indirection. The script specifies a purpose oriented color or font (e.g. foregroundColor, urgentColor, titleFont) which the server is responsible for mapping onto something that makes sense in that GUI environment. We also extended the Motif notion of displaying bitmaps to support XPM format multicolor pixmaps for those widgets that support pixmaps. This obviously doesn't work in the character GUI where these are ignored though the script developer may provide an equivalent handling or mapping for that case.
Over and above the widgets we also provide a point help capability and support access to the platform Help system (optimised toward ScoHelp) both explicitly as a script command or implicitly from help Keys, a Help Menu or from Help buttons. This is decoupled from the scripting system.
On the interpreter side the extension supports an event driven state, callbacks firing as a result of a timeout or when nothing else is happening or when an I/O source or sink comes ready.
Vtcl is a client-server application. We have the notion of a GUI engine or Widget Server engine running autonomously serving out requested user interface applet interfaces as requested by a user. Multiple clients (usually Vtcl interpreters) can connect to and synchronously communicate with the server in order to get a GUI created and rendered.
User Interaction is fed back to the (interpreter) clients at 'interesting' points as messages that eventually fire script callbacks. Additional data is passed to these callbacks providing useful information associated with the callback. In this model variant GUI's are easily supported by changing the GUI engine that the clients connect to.
The Server keeps its own notion of the separate applet clients on a per connection basis. It does some initial applet hierarchy setup on connection, cleanup on client closedown or on losing the client connection and otherwise responds to client commands by creating widgets or modifying widget or application hierarchy state. The server can be thought of as just a rather dumb high level display device, It does not currently contain an embedded interpreter. This is largely a legacy decision and while we may possibly simplify the server implementation we believe we would lose (or face providing server interpreter hooks which would cause loss of) a lot of the current scripting simplicity by doing so.
Client server communication is done using a simple non textual protocol that is basically just a breakdown of the command and options (to the server) and a text string or array of text strings in reply (Callback, errors and command acknowledgement or result ). As an implementation detail this was originally over named pipes, We now have the capapibility of supporting both that and remote socket connections.
Vtcl is obviously based on a core tcl interpreter [1] - this has been extended with two extra packages:
On top of this we provide a package library of convenience routines that encapsulate some common functionality, provide a yet higher level interface to some of the widgets and do commonly required things. In many cases the applications using Vtcl also provide their own package libraries that encapsulate Vtcl components to give their own application specific functionality. For the System administration functions the interpreter itself is augmented with additional commands to provide access to our System administration Framework.
With regard to Vtcl alone, GUI manipulation is connection oriented: GUI creation, manipulation and interaction happens within the bounds of a VtOpen and VtClose command (or the exit of theinterpreter). There are no concurrent connections from a single interpreter though multiple sequential connections are possible There is a single command for creation of each type of Component and two additional commands for generic access and setting of component state. Some components have additional commands for accessing or changing that specific components state.
Heres a script example to give some idea of the 'flavor' .
#!/u1/hops/bin/vtcl
set app [ VtOpen Example ] ;# Open Server connection
# Make a dialog with single default controlling button
set f [ VtFormDialog $app.f -title "Tcl/Tk WkShop eg" \
-okCallback myQuitCB -okLabel Exit \ ]
# declare the callback proc for the 'interesting' action done on the combo box
proc myQuitCB cbs {
global combo
echo [VtGetValues $combo -value] ;# same info avail in cbs structure
VtClose ;# Close connection - shutdown application
exit ;# exit script
}
# Create a ComboBox in the dialog - list contains 3 items
set combo [ VtComboBox $f.combo -itemList { "banana" "plum" "apple" } ]
VtShow $f ;# Display the Dialog for the first time
VtMainLoop ;# Accept events
Looks like this when displayed using the X/Motif Display engine
Heres the same script displayed using the character engine
Interaction is 'event' driven from the point a script enters the Mainloop, prior to that the script usually describes the widget hierarchy and state, sets up callback actions and makes the hierarchy visible. Tcl Commands (usually procs) are registered on the widgets for feedback to the application of interesting actions - These are not presented directly as user events though in most cases thats where they come from. All widgets that allow callbacks support a "-callback" option which is the callback for the 'normal' or 'usual' interesting action of that widget. e.g button pressed, list element selected.
There are other callback options depending on the widgetClass - these either allow access to a variant action or to somewhat lower level of interaction. There are also a number of additional callbacks that are not directly widget or user interaction related which allow added control over the hooks into the help system and trapping of errors in callback commands.
All callbacks are passed a TclX keyed list containing information pertaining to the callback: Common items are usually
Additional or variant items may also exist depending on the callback
Widget naming is hierarchical with each level dot separated. A
widget name includes its entire parent hierarchy
e.g. .app.form.btn
and if its required to be referenced later should be unique. The
server enforces this though it can be told to allow a name as
a duplicate of an existing widget.
Each Widget Command returns the name of the widget created given the new widgets parent and desired name. The server may add its own (internal) widgets into the hierarchy as part of a widget command (this is an irritating Motif-ism we haven't attempted to hide and is most noticeable for Motif Dialogs) which means that the name you give a widget may not always be the name you end up with.
GUI Commands are action-object oriented and generally of the form :
Command objectName \ [-option [arg] ... ]
We've tried to provide a high level interface to components with a lot of functionality that the options trigger. There are no Key or event binding commands for example - we rely on the bindings of the native GUI component and try and present the widgets as a 'black box' with actions at a functional rather than user level.
The commands breakout into 20 Widget Commands (2 Managers), 21 Widget Manipulation commands, 3 Connection Oriented, 7 miscellaneous and 7 Interpreter only.
Widget options are used to trigger or describe widget actions or state (You could think of the Widget as a Class and the options as setting attributes and notification methods on that class though we make no effort to present it in those terms).The options are categorized into several groups depending on their argument types. Options are recognized and filtered, and option arguments are validated for syntax by the interpreter, Semantics are recognized (or ignored) by the server. This implies unrecognized options are discovered early without server interaction, misplaced options rely on the server to handle.
The options (and widget capability) are presented as a Class hierarchy from a base OBJECT class supporting common options with metaclasses for DIALOG, FORM, ROWCOLUMN, LIST, LABEL and BUTTON.
Form geometry management capabilities are presented as an inheritance of the GEOMETRY metaclass on the forms children. These specify where the children are positioned in the form relative to the form and their siblings.
These are tcl procs in a package library and provide the following:
As a result of an earlier project we had the beginnings of an X/Xt/Motif compatible library called CHARM (Character Motif) that mapped most of Motif and a subset of X/Xt onto a Curses interface. This library also mapped keyboard input and redraw processing into an Xlib like model. We managed to avoid a character specific version of the server code by using this library and writing the server to a single source code base as a normal X/Motif application then addressing any differences as they arose.
These mainly entailed augmenting the CHARM library and headers where necessary to provide a more compatible Motif/X API and special casing any places where we couldn't map or hide any differences in the two interfaces. There is a single code base for both Character and Motif Servers of which about 5% is special case code for either of the two environments. The majority of this is additional code for utilizing graphical capability not applicable to a character environment.
The graphical server invocation is unique per uid/LANG/DISPLAY triplet. This server persists until explicitly shutdown or the user logs out. The character server invocation is unique per uid/LANG/tty - its difficult to get coarser granularity than this without building some form of tty redirection capability onto curses. This gives Vtcl programs spawned from each other (on the same tty) use of the same server.
On VtOpen the interpreter attempts to detect its corresponding Server for the environment its running in and GUI desired (by testing a "well known point" - existence of a named pipe or success in binding to a socket). If a Server is detected the interpreter sends an Open message to the detection point. The server, once it is up and running, listens on a 'well known point' for connections. On a connection the server saves some internal state (communication file descriptors) and creates a toplevel shell for the application which is returned to the client (interpreter).
If the server is not detected the interpreter forks and execs it (for the local case). There is an extra level of handshaking here while the server opens a Display connection, After it has done so it sends an ack (or fail) message to the client and continues on with the normal applet open sequence.
So far we've punted on the remote execution server startup and assumed its already running. A better solution would be some sort of host local registry and execution server running from inetd or equivalent that records what servers it may have started on that host and passes back their connection information or starts ones that don't currently exist...
A client close message or losing the client connection (assumed due to the client exiting) causes destruction of the applet topLevel shell. All of the applet connection state is associated with this shell so cleanup is relatively simple and done with the Shell Destroy Callback handler.
There isn't any. Apparent multiple applet threading is done using event handlers on the file descriptor for the (server) input connection. The event handler parses and services the request and the result is sent back to the client interpreter. There is no applet state shared between applets beyond that embedded in the use of the Motif and X libraries.
Apart from seven interpreter local commands (which we can now lose since they're now available in tcl7.5) most of the Vtcl commands messages are implemented as a stub that does nothing more than send a message to the server, wait for a reply (status and optional string) and passes that back to the interpreter. The Close and DestroyWidget commands require some additional handling to clean up any pending callbacks for those widgets that may not have been serviced yet. FileSelectionBox also has additional handling to set the server side applets notion of the current working directory to the same as that of the interpreter. The major effort of the command stub is to parse the command (this is driven though a table of commands and a table of options) and marshall command and options data to (or from) a form conformant with the wire protocol.
Callbacks are implemented as an asynchronous message from the server containing a string (the argument to the widgets 'callback' option) and some data detailing the callback information. The callback data is assembled into a TclX keyed list and appended to the callback string. This string is then evaluated in the interpreter at global scope. Any errors in the evaluation result in a search for a registered error callback and its evaluation with some information about the error or, if none is found, return of an error status to the interpreter which causes an error return from the MainLoop command. Callbacks received while a command is waiting for a synchronous reply are queued for execution the next time the Mainloop code is executed.
All interpreter side user interaction is serviced from the VtMainLoop command which is essentially a select call inside a loop. It fires any pending queued callbacks, waits for incoming fd activity and invokes the callback, handles signals and connection errors or fires workproc callbacks.
This is (for historical reasons) a little gnarly - since its integral to the entire system it also has a lot of inertia which impedes changing it. We use a mixed binary and text protocol organized for the convenience of the Server in breaking out commands, options and arguments. Each protocol message is typed (COMMAND, COMMAND_RETURN, CALLBACK, ERROR) and contains a field count and number of fields count. Each field is also typed and possibly counted (arrays). Field types correspond almost directly to the allowed option types - INTEGER, STRING, STRING_ARRAY, STRING_2D_ARRAY, INTEGER_ARRAY, OPTION (Special sort of string).
This format doesn't give us much beyond some parsing convenience and entails worrying about byte ordering and word sizes. We're debating changing to a fully textual protocol (which would alleviate the above and give some debugging benefits at the possible cost of some decreased security).
The protocol stream is currently relatively easily snooped (for remote connections at least). We're examining the requirement to allow this to be encrypted in some form to protect information being passed across a network. One latter request has been for encrypting scripts in such a way that they are protected from casual browsing and modification. We have a modified interpreter (and encryption utility) that handles (and creates) such scripts and refuses to run unencrypted files. The encryption algorithm is fairly laughable in terms of uncrackability for anyone who has any idea of what they're doing but the level provided is acceptable for the requirements given.
Eventually I anticipate that the provision of a Tcl compiler will enable us to sidestep this to some extent.
There is still constant tension in this process - some users want more flexibility, more access to capabilities we have not exposed, None want to lose the current level of simplicity or have to involve themselves in more detail. We cheated on this and give a fallout into the underlying toolkit. We've been working on the premise that if a lot of users are using the fall through capability for a particular capability we should promote it into a core supported option - so far it seems to be working.
Separation of callback execution from User interaction opened up a race condition between User interaction and script modification (disabling) of UI components. This we kludged around with some callback and app locking commands/options. The solution isn't ideal from the script developers point of view since it exposes some recognition of the separation of interaction and execution but the users liked the problem even less and the solution is adequate if not elegant.
We initially treated the character engine as the limiting factor for features/functionality we had to relax this somewhat in order to be able to generate adequate Graphical interfaces (e.g. support of pixmaps). Its turned out that ignoring functionality due to graphical constraints (e.g. pictures in a character environment) and allowing script override works outwell enough. The supporting library we use instead of Xt and Motif in this environment was originally implemented as 'Motif on Character'. This model turns out to be inadequate without a mouse and being character the user expectations of capability (especially with respect to navigation) are different from Motif. The library is migrating more towards a character interface with a Motif API.
In hindsight we believe we made some mistakes in our presentation of the user model and functionality: The GUI command and option model is very X/Motif-centric in orientation (due mainly the use of interface libraries based around Motif).e.g. menu construction is an exact copy of Motifs. In many cases the naming of options is also similar to Motifs somewhat obscure choice of resource names.
We would have been better off providing specific widget commands for some of these concepts and being a bit more generic about some of our naming. We also ended heavily 'convenience' command oriented principally with List manipulation. A more consistent use of value setting options or better subcommand orientation would have reduced the initial number of commands to become familiar with. Due to some late decisions we have somewhat inconsistent levels of abstractions between font and color and pixmap handling, the latter is not really abstracted at all (being file oriented) yet.
There is a large area of totally missing functionality with Window Manager style manipulation and access to selections and the clipboard. Most users also have their opinion of additional widgets we should support (fortunately there are a few points of common agreement).
Finally one thing we totally missed was providing more compatibility in command model with Tk when we switched to the Tk-like naming scheme. We should have also switched (or at least supported) something more object+option oriented in order to make knowledge transfer between the tcl GUIs much easier.
As mentioned earlier Vtcl is currently deployed as part of the base technology in SCO Open Server 5. Ports (of the graphical system only unfortunately) are available for SunOS/Solaris, HP/UX, IBM AIX, SGI IRIX, and Digital UNIX. It will also continue to be base Technology in the Merged UnixWare/OpenServer OS.
Apart from addressing some of the deficiencies mentioned above we're also working on some of the following areas:
Vtcl is provided on SCO Open Server 5.0 The ports are available as TLS's from sosco.sco.com or as part of Premier Motif.
Theres a Vtcl Web Page at
https://www.sco.com/Products/vtcl/
giving somewhat more introductory material than I've covered above,
links to the TLS's and ports and links to other presentations
and tutorials and general information. It'll probably also have
a copy of this paper.
(The above web page is also referenced from the tcl Web page at
https://www.sco.com/Technology/tcl/Tcl.html
).
I can be contacted for comments, opinions or technical info as hops@sco.com.
Source code is freely available on signing a license form by contacting chrisr@sco.com.