Pp. 109–120 of the Proceedings

XML Support For Tcl

Steve Ball
Zveno Pty Ltd
https://www.zveno.com/
Steve.Ball@zveno.com

Abstract

The DOM is a language-independent specification of how an application accesses and manipulates the document structure. TclDOM is a Tcl language binding for the DOM. The TclDOM specification provides a standard API for Tcl applications to process a XML or HTML document.

TclXML is a Tcl package which provides a sample implementation of TclDOM. It provides XML parsers along with the tools needed to create a hierarchical representation of documents which can be conveniently processed by a Tcl script. There are also facilities to check the validity of a document, along with commands to produce document output. TclXML provides a framework for parser and validator modules which allows some or all of the various components to be implemented in an extension language.

Keywords:

A Brief Introduction To XML and the DOM

XML

XML has been designed with the goal of being easier for tool developers to write software for processing documents, so that there will be plentiful applications available for handling XML. The syntax is far more restricted than SGML, with most optional features removed. For example, tag minimisation and omission are not allowed so it is easy to detect the end of an element. Empty elements must be explicitly marked.

There are two levels of conformity for a XML document. At the very least, a XML document must be well-formed. A well-formed XML document adheres to all of the syntactic rules of the XML specification. All elements must have an end tag, unless they are empty elements in which case the tag must include a trailing slash, such as <BR/>. Element attributes must have a value and the value must be quoted. There may be no occurrances of illegal characters, and so on. XML has been designed so that a program can check that a document is well-formed without having to use the document's DTD.

In addition to being well-formed, a XML document may also be valid. A valid XML document conforms to the rules laid out in its associated DTD. This means that all elements have content which is allowed according to the element's definition of its content model in the document's DTD. All attributes used in elements must also be permitted for that element and their value must be of the correct type. All attribute values which are used for identification must be unique, and so on. Validity is a much stronger assertion for a document than well-formedness, but requires access to the document's DTD and more processing time to check the various rules.

XML is internationalised and support for Unicode is a requirement of XML, so Tcl version 8.1 is well positioned to be used for software applications which handle and process XML documents.

The following document is a small example of a XML document instance:

<?xml version="1.0">
<!DOCTYPE paper SYSTEM "https://www.zveno.com/dtds/paper.dtd">
<paper>
<title>TclXML Example</title>
<abstract>This is an example of a XML document instance.
</abstract>
<introduction>XML uses the 
<acronym><short>DTD</short><full>Document Type Definition</full></acronym>
to define classes of documents.  Such a document is known as a 
<definition>document instance</definition>.
</introduction>
<point>A DTD is used to define the elements and entities that are allowed to appear in a document.
Empty elements have a trailing slash: <figure image="figure-1.png"/>
</point>
<point>XML specifies only the structure of a document.
Authors will use XLL and XSL for linking and presentation.
</point>
<conclusion>XML is way better than HTML.</conclusion>
<references></references>
</paper>

DOM

DOM already has language bindings for Java and ECMAscript. Many tools are being written in Java for processing XML, and standard interfaces are starting to emerge for Java components, such as XAPI-J and SAXDOM. A language binding for Tcl is discussed in this paper.

Other Proposed Standards

In order to provide certain semantics for the elements used in a XML document other facilities are required. A number of languages have been proposed for these purposes. The XML Link Language, XLL, provides the hyperlinking model for XML. XLL has been derived from TEI and HyTime and as a result will be a much richer model than HTML's. For example, XLL will allow bidirectional and multi-destination links, as well as allowing link targets to specify and use the structure of the destination resource. In order to render XML documents for viewing or printing a stylesheet language will be used. Either Cascading Style Sheets (CSS) [CSS] will be used or the new proposed XML Stylesheet Language (XSL). XSL will also include scripting.

Tcl Support For XML

Since applications will have varying requirements, different supporting libraries may be used to process documents. Some of these may be implemented entirely in Tcl, for cross-platform portability, some may use C or C++ extensions to improve performance and some may interface with Java components. Ideally, there will be a standard programming interface for applications written in Tcl to access the document structure for processing. It is the goal of TclDOM to provide that standard interface, based upon the DOM specification.

XML support in Tcl has two distinct parts. Firstly, TclDOM is a language binding for the DOM. TclDOM provides an implementation-independent specification of a Tcl API of the DOM for Tcl scripts to access and manipulate XML and HTML documents. Secondly, TclXML is a sample implementation of TclDOM which provides XML parsers and generators.

Zveno is making the TclDOM specification and TclXML freely available. Both the TclDOM specification and the TclXML distribution may be found on the Zveno website [TCLXML]:
https://www.zveno.com/zm.cgi/in-tclxml/

TclDOM

Access to all of the features of the DOM has the advantage of familiarity to developers who have a previous knowledge of the DOM. The DOM is also a collaborative design effort which ensures that the interface will be comprehensive, allowing sufficient expressiveness of the interface for a general-purpose document scripting library such as TclXML.

Providing all of the DOM features goes a long way towards allowing access to DOM implementations in other languages, because there is then a straightforward mapping between the implementations. It is fair to say that the majority of work in writing XML processors is being done using Java, so an interface to Java is essential for TclXML to leverage this activity. Fortunately, TclBlend and Jacl give easy access to Java classes from Tcl scripts.

TclDOM Interface Design

Previous Work

Uhler's html_library

html_library translates a HTML or XML document into a series of calls to a Tcl procedure. Hence, it is an event-based parser where the application is notified of the start and end of elements. The parser does no checking for well-formedness or validity - all further processing is left to the application. This parser does not provide a good model for TclDOM, since the DOM uses a tree-based view of the document structure.

Plume

parse:pi xml {version 1.0} {}
parse:pi DOCTYPE paper {SYSTEM https://www.zveno.com/dtds/paper.dtd}
parse:element paper {} {
    parse:element title {} {
        parse:text {TclXML Example} {} {}
    }
    parse:element abstract {} {
        parse:text {This is an example of a XML document instance.}
    }
    parse:element introduction {} {
        parse:text {XML uses the }
        parse:element acronym {} {
            parse:element short {} {
                parse:text DTD {} {}
            }
            parse:element full {} {
                parse:text {Document Type Definition}
            }
        }
        parse:text {to define classes of documents.
Such a document is known as a } {} {}
        parse:element definition {} {
            parse:text {document instance} {} {}
        }
        parse:text . {} {}
    }
    parse:element point {} {
        parse:text {A DTD is used to define the elements
and entities that are allowed to appear in a document.
Empty elements have a trailing slash: }
        parse:element figure {image figure-1.png} {}
    }
    parse:element point {} {
        parse:text {XML specifies only the structure of a document.
Authors will use XLL and XSL for linking and presentation.}
    }
    parse:element conclusion {} {
        parse:text {XML is way better than HTML.}
    }
    parse:element references {} {}
}

To facilitate access to the data structure using Tcl list commands, in particular the foreach command, dummy arguments are appended to the entries used for text and processing instructions. This allows a construct to be used such as:

foreach {type arg1 arg2 arg3} [xml::parse $MyDocument] {
    switch $type {
        parse:element {
            #   Process element
            # arg1 is the tag name
            # arg2 is the attribute list
            # arg3 is the element content
        }
        parse:text {
            #   Process character data
            # arg1 is the data
            # arg2 and arg3 are unused
        }
        parse:pi {
            #   A processing instruction
            # arg1 is the PI name
            # arg2 is data for the PI
            # arg3 is unused
        }
    }
}

parse:element

To denote an element, option -elementcommand.

parse:text

To denote character data, option -textcommand.

parse:pi

To denote a processing instruction, option -picommand.

parse:comment

To denote a comment, option -commentcommand.

proc parse:element {name attributes content} {
    eval $content
}
proc parse:text {text unused unused} {
    puts $text
}

eval [xml::cvtscript [xml::parse $MyDocument]]

The disadvantage of using an explicit Tcl representation for the parsed data structure is that the opportunity is lost to implement the document tree using another language, such as C or Java. Accessing and manipulating the tree structure may be quite slow using Tcl. Also, it can be difficult to dynamically modify the document structure, for example for document editing purposes, and to navigate the tree from an arbitrary starting point, for example if a tree node is passed as an argument.

CoST

CoST's API is not directly suitable for use with TclDOM. CoST has the notion of a document node, but nodes are not exposed to the application and there is no equivalent to a DOM "Iterator" for traversing sequences or trees of nodes. Instead, CoST supplies explicit methods for performing iterative operations on document tree nodes, such as withNode or foreachNode. In addition, CoST does not provide support for creating or modifying documents, nor does it provide access to the document's DTD.

Another difference between DOM and CoST is the notion of a current node. In CoST there is always a node which is current, and operations may be performed upon that node by CoST methods. However, in the DOM no actual node is specified as being current, instead there is a pointer to the position between two nodes. The DOM then allows the node before or after the current pointer to be retrieved. The DOM specifies node positions in this way to allow traversal of a dynamically changing document tree. With the DOM there is no danger of the current node being deleted and so the current node reference becoming invalid.

The TclDOM API

DOM is defined using the OMG IDL interface specification, a language-neutral definition language. This presents some difficulties for defining the TclDOM specification because IDL is object-oriented, whereas Tcl is not. Common Tcl conventions are used to overcome this problem, by defining class creation commands and object instance commands. TclDOM defines a number of commands within the dom namespace which correspond to the IDL interfaces defined in the DOM specification. These commands produce and accept "tokens" as arguments for referring to nodes in the document tree. The DOM implementation may use these tokens to lookup the node in an internal data structure. This allows for efficient implementation in extension languages.

The DOM specification provides a class for accessing a list of nodes and a class to manipulate strings. These are unnecessary in Tcl, as Tcl already has a rich set of primitives for manipulating lists and strings, but these interfaces may be emulated for the sake of compatibility. TclDOM defines methods for retrieving a list of the children of a node, the parent of a node, the list of attributes for an element, and so on. An application may use these methods to traverse the document tree, much like a Tk script traverses the widget hierarchy. Tcl lists are ordered, which is an important property for representing the children of an element node. One potential problem with this approach is that in an application where the document is being dynamically updated the document tree may change after a list of nodes has been generated. However, this may also be seen as an advantage when compared to Plume's nested list approach where changing the document tree can be cumbersome.

Attribute lists are also defined in terms of a Tcl list, but are represented as name/value pairs. This is convenient for use in conjunction with the array set Tcl command for accessing attributes via a Tcl array. Attribute lists are unordered, so storing these in a Tcl array is satisfactory.

The command necessary to perform parsing and serialisation a XML document instance is not explicitly provided by the DOM specification. For TclDOM these functions may be made implicit by defining that XML documents are stored as an internal representation of a Tcl Object. In this way, a XML document will be stored initially as a string, but when accessed by a DOM function it will be parsed into the implementation's internal data structure. When the string representation is required the internal structure is serialised. The only problem with this approach is that an implementation would be difficult to write as a pure Tcl script, since the internal representation of a Tcl Object cannot be accessed from the Tcl script level.

Example

# Implicitly parse a document
set text [dom::text cget -nodeValue \
    [lindex [dom::node children {<Example>Sample Text</Example>}] 0] \
]

# Create a document in memory

set docRoot [dom::document createDocumentFragment]
set paper [dom::document createElement $docRoot paper]
set title [dom::document createElement $paper title]
dom::document createTextNode $title {TclXML Example}
dom::document createTextNode \
    [dom::document createElement $paper abstract] \
    {This is an example of a XML document instance}
set intro [dom::document createElement $paper introduction]
dom::document createTextNode $intro {XML uses the }
set acronym [dom::document createElement $intro acronym]
dom::document createTextNode \
    [dom::document createElement $acronym short] \
    DTD
dom::document createTextnode \
    [dom::document createElement $acronym full] \
    {Document Type Definition}
dom::document createTextNode $intro {to define classes of documents.  \
Such a document is known as a }
dom::document createTextNode \
    [dom::document createElement $intro definition] \
    {document instance}
dom::document createTextNode $intro .

# Now add an ID attribute to the <abstract> element

# Search for the element.  There is a method to do this.
foreach child [dom::node children $paper] {
    if {[dom::node cget -nodeType] == "element"} {
        if {[dom::node cget -nodeName] == "abstract"} {
            dom::element configure -attributes {ID abc123}
            break
        }
    }
}

# Write out the XML document

set ch [open example.xml w]

# NB. There is no need to explicitly serialise the document
puts $ch $docRoot

close $ch

TclXML

Applications wishing to use TclXML will have different requirements, both in terms of functionality and in terms of performance. Some applications may require an event-based parser, others may require a tree-based representation. An application may only need to check that a document is well-formed, whereas another may need to validate its documents. Figure 1 outlines the various requirements needed.

Figure 1: XML Processing Requirements of Applications

All of these facilities have been exposed to the application developer, since the different parts may be useful for different applications. Some application may process a XML document as a stream, in which an event-based parser is most useful. Other applications may need to traverse the document structure, in which case a tree-based parser is required. Also, the modular construction of the TclXML toolkit, makes it a simple matter to replace components of it with packages written in other languages. TclXML already includes expat, which is written in C, as an example.

Figure 2: TclXML Modular Structure

Compliant Parsers

-elementstartcommand name attributes

Invoked when an element starts. Arguments are the name of the element and its attribute list, given as a name-value paired list.

-elementendcommand name

Invoked when an element ends. The name of the element is appended.

-datacommand data

Invoked when character data is encountered. The data is appended.

-processinginstructioncommand name data

Invoked when a processing instruction is encountered. Arguments are the name of the processing instruction and the instruction's data.

-defaultcommand data

This callback is invoked when data is encountered for which there are no other handlers registered.

An example of using TclExpat is as follows:

package require expat

set parser [expat xmlParser]
$parser configure -elementstartcommand CountElements
$parser configure -processinginstructioncommand PI

proc CountElements args {
    incr ::count
}

proc PI {name args} {
    if {[regexp {break|continue|error} $name]} {
        return -code $name "$name due to processing instruction"
    }
}

set result {}

set count 0
$parser parse {<?xml version="1.0"?>
<!DOCTYPE Document SYSTEM "Document.dtd">
<Document>
<Visible>This element is processed</Visible>
<Interrupted>This element is skipped <?break?>
<NotCounted/>
</Interrupted>
<NotCounted/>
</Document>
}
lappend result $count

set count 0
$parser reset
$parser parser {<?xml version="1.0"?>
<!DOCTYPE Document SYSTEM "Document.dtd">
<Document>
<Visible>This element is processed</Visible>
<Interrupted>This element is skipped <?continue?>
<NotCounted/>
</Interrupted>
<Counted/>
</Document>
}
lappend result $count

puts $result

Document Generation

It is proposed that TclDOM-compliant implementations store their documents as the internal representation of a Tcl Object. When a Tcl script requires such an object in a string format, the object will serialise the document, thus generating a XML document. This process may apply to an entire document, a document fragment or an element hierarchy. After serialisation the document is just a plain Unicode string and the application may write it to a file, a network channel and so on.

Performance

Figure 3: Performance Comparison of C Program with TclExpat and TclXML.

The tests indicate that the Tcl script using the TclExpat extension is approximately 10 times slower than the equivalent C code, and the full Tcl version is 20 times slower. Given that the only difference between the two versions of the Tcl program are that the all-Tcl version must also parse the XML document in Tcl, it would appear that the performance loss in the TclExpat version is due to the task of writing the normalised XML document. A 10 to 20 times performance loss for Tcl scripts when compared to C code is acceptable. From this we can conclude that the TclExpat extension adds minimal overhead to expat for parsing XML documents. Further testing on more complex applications will be required to draw conclusions on the feasibility and/or efficiency of using Tcl to process XML documents, but this comparison test is encouraging.

An important point to note is that the C program included with the expat distribution is 653 lines of code. The Tcl script developed using the TclExpat extension or TclXML package is only 89 lines of code. This indicates that writing document processing applications is significantly more productive using Tcl.

Application Examples

To meet all of these requirements the book was written in XML, with markup used to identify code examples along with other structural features. Two simple scripts were written, one to translate the documents into HTML and another to output plain text while at the same time extracting the code examples. The tree-based document representation allowed these scripts to be written quickly and easily. These scripts used the Plume XML support library, which provided some valuable experience to guide the development of the TclDOM specification.

Another application is an Intranet development for the Department of Foreign Affairs and Trade of the Australian Federal Government. For this project it was decided to store corporate documents in XML format, to allow rich structuring of the information for improved searching and vendor- and platform-independence of the data. To support the current generation of Web browsers, the XML documents are translated into HTML on-the-fly at the time of document delivery. If the translation required is straightforward an event-based parser is used. An event-based parser allows large documents to be delivered incrementally, thus reducing the latency of document delivery and improving the perceived responsive of the system to the user. More complex translations use the tree-based representation of the document, which allows advanced formatting depending on the content of elements. These technologies are augmented with Tcl microscripting, using the Tcl Web Server [Tclhttpd], to provide a highly dynamic Web site.

Conclusion

A Tcl package has been developed, called TclXML, which provides a sample implementation of the TclDOM. TclXML includes a non-validating, event-based XML parser written in Tcl, as well as a Tcl interface to the expat parser. Modules are, or will be, provided to create the document tree, validate documents and produce XML or HTML output. TclXML is being designed to take advantage of the significant number of Java packages that are available for processing XML documents.

XML, DOM and other related standards are, or will be, Recommendations of the World Wide Web Consortium which will have a major impact on the World Wide Web, as well as making inroads into application areas which currently use SGML. Many network protocols and document formats are proposing to use XML as their data syntax, such as CDF, RDF and PGML to name but a few. This means that it is very important for the Tcl language to be able to efficiently process XML documents, in order to be able to support the development of applications which make use of XML. Tcl 8.1 has the advantage that it supports Unicode, though Tcl 8.0 may also be used for western character sets.

Given the increasing importance of XML for a wide range of application development it is the author's opinion that XML support of some form must be included in the core Tcl distribution. TclXML is being distributed under a license that permits it to be added to Tcl. It is interesting to note that Perl is also undergoing development to include Unicode and XML support [Perl]. Tcl has a significant advantage in that it is already Unicode-enabled, and so has a lead on Perl development.

References

[TCLXML]
Steve Ball. Tcl Support For XML.
Zveno Pty Ltd.
https://www.zveno.com/zm.cgi/in-tclxml/

[XML]
Tim Bray, et al. Extensible Markup Language Specification.
World Wide Web Consortium.
https://www.w3c.org/TR/

[W3C]
World Wide Web Consortium.
https://www.w3c.org/

[DOM]
Lauren Wood, et al. Document Object Model.
World Wide Web Consortium.
https://www.w3c.org/TR/

[CSS]
Hakon Lie, et al. Cascading Style Sheets.
World Wide Web Consortium.
https://www.w3c.org/Style/

[Ball98A]
Steve Ball. The Plume WWW Browser.
https://plume.browser.org/

[Ball98B]
Steve Ball. Web Tcl Complete
McGraw-Hill, New York. https://www.zveno.com/zm.cgi/in-wtc/

[Uhler95]
Uhler, S. html_library.
Sun Microsystems Laboratories.
ftp://ftp.smli.com/research/tcl/html_library.tar.gz

[Clark98]
James Clark. Expat XML Library.
https://www.jclark.com/xml/

[English96]
Joe English. CoST.
https://www.art.com/cost/

[PM-R96]
Peter Murray-Rust. CoSTWish.
https://www.venus.co.uk/omf/costwish/

[Nicol98]
Gavin Nicol, Inso Corporation.
mailto:gtn@inso.com

[Tclhttpd]
Brent Welch, Scriptics Corporation.
https://www.scriptics.com/

[Perl]
Larry Wall.
https://perl.oreilly.com/