Experiences with modularizing the Tcl core:
The Workbench

This workbench provides anything which is required to perform your own experiments on amodular core. It is organized as described below.

• Work Area
• Supporting scripts
• Stack Recorder

work/src

This directory contains the sources of three different variants of the Tcl core, all derived from version 8.3.4. To use them unpack the provided archives in the directory.

tcl834_baseline.tar.gz

is the unmodified Tcl 8.3.4 core.

tcl834_cut.tar.gz

is the modular Tcl 8.3.4 core. Slice and dice as seen fit.

tcl834_stkr.tar.gz

is a copy of the modular Tcl 8.3.4 core, additionally instrumented for recording of stack usage information. This is also the staging area for changes affecting the consumption of stack. In other words: Experiment here before making changes to tcl834_cut.

work/build

Each subdirectory in this directory represents a build variant.

The configuration information for each variant is stored in the file cdrive.cfg. It contains a reference to the sources to use (relative to work/src) and the compilation flags to give to the build system. Subdirectories not containing cdrive.cfg are ignored. If the file STACK is present in the configuration directory this configuration will be recognized as one for which a stack-test can be run.

Five configurations are predefined for your convenience.

Baseline

The unadorned 8.3.4 core.

Baseline (-O)

The 8.3.4 core, compiled with -O.

Stack (Zero)

The modular core, compiled with -O and no additional stack reduction flags. Essentially only the NRE is active. See Table 6 in the paper.

Stack (Base)

See Stack (Zero), with a basic set of stack reduction flags active. See Table 6 in the paper.

Stack (All)

See Stack (Zero), with all stack reduction flags active. See Table 6 in the paper.

work/install

Each build variant will install the results of its compilation into a subdirectory of this directory. The subdirectory will have the same name as the build variant, which is also the name of the subdirectory in work/build.

work/data

Each build variant will store the statistical information collected for it in a subdirectory of this directory. This encompasses the static size of the library, the list of available tests, stack-test information, and more.

drive.compile ?config?

Compiles all valid configurations in work/build if called without argument. If an argument is present it compiles only the specified configuration.

report-gen.static-lib-size

Goes through all valid configurations, collects the sizes of libraries and objects files and generates a nice report comparing the configurations. The report is written to stdout, its format is HTML. An example is available.

drive.stacktest

Does a stack test for all valid configurations in work/build marked with STACK (see above).

stacktest.config config

Helper to drive.stacktest. Runs a single configuration through all tests in the testsuite and measures their stack consumption.

stacktest.series config file_of_test test_id

Helper to stacktest.config. Runs a single test repeatedly, with decreasing stack space. Records the size of the stack causing the test to crash for the first time.

report-gen.stack-usage.avg

Goes through all stack.usage files generated by a run of drive.stacktest (and stored under work/data) and computes per configuration the average number of kilobytes at which a test crashed. The lower the better. The results are written to stdout. They are in plain text, no HTML.

report-gen.stack-usage stkr_log

Reads the stack recorder log file specified as the single argument, processes it and then generates a report containing the stack usage for the functions listed in the log. The report is written to stdout, its format is HTML. An example is available. Note that the script contains a hardwired list of functions to ignore. This is used to reduce the report and to skip over functions not relevant to cisco (like regex and filesystem functionality).

stack.compare usage ...

Takes one or more stack.usage files and generates a table with statistics about them. Tables generated by this script were the base for Table 5 of the paper.

get.config cdrive_file

Extracts the compilation flags from the specified configuration file and returns them on stdout.

instrument.c-file

instrument.c-file.dryrun

instrument.config

instrument.config.dryrun

instrument.postcheck.config

instrument.postcheck.file

Scripts which were used to generate the initial instrumented version of the core. This version is now heavily modified manually for improved and correcter results.

The implementation of the recorder can be found in the work/src/tcl834_stkr sources. It consists of four functions, all implemented in the file unix/tclUnixInit.c (at the bottom).

Four additional macros, declared in generic/tcl.h allow us to compile the core with and without stack recording. The controlling macro is TCL_RECORD_STACK_SIZES. Its presence activates the recorder.

These four macros are

#define TCL_STACK_INIT TclStackRecorderInit ()

Initialization of the recorder.

#define TCL_STACK_MARK(str) TclStackRecorderMark (str, __FILE__, __LINE__)

Mark entry into a function (or block).

#define TCL_STACK_POP(str) TclStackRecorderPop (str, __FILE__, __LINE__)

Execute whereever a function returns to its caller. The function checks that the name of the function popping a frame matches the name recorded in the top frame and bails out if it doesn't. A non-match indicates that a function goes through a path of execution which has no POP statement at its end.

#define TCL_STACK_STOP(str) TclStackRecorderStopOn (str, __FILE__, __LINE__)

Check for a particular call frame on the stack and assert if that happens. Helper for debugging the instrumentation. If something is off, use stop to force a core dump, thus enabling a full inspection of the true stack.

To reduce the size of the output the recorder maintains a hashtable which contains lines already written as its keys. Known strings are not written again.

The written lines contain the following information:

Called function, name, file it is line, line we are at.

Calling function.

Number of bytes on the stack used by the called function.

Use the script report-gen.stack-usage to convert such a log into a readable report.