TIP #85: Custom Comparisons in Tcltest
_____________________________________________________________________________
TIP:	85
Title:	Custom Comparisons in Tcltest
Version:	$Revision: 1.14 $
Authors:	Arjen Markus <arjen.markus@wldelft.nl>
	Don Porter <dgp@users.sf.net>
State:	Final
Type:	Project
Tcl-Version:	8.4
Vote:	Done
Created:	Thu Jan 31 00:00:00 UTC 2002
Keywords:	test, string comparison, floating-point
Abstract
This TIP proposes a simple mechanism to make the tcltest package an even more flexible package than it already is by allowing the programmer to define his or her own comparison procedures. Such procedures can deal with issues like allowing a (small) tolerance in floating-point results.
Rationale
The test command of the package tcltest 2.0 supports the comparison of the actual result with the expected result by a number of methods: exact matching, glob-style matching and matching via a regular expression, according to the -match option. The flexibility is indeed enhanced over the package tcltest 1.0, as it is now much easier to allow for small variations in string results. But it is nearly impossible to define an accurate test that checks if floating-point results are the "same" - exact matching will seldom suffice due to platform-specific round-off errors or differences in formatting a floating-point number (0.12 versus .12 for instance).
It is also impossible to compare results that are not easily expressed as strings, for instance an application that produces binary files that need to be compared or simply very long strings - these could easily be stored in an external file, but would be awkward in a file with a large number of such tests.
Proposal
The package tcltest 2.0.2 defines an internal comparison procedure, CompareStrings that performs matching according to the three built-in -match options of test. This procedure can easily be replaced by one that invokes registered commands or procedures. Such a command or procedure takes two arguments and returns 1 for a match and a 0 for failure, just as CompareStrings does in the current implementation:
 proc myMatchProc  expected actual   
   if  $expected (is somehow equal) $actual  
      return 1
    else
      return 0

A new public command customMatch is proposed for the purpose of registering these matching commands. It can register a procedure, such as myMatchProc defined above:
 ::tcltest::customMatch mytype myMatchProc
or, as in the sample implementation, an incomplete command:
 ::tcltest::customMatch exact [list ::string equal]
When the test command is called with the -match mytype option, the command myMatchProc will be completed with two arguments, the expected and actual results, and will be evaluated in the global namespace to determine whether the test result matches the expected result. Likewise, the test option -match exact will cause matching to be tested by the command ::string equal. The default value of the -match option will continue to be exact.
Allowing procedures to be invoked by their type names gives us the flexibility to register as many such procedures or commands as required.
Because this proposal adds a new public command to the tcltest package, the version will be incremented to 2.1.
A patch to the current HEAD that implements this proposal is available as Tcl Patch 521362 at the Tcl project at SourceForge. http://sf.net/tracker/?func=detail&aid=521362&group_id=10894&atid=310894
Two Examples
To show how this works, we include two simple examples:
Testing a package for calculating mathematical functions like Bessel functions.
Testing for negative results, as when providing an alternative, but incompatible implementation of a feature.
First, suppose you have defined a package for calculating the value of a general Bessel function, just the sort of function that returns floating-point numbers. Then the results may be imprecise due to rounding-off errors, different values of tcl_precision or, even more banally, differences in the formatting of floating-point numbers (0.12 versus .12 for instance). 
The following shows how to do this:
 #
 # Test implementation of Bessel functions
 # (Table only provides 4 decimals)
 #
 customMatch 4decimals matchFloat4Decimals

 proc matchFloat4Decimals  expected actual  
    return [expr abs($expected-$actual) <= 0.5e-4]

 test "J0-1.1" "J0 for x=1.0" -match 4decimals -body 
    J0 1.0
  -result 0.7652

 test "J1-1.1" "J0 for x=1.0" -match 4decimals -body 
    J1 1.0
  -result 0.4401
The second example occurs for instance when testing alternative implementations: you want to check that the original standard feature is failing whereas the new but incompatible alternative gets it right. Then:
 proc matchNegative  expected actual  
    set match 0
    foreach a $actual e $expected 
       if  $a != $e  
          set match 1
          break

    return $match

 customMatch negative matchNegative

 #
 # Floating-point comparisons are imprecise. The following
 # test returns typically such a list as 643 1357 1921 79 781 1219
 # so nothing even close to the expected values.
 # 
 test "ManyCompares-1.2" "Compare fails - naive comparison" \par    -match negative -body 
    set naiv_eq 0
    set naiv_ne 0
    set naiv_ge 0
    set naiv_gt 0
    set naiv_le 0
    set naiv_lt 0

    for  set i -1000   $i <= 1000   incr i  
       if  $i == 0  continue

       set x [expr 1.01/double($i)]
       set y [expr (2.1*$x)*(double($i)/2.1)]

       if  $y == 1.01   incr naiv_eq 
       if  $y != 1.01   incr naiv_ne 
       if  $y >= 1.01   incr naiv_ge 
       if  $y >  1.01   incr naiv_gt 
       if  $y <= 1.01   incr naiv_le 
       if  $y <  1.01   incr naiv_lt 
    
    set result [list $naiv_eq $naiv_ne $naiv_ge $naiv_gt $naiv_le $naiv_lt]
  -result 2000 0 2000 0 2000 0
makes sure that a mismatch is treated as the expected outcome.
Alternatives and objections
Of course, it is possible to achieve these effects within the current framework of tcltest, by putting these match procedures inside the body of the test case. No extra user command would be necessary then.
There are at least two drawbacks to this approach:
The result against which we want to match is hidden in the code
If the test fails, the actual result is not printed (at least not by the tcltest framework).
As a matter of fact, the proposed mechanism actually simplifies the current implementation of the three match types to a certain degree by turning a switch between the three types into an array index.
See Also
Tcl Feature Request 490298. http://sf.net/tracker/?func=detail&aid=490298&group_id=10894&atid=360894
History
Cameron Laird was quite enthousiastic about the idea of providing custom match procedures.
Mo DeJong requested the explicit examples (the second is actually the situation that triggered this TIP in the first place).
Don Porter <dgp@users.sf.net> revised the registration mechanism such that an arbitrary set of matching commands or procedures can be supported. His suggestions led to a revision of the TIP. He also revised the draft implementation.
Copyright
This document is placed in the public domain.
_____________________________________________________________________________
Colophon
TIP AutoGenerator - written by Donal K. Fellows