TIP #409: UDP in Tcl
_____________________________________________________________________________
TIP:	409
Title:	UDP in Tcl
Version:	$Revision: 1.19 $
Authors:	Alexandre Ferrieux <alexandre.ferrieux@gmail.com>
	Colin McCormack <colin@chinix.com>
State:	Draft
Type:	Project
Tcl-Version:	8.7
Vote:	Pending
Created:	Fri Aug 17 00:00:00 UTC 2012
Keywords:	udp, datagram, message
Abstract
This TIP adds support for UDP in Tcl, with a pure event approach to packet reception (as opposed to hijacking the existing channel infrastructure).
Rationale
UDP support is a long-awaited feature in Tcl. Some extensions have traditionally supported it through the channel abstraction. This is of course doable, but there is a non-negligible cost in both complexity and performance due to the impedance mismatch between the "fluid" model behind channels and the "granular" world of datagrams (where boundaries are significant). Another discrepancy is the existence of (per-packet) metadata, like the source address and port of an incoming UDP packet, which do not fit nicely in the (per-connection) options of the channel API (via fconfigure).
Once this mismatch is acknowledged, it is easy to identify a better home for UDP in Tcl: let it be a direct event source (for the receive side), just like Tk or Snack.
Indeed, hooking a callback for packet reception is a natural fit with Tcl's event-driven tradition, while preserving packet boundaries and easily exposing per-packet metadata.
Sending is trivially handled by a direct per-packet call (but not disguised as a puts). Again, this naturally allows for boundary preservation and metadata specification.
This TIP asks for UDP-in-the core for two strong reasons:
 - it enables another important core feature: asynchronous (event-driven) DNS lookups
 - the C runtime supports it without extra libraries (unlike USB or Bluetooth)
We then believe that the ubiquity of the need and the lack of external dependencies justify for UDP the first-class status that TCP enjoys (regardless of the channel vs event models). Note that an acceptable compromise would be a bundled extension, ie distributed-with-the-core (in "pkgs/").
Specification
The new udp ensemble hosts three subcommands: new, create, and names
udp new ?options?
udp create name ?options?
both create an UDP endpoint ;
udp names
returns the list of existing such endpoints.
Lifecycle
Following the traditions of Tk and Snack, udp new and udp create return a Tcl command, which takes subcommands implementing the needed verbs. By analogy with the corresponding TclOO constructors,
new generates the command name internally, in the ::tcl::udp namespace
create takes an extra name argument providing the target command name, either fully qualified, or relative to the current namespace.
 An UDP endpoint is thus created by:
   set u [udp new ...]
or
   udp create foo ...
 Once created, its configuration can be tweaked by:
$u configure -option value
and retrieved with
set value [$u configure -option]
or 
set fullconf [$u configure]
To destroy the endpoint, use:
$u destroy
or, as in Tk, destroy the command:
rename $u 
or the whole namespace.
Status of options
Among options, a few may only be specified on creation:
- local interface+port
- (optional) connect() target's address+port
- reuse flag (described in Advanced Features)
while all the other options may be specified both on creation or through [[$u configure]].
Addresses and Ports
The first two creation-only options have the following syntax:
set u [[udp new ?-local address_port? ]]
set u [[udp new ?-remote address_port? ]]
where address_port pairs are represented as 2-element Tcl lists.
set address_port [[list $address $port]]
Addresses can be given either as numeric IP (v4 or v6) addresses, or as hostnames; in the latter case, a synchronous DNS resolution is performed before actual use, just like for socket.
In the case of -local, $address can be specified as "*", meaning INADDR_ANY, and $port can be 0, asking for the OS to select a free port. Thus a dynamic port on all interfaces can be requested with
   set u [udp new -local [list * 0]]
In case of port 0, after creation of the endpoint, the actual port chosen by the OS can be retrieved with [[$u configure]]:
   puts "Local port is: [lindex [$u configure -local] 1]"
In the case of -remote, both address and port must be fully specified. The semantics, as is well known, is to tell the kernel (a) to forbid sending to any other destination, and (b) to discard all incoming packets sent by another source.
Sending a Message
Sending is done, unsurprisingly, with the send verb:
$u send payload ?dest_addr_port?
where dest_addr_port is a pair as above.
Its blocking semantics is that of the underlying send()/sendto() system calls: it typically returns immediately, though the hardware may buffer the data for some time, and delivery is not guaranteed. The payload is interpreted as a byte array that holds the entire content of the UDP message to send.
The destination parameter can be omitted if the endpoint has been created with the -remote option (connected mode).
Receiving a message asynchronously
Asynchronous (i.e. event-driven) message reception is done by specifying a listener callback to new/create or configure:
$u configure -listener command_prefix
Subsequently, when an incoming packet hits Tcl in the event loop, the command_prefix is called with the endpoint identifier, payload and metadata:
*command_prefix $u payload metadata_dict
where payload is the byte array of the UDP payload and metadata_dict is a dict containing at least two options:
-remote address_port
-local address_port
When read from options or metadata, all address components apart from "*" are returned in numeric form; no reverse DNS is ever performed.
Note that -local may carry more information than a configured -local where the address part is "*", by identifying which of the system's several interfaces was targeted.
Also note that command_prefix is a single command, possibly with arguments, that will be expanded on invocation (hence it must be a proper list); it is *not* an arbitrary script as in Tk's 'bind tradition.
When command_prefix is the empty list (which is the default), the notifier gives up interest in the underlying UDP socket; this allows to keep the port bound while letting the OS buffer any incoming packets (up to a configurable limit) without any script-level handling, while leaving the event loop active. This is similar to setting a fileevent on a channel to the empty string.
Receiving a message synchronously
Synchronous message reception is done with the recv verb:
set payload [$u recv metadata_dict_var]
The payload is returned as a byte array, and the variable passed as argument receives the metadata dict just described.
Advanced features
The following few verbs and options control extra IP features that are typically useful in popular UDP applications like media streaming:
$u configure -sendbuffer buffersize
$u configure -receivebuffer buffersize
Set the OS's send (resp. receive) buffer sizes to the given values (in bytes).
$u join multiaddr ?-source sourceaddr?
Joins the multicast group multiaddr, optionally using IGMPv3's source membership target sourceaddr.
$u leave multiaddr ?-source sourceaddr?
Leaves the same multicast group with or without source membership.
set u [udp new -reuse 1]
Sets the SO_REUSEADDR socket option, so that multiple processes may bind to the same local port and interface, and receive the same packets. Creation-time only.
$u configure -ttl ttl
Sets the TTL of subsequent outgoing packets. Some operating systems typically reduce the default TTL to one for multicast packets; this override may thus come in handy.
Rejected Alternatives
Token-based API (like for channels) instead of commands: would need extra machinery (a new Tcl_ObjType). Moreover, the command style gives reflection for free: a simple proc can emulate an UDP endpoint, and is compatible with all OO schemes.
Address+Port pairs as dicts or urls: would hamper performance for no added value
Listener as specific subcommand instead of option: R/W semantics is actually that of an option; a subcommand would preclude the direct specification of the listener on creation ; analogy with the -command of fcopy, http, or Tk widgets. 
Sources of Inspiration
This TIP builds on experience with previous attempts at UDP extensions: TclUDP, ceptcl, Duft, along with intensive practice with unpublished work, and borrows ideas from all of them.
Copyright
This document has been placed in the public domain.
_____________________________________________________________________________
Colophon
TIP AutoGenerator - written by Donal K. Fellows