TIP #412: Dynamic Locale Changing for msgcat with On-Demand File Load


TIP:412
Title:Dynamic Locale Changing for msgcat with On-Demand File Load
Version:$Revision: 1.9 $
Authors: Harald Oehlmann <harald dot oehlmann at elmicron dot de>
Harald Oehlmann <oehhar at users dot sourceforge dot net>
State:Final
Type:Project
Tcl-Version:8.6
Vote:Done
Created:Tuesday, 27 March 2012
Obsoletes:TIP #399
Keywords:Tcl, localization, msgcat

Abstract

This TIP adds dynamic locale switching capabilities to the msgcat package.

Rationale

Dynamic Locale Switching

Within a multi-language application like a web-server, one may change the locale quite frequently, for example if users with different locales are requesting pages. Unfortunately, this does not fit well with the model adopted by the msgcat package, which assumes that all code follows this sequence:

  1. Set locale list: mclocale locale

  2. Load language files with other package load: mcload msg-folder

  3. Translate strings: mc key args...

Note that if the locale should be changed after other packages are loaded, one must restart at step 2. This requires reloading all packages which is mostly not practical.

The aim of this TIP is to extend the package by dynamic locale change capabilities.

msgcat will reload any missing message catalog files of all currently loaded packages on a locale change. In addition, any package may register to get informed to a locale change. Other packages may do changes to reflect the locale change like rebuilding the GUI.

This TIP compares to TIP #399 that the package is able to load message catalog files on demand, e.g. specially on a locale change.

package locale

If the clock command gets called with the argument "-locale", the locale is changed using msgcat::mclocale. After processing, the initial value is restored. The package keeps track, which locales where already used and calls msgcat::mcload for any new locale. The locale is restored after processing.

This is an implementation of dynamic locales but conflicts with the new features described above. Other packages may be informed to change the locale and may trigger expensive operations like a rebuild of the GUI.

In consequence, each package may define a package locale which is independent of the default locale.

Overview of the proposed solution

Proposed changes in brief:

Dynamically load message catalog files

if the locale is changed by mclocale locale, the message file load process is executed for every present package.

Package locale

A package may install a package local locale which is independent to the global locale.

Locale change callback

A callback may be registered to get informed about the change of locale. A use case is to refresh a GUI if the locale changed.

Non message file operation

A program may use message files to issue mcset commands or may issue them by other means, if the message catalogs are, for example, stored in a data base.

Each package may register a callback to get informed that a certain locale should be loaded and may issue the corresponding mcset commands.

Package mcunknown

A package may have a certain way to provide translations for message keys not included in the message catalog. Thus, it may register an own package message unknown callback to provide a translation.

Specification

Package Equals Client Namespace

A client package is a package which uses msgcat. A unique namespace is required for each client package. Within msgcat, namespace and package is always connected.

Up to now, the msgcat package used this namespace as an identifier to store the catalog data of a certain package. This is now extended to additional properties which are stored for a package.

Package locale

A package locale may be used by a package instead the default locale set by msgcat::mclocale. A package may choose to use a package locale or the default locale.

Default and Package State

Some state values (like the locale) are available as default (global) values. In addition, each package may choose to use a package locale state.

The used naming is:

default

state

valid for all packages which do not set a package state.

package

state

only valid for one package if it has set a package state.

The following state values are present as default state and may be set individually per package:

The locale like "de_ch". The preferences property is a list of locales in their preference order and is automatically computed from locale. Example locale = "de_ch" -> preferences = "de_ch de {}". The loadedlocales state value is the list of currently loaded locales.

Default State

The following standard methods exist to get or set the default state:

msgcat::mclocale

The default locale. It may be read using msgcat::mclocale.

It may be set using msgcat::mclocale locale. This command is extended, that the message catalogs of all missing locales for all packages not having set a package state are loaded.

msgcat::mcpreferences

Get the default preferences (derived from the default locale).

msgcat::mcloadedlocales

The following new command may be used to deal with the default state:

msgcat::mcloadedlocales subcommand ?locale?

The parameter locale is mandatory for the subcommand present.

The following subcommands are available:

Subcommand "get"

Get the list of current loaded locales

Subcommand "present"

Returns true, if the given locale is loaded

Subcommand "clear"

The list of currently loaded locales is set to mcpreferences and all message catalog keys of packages without a package locale set and with locales not in mcpreferences are unset.

Package Configuration

The package configuration of the calling package may be changed using the following new command:

msgcat::mcpackagelocale subcommand ?locale?

The parameter locale is mandatory for the subcommands set and present.

Available subcommands are:

Subcommand "set"

Set or change the package locale.

The global state values are copied, if there were no package locale set before.

The package locale is changed to the optional given new package locale.

Subcommand "get"

Return the package locale or the default locale, if no package locale set.

Subcommand "preferences"

Return the package preferences or the default preferences, if no package locale set.

Subcommand "loaded"

The list of locales loaded for this package is returned.

Subcommand "isset"

Returns true, if a package locale is set.

Subcommand "unset"

Unset the package locale and use the default state for the package. Load all message catalog files of the package for locales, which were not present in the package loadedlocales list and are present in the default list.

Subcommand "present"

Returns true, if the given locale is loaded

Subcommand "clear"

Set the current loaded locales list of the package to preferences and unset all message catalog keys of the package with locales not included in the package preferences.

Package Configuration Options

Each package may have a set of configuration options set to invoke certain actions. They may be retrieved or changed with the following new command:

msgcat::mcpackageconfig subcommand option ?value?

Available subcommands are:

get

Get the current value of the option or an error if not set.

isset

Returns true if option is set.

set

Set the given value to the option. May have additional consequences and return values as described in the option section.

unset

Unset the option.

Available options are:

Package Option "mcfolder"

This is the message folder of the package. This option is set by mcload and by the subcommand set. Both are identical and both return the number of loaded message catalog files.

Setting or changing this value will load all locales contained in the preferences valid for the package. This implies also to invoke any set loadcmd (see below).

Unsetting this value will disable message file load for the package.

If the locale valid for this package changes, this value is used to eventually load message catalog files.

Message catalog files are always sourced in the namespace of the package registering the value.

Package Option "loadcmd"

This callback is invoked before a set of message catalog files are loaded for the package which has this property set.

This callback may be used to do any preparation work for message file load or to get the message data from another source like a data base. In this case, no message files are used (mcfolder is unset).

See chapter callback invocation below. The parameter list appended to this callback is the list of locales to load.

If this callback is changed, it is called with the preferences valid for the package.

Package Option "changecmd"

This callback is invoked when a default local change was performed. Its purpose is to allow a package to update any dependency on the default locale like showing the GUI in another language.

Tk may be extended to register to this callback and to invoke a virtual event.

See the callback invocation section below. The parameter list appended to this callback is mcpreferences. All registered packages are invoked in no particular order.

Package Option "unknowncmd"

Use a package locale mcunknown procedure instead of the standard version supplied by the msgcat package (msgcat::mcunknown).

The called procedure must return the formatted message which will finally be returned by msgcat::mc.

A generic unknown handler is used if set to the empty string. This consists in returning the key if no arguments are given. With given arguments, format is used to process the arguments.

See chapter callback invocation below. The appended arguments are identical to mcunknown.

Callback Invocation

Callbacks are invoked under the following conditions:

the callback command is set, the command is not the empty string, the registration namespace exists.

Any error within the callback stops the operation which invoked the callback. This might be surprising, as the error might be in another package.

Test if Message Key is Set

Message catalog keys may be expensive to calculate and thus may be set on demand.

The following new procedure returns false, if mc would call mcunknown for a key:

msgcat::mcexists src

There are two options, to limit the key search to just the current namespace (don't search in parent namespaces) and just the current locale (don't search the preferences but the first item):

msgcat::mcexists ?-exactnamespace? ?-exactlocale? src

forget package

A package may clear all its keys and state using the new command:

msgcat::mcforgetpackage

Locale and Preferences Format

Locales set by mcset may eventually not correspond to the current preferences, as the preferences are treated as follows:

put to lower case, remove any multiple "_" and any "_" at the beginning or at the end of the

locale.

It is proposed, that:

the locale and the first preferences element is always identical to the

lowercase passed locale,

any multiple "_" are seen as one separator.

Example: preferences of locale "sy__cyrl_win"

current preferences

"sy_cyrl_win sy_cyrl sy" proposed preferences: "sy__cyrl_win sy__cyrl sy".

Alternatively, all locales may normalized using the upper algorithm, which felt heavy in computation with little gain.

Example Usage

Example from TIP #399

Imagine an application which supports the current user language and French, German and English. An external package tp is used. The package uses msgcat and installs itself during the package require tp call:

package require msgcat
msgcat::mcload [file join [file dirname [info script]] msgs]

An implementation of the application with the current msgcat 1.5.0 would require the following initialization sequence:

package require msgcat
package require np

and the following code to change the locale to French:

package forget np
msgcat::mclocale fr
package require np

Using the extension of this TIP, one may load as usual:

package require msgcat
package require np

and to change to french locale:

msgcat::mclocale fr

The first time, a locale is required, all corresponding message files of all packages which use msgcat get loaded. This might be a heavy operation.

If a locale is reactivated (and the message catalog data was not cleared), it is a quick operation.

Without this TIP, it is computational expensive (if possible, as many packages are not reloadable or a reload may disturb current processing, e.g., by forcing the closing of sockets, etc.).

Change with No Need to Come Back

If it is certain that a locale is changed and the then obsolete data is of no use, one may clear unused message catalog items:

msgcat::mclocale fr
msgcat::mcloadedlocale clear

Use a Callback to be Notified About a Locale Change

Packages which display a GUI may update their widgets when the locale changes. To register to a callback, use:

namespace eval gui {
 msgcat::mcpackageconfig changecmd updateGUI

 proc updateGui args {
 puts "New locale is '[lindex $args 0]'."
 }
}

% msgcat::mclocale fr
fr
% New locale is 'fr'.

To Use Another Locale Source than Message Catalog Files

If locales (or additional locales) are contained in another source like a data base, a package may use the load callback and not mcload:

namespace eval db {
 msgcat::mcpackageconfig loadcmd loadMessages
 msgcat::mcconfig loadedpackages\
 [concat [msgcat::mcconfig loadedpackages] namespace current]

 proc loadMessages args {
 foreach locale $args {
 if {[LocaleInDB $locale]} {
 msgcat::mcmset $locale [GetLocaleList $locale]
 }
 }
 }
}

Use a package locale

The reference implementation also contains a changed clock command which uses a package locale. Here are some sketches from the implementation.

First, a package locale is initialized and the generic unknown function is activated:

msgcat::mcpackagelocale set
msgcat::mcpackageconfig unknowncmd ""

If the user requires the week day in a certain locale, it is changed:

clock format clock seconds -format %A -locale fr

and the code:

msgcat::mcpackagelocale set $locale
return [lindex [msgcat::mc DAYS_OF_WEEK_FULL] $day]
### Returns "mercredi"

Some message-catalog items are heavy in computation and thus are dynamically cached using:

proc ::tcl::clock::LocalizeFormat { locale format } {
 set key FORMAT_$format
 if { [::msgcat::mcexists -exactlocale -exactnamespace $key] } {
 return [mc $key]
 }
 #...expensive computation of format clipped...
 mcset $locale $key $format
 return $format
}

Reference Implementation

See Tcl fossil tag msgcat_dyn_locale TIP #1.

Compatibility

Imagined incompatibilities:

If packages call mcload multiple times with different folders, the

data was currently appended. This is still the case, but only the last folder is used for any reload. The property mcfolder may be transformed to a list to cover this case.

The return value of mcload (file count) may be much higher as there

may be loaded much more files. I suppose, this value is only used by the test suite to verify functionality and is not for big general use.

Message files may not be aware, that they may be loaded at any moment and

not only after their own mcload. I suppose, this is the biggest issue but I think, there is no alternative.

Message files do not get reloaded any more, if a second mcload is

issued with the same path argument.

Package which temporary change the default locale trigger any callback

and may lead to user visible side effects.

Issues

Known issues:

Packages might not be aware of a locale change and may buffer

translations outside of msgcat. Packages should not buffer msgcat messages if they are used in a dynamic locale application (like tklib tooltip does for example).

The clock command currently has a small dynamic patch for msgcat

implemented. This must be removed in favor to new msgcat features due to the temporarily change of the default locale.

Extensions

Expose the function to calculate the preference list from a given locale. Load a message catalog file for a given locale without changing the

default/package locale.

Methods isloaded to check if a locale is currently loaded. Access message catalog with specified namespace, locale and search

behavior.

Alternatives

The alternative is the former TIP #399, but that is problematic because the list of locales must be known before any package load. The additional complexity of this TIP is a justifiable trade-off against the greatly improved flexibility in the loading and locale selection order.

Copyright

This document has been placed in the public domain.


Powered by Tcl[Index] [History] [HTML Format] [Source Format] [LaTeX Format] [Text Format] [XML Format] [*roff Format (experimental)] [RTF Format (experimental)]

TIP AutoGenerator - written by Donal K. Fellows