Legacy Document

Inside Macintosh: Text /

Appendix B - International Resources / Tokens Resource (Type 'itl4')

The Untoken Table

The untoken table provides a Pascal string for any type of fixed token. A fixed token is a token whose representation is unvarying, like punctuation. Alphabetic and numeric tokens are not fixed; specifying the token does not specify the string it represents.

The untoken table contains standard representations for the fixed tokens. You can use
it to display the canonical format for any fixed token in the script system of the
tokens resource.

The unTokenTable data type describes the format of the untoken table:

TYPE
   UntokenTable = 
   RECORD
      len:        Integer;       {length of untoken table}
      lastToken:  Integer;       {maximum token code to be used}
      index:      ARRAY[0..255] OF Integer;
                                 {offsets to Pascal strings for }
                                 { tokens; last entry = lastToken}
   END;

   UntokenTablePtr         =  ^UntokenTable;
   UntokenTableHandle      =  ^UntokenTablePtr;

Field Description

len

The length in bytes of the untoken table.

lastToken

The highest token code used in this table (for range-checking).

index

An array of byte offsets from the beginning of the untoken table to Pascal strings--one for each possible token type--that give the canonical format for each fixed token type. The entries in the array correspond, in order, to token code values from 0 to lastToken. For example, the offset to the Pascal string for tokenColonEqual (token code = 39) is found at offset 39 in the array.

The string data directly follows the index array. It is a simple concatenation of Pascal strings; for example, the Pascal string for the token type tokenColonEqual may consist of a length byte (of value 2) followed by the characters ":=".