Inside Macintosh: Text /

Appendix B - International Resources

Tokens Resource (Type 'itl4')

The tokens resource (resource type'itl4') contains information needed to convert text into a series of language-independent tokens. Compilers, interpreters, and other expression evaluators convert source text to tokens as an initial step in their processing. The Script Manager IntlTokenize function uses information in the tokens resource to produce tokens from source text.

The tokens resource also contains tables for converting tokens into text, for formatting numbers, and for determining whitespace characters.

Each enabled script system has one or more tokens resources. The resource ID for each one is within the range of resource ID numbers for that script system. The default 'itl4' resource for a script is specified in the itlbToken field of the script's international bundle ('itlb') resource.

Each tokens resource contains the following:

• Header
• Tokenizing tables and code
• Untoken table
• Number parts table
• Whitespace table

The tokenizing tables and code are used by the IntlTokenize function. The untoken table is used by the IntlTokenize function and by applications that want to convert tokens to strings. The number parts table is used by the Text Utilities number-formatting routines and by applications that do their own number formatting. The whitespace table is available for application use.

---

Subtopics

The NItl4Rec Data Type

The Token Table

The Extension-Fetching Routine

The Token-String Copy Routine

The Untoken Table

The Number Parts Table

The Whitespace Table