Legacy Document

Inside Macintosh: Text /

Appendix B - International Resources / Tokens Resource (Type 'itl4')

The Extension-Fetching Routine

The IntlTokenize function uses the extension-fetching routine to retrieve the second byte of a 2-byte character and determine what type of token should represent it. When IntlTokenize encounters a byte value in the source text that is represented by a -1 in the token table, IntlTokenize calls the extension-fetching routine with register A0 pointing to the second byte of the 2-byte character.

The extension-fetching routine consults internal, script-specific tables and returns the token code associated with the byte pair. IntlTokenize adds that token to the token list, and continues processing with the first byte following the byte pair.