Legacy Document

Inside Macintosh: Text /

Chapter 6 - Script Manager / Script Manager Reference

Data Structures

Token Record

The token record (data type TokenRec) holds the results of the conversion of a sequence of characters to a token by the IntlTokenize function. When it analyzes text, IntlTokenize generates a token list, which is a sequence of token records.

TYPE 
   TokenRec = 
   RECORD
      theToken:         TokenType;  {numeric code for token}
      position:         Ptr;        {pointer to source text from }
                                    { which token was generated}
      length:           LongInt;    {length of source text from }
                                    { which token was generated}
      stringPosition:   StringPtr;  {pointer to Pascal string }
                                    { generated from token}
   END;
   TokenRecPtr = ^TokenRec;

The fields in the token record are described under the routine description for IntlTokenize, on page 6-95.