Important: The information in this document is obsolete and should not be used for new development.
Token Record
The token record (data type TokenRec) holds the results of the conversion of a sequence of characters to a token by theIntlTokenize
function. When it analyzes text,IntlTokenize
generates a token list, which is a sequence of token records.
TYPE TokenRec = RECORD theToken: TokenType; {numeric code for token} position: Ptr; {pointer to source text from } { which token was generated} length: LongInt; {length of source text from } { which token was generated} stringPosition: StringPtr; {pointer to Pascal string } { generated from token} END; TokenRecPtr = ^TokenRec;The fields in the token record are described under the routine description forIntlTokenize
, on page 6-95.