Enumeration

CFStringTokenizerTokenType

Declaration

typedef enum CFStringTokenizerTokenType : CFOptionFlags {
    ...
} CFStringTokenizerTokenType;

Overview

See http://www.unicode.org/reports/tr29/#Word_Boundaries for a detailed description of word boundaries.

Topics

Constants

kCFStringTokenizerTokenHasSubTokensMask

Compound token which may contain subtokens but with no derived subtokens.

kCFStringTokenizerTokenHasDerivedSubTokensMask

Compound token which may contain derived subtokens.

kCFStringTokenizerTokenHasHasNumbersMask

Appears to contain a number.

kCFStringTokenizerTokenHasNonLettersMask

Contains punctuation, symbols, and so on.

kCFStringTokenizerTokenIsCJWordMask

Contains kana and/or ideographs.