CFStringTokenizer Reference

Overview

CFStringTokenizer allows you to tokenize strings into words, sentences or paragraphs in a language-neutral way. It supports languages such as Japanese and Chinese that do not delimit words by spaces, as well as de-compounding German compounds. You can obtain Latin transcription for tokens. It also provides language identification API.

You can use a CFStringTokenizer to break a string into tokens (sub-strings) on the basis of words, sentences, or paragraphs. When you create a tokenizer, you can supply options to further modify the tokenization—see “Tokenization Modifiers.”

In addition, with CFStringTokenizer:

• You can de-compound German compounds

• You can identify the language used in a string (using CFStringTokenizerCopyBestStringLanguage)

• You can obtain Latin transcription for tokens

To find a token that includes the character specified by character index and set it as the current token, you call CFStringTokenizerGoToTokenAtIndex. To advance to the next token and set it as the current token, you call CFStringTokenizerAdvanceToNextToken. To get the range of current token, you call CFStringTokenizerGetCurrentTokenRange. You can use CFStringTokenizerCopyCurrentTokenAttribute to get the attribute of the current token. If the current token is a compound, you can call CFStringTokenizerGetCurrentSubTokens to retrieve the subtokens or derived subtokens contained in the compound token. To guess the language of a string, you call CFStringTokenizerCopyBestStringLanguage.

Functions by Task

Creating a Tokenizer

• CFStringTokenizerCreate

Setting the String

• CFStringTokenizerSetString

Changing the Location

• CFStringTokenizerAdvanceToNextToken
• CFStringTokenizerGoToTokenAtIndex

Getting Information About the Current Token

• CFStringTokenizerCopyCurrentTokenAttribute
• CFStringTokenizerGetCurrentTokenRange
• CFStringTokenizerGetCurrentSubTokens

Identifying a Language

• CFStringTokenizerCopyBestStringLanguage

Getting the CFStringTokenizer Type ID

• CFStringTokenizerGetTypeID

Functions

CFStringTokenizerAdvanceToNextToken

Advances the tokenizer to the next token and sets that as the current token.

CFStringTokenizerTokenType CFStringTokenizerAdvanceToNextToken (
   CFStringTokenizerRef tokenizer
);

CFStringTokenizerCopyBestStringLanguage

Guesses a language of a given string and returns the guess as a BCP 47 string.

CFStringRef CFStringTokenizerCopyBestStringLanguage (
   CFStringRef string,
   CFRange range
);

CFStringTokenizerCopyCurrentTokenAttribute

Returns a given attribute of the current token.

CFTypeRef CFStringTokenizerCopyCurrentTokenAttribute (
   CFStringTokenizerRef tokenizer,
   CFOptionFlags attribute
);

CFStringTokenizerCreate

Returns a tokenizer for a given string.

CFStringTokenizerRef CFStringTokenizerCreate (
   CFAllocatorRef alloc,
   CFStringRef string,
   CFRange range,
   CFOptionFlags options,
   CFLocaleRef locale
);

CFStringTokenizerGetCurrentSubTokens

Retrieves the subtokens or derived subtokens contained in the compound token.

CFIndex CFStringTokenizerGetCurrentSubTokens (
   CFStringTokenizerRef tokenizer,
   CFRange *ranges,
   CFIndex maxRangeLength,
   CFMutableArrayRef derivedSubTokens
);

CFStringTokenizerGetCurrentTokenRange

Returns the range of the current token.

CFRange CFStringTokenizerGetCurrentTokenRange (
   CFStringTokenizerRef tokenizer
);

CFStringTokenizerGetTypeID

Returns the type ID for CFStringTokenizer.

CFTypeID CFStringTokenizerGetTypeID (
   void
);

CFStringTokenizerGoToTokenAtIndex

Finds a token that includes the character at a given index, and set it as the current token.

CFStringTokenizerTokenType CFStringTokenizerGoToTokenAtIndex (
   CFStringTokenizerRef tokenizer,
   CFIndex index
);

CFStringTokenizerSetString

Sets the string for a tokenizer.

void CFStringTokenizerSetString (
   CFStringTokenizerRef tokenizer,
   CFStringRef string,
   CFRange range
);

Data Types

CFStringTokenizerRef

A reference to a CFStringTokenizer object.

typedef struct __CFStringTokenizer * CFStringTokenizerRef;

Constants

Tokenization Modifiers

Tokenization options are used with CFStringTokenizerCreate to specify how the string should be tokenized

enum {
   kCFStringTokenizerUnitWord      = 0,
   kCFStringTokenizerUnitSentence  = 1,
   kCFStringTokenizerUnitParagraph = 2,
   kCFStringTokenizerUnitLineBreak = 3,
   kCFStringTokenizerUnitWordBoundary = 4,
   kCFStringTokenizerAttributeLatinTranscription = 1L << 16,
   kCFStringTokenizerAttributeLanguage           = 1L << 17
};

Constants

kCFStringTokenizerUnitWord

Specifies that a string should be tokenized by word. The locale parameter of CFStringTokenizerCreate is ignored.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerUnitSentence

Specifies that a string should be tokenized by sentence. The locale parameter of CFStringTokenizerCreate is ignored.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerUnitParagraph

Specifies that a string should be tokenized by paragraph. The locale parameter of CFStringTokenizerCreate is ignored.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerUnitLineBreak

Specifies that a string should be tokenized by line break. The locale parameter of CFStringTokenizerCreate is ignored.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerUnitWordBoundary

Specifies that a string should be tokenized by locale-sensitive word boundary.

You can use this constant in double-click range detection and whole word search. It is locale-sensitive. If the locale is en_US_POSIX, a colon (U+003A) is treated as a word separator. If the locale parameter of CFStringTokenizerCreate is NULL, the locale from the global AppleTextBreakLocale preference is used if it is available; otherwise the locale defaults to the first locale in AppleLanguages.

kCFStringTokenizerUnitWordBoundary also returns space between words as a token.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerAttributeLatinTranscription

Used with kCFStringTokenizerUnitWord, tells the tokenizer to prepare the Latin transcription when it tokenizes the string.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerAttributeLanguage

Tells the tokenizer to prepare the language (specified as an RFC 3066bis string) when it tokenizes the string.

Used with kCFStringTokenizerUnitSentence or kCFStringTokenizerUnitParagraph.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

CFStringTokenizerTokenType

Token types returned by CFStringTokenizerGoToTokenAtIndex and CFStringTokenizerAdvanceToNextToken.

enum {
   kCFStringTokenizerTokenNone                    = 0,
   kCFStringTokenizerTokenNormal                  = 1,
   kCFStringTokenizerTokenHasSubTokensMask        = 1L << 1,
   kCFStringTokenizerTokenHasDerivedSubTokensMask = 1L << 2,
   kCFStringTokenizerTokenHasHasNumbersMask       = 1L << 3,
   kCFStringTokenizerTokenHasNonLettersMask       = 1L << 4,
   kCFStringTokenizerTokenIsCJWordMask            = 1L << 5
};
typedef CFOptionFlags CFStringTokenizerTokenType;

Constants

kCFStringTokenizerTokenNone

Has no token.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenNormal

Has a normal token.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenHasSubTokensMask

Compound token which may contain subtokens but with no derived subtokens.

You can obtain subtokens by calling CFStringTokenizerGetCurrentSubTokens.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenHasDerivedSubTokensMask

Compound token which may contain derived subtokens.

You can obtain subtokens and derived subtokens by calling CFStringTokenizerGetCurrentSubTokens.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenHasHasNumbersMask

Appears to contain a number.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenHasNonLettersMask

Contains punctuation, symbols, and so on.

Given the way Unicode word break works, this means it is a standalone punctuation or symbol character, or a string of such.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.

kCFStringTokenizerTokenIsCJWordMask

Contains kana and/or ideographs.

Available in iOS 3.0 and later.

Declared in CFStringTokenizer.h.