# CFStringTokenizer ``` class CFStringTokenizer ``` ## Overview CFStringTokenizer allows you to tokenize strings into words, sentences or paragraphs in a language-neutral way. It supports languages such as Japanese and Chinese that do not delimit words by spaces, as well as de-compounding German compounds. You can obtain Latin transcription for tokens. It also provides language identification API. You can use a CFStringTokenizer to break a string into tokens (sub-strings) on the basis of words, sentences, or paragraphs. When you create a tokenizer, you can supply options to further modify the tokenization—see [Tokenization Modifiers](/documentation/CoreFoundation/1588024-tokenization-modifiers). In addition, with CFStringTokenizer: - You can de-compound German compounds - You can identify the language used in a string (using [`CFStringTokenizerCopyBestStringLanguage(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyBestStringLanguage(_:_:))) - You can obtain Latin transcription for tokens To find a token that includes the character specified by character index and set it as the current token, you call [`CFStringTokenizerGoToTokenAtIndex(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGoToTokenAtIndex(_:_:)). To advance to the next token and set it as the current token, you call [`CFStringTokenizerAdvanceToNextToken(_:)`](/documentation/CoreFoundation/CFStringTokenizerAdvanceToNextToken(_:)). To get the range of current token, you call [`CFStringTokenizerGetCurrentTokenRange(_:)`](/documentation/CoreFoundation/CFStringTokenizerGetCurrentTokenRange(_:)). You can use [`CFStringTokenizerCopyCurrentTokenAttribute(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyCurrentTokenAttribute(_:_:)) to get the attribute of the current token. If the current token is a compound, you can call [`CFStringTokenizerGetCurrentSubTokens(_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGetCurrentSubTokens(_:_:_:_:)) to retrieve the subtokens or derived subtokens contained in the compound token. To guess the language of a string, you call [`CFStringTokenizerCopyBestStringLanguage(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyBestStringLanguage(_:_:)). ## Topics ### Creating a Tokenizer [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) Returns a tokenizer for a given string. ### Setting the String [`CFStringTokenizerSetString(_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerSetString(_:_:_:)) Sets the string for a tokenizer. ### Changing the Location [`CFStringTokenizerAdvanceToNextToken(_:)`](/documentation/CoreFoundation/CFStringTokenizerAdvanceToNextToken(_:)) Advances the tokenizer to the next token and sets that as the current token. [`CFStringTokenizerGoToTokenAtIndex(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGoToTokenAtIndex(_:_:)) Finds a token that includes the character at a given index, and set it as the current token. ### Getting Information About the Current Token [`CFStringTokenizerCopyCurrentTokenAttribute(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyCurrentTokenAttribute(_:_:)) Returns a given attribute of the current token. [`CFStringTokenizerGetCurrentTokenRange(_:)`](/documentation/CoreFoundation/CFStringTokenizerGetCurrentTokenRange(_:)) Returns the range of the current token. [`CFStringTokenizerGetCurrentSubTokens(_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGetCurrentSubTokens(_:_:_:_:)) Retrieves the subtokens or derived subtokens contained in the compound token. ### Identifying a Language [`CFStringTokenizerCopyBestStringLanguage(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyBestStringLanguage(_:_:)) Guesses a language of a given string and returns the guess as a BCP 47 string. ### Getting the CFStringTokenizer Type ID [`CFStringTokenizerGetTypeID()`](/documentation/CoreFoundation/CFStringTokenizerGetTypeID()) Returns the type ID for CFStringTokenizer. ### Constants [Tokenization Modifiers](/documentation/CoreFoundation/1588024-tokenization-modifiers) Tokenization options are used with [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) to specify how the string should be tokenized [`CFStringTokenizerTokenType`](/documentation/CoreFoundation/CFStringTokenizerTokenType) Token types returned by [`CFStringTokenizerGoToTokenAtIndex(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGoToTokenAtIndex(_:_:)) and [`CFStringTokenizerAdvanceToNextToken(_:)`](/documentation/CoreFoundation/CFStringTokenizerAdvanceToNextToken(_:)). ## See Also [String Programming Guide for Core Foundation](https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFStrings/introCFStrings.html#//apple_ref/doc/uid/10000131i) --- Copyright © 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)