<!--
{
  "documentType" : "article",
  "framework" : "CoreFoundation",
  "identifier" : "/documentation/CoreFoundation/1588024-tokenization-modifiers",
  "metadataVersion" : "0.1.0",
  "role" : "collectionGroup",
  "title" : "Tokenization Modifiers"
}
-->

# Tokenization Modifiers

Tokenization options are used with [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) to specify how the string should be tokenized

## Discussion

You use the tokenization unit options with [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) to specify how a string should be tokenized.

You use the modifiers together with a tokenization unit to modify the way the string is tokenized.

You use the attribute specifiers to tell the tokenizer to prepare the specified attribute when it tokenizes the given string. You can retrieve the attribute value by calling [`CFStringTokenizerCopyCurrentTokenAttribute(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCopyCurrentTokenAttribute(_:_:)) with one of the attribute options.

The locale sensitivity of the tokenization unit options may change in a future release.

## Topics

### Constants

[`kCFStringTokenizerUnitWord`](/documentation/CoreFoundation/kCFStringTokenizerUnitWord)

Specifies that a string should be tokenized by word. The `locale` parameter of [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) is ignored.

[`kCFStringTokenizerUnitSentence`](/documentation/CoreFoundation/kCFStringTokenizerUnitSentence)

Specifies that a string should be tokenized by sentence. The `locale` parameter of [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) is ignored.

[`kCFStringTokenizerUnitParagraph`](/documentation/CoreFoundation/kCFStringTokenizerUnitParagraph)

Specifies that a string should be tokenized by paragraph. The `locale` parameter of [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) is ignored.

[`kCFStringTokenizerUnitLineBreak`](/documentation/CoreFoundation/kCFStringTokenizerUnitLineBreak)

Specifies that a string should be tokenized by line break. The `locale` parameter of [`CFStringTokenizerCreate(_:_:_:_:_:)`](/documentation/CoreFoundation/CFStringTokenizerCreate(_:_:_:_:_:)) is ignored.

[`kCFStringTokenizerUnitWordBoundary`](/documentation/CoreFoundation/kCFStringTokenizerUnitWordBoundary)

Specifies that a string should be tokenized by locale-sensitive word boundary.

[`kCFStringTokenizerAttributeLatinTranscription`](/documentation/CoreFoundation/kCFStringTokenizerAttributeLatinTranscription)

Used with `kCFStringTokenizerUnitWord`, tells the tokenizer to prepare the Latin transcription when it tokenizes the string.

[`kCFStringTokenizerAttributeLanguage`](/documentation/CoreFoundation/kCFStringTokenizerAttributeLanguage)

Tells the tokenizer to prepare the language (specified as an RFC 3066bis string) when it tokenizes the string.


---

Copyright &copy; 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)