<!--
{
  "documentType" : "article",
  "framework" : "Foundation",
  "identifier" : "/documentation/Foundation/tokenizing-natural-language-text",
  "metadataVersion" : "0.1.0",
  "role" : "article",
  "title" : "Tokenizing Natural Language Text"
}
-->

# Tokenizing Natural Language Text

Enumerate the words in a string.

## Discussion

When you work with natural language text, it’s often useful to tokenize the text into individual words. Using [`NSLinguisticTagger`](/documentation/Foundation/NSLinguisticTagger) to enumerate words, rather than simply splitting components by whitespace, ensures correct behavior in multiple scripts and languages. For example, neither Chinese nor Japanese uses spaces to delimit words.

The example and accompanying steps below show how you use [`NSLinguisticTagger`](/documentation/Foundation/NSLinguisticTagger) to enumerate over the words in natural language text.

```swift
let text = """
All human beings are born free and equal in dignity and rights.
They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
"""

let tagger = NSLinguisticTagger(tagSchemes: [.tokenType], options: 0)
tagger.string = text

let range = NSRange(location: 0, length: text.utf16.count)
let options: NSLinguisticTagger.Options = [.omitPunctuation, .omitWhitespace]
tagger.enumerateTags(in: range, unit: .word, scheme: .tokenType, options: options) { _, tokenRange, _ in
    let word = (text as NSString).substring(with: tokenRange)
    print(word)
}
```

1. Create an instance of [`NSLinguisticTagger`](/documentation/Foundation/NSLinguisticTagger), specifying [`tokenType`](/documentation/Foundation/NSLinguisticTagScheme/tokenType) as the tag scheme to be used.
2. Set the [`string`](/documentation/Foundation/NSLinguisticTagger/string) property of the linguistic tagger to the natural language text.
3. Enumerate over the entire range of the string by calling the [`enumerateTags(in:unit:scheme:options:using:)`](/documentation/Foundation/NSLinguisticTagger/enumerateTags(in:unit:scheme:options:using:)) method, specifying [`NSLinguisticTaggerUnit.word`](/documentation/Foundation/NSLinguisticTaggerUnit/word) as the tag unit and [`tokenType`](/documentation/Foundation/NSLinguisticTagScheme/tokenType) as the tag scheme to enumerate, and omitting any punctuation or whitespace.
4. In the enumeration block, take a substring of the original text at `tokenRange` to obtain each word.
5. Run this code to print out each word in `text` on a new line.

## See Also

[Identifying Parts of Speech](/documentation/Foundation/identifying-parts-of-speech)

Classify nouns, verbs, adjectives, and other parts of speech in a string.

[Identifying People, Places, and Organizations](/documentation/Foundation/identifying-people-places-and-organizations)

Use a linguistic tagger to perform named entity recognition on a string.


---

Copyright &copy; 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)