<!--
{
  "availability" : [
    "iOS: 12.0.0 -",
    "iPadOS: 12.0.0 -",
    "macCatalyst: 13.1.0 -",
    "macOS: 10.14.0 -",
    "tvOS: 12.0.0 -",
    "visionOS: 1.0.0 -",
    "watchOS: 5.0.0 -"
  ],
  "documentType" : "symbol",
  "framework" : "NaturalLanguage",
  "identifier" : "/documentation/NaturalLanguage/NLTokenizer",
  "metadataVersion" : "0.1.0",
  "role" : "Class",
  "symbol" : {
    "kind" : "Class",
    "modules" : [
      "Natural Language"
    ],
    "preciseIdentifier" : "c:objc(cs)NLTokenizer"
  },
  "title" : "NLTokenizer"
}
-->

# NLTokenizer

A tokenizer that segments natural language text into semantic units.

```
class NLTokenizer
```

## Overview

[`NLTokenizer`](/documentation/NaturalLanguage/NLTokenizer) creates individual units from natural language text. Define the desired unit (word, sentence, paragraph, or document as declared in the [`NLTokenUnit`](/documentation/NaturalLanguage/NLTokenUnit)) for tokenization, and then assign a string to tokenize. The [`enumerateTokensInRange:usingBlock:`](/documentation/NaturalLanguage/NLTokenizer/enumerateTokensInRange:usingBlock:) method provides the ranges of the tokens in the string based on the tokenization unit.

For more information, see [Tokenizing natural language text](/documentation/NaturalLanguage/tokenizing-natural-language-text).

> Important:
> Use an ``doc://com.apple.naturallanguage/documentation/NaturalLanguage/NLTokenizer`` instance on one thread or one dispatch queue at a time. You do this by either serializing method calls to the tokenizer, or by creating a separate tokenizer instance for each thread and dispatch queue.

## Topics

### Creating a tokenizer

[`init(unit:)`](/documentation/NaturalLanguage/NLTokenizer/init(unit:))

Creates a tokenizer with the specified unit.

### Configuring a tokenizer

[`string`](/documentation/NaturalLanguage/NLTokenizer/string)

The text to be tokenized.

[`setLanguage(_:)`](/documentation/NaturalLanguage/NLTokenizer/setLanguage(_:))

Sets the language of the text to be tokenized.

[`unit`](/documentation/NaturalLanguage/NLTokenizer/unit)

The linguistic unit that this tokenizer uses.

[`NLTokenizer.Attributes`](/documentation/NaturalLanguage/NLTokenizer/Attributes)

Hints about the contents of the string for the tokenizer.

### Enumerating the tokens

[`enumerateTokens(in:using:)`](/documentation/NaturalLanguage/NLTokenizer/enumerateTokens(in:using:))

Enumerates over a given range of the string and calls the specified block for each token.

[`enumerateTokensInRange:usingBlock:`](/documentation/NaturalLanguage/NLTokenizer/enumerateTokensInRange:usingBlock:)

Enumerates over a given range of the string and calls the specified block for each token.

[`tokens(for:)`](/documentation/NaturalLanguage/NLTokenizer/tokens(for:))

Tokenizes the string within the provided range.

[`tokensForRange:`](/documentation/NaturalLanguage/NLTokenizer/tokensForRange:)

Tokenizes the string within the provided range.

[`tokenRange(at:)`](/documentation/NaturalLanguage/NLTokenizer/tokenRange(at:))

Finds the range of the token at the given index.

[`tokenRangeAtIndex:`](/documentation/NaturalLanguage/NLTokenizer/tokenRangeAtIndex:)

Finds the range of the token at the given index.

[`tokenRange(for:)`](/documentation/NaturalLanguage/NLTokenizer/tokenRange(for:))

Finds the entire range of all tokens contained completely or partially within the specified range.

[`tokenRangeForRange:`](/documentation/NaturalLanguage/NLTokenizer/tokenRangeForRange:)

Finds the entire range of all tokens contained completely or partially within the specified range.


---

Copyright &copy; 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)