<!--
{
  "availability" : [
    "iOS: -",
    "iPadOS: -",
    "macCatalyst: -",
    "macOS: -",
    "tvOS: -",
    "visionOS: -",
    "watchOS: -"
  ],
  "documentType" : "symbol",
  "framework" : "CoreFoundation",
  "identifier" : "/documentation/CoreFoundation/CFStringTokenizerTokenType",
  "metadataVersion" : "0.1.0",
  "role" : "Structure",
  "symbol" : {
    "kind" : "Structure",
    "modules" : [
      "Core Foundation"
    ],
    "preciseIdentifier" : "c:@E@CFStringTokenizerTokenType"
  },
  "title" : "CFStringTokenizerTokenType"
}
-->

# CFStringTokenizerTokenType

Token types returned by [`CFStringTokenizerGoToTokenAtIndex(_:_:)`](/documentation/CoreFoundation/CFStringTokenizerGoToTokenAtIndex(_:_:)) and [`CFStringTokenizerAdvanceToNextToken(_:)`](/documentation/CoreFoundation/CFStringTokenizerAdvanceToNextToken(_:)).

```
struct CFStringTokenizerTokenType
```

## Overview

See <http://www.unicode.org/reports/tr29/#Word_Boundaries> for a detailed description of word boundaries.

## Topics

### Constants

[`kCFStringTokenizerTokenNone`](/documentation/CoreFoundation/CFStringTokenizerTokenType/kCFStringTokenizerTokenNone)

Has no token.

[`kCFStringTokenizerTokenNormal`](/documentation/CoreFoundation/CFStringTokenizerTokenType/normal)

Has a normal token.

[`kCFStringTokenizerTokenHasSubTokensMask`](/documentation/CoreFoundation/CFStringTokenizerTokenType/hasSubTokensMask)

Compound token which may contain subtokens but with no derived subtokens.

[`kCFStringTokenizerTokenHasDerivedSubTokensMask`](/documentation/CoreFoundation/CFStringTokenizerTokenType/hasDerivedSubTokensMask)

Compound token which may contain derived subtokens.

[`kCFStringTokenizerTokenHasHasNumbersMask`](/documentation/CoreFoundation/CFStringTokenizerTokenType/hasHasNumbersMask)

Appears to contain a number.

[`kCFStringTokenizerTokenHasNonLettersMask`](/documentation/CoreFoundation/CFStringTokenizerTokenType/hasNonLettersMask)

Contains punctuation, symbols, and so on.

[`kCFStringTokenizerTokenIsCJWordMask`](/documentation/CoreFoundation/CFStringTokenizerTokenType/isCJWordMask)

Contains kana and/or ideographs.

## Relationships

### Conforms To

[`Sendable`](/documentation/Swift/Sendable)

[`BitwiseCopyable`](/documentation/Swift/BitwiseCopyable)

[`ExpressibleByArrayLiteral`](/documentation/Swift/ExpressibleByArrayLiteral)

[`RawRepresentable`](/documentation/Swift/RawRepresentable)

[`SendableMetatype`](/documentation/Swift/SendableMetatype)

[`SetAlgebra`](/documentation/Swift/SetAlgebra)

[`Equatable`](/documentation/Swift/Equatable)

[`OptionSet`](/documentation/Swift/OptionSet)

---

Copyright &copy; 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)