ADC Home > Reference Library > Reference > Text & Fonts > Text Manipulation > Latent Semantic Mapping Reference

LatentSemanticMapping.h

Include Path:	<LatentSemanticMapping/LatentSemanticMapping.h>
Path:	/System/Library/Frameworks/LatentSemanticMapping.framework/Versions/A/Headers/LatentSemanticMapping.h
Includes:	<CoreFoundation/CoreFoundation.h> <CoreServices/CoreServices.h> <Carbon/Carbon.h> <stdio.h> <stdint.h>

Overview

This header contains the Latent Semantic Mapping API, which supports the classification of text and other token-based content, based on latent semantic information. For example, you can use the functions of the LSM API to classify an email message as consistent or inconsistent with a user's interests.

The LSM API defines the following entities:

A map, which includes a language-analysis algorithm and a repository of knowledge about the language characteristics that are associated with a number of specific categories.
A category, which represents a developer-supplied set of rules used to classify text and other token-based content.

To begin the process of classification, you first create an LSM map and train it by giving it text samples that exhibit the characteristics of your selected categories. The more text samples you use, the more you refine the map's ability to analyze and classify arbitrary input. After you have trained the map you compile it, which transforms the map into an executable object. To classify text and other token-based content, you use the LSMResultCreate function to get either the categories or the words that best match the input text.

The executable form of an LSM map can be stored to and loaded from a file. A stored map preserves its training data, and it's possible (depending on the options you set when you store the map) to give the map additional training input and recompile it.

Functions

LSMMapAddCategory: Adds a category to the specified map.
LSMMapAddText: Adds a training text to the specified category.
LSMMapAddTextWithWeight: Adds a training text to the given category, with a weight other than 1.
LSMMapApplyClusters: Groups categories, words, or tokens into the specified sets of clusters.
LSMMapCompile: Compiles the map into executable form.
LSMMapCreate: Creates a new LSM map.
LSMMapCreateClusters: Computes a set of clusters, grouping similar categories, words, or tokens.
LSMMapCreateFromURL: Loads a map from the specified file.
LSMMapGetCategoryCount: Returns the number of categories in the specified map.
LSMMapGetProperties: Gets the dictionary of properties for the map.
LSMMapGetTypeID: Returns the Core Foundation type identifier for LSM maps.
LSMMapSetProperties: Sets a dictionary of properties for the map.
LSMMapSetStopWords: Specifies words to be omitted from all classification efforts.
LSMMapStartTraining: Puts the map into training mode, preparing it for the addition of more categories or texts or both.
LSMMapWriteToStream: Writes information about a map or a text or both to a byte stream in text form.
LSMMapWriteToURL: Compiles the map if necessary and stores it into the specified file.
LSMResultCopyToken: Returns the token for the n-th best (zero-based) result.
LSMResultCopyTokenCluster: Returns the cluster of tokens for the n-th best (zero-based) result.
LSMResultCopyWord: Returns the word for the n-th best (zero-based) result.
LSMResultCopyWordCluster: Returns the cluster of words for the n-th best (zero-based) result.
LSMResultCreate: Returns, in decreasing order of likelihood, the categories or words that best match when a text is mapped into a map.
LSMResultGetCategory: Returns the category of the n-th best (zero-based) result.
LSMResultGetCount: Returns the number of results associated with the specified result.
LSMResultGetScore: Returns the likelihood of the n-th best (zero-based) result.
LSMResultGetTypeID: Returns the Core Foundation type identifier for LSM results.
LSMTextAddToken: Adds an arbitrary binary token to the text.
LSMTextAddWord: Adds a word to the text.
LSMTextAddWords: Breaks a string into words using the specified locale, and adds the words to the text.
LSMTextCreate: Creates a new text.
LSMTextGetTypeID: Returns the Core Foundation type identifier for LSM texts.

LSMMapAddCategory

Adds a category to the specified map.

LSMCategory LSMMapAddCategory(
    LSMMapRef mapref);

Parameters

mapref: An LSM map.

Return Value

The category identifier of the category added to the map. See Error_Codes for error codes that might be returned.

LSMMapAddText

Adds a training text to the specified category.

OSStatus LSMMapAddText(
    LSMMapRef mapref,
    LSMTextRef textref,
    LSMCategory category);

Parameters

mapref: An LSM map.
textref: The input text.
category: The category to which to add the text.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

The textref parameter is not needed after this call, and can be released.

LSMMapAddTextWithWeight

Adds a training text to the given category, with a weight other than 1.

OSStatus LSMMapAddTextWithWeight(
    LSMMapRef mapref,
    LSMTextRef textref,
    LSMCategory category,
    float weight);

Parameters

mapref: An LSM map.
textref: The input text.
category: The category to which to add the text.
weight: The weight of the text (must not equal 1).

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

Although the value of the weight parameter may be negative, no word will receive a total count less than 0. The textref parameter is not needed after this call, and can be released.

LSMMapApplyClusters

Groups categories, words, or tokens into the specified sets of clusters.

OSStatus LSMMapApplyClusters(
    LSMMapRef mapref,
    CFArrayRef clusters);

Parameters

mapref: An LSM map.
clusters: An array of clusters.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

LSMMapCompile

Compiles the map into executable form.

OSStatus LSMMapCompile(
    LSMMapRef mapref);

Parameters

mapref: An LSM map.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

This function puts the map into mapping mode and prepares it for the classification of texts and other token-based content. Note that this function is computationally expensive.

LSMMapCreate

Creates a new LSM map.

LSMMapRef LSMMapCreate(
    CFAllocatorRef alloc,
    CFOptionFlags flags);

Parameters

alloc: An application-specified allocator.
flags: Any of the options that specify mapping accuracy, defined in Map_Flags.

Return Value

An LSM map. See Error_Codes for error codes that might be returned.

Discussion

Call CFRelease to dispose of the map.

LSMMapCreateClusters

Computes a set of clusters, grouping similar categories, words, or tokens.

CFArrayRef LSMMapCreateClusters(
    CFAllocatorRef alloc, 
    LSMMapRef mapref,
    CFArrayRef subset, 
    CFIndex numClusters,
    CFOptionFlags flags);

Parameters

alloc: A user-specified allocator.
mapref: An LSM map.
subset: An array of categories, words, or tokens on which to perform clustering.
numClusters: The number of clusters desired.
flags: An option that specifies what should be clustered (categories, words, or tokens) and how the clusters should be created (using the k-Means or agglomerative algorithm). See Clustering_Flags for specific values.

Return Value

An array containing the specified number of clusters.

Discussion

If subset is non-NULL, only perform clustering on the categories, words, or tokens listed.

LSMMapCreateFromURL

Loads a map from the specified file.

LSMMapRef LSMMapCreateFromURL(
    CFAllocatorRef alloc,
    CFURLRef file, 
    CFOptionFlags flags);

Parameters

alloc

A user-specified allocator.

file

The file from which to load the map.

flags

A flag that determines loading options. You can pass one of the following flags:

kLSMMapDiscardCounts (defined in Storage_Flags). Note that if you pass this flag, the map will need to be reloaded with the kLSMMapLoadMutable option instead before calling LSMMapStartTraining.
kLSMMapLoadMutable (defined in Storage_Flags)

Return Value

The LSM map loaded from the specified file. See Error_Codes for error codes that might be returned.

LSMMapGetCategoryCount

Returns the number of categories in the specified map.

CFIndex LSMMapGetCategoryCount(
    LSMMapRef mapref);

Parameters

mapref: An LSM map.

Return Value

The number of categories in the map. See Error_Codes for error codes that might be returned.

LSMMapGetProperties

Gets the dictionary of properties for the map.

CFDictionaryRef LSMMapGetProperties(
    LSMMapRef mapref);

Parameters

mapref: An LSM map.

Return Value

A CFDictionary of properties associated with the map. See Error_Codes for error codes that might be returned.

Discussion

Because LSM retains ownership of the dictionary this function returns, do not release it. See LSM_Map_Properties for information on these properties.

LSMMapGetTypeID

Returns the Core Foundation type identifier for LSM maps.

CFTypeID LSMMapGetTypeID(
    void);

LSMMapSetProperties

Sets a dictionary of properties for the map.

void LSMMapSetProperties(
    LSMMapRef mapref,
    CFDictionaryRef properties);

Parameters

mapref: An LSM map.
properties: A CFDictionary of map properties (see LSM_Map_Properties for these properties).

Discussion

Because LSM makes its own copy of these properties, there's no need to retain them past this call.

LSMMapSetStopWords

Specifies words to be omitted from all classification efforts.

OSStatus LSMMapSetStopWords(
    LSMMapRef mapref,
    LSMTextRef textref);

Parameters

mapref: An LSM map.
textref: The input text.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

If you use this function, you must call it before any calls to LSMMapAddText. The textref parameter is not needed after this call, and can be released.

LSMMapStartTraining

Puts the map into training mode, preparing it for the addition of more categories or texts or both.

OSStatus LSMMapStartTraining(
    LSMMapRef mapref);

Parameters

mapref: An LSM map.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

This function is somewhat expensive, because it requires substantial data structure reorganization.

LSMMapWriteToStream

Writes information about a map or a text or both to a byte stream in text form.

OSStatus LSMMapWriteToStream(
    LSMMapRef mapref,
    LSMTextRef textref, 
    CFWriteStreamRef stream,
    CFOptionFlags options);

Parameters

mapref: An LSM map.
textref: A text (can be NULL).
stream: A writable stream object (this object must be opened with CFWriteStreamOpen).
options: Currently not used.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

LSMMapWriteToURL

Compiles the map if necessary and stores it into the specified file.

OSStatus LSMMapWriteToURL(
    LSMMapRef mapref,
    CFURLRef file,
    CFOptionFlags flags);

Parameters

mapref

An LSM map.

file

The file into which to store the map.

flags

A flag that determines storage options. You can pass one of the following flags:

kLSMMapDiscardCounts (defined in Storage_Flags). Note that if you pass this flag you save disk space, but the map as stored can't be retrained (you can't call LSMMapStartTraining on it).
kLSMMapHashText (defined in Map_Flags). Note that if you pass this flag, the map will be hashed if it hasn't been hashed yet.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

LSMResultCopyToken

Returns the token for the n-th best (zero-based) result.

CFDataRef LSMResultCopyToken(
    LSMResultRef mapref,
    CFIndex n);

Parameters

mapref: An LSM map.
n: The rank of a result.

Return Value

The token for the n-th best result. See Error_Codes for error codes that might be returned.

LSMResultCopyTokenCluster

Returns the cluster of tokens for the n-th best (zero-based) result.

CFArrayRef LSMResultCopyTokenCluster(
    LSMResultRef mapref,
    CFIndex n);

Parameters

mapref: An LSM map.
n: The rank of a result.

Return Value

An array containing the cluster of tokens for the n-th best result. See Error_Codes for error codes that might be returned.

LSMResultCopyWord

Returns the word for the n-th best (zero-based) result.

CFStringRef LSMResultCopyWord(
    LSMResultRef result,
    CFIndex n);

Parameters

result: A result.
n: The ranking of the result.

Return Value

The word for the n-th best result. See Error_Codes for error codes that might be returned.

LSMResultCopyWordCluster

Returns the cluster of words for the n-th best (zero-based) result.

CFArrayRef LSMResultCopyWordCluster(
    LSMResultRef result,
    CFIndex n);

Parameters

result: A result.
n: The rank of a result.

Return Value

An array containing the cluster of words for the n-th best result. See Error_Codes for error codes that might be returned.

LSMResultCreate

Returns, in decreasing order of likelihood, the categories or words that best match when a text is mapped into a map.

LSMResultRef LSMResultCreate(
    CFAllocatorRef alloc, 
    LSMMapRef mapref,
    LSMTextRef textref, 
    CFIndex numResults,
    CFOptionFlags flags);

Parameters

alloc: An application-defined allocator.
mapref: An LSM map.
textref: The text to be categorized.
numResults: The highest number of categories or words that best match the text.
flags: A value that determines whether the function result represents categories or words (pass kLSMResultBestWords, defined in Result_Flags to find words).

Return Value

An LSMResultRef value. See Error_Codes for error codes that might be returned.

Discussion

This function categorizes the input text and returns an LSMResultRef value that represents up to numResults categories (or, optionally, numResults words) that best match, in decreasing order.

LSMResultGetCategory

Returns the category of the n-th best (zero-based) result.

LSMCategory LSMResultGetCategory(
    LSMResultRef result,
    CFIndex n);

Parameters

result: A result.
n: The ranking of the result.

Return Value

The LSM category of the n-th best (zero-based) result. See Error_Codes for error codes that might be returned.

LSMResultGetCount

Returns the number of results associated with the specified result.

CFIndex LSMResultGetCount(
    LSMResultRef result);

Parameters

result: A result.

Return Value

The number of LSM results actually created. See Error_Codes for error codes that might be returned.

LSMResultGetScore

Returns the likelihood of the n-th best (zero-based) result.

float LSMResultGetScore(
    LSMResultRef result,
    CFIndex n);

Parameters

result: A result.
n: The ranking of the result.

Return Value

The likelihood of the n-th best result, as a floating point value. See Error_Codes for error codes that might be returned.

LSMResultGetTypeID

Returns the Core Foundation type identifier for LSM results.

CFTypeID LSMResultGetTypeID(
    void);

LSMTextAddToken

Adds an arbitrary binary token to the text.

OSStatus LSMTextAddToken(
    LSMTextRef textref,
    CFDataRef token);

Parameters

textref: The text in which to add the token.
token: An application-defined token.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

The order of tokens is significant if the map uses pairs or triplets. The count of tokens is always significant.

LSMTextAddWord

Adds a word to the text.

OSStatus LSMTextAddWord(
    LSMTextRef textref,
    CFStringRef word);

Parameters

textref: The text to which to add the word.
word: The word to add to the text.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

Discussion

The order of words is significant if the map uses pairs or triplets. The count of words is always significant.

LSMTextAddWords

Breaks a string into words using the specified locale, and adds the words to the text.

OSStatus LSMTextAddWords(
    LSMTextRef textref,
    CFStringRef words, 
    CFLocaleRef locale,
    CFOptionFlags flags);

Parameters

textref: The text to which to add the words.
words: The string of words to add to the text.
locale: A CFLocaleRef object that specifies language-specific or region-specific information about the words string. Pass NULL to get the default locale.
flags: Options that govern how the words in words should be mapped. See Parsing_Flags for available options.

Return Value

NoErr if successful. See Error_Codes for error codes that might be returned.

LSMTextCreate

Creates a new text.

LSMTextRef LSMTextCreate(
    CFAllocatorRef alloc,
    LSMMapRef mapref);

Parameters

alloc: A user-defined allocator.
mapref: An LSM map.

Return Value

The text created from the specified map. See Error_Codes for error codes that might be returned.

LSMTextGetTypeID

Returns the Core Foundation type identifier for LSM texts.

CFTypeID LSMTextGetTypeID(
    void);

Typedefs

LSMCategory

typedef uint32_t LSMCategory;

Discussion

An integral type representing a category.

LSMMapRef

typedef struct __LSMMap * LSMMapRef;

Discussion

An opaque Core Foundation type representing an LSM map (mutable).

LSMResultRef

typedef struct __LSMResult * LSMResultRef;

Discussion

An opaque Core Foundation type representing the result of a lookup (immutable).

LSMTextRef

typedef struct __LSMText * LSMTextRef;

Discussion

An opaque Core Foundation type representing an input text (mutable).

Enumerations

Clustering_Flags

Options for LSMMapCreateClusters.

enum { 
    kLSMClusterCategories = 0, 
    kLSMClusterWords = 1, 
    kLSMClusterTokens = 2,  
    kLSMClusterKMeans = 0, 
    kLSMClusterAgglomerative = 4 
};

Constants

Discussion

The first 3 flags specify the type of cluster and the last 2 specify the algorithm to be used. In LSMMapCreateClusters, you should OR a cluster-type flag with an algorithm flag for the flags parameter.

Error_Codes

Error codes that may be returned from LSM routines.

enum { 
    kLSMMapOutOfState = -6640, 
    kLSMMapNoSuchCategory = -6641, 
    kLSMMapWriteError = -6642, 
    kLSMMapBadPath = -6643, 
    kLSMMapBadCluster = -6644 
};

Constants

Map_Flags

Options that can be specified for LSMMapCreate.

enum { 
    kLSMMapPairs = 1, 
    kLSMMapTriplets = 2,  
    kLSMMapHashText = 256 
};

Constants

Discussion

These options can improve mapping accuracy, at a potentially significant increase in memory usage.

Parsing_Flags

Options you can specify for LSMTextAddWords.

enum { 
    kLSMTextPreserveCase = 1, 
    kLSMTextPreserveAcronyms = 2, 
    kLSMTextApplySpamHeuristics = 4 
};

Constants

Result_Flags

enum { 
    kLSMResultBestWords = 1, 
};

Constants

Discussion

Options for LSMResultCreate.

Storage_Flags

enum { 
    kLSMMapDiscardCounts = 1, 
    kLSMMapLoadMutable = 2 
};

Constants

Discussion

Storage and loading options for LSMMapCreateFromURL and LSMMapWriteToURL.

#defines

kLSMAlgorithmDense

Predefined keys and properties for LSM maps.

See Also:

#define kLSMAlgorithmDense CFSTR("LSMAlgorithmDense")

Discussion

Perform an SVD on a dense map (in a dense map, most words occur in most categories).

kLSMAlgorithmKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMAlgorithmKey CFSTR("LSMAlgorithm")

Discussion

The algorithm to be used.

kLSMAlgorithmSparse

Predefined keys and properties for LSM maps.

See Also:

#define kLSMAlgorithmSparse CFSTR("LSMAlgorithmSparse")

Discussion

Perform an SVD on a sparse map (in a sparse map, most words occur in only a small subset of categories).

kLSMDimensionKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMDimensionKey CFSTR("LSMDimension")

Discussion

The maximum number of dimensions to use, as a CFNumber value (this defaults to the number of categories). Often, a lower dimension is appropriate, especially for the sparse algorithm.

kLSMIterationsKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMIterationsKey CFSTR("LSMIterations")

Discussion

The number of iterations to use in the sparse algorithm, as a CFNumber value (this defaults to a number based on the number of dimensions in the map and rarely needs to be changed).

kLSMPrecisionDouble

Predefined keys and properties for LSM maps.

See Also:

#define kLSMPrecisionDouble CFSTR("LSMPrecisionDouble")

Discussion

Use double precision floating point when performing computations (default for sparse map).

kLSMPrecisionFloat

Predefined keys and properties for LSM maps.

See Also:

#define kLSMPrecisionFloat CFSTR("LSMPrecisionFloat")

Discussion

Use single precision floating point when performing computations (default for a dense map).

kLSMPrecisionKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMPrecisionKey CFSTR("LSMPrecision")

Discussion

The precision to be used when performing computations.

kLSMSweepAgeKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMSweepAgeKey CFSTR("LSMSweepAge")

Discussion

The number of days between sweeping generations (this defaults to 7).

kLSMSweepCutoffKey

Predefined keys and properties for LSM maps.

See Also:

#define kLSMSweepCutoffKey CFSTR("LSMSweepCutoff")

Discussion

When used, a CFNumber value that causes the map to be scanned every kLSMSweepAgeKey days, removing each entry older than 3 times the value of kLSMSweepAgeKey, whose total weight in the map is less than kLSMSweepCutoffKey.

LSM_Map_Properties

Predefined keys and properties for LSM maps.

See Also:

#define kLSMAlgorithmKey CFSTR( "LSMAlgorithm") //! The algorithm to be used.
#define kLSMAlgorithmDense CFSTR( "LSMAlgorithmDense") //! Perform an SVD on a dense map ( in a dense map, most words occur in most categories).
#define kLSMAlgorithmSparse CFSTR( "LSMAlgorithmSparse") //! Perform an SVD on a sparse map ( in a sparse map, most words occur in only a small subset of categories).
#define kLSMPrecisionKey CFSTR("LSMPrecision") //! The precision to be used when performing
    computations.
#define kLSMPrecisionFloat CFSTR("LSMPrecisionFloat") //! Use single precision floating point
    when performing computations ( default for a dense map).
#define kLSMPrecisionDouble CFSTR("LSMPrecisionDouble") //! Use double precision floating 
    point when performing computations ( default for sparse map).
#define kLSMDimensionKey CFSTR( "LSMDimension") //! The maximum number of dimensions to use, as a CFNumber value ( this defaults to the number of categories). Often, a lower dimension is appropriate, especially for the sparse algorithm.
#define kLSMIterationsKey CFSTR("LSMIterations") //! The number of iterations to use in the 
    sparse algorithm, as a CFNumber value (this defaults to a number based on the number of dimensions 
    in the map and rarely needs to be changed).
#define kLSMSweepAgeKey CFSTR("LSMSweepAge") //! The number of days between sweeping 
    generations ( this defaults to 7).
#define kLSMSweepCutoffKey CFSTR( "LSMSweepCutoff") //! When used,a CFNumber value that causes the map to be scanned every 
    kLSMSweepAgeKey days,removing each entry older than 3 times the value of 
    kLSMSweepAgeKey,whose total weight in the map is less than 
    kLSMSweepCutoffKey.

Discussion

A CFDictionary of properties may be associated with an LSM map. These properties specify map settings, such as which algorithm should be used, with what precision should the computations be performed, and how many dimensions to use.

In addition, this API defines two keys that allow you to prune from a map those entries that are seen only infrequently. As you add new data to a map, the map sometimes will have a large number of entries that were encountered only a small number of times. To remedy this, you can specify a weight value for the kLSMSweepCutoffKey. When this key-value pair is present in a map's dictionary, the map is scanned at the interval specified by kLSMSweepAgeKey (by default, every 7 days). In each scan, entries are removed if they are older than three times this interval (by default, 21 days) and their weight in the map is less than the value specified by kLSMSweepCutoffKey.

The following keys and properties currently are interpreted by LSM (all other keys starting with LSM are reserved).

Did this document help you?

Yes: Tell us what works for you.

It’s good, but: Report typos, inaccuracies, and so forth.

It wasn’t helpful: Tell us what would have helped.

Last Updated: 2008-03-11