FoundationModels coding

Machine Learning & AI Apple Intelligence
joelesler OP
Created 2h
Replies 1
Boosts 0
Views 17
Participants 2

I am writing an app that parses text and conducts some actions. I don't want to give too much away ;)

However, I am having a huge problem with token sizes. LanguageModelSession will of course give me the on device model 4096 available, but when you go over 4096, my code doesn't seem to be falling back to PCC, or even the system configured ChatGPT. Can anyone assist me with this? For some reason, after reading the docs, it's very unclear how this transition between the three takes place.

Replies  1
Boosts  0
Views  17
Participants  2
DTS Engineer OP
Apple
20m

Hi @joelesler, currently the context window limit for Foundation Models is around 4k tokens, as you mentioned. When your app exceeds that limit, it will throw an .exceededContextWindowSize error, which your app can then handle appropriately.

Currently the framework does not support a way to "fall back" to cloud-based providers. Please take a look at this Technote which may prove useful:

Managing the on-device foundation model’s context window

Best,

-J

0
FoundationModels coding
First post date Last post date
 
 
Q