Foundation Models inside of DeviceActivityReport?

Pretty much as per the title and I suspect I know the answer. Given that Foundation Models run on device, is it possible to use Foundation Models framework inside of a DeviceActivityReport? I've been tinkering with it, and all I get is errors and "Sandbox restrictions". Am I missing something? Seems like a missed trick to utilise on device AI/ML with other frameworks.

Answered by DTS Engineer in 861835022

Yeah, I see the following error when trying to prompt the on-device foundation model from within a device activity report extension (DeviceActivityReportExtension), which I believe is the same as what you mentioned:

establishment of session failed with Unrecognized underlying error: Underlying connection was invalidated. Reason: failed at lookup with error 159 - Sandbox restriction

ModelManagerError: got unrecognized error Underlying connection was invalidated. Reason: failed at lookup with error 159 - Sandbox restriction

I am pretty sure that is as-designed.

To give you more context, when responding your prompt, the model needs to communicate with the model manager, which is a system daemon that manages the inference requests. To protect the user’s privacy, however, the system doesn't allow a device activity report extension to move data outside the extension’s address space, and that prevents the model from creating a connection to the model manager.

Having said that, if you have a compelling case that needs to use the foundation model from within your device activity report extension, please feel free to file a feedback report.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Yeah, I see the following error when trying to prompt the on-device foundation model from within a device activity report extension (DeviceActivityReportExtension), which I believe is the same as what you mentioned:

establishment of session failed with Unrecognized underlying error: Underlying connection was invalidated. Reason: failed at lookup with error 159 - Sandbox restriction

ModelManagerError: got unrecognized error Underlying connection was invalidated. Reason: failed at lookup with error 159 - Sandbox restriction

I am pretty sure that is as-designed.

To give you more context, when responding your prompt, the model needs to communicate with the model manager, which is a system daemon that manages the inference requests. To protect the user’s privacy, however, the system doesn't allow a device activity report extension to move data outside the extension’s address space, and that prevents the model from creating a connection to the model manager.

Having said that, if you have a compelling case that needs to use the foundation model from within your device activity report extension, please feel free to file a feedback report.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Foundation Models inside of DeviceActivityReport?
 
 
Q