AI framework usage without user session

Question

Created 3w

Replies 6

Boosts 0

Participants 3

We are evaluating various AI frameworks to use within our code, and are hoping to use some of the build-in frameworks in macOS including CoreML and Vision. However, we need to use these frameworks in a background process (system extension) that has no user session attached to it. (To be pedantic, we'll be using an XPC service that is spawned by the system extension, but neither would have an associated user session).

Saying the daemon-safe frameworks list has not been updated in a while is an understatement, but it's all we have to go on. CoreGraphics isn't even listed--back then it part of ApplicationServices (I think?) and ApplicationServices is a no go. Vision does use CoreGraphics symbols and data types so I have doubts.

We do have a POC that uses both frameworks and they seem to function fine but obviously having something official is better. Any Apple engineers that can comment on this?

Boost

Answer 1

DTS Engineer OP

Apple

3w

You are definitely breaking new ground here, so I’m gonna to do some research and get back to you.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

2

Answer 2

DTS Engineer OP

Apple

1w

I’m still working on this. It’s proving to be challenging, not helped by the fact that WWDC is looming large on everyone’s calendar.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 3

eddiewangyw OP

1w

We ran CoreML inference from a launch daemon (no user session) for about a year — it works but with caveats. ANE access is unreliable without a session, so you'll likely fall back to CPU/GPU compute units. Vision framework calls that touch CoreGraphics can deadlock if there's no window server connection. Our workaround was forcing .cpuOnly for the daemon path and keeping the GPU/ANE path for the user-facing XPC.

0

Answer 4

DTS Engineer OP

Apple

3d

eddiewangyw, Thanks for sharing your experience with this.

ANE access is unreliable without a session

Yeah, that was one of my concerns. I’m also concerned about the GPU itself, because it’s not clear to me whether that’ll be affected by the window server lifecycle.

Earlier I wrote:

I’m still working on this.

I’m starting to make some progress here, but I’ve still nothing concrete to share.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 5

boilerla220 OP

14h

Thanks Quinn and eddiewangyw for the updates. One thing I guess worth clarifying also is what is meant by "user session". Our service(s) have no connection to CGS since they are root session only but other Apple daemons/services may—and I assume that's the real crux of the issue may make a difference. But if how a user is logged in—via physical console or SSH or VNC etc makes a difference could also be interesting.

In our testing, the ANE and GPU were reliable when a user was logged in at the physical console. It's much harder to trigger these specific flows without some GUI console access (e.g. SSH would be hard) but over VNC it's definitely possible. Then I began to question if there's a difference with somebody logged in but at loginwindow, etc.

0

Answer 6

DTS Engineer OP

Apple

1h

I guess worth clarifying also is what is meant by "user session".

Right. I generally try to follow the terminology from TN2083 Daemons and Agents. A system extension [1] is effectively a launchd daemon and thus runs in the global context. It can’t reliably access the window server. And any XPC service it calls upon will be similarly restricted. However, the GPU lifecycle isn’t tied directly to the window server lifecycle, so there is some wiggle room there, and that’s the point I’m still working to clarify internally.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Non-DEXT system extensions. DEXTs are weird (-:

0