Title: Hypervisor.framework — vCPU cessation guarantee after VM-owning process termination

Title: Hypervisor.framework — vCPU cessation guarantee after VM-owning process termination

Hello,

I'm investigating the failure-containment guarantees of Hypervisor.framework on Apple Silicon, specifically for an application that uses QEMU's HVF accelerator.

In the QEMU 11.1.2 ARM64 implementation:

  • hv_vm_create() creates the VM within the QEMU process.
  • Each vCPU is created and executed on a QEMU pthread using hv_vcpu_create() and hv_vcpu_run().
  • No separate userspace VM-execution helper was identified in the reviewed execution path.

The security requirement is to establish guest-execution cessation independently when the VM-owning application fails.

My primary question:

If a supervising parent successfully observes that the QEMU process has actually terminated through waitpid() — whether normally or following SIGKILL — is that a supported guarantee that every Hypervisor.framework vCPU belonging to that process has ceased guest execution and cannot resume?

I am specifically asking about irreversible cessation of guest execution, not necessarily completion of all VM resource reclamation.

A few related questions:

  1. Does an EVFILT_PROC/NOTE_EXIT notification establish the same guarantee, and is there a documented ordering relationship between that notification and cessation of all associated vCPUs?

  2. If a thread is executing hv_vcpu_run() when its process receives SIGKILL or SIGSTOP, what cessation or suspension guarantees apply?

  3. Is there a documented upper bound for cessation following process termination, or could kernel/framework blockage delay it without a specified bound?

  4. Can any other userspace process retain or acquire resources allowing those vCPUs to continue executing after the original owner has terminated?

  5. If the original supervising parent also crashes, is there a supported mechanism for a successor controller to obtain equivalent positive evidence of termination and vCPU cessation for the original execution instance?

The distinction between a process merely receiving a termination request and the operating system confirming actual process termination is important here.

I'm trying to establish which guarantees are supported by the framework rather than relying on experimental observations or assumptions about kernel cleanup.

Any clarification would be greatly appreciated.

Thanks, Gary

Answered by DTS Engineer in 910398022

Virtualization and Hypervisor are very different when it comes to process management. In Virtualization, the OS is responsible for managing the process that runs the VM. This is something we’ve talked about in your previous thread.

In Hypervisor that’s not the case. Rather, you are responsible for running the VM. Typically that means starting your own thread to run each vCPU by calling hv_vcpu_run(_:).

Given that architecture, your question boils down to “What does QEMU do?”, and that’s not something I can answer. You can either look at the QEMU source or escalate your question via QEMU’s support channel.

Oh wait, one of your questions still make sense given the above:

4- Can any other userspace process retain or acquire resources allowing those vCPUs to continue executing after the original owner has terminated?

And the answer to that is “No.”

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Virtualization and Hypervisor are very different when it comes to process management. In Virtualization, the OS is responsible for managing the process that runs the VM. This is something we’ve talked about in your previous thread.

In Hypervisor that’s not the case. Rather, you are responsible for running the VM. Typically that means starting your own thread to run each vCPU by calling hv_vcpu_run(_:).

Given that architecture, your question boils down to “What does QEMU do?”, and that’s not something I can answer. You can either look at the QEMU source or escalate your question via QEMU’s support channel.

Oh wait, one of your questions still make sense given the above:

4- Can any other userspace process retain or acquire resources allowing those vCPUs to continue executing after the original owner has terminated?

And the answer to that is “No.”

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Quinn,

Thank you for clarifying the difference between Virtualization.framework and Hypervisor.framework, and for confirming that another userspace process cannot retain or acquire the original vCPU execution resources after their owner terminates.

I understand that QEMU-specific thread and process management is our responsibility, and we're examining that separately.

I have one remaining question about Hypervisor.framework itself, independent of QEMU's implementation:

When the process owning a Hypervisor.framework VM has actually terminated, and its supervising parent obtains terminal status through waitpid() (WIFEXITED or WIFSIGNALED), is it a supported guarantee that every vCPU executed by that process has already ceased executing guest instructions and cannot resume?

This includes abnormal process termination while a thread was executing hv_vcpu_run().

I'm deliberately distinguishing irreversible cessation of guest execution from complete VM resource reclamation. I am not asking whether sending SIGKILL guarantees immediate termination, only what is established once actual process termination has been confirmed.

If Hypervisor.framework provides no documented guarantee for that ordering, confirmation of that limitation would also be helpful.

Thanks again for your guidance.

Gary

I think you’ve misunderstood how Hypervisor framework works. When a thread calls hv_vcpu_run, it’s that thread that’s running the guest. So your question is equivalent to:

When I’m notified of a process exiting, are any of its threads still running?

and you already know the answer to that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"