File Handle Exhaustion Issue with com.apple.Virtualization.VirtualMachine in VZ Environment

We are currently utilizing VZ with Lima (details: Lima VM and VZ) for our development environment. However, we're encountering a critical issue with the com.apple.Virtualization.VirtualMachine process leading to open file handle exhaustion.

When mounting our programming languages dependency cache folder (Which can have a lot of files) into the VZ VM, we encounter an operating system error related to open file limits:

/gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/object/patch.go:14:2: open /gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/format/diff/unified_encoder.go: too many open files in system

Further investigation revealed an abnormally high number of open files associated with the com.apple.Virtualization.VirtualMachine process. A significant portion of these files are not actively used but remain open.

Example Case:

A file (/Users/rcurrah/test.txt) created on the Mac host and listed (ls) in the VM remains open even 20 minutes later, as evidenced by the following command output:

❯ lsof | grep 11208 | grep test.txt

COMMAND     PID    USER   FD      TYPE             DEVICE     SIZE/OFF                NODE NAME
com.apple 11208 rcurrah 4823r      REG               1,13            0            46200882 /Users/rcurrah/test.txt

Steps to Reproduce the Issue:

To reproduce the file handle exhaustion follow the below steps. This process will create a large number of files on the Mac host, listing them on the VZ VM, and then verifying their open status using lsof.

Setup the VZ Environment with Sharing:

Create a VZ VM with your home directory shared to the VM.

Create a Test Directory on the Mac Host:

Create a new directory on your Mac host, e.g., mkdir ~/test-file-exhaustion.

Generate a Large Number of Files:

Navigate to the created directory: cd ~/test-file-exhaustion. Use a loop to create a large number of files, e.g., for i in {1..10000}; do touch "file_${i}.txt"; done. This will create 10,000 files named file_1.txt, file_2.txt, etc.

List Files in the VM:

Access the VZ VM shell. Navigate to the mounted directory and list the files using the ls command, e.g., ls /path/to/mounted/test-file-exhaustion.

Check Open Files on Mac Host:

Exit the VM and return to your Mac host terminal.

Use the lsof command to check for open files related to the com.apple.Virtualization.VirtualMachine process: lsof | grep "$(pgrep com.apple.Virtualization.VirtualMachine)" | grep 'test-file-exhaustion' | wc -l.

Document the Output:

Record the output of the lsof command. Note the number of open files.

Verify File Closure (or Lack Thereof):

After a certain period, e.g., 20 minutes, repeat the lsof command to see if the files are still open, indicating that they haven’t been closed properly by the process.

Given these observations, we have a couple of questions:

Is this behavior of com.apple.Virtualization.VirtualMachine retaining open file handles a known issue or a bug?

Should VZ be managing the closure of these file handles more efficiently, especially when they are no longer in use?

This issue is impacting our development workflow significantly. Any guidance or insights on resolving this would be highly appreciated.

Thank you for your attention to this matter.

Best regards,

Ryan

System Info:

Could you please try on macOS 14.1 or later? Virtualization attempts to recycle non-opened files for which there is an entry in VirtioFS. Some of those cases were improved in macOS 14.

@BenjaminApple thank you for your prompt response. I have upgraded to MacOS Sonoma as suggested this morning and conducted further testing to investigate the issue.

File Creation: I generated a substantial number of files (70,000) on the Mac host using the command for i in {1..70000}; do touch "file_${i}.txt"; done.

Listing Files in VZ VM: I navigated to the directory (cd ~/test-file-exhaustion) and listed all the files within the VZ VM.

Initial File Handle Count: Upon checking with lsof, the initial count of file handles associated with com.apple.Virtualization.VirtualMachine was high:

❯ lsof | grep "$(pgrep com.apple.Virtualization.VirtualMachine)" | wc -l
58625

Subsequent Check after 30 Minutes: After a wait of 30 minutes, I rechecked the file handles. Only 482 file handles were closed by com.apple.Virtualization.VirtualMachine:

❯ lsof | grep "$(pgrep com.apple.Virtualization.VirtualMachine)" | wc -l
58143

This test indicates that the issue of excessive open file handles persists even on the latest MacOS version and can be consistently reproduced.

I look forward to any guidance or suggestions you might have regarding this matter.

Related issue on Docker For Mac https://github.com/docker/for-mac/issues/6783

The large number of open file descriptor isn't a symptom by itself. Unless the guest sends FUSE_FORGET, there is no harm in keeping them around for performance. When the number gets close to the process limits, they are being recycled.

Do you still see the guest running out of file descriptor on macOS Sonoma? If that happens, that could indicate a problem with the recycling of file descriptors.

Thank you for your insights. I conducted a test on macOS Sonoma and can confirm that the issue persists. Despite the maximum file descriptors being capped at 65,535, my experiment involved opening 70,000 file descriptors. While the count of open file descriptors never exceeded the maximum limit, the system still produced too many open files in system errors.

This issue became evident when other processes on my macOS, such as iTerm, were affected, displaying the same error message. This leads me to believe that your suggestion about potential issues with the recycling of file descriptors might be accurate. It seems that the system isn’t effectively managing or recycling these descriptors, hence leading to the observed problem.

@BenjaminApple looking forward to any guidance you have on this matter. Thank you.

Sorry for the delay. That definitely sounds like a bug. Can you please file a report with Feedback Assistant, noting the version of macOS? If possible, can you describe how you reproduce the issue? The bug could be specific to certain paths in FUSE.

Thanks Benjamin I have submitted feedback, FB13640480.

Hi, I'm just chiming in to see if there's been any updates on this. I can't seem to access the feedback linked above. And it looks like the Apple engineer that was on this thread is no longer active? 😬 I'm really hoping this gets some attention soon. Many thanks!

File Handle Exhaustion Issue with com.apple.Virtualization.VirtualMachine in VZ Environment
 
 
Q