More vCPUs, lower build performance (macOS VMs)

How much RAM is available on the host? And how much are you allocating to the VM?

This matters because adding more CPUs is likely to cause your build system to go ‘wider’. If each parallel job needs a certain amount of memory, then more CPUs will hurt rather than help.

I was discussing this with some folks internally and they recounted a story where they found that building a large open source project needed 2 GiB of per CPU. If you added more CPUs, the performance tanked because the build system went wider than the available RAM could support.

But that’s just one example. Optimising build performance is definitely a dark art.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks for the suggestion.

The host has 96 GB RAM. During testing I assigned approximately 80% of the host memory to the VM, and later also tested with 90% allocation.

Based on the metrics I collected so far, memory does not appear to be the limiting factor. While memory usage increases during the build, the host still has roughly 20 GB of free RAM available and I only observed moderate swap usage (around 5 GB).

My understanding of your comment is that Xcode may schedule more compilation tasks in parallel as additional vCPUs become available, increasing the overall memory requirements of the build. That certainly sounds plausible in general.

I also tracked RAM usage during these tests and compared VM builds against builds running directly on the host. From what I've seen so far, the memory footprint looks reasonable in both cases, and I haven't found an obvious indication that memory pressure is causing the slowdown.

That said, I haven't ruled it out completely yet. I'll take another look at the memory metrics during the runs with higher vCPU counts to see if there's something I may have missed. Most of the data I've reviewed so far comes from the host side, so it would be worth digging deeper into the memory usage inside the VM itself during the build to verify that I'm not overlooking something there.

The host has 96 GB of RAM. During testing, I assigned approximately 80% of the host memory to the VM, and later also tested with a 90% allocation.

Based on the metrics I collected so far, memory does not appear to be the limiting factor. While memory usage increases during the build, the host still has roughly 20 GB of free RAM available, and I only observed moderate swap usage (around 5 GB).

I haven't specifically looked at this, but I STRONGLY suspect that there are mitigations [1] in place that limit/reduce the host’s willingness to compress (and swap) memory that’s been assigned to the VM. The problem here is that the VM is running its own virtual memory system, so if we host start applying its own compression, it's going to end up trying to compress the pages the VM system ALREADY compressed, which does nothing but waste space. A similar issue applies with swap, except now the VM can't really "see" what's already been swapped, leading to all sorts of weird "churn".

[1] From a brief look in XNU, it’s possible that the VM system is in fact wiring pages down (locking their position and state in physical memory), which is the strongest possible "mitigation".

In terms of how this affects the behavior here:

While memory usage increases during the build, the host still has roughly 20 GB of free RAM available.

I'd need to work out the math, but just like the host, the VM is going to keep a "reserve" of free memory, which is what it uses for things like servicing temporary “spikes" or presenting the UI that says "Please kill some App now before I'm forced to murder processes and/or panic". Unless you REALLY put the system into a bind, you won't really "see" it ever use that cushion.

and I only observed moderate swap usage (around 5 GB).

I don't think VM activity should really generate swap activity at the host level, as doing so opens the door to ugly I/O anti-patterns like:

1. Host swaps memory to disk.

2. VM agrees with that decision and swaps the same memory to disk.

3. VM deletes memory from host "view" (since that’s what happens when the VM swaps out...).

4. Host reads pages off of disk, deletes pages from swap, and then memory.

5. VM reads its swap off of disk, loading pages back into host memory.

...and the whole cycle starts over again. Basically, it's really easy for a naive implementation to generate a lot of pointless, duplicate I/O.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi, thanks for the comment. Fortunately, I was able to create a broader test setup to verify whether RAM was affecting build performance.

I tested two hosts:

• Host A: 32 CPU cores, 96 GB RAM
• Host B: 24 CPU cores, 128 GB RAM

For both hosts, I ran multiple VM configurations with different CPU and RAM allocations (80% and 90% of host memory).

Result: The build times, and more important the CPU usage were almost identical across all test cases. It did not seem to matter whether the build ran on a machine with more RAM but fewer CPU cores. The overall behavior was very similar on both hosts. The dropping CPU usage during the build-phase was reproducible.

One interesting observation is that in the most aggressive configuration (all CPUs assigned and 90% RAM allocated to the VM), the 96 GB host showed some swap activity, while the 128 GB host did not. However, this had no noticeable impact on build performance (in both cases the CPU usage dropped).

Also, increasing the VM's RAM allocation did not increase actual memory usage during the build. The workload simply did not consume the additional memory, which suggests that RAM is probably not the limiting factor in this case.

Based on these results, I think we can largely rule out RAM as the cause of the slowdown. Thanks again for the suggestion.

Attached two screenshots showing the same build in the worst- and best setup.

Host A - VM specs: 32 vCPU / 86 GB RAM:

Host A - VM specs: 24 vCPU / 86 GB RAM: