While heap allocations are very important (that's where all your Swift and Objective-C objects are created), you can't ignore the importance of VM memory for most apps. I recommend investigating that category because it can grow rather large, and the memory being allocated there "by the operating system" is going to be in response to some task you've asked your application to perform.
For instance, when you allocate a large CGImage, a tiny object is allocated on the heap, but the many megabytes of image data is allocated in a separate VM region. In an image-heavy application, your heap may be relatively small but your VM regions may be rather large. In that case, you'd need to take a closer look at the size and number of images your application has loaded at any given moment.
So in general, you need to ensure you are optimizing *all* of your memory usage.