Custom Instruments Probes & Exporting

Hey folks,

First off- long time fan of the performance tooling on darwin, thank you so much for all your guys' hard work.

Recently I've been exploring custom instruments- these are really cool! In the past I've heavily leaned on dtrace to do a lot of my performance analysis, but I was hoping to replace that usage with custom instruments moving forward[1]. Right now there are 2 features that dtrace gives me that custom instruments doesn't (at least afaict):

- The ability to horizontally inject probes into software

- The ability to export data, and more specifically, write scripts around the exported data to validate behavior in an automated fashion

---

Probes

---

In dtrace a user is able to inject their own instrumentation without recompiling the app. When doing performance investigations its incredibly valuable to dynamically inject probes at runtime- analyses I do are often organic, thus it's unknown upfront where I'll need instrumentation and where I won't. In order to address this issue (without using dtrace) I've introduced probes everywhere I can think might be useful ahead of time so that I can avoid recompiling my software. This technique has the following detriments: (a) we collect a ton of data, most of which we don't need (b) if we didn't think hard enough up front then it's still possible we'll need to add more instrumentation later.

In addition to userland probes, kernel probes have provided a ton of value. Depending on the performance problem I experience there are different subsystems within the kernel that I want to dig into. For example if I sense that there's an memory I/O problem, it's often useful to turn on various probes within the vminfo provider in order to better understand how our app is working with the VM system. I do see that there are a bunch of kdebug probes statically compiled in, however in my experience (this might be outdated as I haven't compared in a little while) there are more dtrace probes than kdebug probes- and even further the dtrace probes provide more information for the user to work with.

Concisely put my feature request is as follows: can we introduce custom probes, like with dtrace, in custom instruments? Notably, just saying "this function started" or "this function ended" wouldn't be enough- exposing the arguments as well (like dtrace) would be really useful.

Export Data

---

Before I address exporting data directly let me discuss memory briefly: memory I/O patterns, in my experience, are pretty impactful on an apps performance- so impactful that my most common workflow with dtrace/systrace is to analyze I/O patterns, extract them and leverage them to later make my software perform better (e.g. PGO, madvise etc.). Currently if I want to extract these patterns I either:

(a) Leverage dtrace

(b) Use Instruments and manually copy/paste the data out of the Virtual Memory Trace detail view

Since the former seems to be going away, I'll need to rely on the latter. Unfortunately the latter cannot be automated, and thus any I/O improvements would need to be done manually. I see that there's an `instruments` CLI tool. Would it be possible to introduce exporting options for that tool so that we can export various tables that are created by custom instruments? This would allow us to aggregate performance traces and provide us data we can leverage to make the app faster (e.g. PGO etc.) the next time we compile.

[1]: It seems dtrace might not be supported in newer versions of macOS. Currently you either need to disable SIP or enable SIP without dtrace support (the former I'm guessing Apple obviously doesn't like and the latter causes `csrutil` to spit out an error message stating that SIP without dtrace resrtictions is an unsupported configuration).