Guides and Sample Code

Developer

Instruments User Guide

On This Page

Create Custom Instruments

Built-in instruments provide a great deal of information about the inner workings of your app. Sometimes, though, you may want to tailor this information more closely to your own code. For example, instead of gathering data every time a function is called, you might set conditions on when data is gathered. Alternatively, you might want to dig deeper into your own code than the built-in instruments allow.

About Custom Instruments

Custom instruments use DTrace for their implementation. DTrace is a dynamic tracing facility originally created by Sun and ported to OS X. Because DTrace taps into the operating system kernel, it allows you to access low-level information about the kernel itself and the user processes running on your computer. Many of the built-in instruments are already based on DTrace. DTrace is itself a complex tool, but Instruments provides a simple interface that lets you access the power of DTrace without the complexity.

DTrace has not been ported to iOS, so it is not possible to create custom instruments for devices running iOS.

Custom instruments are built using DTrace probes. A probe is like a sensor that you place in your code. It corresponds to a location or event, such as a function entry point, to which DTrace can bind. When the function executes or the event is generated, the associated probe fires, and DTrace runs whatever actions are associated with the probe. Most DTrace actions simply collect data about the operating system and user app behavior at that moment. It is possible, however, to run custom scripts as part of an action. Scripts let you use the features of DTrace to fine tune the data you gather.

Probes fire each time they are encountered, but the action associated with the probe doesn’t need to run every time the probe fires. A predicate is a conditional statement that allows you to restrict when the probe’s action is run. For example, you can restrict a probe to a specific process or user, or you can run an action when a specific condition in your instrument is true. By default, probes do not have any predicates, meaning that the associated action runs every time the probe fires. You can add any number of predicates to a probe, however, and link them together using AND and OR operators to create complex decision trees.

A custom instrument consists of the following blocks:

  • A description block, containing the name, category, and description of the instrument

  • One or more probes, each containing its associated actions and predicates

  • A Data declaration area, for optionally declaring global variables shared by all probes

  • A Begin script, which optionally initializes global variables and performs startup tasks required by the instrument

  • An End script, which optionally performs final cleanup actions

A custom instrument must have at least one probe with associated actions. Similarly, a custom instrument should have an appropriate name and description that identifies it to Instruments users. Instruments displays this descriptive information in the Library palette. Providing good information makes it easier to remember what the instrument does and how it should be used.

Probes are not required to have a global Data declaration, or Begin and End scripts. These elements are used in advanced instrument design when you want to share data among probes or provide some sort of initial configuration for your instrument. The creation of Data, Begin, and End blocks is described in Write Custom Scripts.

Create a Custom Instrument

To create a custom DTrace instrument, choose Instrument > Build New Instrument. This command displays the instrument configuration dialog, shown in Figure 25-1. You use this sheet to specify your instrument information, including any probes and custom scripts.

Figure 25-1The instrument configuration dialog image: ../Art/instruments_custominstrument_dialog_empty_2x.png

At a minimum, provide the following information for every custom instrument you create:

  • Name. The name associated with your custom instrument in the library.

  • Category. The category in which your instrument appears in the library. You can specify the name of an existing category—such as Memory—or create your own.

  • Description. The instrument description, used in both the Library palette and in the instrument’s tooltip.

  • Probe provider. The probe type and the details of when it should fire. Typically, this involves specifying the method or function to which the probe applies. See Specify the Probe Provider.

  • Probe action. The data to record or the script to execute when your probe fires; see Add Actions to a Probe.

An instrument should contain at least one probe and may contain more than one. The probe definition consists of the provider information, predicate information, and action. All probes must specify the provider information at a minimum, and nearly all probes define some sort of action. The predicate portion of a probe definition is optional but can be a very useful tool for focusing your instrument on the correct data.

Add and Delete Probes

Every new instrument comes with one probe that you can configure. To add more probes, click the Add button (+) at the bottom of the instrument configuration dialog. See Figure 25-2.

To remove a probe from your instrument, click the probe to select it and click the Remove button (-) at the bottom of the instrument configuration dialog.

Figure 25-2Adding or removing probes in the instrument configuration dialog image: ../Art/instruments_custominstrument_dialog_addremove_probe_buttons_2x.png

When adding a probe, it is a good idea to provide a descriptive name for the probe. By default, Instruments assigns each probe a sequentially numbered name, such as Probe 1 and Probe 2.

Specify the Probe Provider

To specify the location point or event that triggers a probe, associate the appropriate provider with the probe. Providers are kernel modules that act as agents for DTrace, providing the instrumentation necessary to create probes. You do not need to know how providers operate to create an instrument, but you do need to know the basic capabilities of each provider. Table 25-1 lists the providers that are supported by the Instruments app and available for use in your custom instruments. The Provider column lists the name displayed in the instrument configuration dialog, and the DTrace provider column lists the actual name of the provider used in the corresponding DTrace script.

Table 25-1DTrace providers

Provider

DTrace provider

Description

User Process

pid

The probe fires on entry (or return) of the specified function in your code. You must provide the function name and the name of the library that contains it.

Objective-C

objc

The probe fires on entry (or return) of the specified Objective-C method. You must provide the method name and the class to which it belongs.

System Call

syscall

The probe fires on entry (or return) of the specified system library function.

DTrace

DTrace

The probe fires when DTrace itself enters a Begin, End, or Error block.

Kernel Function Boundaries

fbt

The probe fires on entry (or return) of the specified kernel function in your code. You must provide the kernel function name and the name of the library that contains it.

Mach

mach_trap

The probe fires on entry (or return) of the specified Mach library function.

Profile

profile

The probe fires regularly at the specified time interval on each core of the machine. Profile probes can fire with a granularity that ranges from microseconds to days.

Tick

tick

The probe fires at periodic intervals on one core of the machine. Tick probes can fire with a granularity that ranges from microseconds to days. You might use this provider to perform periodic tasks that are not required to be on a particular core.

I/O

io

The probe fires at the start of the specified kernel routine. For a list of functions monitored by this probe, use the dtrace -l command from Terminal to get a list of probe points. You can then search this list for probes monitored by the io module.

Kernel Process

proc

The probe fires on the initiation of one of several kernel-level routines. For a list of functions monitored by this probe, use the dtrace -l command from Terminal to get a list of probe points. You can then search this list for functions monitored by the proc module.

User-Level Synchronization

plockstat

The probe fires at one of several synchronization points. You can use this provider to monitor mutex and read-write lock events.

CPU Scheduling

sched

The probe fires when CPU scheduling events occur.

After selecting the provider for your probe, specify the information needed by the probe. For example, for some function-level probes, providers may need function or method names, along with your code module or else the class containing your module. Other providers may only need you to select appropriate events from a pop-up menu.

After you have configured a probe, you can add additional predicates to it (to determine when it should fire) or you can go ahead and define the action for that probe.

Add Predicates to a Probe

Predicates give you control over when a probe’s action is executed by Instruments. You can use predicates to prevent Instruments from gathering data when you don’t want it or think the data might be erroneous. For example, if your code exhibits unusual behavior only when the stack reaches a certain depth, use a predicate to specify the minimum target stack depth. Every time a probe fires, Instruments evaluates the associated predicates. Only if they evaluate to true does DTrace perform the associated actions.

To add a predicate to a probe
  1. Click the Add button (+) in the probe conditions.

  2. Select the type of predicate.

  3. Define the predicate values.

image: ../Art/instruments_custominstrument_dialog_configuringaprobe_2x.png

You can add subsequent predicates using the Add buttons (+) of either the probe or the predicate. To remove a predicate, click the Remove button (-) next to the predicate.

Instruments evaluates predicates from top to bottom in the order in which they appear. To rearrange predicates, click the predicate’s row and drag it to a new location in the table. You can link predicates using AND and OR operators, but you cannot group them to create nested condition blocks. Instead, order your predicates carefully to ensure that the appropriate conditions are checked.

Use the first pop-up menu in a predicate row to choose the data to inspect as part of the condition. Table 25-2 lists the standard variables defined by DTrace that you can use in your predicates or script code. The Variable column lists the name as it appears in the instrument configuration panel, and the “DTrace variable” column lists the actual name of the variable used in corresponding DTrace scripts. In addition to testing the standard variables, you can test against custom variables and constants from your script code by specifying the Custom variable type in the predicate field.

Table 25-2DTrace variables

Variable

DTrace variable

Description

Caller

caller

The value of the current thread’s program counter just before entering the probe. This variable contains an integer value.

Chip

chip

The identifier for the physical chip executing the probe. This is a 0-based integer indicating the index of the current core. For example, a four-core machine has cores 0 through 3.

CPU

cpu

The identifier for the CPU executing the probe. This is a 0-based integer indicating the index of the current core. For example, a four-core machine has cores 0 through 3.

Current Working Directory

cwd

The current working directory of the current process. This variable contains a string value.

Last Error #

errno

The error value returned by the last system call made on the current thread. This variable contains an integer value.

Executable

execname

The name that was passed to exec to execute the current process. This variable contains a string value.

User ID

uid

The real user ID of the current process. This variable contains an integer value.

Group ID

gid

The real group ID of the current process. This variable contains an integer value.

Process ID

pid

The process ID of the current process. This variable contains an integer value.

Parent ID

ppid

The process ID of the parent process. This variable contains an integer value.

Interrupt Priority Level

ipl

The interrupt priority level on the current CPU at the time the probe fired. This variable contains an unsigned integer value.

Function

probefunc

The function name part of the probe’s description. This variable contains a string value.

Module

probemod

The module name part of the probe’s description. This variable contains a string value.

Name

probename

The name portion of the probe’s description. This variable contains a string value.

Provider

probeprov

The provider name part of the probe’s description. This variable contains a string value.

Root Directory

root

The root directory of the process. This variable contains a string value.

Stack Depth

stackdepth

The stack frame depth of the current thread at the time the thread fired. This variable contains an unsigned integer value.

User Stack Depth

ustackdepth

The stack frame depth for user frames (omitting kernel mode frames) of the current thread at the time the thread fired. This variable contains an unsigned integer value.

Relative Timestamp

timestamp

The current value of the system’s timestamp counter, in nanoseconds. Because this counter increments from an arbitrary point in the past, use it to calculate only relative time differences. This variable contains an unsigned 64-bit integer value.

Virtual Timestamp

vtimestamp

The amount of time the current thread has been running, in nanoseconds. This value does not include time spent in DTrace predicates and actions. This variable contains an unsigned 64-bit integer value.

Timestamp

walltimestamp/1000

The current number of nanoseconds that have elapsed since 00:00 Universal coordinated Time, January 1, 1970. This variable contains an unsigned 64-bit integer value.

arg0 through arg9

arg0 through arg9

The first 10 arguments to the probe, represented as raw 64-bit integers. If fewer than ten arguments were passed to the probe, the remaining variables contain the value 0.

Custom

The name of your variable

Use this option to specify a variable or constant from one of your scripts.

In addition to specifying a variable to inspect, you must specify a comparison operator (equals, does not equal, etc.) and a value to use for comparison against the variable.

Add Actions to a Probe

When a probe point defined by your instrument is hit and the probe’s predicate conditions evaluate to true, DTrace runs the actions associated with the probe. You use your probe’s actions to gather data or to perform additional processing. For example, if your probe monitors a specific function or method, you could have it return the caller of that function and any stack trace information to Instruments. If you wanted a slightly more advanced action, you could use a script variable to track the number of times the function was called and report that information as well. And if you wanted an even more advanced action, you could write a script that uses kernel-level DTrace functions to determine the status of a lock used by your function. In this latter case, your script code might also return the current owner of the lock (if there is one) to help you determine the interactions among your code’s different threads.

Figure 25-3 shows the portion of the instrument configuration dialog where you specify your probe’s actions. The script portion simply contains a text field for you to type in your script code. (Instruments does not validate your code before passing it to DTrace, so check your code carefully.) The bottom section contains controls for specifying the data you want DTrace to return to Instruments. You can use the pop-up menus to configure the built-in DTrace variables you want to return. When the first pop-up menu is set to Record in Instruments, you can optionally choose Custom from the second pop-up menu and return one of your script variables.

Figure 25-3Configuring a probe’s action image: ../Art/instruments_custominstrument_dialog_configuringaprobe_action_2x.png

When you configure your instrument to return a custom variable, Instruments asks you to provide the following information:

  • The script variable containing the data

  • The name to apply to the variable in your instrument interface

  • The type of the variable

Any data your probe returned to Instruments is collected and displayed in your instrument’s detail pane. The detail pane displays all data variables regardless of type. If stack trace information is available for a specific probe, Instruments displays that information in the extended detail area of the inspector pane for your instrument. In addition, Instruments automatically looks for integer data types returned by your instrument and adds those types to the list of statistics your instrument can display in the track pane.

Because DTrace scripts run in kernel space and the Instruments app runs in user space, if you want to return the value of a custom pointer-based script variable to Instruments, you must create a buffer to hold the variable’s data. The simplest way to create a buffer is to use the copyin or copyinstr subroutines found in DTrace. The copyinstr subroutine takes a pointer to a C string and returns the contents of the string in a form you can return to Instruments. Similarly, the copyin subroutine takes a pointer and size value and returns a buffer to the data, which you can later format into a string using the stringof keyword. Both of these subroutines are part of the DTrace environment and can be used from any part of your probe’s action definition. For example, to return the string from a C-style string pointer, you simply wrap the variable name with the copyinstr subroutine, as shown in Figure 25-4.

Figure 25-4Returning a string pointer image: ../Art/instruments_custominstrument_dialog_returnstringpointer_2x.png

For a list of the built-in variables supported by Instruments, see Table 25-2. For more information on scripts and script variables, see Write Custom Scripts. For more information on DTrace subroutines, including the copyin and copyinstr subroutines, see the Solaris Dynamic Tracing Guide, available from the Oracle Technology Network.

Write Custom Scripts

You write DTrace scripts using the D scripting language, whose syntax is derived from a large subset of the C programming language. The D language combines the programming constructs of the C language with a special set of functions and variables to help you trace information in your app.

The following sections describe common ways to use scripts in your custom instruments. These sections do not provide a comprehensive overview of the D language or the process for writing DTrace scripts. For information about scripting and the D language, see the Solaris Dynamic Tracing Guide, available from the Oracle Technology Network.

Write Begin and End Scripts

If you want to do more than return the information in DTrace’s built-in variables to Instruments whenever your action fires, you need to write custom scripts. Scripts interact directly with DTrace at the kernel level, providing access to low-level information about the kernel and the active process. Most instruments use scripts to gather information not readily available from DTrace. You can also use scripts to manipulate raw data before returning it to Instruments. For example, you can use a script to normalize a data value to a specific range if you want to make it easier to compare that value graphically with other values in your instrument’s track pane.

In Instruments, the custom instrument configuration dialog provides several areas where you can write DTrace scripts:

  • The Data section contains definitions of any global variables you want to use in your instrument.

  • The Begin section contains any initialization code for your instrument.

  • Each probe contains script code as part of its action.

  • The End section contains any cleanup code for your instrument.

All script sections are optional. You are not required to have initialization scripts or cleanup scripts if your instrument does not need them. If your instrument defines global variables in its Data section, however, it is recommended that you also provide an initialization script to set those variables to a known value. The D language does not allow you to assign values inline with your global variable declarations, so you must put those assignments in your Begin section. For example, a simple Data section might consist of a single variable declaration, such as the following:

  1. int myVariable;

The corresponding Begin section would then contain the following code to initialize that variable:

  1. myVariable = 0;

If your corresponding probe actions change the value of myVariable, you can use the End section of your probe to format and print out the final value of the variable.

Most of your script code is likely to be associated with individual probes. Each probe can have a script associated with its action. When it comes time to execute a probe’s action, DTrace runs your script code first and then returns any requested data back to Instruments. Because passing data back to Instruments involves copying data from the kernel space back to the Instruments app space, you should always pass data back to Instruments by configuring the appropriate entries in the “Record the following data” section of the instrument configuration dialog. Variables returned manually from your script code may not be returned correctly to Instruments.

Access Kernel Data from Custom Scripts

Because DTrace scripts execute inside the system kernel, they have access to kernel symbols. To look at a global kernel variable and data structure from your custom instrument in your DTrace scripts, precede the name of the variable with the backquote character (`). The backquote character tells DTrace to look for the specified variable outside of the current script.

Listing 25-1 shows a sample action script that retrieves the current load information from the avenrun kernel variable and uses that variable to calculate a one-minute average load of the system. If you create a probe using the Profile provider, you can have this script gather load data periodically and then graph that information in Instruments.

Listing 25-1Accessing kernel variables from a DTrace script
  1. this->load1a = `avenrun[0]/1000;
  2. this->load1b = ((`avenrun[0] % 1000) * 100) / 1000;
  3. this->load1 = (100 * this->load1a) + this->load1b;

Scope Variables Appropriately

DTrace scripts have an essentially flat structure, due to a lack of flow control statements and the desire to keep probe execution time to a minimum. That said, you can scope the variables in DTrace scripts to different levels depending on your need. Table 25-3 lists the scoping levels for variables and the syntax for using variables at each level.

Table 25-3Variable scope in DTrace scripts

Scope

Syntax example

Description

Global

myGlobal = 1;

Global variables are identified by the variable name. All probe actions on all system threads have access to variables in this space.

Thread

self->myThreadVar = 1;

Thread-local variables are dereferenced from the self keyword. All probe actions running on the same thread have access to variables in this space. You might use this scope to collect data over the course of several runs of a probe’s action on the current thread.

Probe

this->myLocalVar = 1;

Probe-local variables are dereferenced using the this keyword. Only the current running probe has access to variables in this space. Typically, you use this scope to define temporary variables that you want the kernel to clean up when the current action ends.

Find Script Errors

If the script code for one of your custom instruments contains an error, Instruments displays an error message in the track pane when DTrace compiles the script. Instruments reports the error after you click the Record button in your trace document but before tracing actually begins. Inside the error message bubble is an Edit button. Clicking this button opens the instrument configuration dialog, which now identifies the probe with the error.

Export and Import DTrace Scripts

Although Instruments provides a convenient interface for gathering trace data, sometimes it is more convenient to gather trace data directly using DTrace. If you are a system administrator or are writing automated test scripts, for example, you might prefer the DTrace command-line interface to launch a process and gather the data. Using the command-line tool requires you to write your own DTrace scripts, which can be time consuming and can lead to errors. If you already have a trace document with one or more DTrace-based instruments, you can use the Instruments app to generate a DTrace script that provides the same behavior as the instruments in your trace document.

Instruments supports exporting DTrace scripts only for documents in which all of the instruments are based on DTrace. This means that your document can include custom instruments and a handful of the built-in instruments, such as the file system-related and Core Data instruments in the Library palette.

To export a DTrace script
  1. Select the trace document.

  2. Choose File > DTrace Script Export.

  3. Enter a name for the DTrace script.

  4. Select a location for the DTrace script.

  5. Click Save.

The DTrace Script Export command places the script commands for your instruments in a text file that you can then pass to the dtrace command-line tool using the -s option. For example, if you export a script named MyInstrumentsScript.d, run it from Terminal using the following command:

  1. sudo dtrace -s MyInstrumentsScript.d

Another advantage of exporting your scripts from Instruments (as opposed to writing them manually) is that after running the script, you can import the resulting data back into Instruments and review it there. Scripts exported from Instruments print a start marker (with the text dtrace_output_begin) at the beginning of the DTrace output. To gather the data, simply copy all of the DTrace output (including the start marker) from Terminal and paste it into a text file, or just redirect the output from the dtrace tool directly to a file. To import the data in Instruments, select the trace document from which you generated the original script, and choose File > DTrace Data Import.