In your development process, Xcode provides a wealth of tools for analyzing, isolating, and eliminating bugs in your applications. But after you've built your application for release, those safety nets disappear, making it difficult to track down bugs found in production. Without debugging information, familiar debugging tools (such as the Xcode debugger and Shark) become nearly useless. If the application should crash, the crash log will contain only hexadecimal addresses—scant information for determining why the application failed.

A simple solution is to include the debugging information in the released application. However, such a solution is typically unacceptable for two reasons. First, debugging information can be many times larger than the executable code itself, making distribution unwieldy and inefficient. Second, including symbol and variable information exposes your application's design to unwanted scrutiny and abuse.

The goal, therefore, is a development workflow that:

• Strips a release build of debugging information, making it suitable for distribution;
• Retains the ability to debug a release build with familiar tools;
• Allows the information in a crash report to be used to pinpoint the original source code that caused the crash.

This article explores Xcode solutions that achieve some or all of these goals. You'll learn various techniques for stripping the symbolic information from your finished application, how to debug a stripped application, and how to interpret the crash logs produced by a stripped application. The first part of this article concentrates on building and debugging the final application. Later sections explore crash dumps with no debugging information. The techniques presented apply to all C, C++, and Objective-C applications.

Note: This articles assumes you are using Xcode version 2.4.x.

Using the 'Out of the Box' Solution

The typical development style is to build an application with minimal optimization and full debugging information. Doing so provides the debugger and performance-analysis tools with the maximum amount of information about the compiled code. When the application is fully debugged, you build it again without that debugging information. This final version is suitable for distribution to users.

The Xcode project templates provide exactly those build configurations. A new project has a Debug configuration, which builds the application with full debugging information and minimal optimization, and a Release build configuration, which produces minimal symbol information and performs a full optimization of the code. While simple and sufficient for many projects, however, these configurations have a few drawbacks.

If you encounter problems with the release version, the code is difficult to debug. Without the debugging information, the GNU Debugger (GDB) and other analysis tools are extremely limited. If the application crashes, the crash log contains method or function names but no source line numbers.

Note that the default release build produces an application with minimal method/function name information—a middle ground between no debugging information and full debugging information. The advantage is that this information might be sufficient for diagnosing a crash report. The disadvantage is that the class and method names embedded in the application might be more information that you want to expose to users.

Adding Debugging Information to the Release Build

The simplest solution is to build your release application twice—once with debugging information and again without it. Select Project > Edit Project Settings. Click the Configurations tab; in the Configurations pane, select the Release configuration and duplicate it. Name the new configuration "Debug Release." In the Targets group of the project, open the Info window for the application target. In the Build pane, change the configuration to Debug Release, and then set the Generate Debug Symbols build setting (found in the Code Generation collection). When you're done, your project build settings should resemble those shown in Table 1.

Table 1: Build Settings in the Debug Release Configuration

Build the project once using the Release configuration and again using the Debug Release configuration. The executable code for the two applications should be identical; the only difference should be that one includes debugging information. You can successfully debug the Debug Release build, and an application crash will be fully symbolized with method names and source file line numbers. To demonstrate the latter, add code to your application that will intentionally cause an application crash when executed, such as the code that Listing 1 shows.

#define CRASH_CODE 1
#if CRASH_CODE
	(void)strlen((const char *)1);
#endif

Listing 1: Source Code to Induce a Crash

Build and launch the application from the Finder. Don't start it from Xcode; Xcode will intercept the crash and start the debugger. Execute the code that will cause your application to crash. To receive a crash report, you might need to enable the Crash Reporter. See Technical Note TN2123 CrashReporter for details.

This approach is simple and satisfies the second goal and some of the first goal. You can easily debug the release version of the application, and a reproducible crash will produce a fully symbolized crash report. However, crashes that the Release build produces are still inscrutable, and it might not be appropriate or practical to provide users with the debug-laden version of the application.

There are other significant disadvantages. Building twice is time-consuming. More importantly, there's no guarantee that the Release and Debug Release applications are identical—which means that the Debug Release build might not have the same bugs as the released version. Worse, there is no way to verify that the two builds are identical. You must carefully maintain the Release and Debug Release build configurations in parallel; any differences could cause the resulting applications to diverge. This requirement might be too fragile for some development environments.

Stripping the Finished Product

Xcode provides several built-in options for stripping executables of their debugging symbols. One of these is the Strip Linked Product build setting. While typically set, it has no effect unless the Deployment Postprocessing setting is also set. Deployment Postprocessing is a master switch that enables the action of a host of other build settings. It's approximately analogous to running the xcodebuild tool with the install command.

Again, open the target build settings and turn on debugging symbols for the Release configuration. Open the project build settings; in the Release configuration, enable both Strip Linked Product (if it isn't on already) and Deployment Postprocessing. Your project settings should now resemble those shown in Table 2.

Table 2: Project Build Settings in the Release Configuration

Stripping the executable finally achieves the first goal; it produces a final executable that's completely devoid of all debugging symbol information. Consequently, this strategy fails to meet the second goal; it can't be debugged and a crash report will contain no program symbol information whatsoever.

By combining these two solutions, you can achieve both goals. Do this by setting Deployment Postprocessing in the Release configuration but not in the Debug Release configuration. However, the combined solution still has all the other disadvantages of the first solution.

Extracting the Debugging Information to a Separate File

It should be obvious by now that you need a build process that strips the final executable file of all debugging information while simultaneously saving that information for future debugging. Instead of merely stripping the debug information from the final executable, a better solution is to extract the debugging information and save it to a separate location. The application is built only once, and the extracted debugging information is available to the developer but is no longer embedded in the final product.

Beginning with version 2.4, Xcode supports the Debugging with Attributed Record Formats (DWARF) with dSYM file format. Setting the Debug Information Format to DWARF with dSYM causes the linker to produce a separate file package, with the dSYM extension, that contains a copy of the symbol information in the executable. A small reference is added to the executable so that debugging tools know that a dSYM file for the program was produced.

The embedded debug information can now be safely stripped from the executable. When loading a stripped executable, the GNU debugger will check for its companion dSYM file. If present, it will automatically use the DWARF debugging information contained therein. A dSYM file must be in the same directory as the product (simple executable or application bundle) for this to occur.

In the Release configuration of the application target, enable dSYM-style debugging information, as Table 3 shows.

Table 3: Target Build Settings in the Release Configuration

Note that the dSYM file format can't be used with ZeroLink and is generally useful only for producing a stripped product while retaining a copy of the debugging information. In your development build configurations, continue to use the regular stabs or DWARF formats.

And, like the previous solution, enable stripping of the final executable in the project build settings, as Table 4 shows.

Table 4: Project Build Settings in the Release Configuration

This solution meets the first and second goals and is easy to maintain. You can archive the finished application for future debugging simply by preserving the dSYM file. You can debug the release build of the application without any special steps; simply start the debugging session with the dSYM file present. The application, sans the dSYM file, can be freely distributed and contains no symbol information.

A Note About Debugging Optimized Code

Be aware that you could encounter subtle problems when trying to debug optimized code. During debugging, code generally isn't optimized because optimization can destroy the one-to-one relationship between the source code and the compiled object code, which could make it impossible to correlate an execution address with a line of code in the original source (or vice versa). Another artifact might be unexpected order of execution (i.e., the statement on line 9 executes before the statement on line 8). This behavior sometimes makes it difficult to set breakpoints or control execution in the debugger.

There's no real solution to this problem. The very nature of optimizers means that the compiler might rewrite code expressed in source code for better performance.

Symbolizing Crash Dumps

Should your application crash, the system crash reporter facility creates a crash log. This crash log contains a stack trace annotated with whatever program symbol information was found in the application and frameworks. The standard system frameworks contain basic symbol information, but if the application has been stripped of its debugging information, no useful information about the application's stack will be logged. Listing 2 shows a portion of the TempConverter.crash.log.

Thread 0 Crashed:
0   com.yourcompany.TempConverter 	0x00002b84 0x1000 + 7044
1   com.apple.Foundation          	0x929ea9c8 _NSSetObjectValueAndNotify + 136
2   com.apple.Foundation          	0x929ea6f4 -[NSObject(NSKeyValueCoding) setValue:forKeyPath:] + 180
…
27  com.apple.AppKit              	0x937f887c NSApplicationMain + 452
28  com.yourcompany.TempConverter 	0x000029dc 0x1000 + 6620
29  com.yourcompany.TempConverter 	0x000026e0 0x1000 + 5856

Listing 2: Excerpt from TempConverter.crash.log

Because the TempConverter executable has no symbol information, program locations in the TempConverter code segment are listed as hexadecimal offsets. You can use the atos ("address to symbol") tool to convert these addresses back to their symbolic names. If enough debugging information is available, the tool also supplies the source file name and line number for the address. Open a Terminal window, and set the current directory to the TempConverter D/build/Release folder. Then, issue the command that Listing 3 shows.

atos -o Debug/TempConverter.app/Contents/MacOS/TempConverter 0x00002b84
-[ConverterApplicationDelegate setCentigradeTemperature:] (in TempConverter) (ConverterApplicationDelegate.m:42)

Listing 3: Using atos to Translate Memory Address to Symbol

The atos tool reads the symbols contained in the TempConverter executable file and uses that information to convert the address 0x00002b84 to its symbolic equivalent, complete with source file name and line number. If you're using DWARF dSYM files, you must be using the version of atos included in Xcode 3 (Mac OS X version 10.5).

The atos command accepts multiple address arguments or can read a list of addresses from stdin. While useful enough for converting a few addresses, atos becomes cumbersome for large stack traces or multiple crash logs. Fortunately, you can automate the process with a script like the one Listing 4 shows.

#!/bin/bash

AWK_SCRIPT=/tmp/symbolizecrashlog_$$.awk
SH_SCRIPT=/tmp/symbolizecrashlog_$$.sh

if [[ $# < 2 ]]
then
	echo "Usage: $0 [ -arch <arch> ] symbol-file [ crash.log, ... ]"
	exit 1
fi

ARCH_PARAMS=''
if [[ "${1}" == '-arch' ]]
then
	ARCH_PARAMS="-arch ${2}"
	shift 2
fi

SYMBOL_FILE="${1}"
shift

cat > "${AWK_SCRIPT}" << _END
/^[0-9]+ +[-._a-zA-Z0-9]+ *\t0x[0-9a-fA-F]+ / {
	addr_index = index(\$0,\$3);
	end_index = addr_index+length(\$3);
	line_legnth = length;
	printf("echo '%s'\"\$(symbolize %s '%s')\"\n",substr(\$0,1,end_index),
		\$3,substr(\$0,end_index+1,line_legnth-end_index));
	next;
	}
{ gsub(/'/,"'\\\\''"); printf("echo '%s'\n",\$0); }
_END

function symbolize()
{
	# Translate the address using atos
	SYMBOL=$(atos -o "${SYMBOL_FILE}" ${ARCH_PARAMS} "${1}" 2>/dev/null)
	# If successful, output the translated symbol. If atos returns an address, output the original symbol
	if [[ "${SYMBOL##0x[0-9a-fA-F]*}" ]]
	then
		echo -n "${SYMBOL}"
	else
		echo -n "${2}"
	fi
}

awk -f "${AWK_SCRIPT}" $* > "${SH_SCRIPT}"

. "${SH_SCRIPT}"

rm -f "${AWK_SCRIPT}" "${SH_SCRIPT}"

Listing 4: The symbolizecrashlog Script

Obtain a crash log from the TempConverter without symbol information and a copy of the executable file that includes its debugging information. Invoke the script with the command Listing 5 shows. (This command assumes that all three files are in the current directory.)

./symbolizecrashlog TempConverter TempConverter.crash.log > TempConverter.symbolized.log

Listing 5: Running the symbolizecrashlog Tool

The symbolizecrashlog script takes as input one or more crash logs and uses the debugging information found in the first argument to translate all stack trace addresses to their symbolic equivalents. Addresses that can't be translated are left unaltered. If no crash log files are given, the script reads the crash log from stdin.

Crash Processor Architecture

Universal binaries contain the executable for multiple processor architectures. Both the atos tool and the symbolizecrashlog script assume that the addresses being converted are for the architecture of the computer on which they're running. For example, symbolizing a crash log that occurred on a Power Mac (PowerPC) using a Power Mac Pro (Intel) produces invalid results, because atos assumes that addresses apply to the Intel executable. You must tell atos to use the debugging symbols information that belongs to the "ppc" executable, instead. Both atos and symbolizecrashlog accept an -arch option to specify the architecture of the system that should be used to interpret the addresses. Listing 6 shows some examples.

atos -arch ppc TempConverter 0x00002b84
symbolizecrashlog -arch i386 TempConverter TempConverter.crash.log > TempConverter.symbolized.log

Listing 6: Specifying the Architecture when Using atos and symbolizecrashlog

Conclusion

Balancing the requirement of producing a finished application without any debugging information and the need to debug and diagnose problems with that same application complicates development. Fortunately, Xcode provides built-in methods for controlling exactly how much debugging information is produced and where it gets stored.

For More Information

• See Technical Note TN2123: CrashReporter
• Read the Xcode 2.4 Build Settings Notes

Posted: 2007-04-02