Technical Note TN2265

Troubleshooting Push Notifications

Describes techniques you can use to resolve issues with sending and receiving push (remote) notifications in iOS and OS X.

Introduction
Issues with Receiving Push Notifications
Issues with Sending Push Notifications
Other Tips and Tricks
References
Downloadables
Document Revision History

Introduction

Many iOS and Mac applications present dynamic content delivered over the Internet. Push notifications (also known as remote notifications) are a way to let users know that new or updated content they're interested in is available.

This technote describes techniques you can use to resolve issues with sending and receiving push notifications.

Issues with Receiving Push Notifications

Registering for Push Notifications

In order to receive push notifications, an application must first register with the Apple Push Notification service (APNs or "push service"). Registration has three stages:

  1. The application calls the registerForRemoteNotificationTypes: method of UIApplication (on iOS) or NSApplication (on OS X).

  2. The application implements the application:didRegisterForRemoteNotificationsWithDeviceToken: method of UIApplicationDelegate (iOS) or NSApplicationDelegate (OS X) to receive the unique device token generated by the push service.

  3. The application implements the application:didFailToRegisterForRemoteNotificationsWithError: method of UIApplicationDelegate (iOS) or NSApplicationDelegate (OS X) to receive an error if the registration failed.

The application passes the device token to your provider as a non-object, binary value. The provider is your server that delivers dynamic content to your application.

Registering for push notifications is straightforward, but there are a few important details you need to be aware of.

No Delegate Callbacks

When the first push-capable app is installed, iOS or OS X attempts to establish a persistent network connection to the push service that will be shared by all push-capable apps on the system. If neither delegate callback application:didRegisterForRemoteNotificationsWithDeviceToken: nor application:didFailToRegisterForRemoteNotificationsWithError: is called, that means that this connection has not yet been established.

This is not necessarily an error condition. The system may not have Internet connectivity at all because it is out of range of any cell towers or Wi-Fi access points, or it may be in airplane mode. Instead of treating this as an error, your app should continue normally, disabling only that functionality that relies on push notifications.

Keep in mind that network availability can change frequently. Once the persistent connection to the push service succeeds, one of the previously-mentioned application delegate methods will be called.

On iOS, push notifications use the cellular data network whenever possible, even if the device is currently using Wi-Fi for other network activity such as web browsing or email. However, the push service will fall back to Wi-Fi if cellular data service isn't available.

If your iOS device is capable of using the cellular data network, check that it has an active cellular data plan. Turn off Wi-Fi in Settings and see if you can still browse the web with Safari, for example. On the other hand, if the push service is using Wi-Fi, any firewalls between your device or computer and the Internet must allow TCP traffic to and from port 5223.

Error Delegate Callback

The first time a push-capable iOS application is run, iOS asks the user if they want to receive push notifications for this app. If the app isn't even offering the option of receiving push notifications, the chances are that it is missing its aps-environment code signing entitlement. This entitlement is taken from the provisioning profile used when building the app, and it controls which push environment the app will connect to in order to receive remote notifications.

If an iOS app is running on a device, or if the app is a Mac app, the error method application:didFailToRegisterForRemoteNotificationsWithError: will be called if the code signing entitlements for accessing the push service are invalid. If you have not implemented this method in your application, you should do that as a way of checking that your aps-environment (iOS) or com.apple.developer.aps-environment (OS X) entitlement is present at runtime.

When using Xcode to submit an app, Xcode will re-sign it using the code signing identity and associated provisioning profile you select. So the signature of the submitted app and its contents might be different than what's in the Xcode archive.

Here's how to check the signature for an iOS app being submitted to the App Store:

  1. In the Xcode Organizer, instead of Submit to the iOS App Store, do Save for Enterprise or Ad-Hoc Deployment. That will create a local copy of the .ipa file that would be submitted to the App Store.

  2. When asked to choose an identity to sign with, select the same distribution identity you use when submitting to the App Store.

  3. When asked to save the package, uncheck Save for Enterprise Distribution, then save the .ipa file.

  4. Find the .ipa file and change its the extension to .zip.

  5. Expand the .zip file. That will produce a Payload folder containing your app bundle.

  6. Use the codesign tool to check the entitlements on the app bundle like this:

    $ codesign -d --entitlements :- "Payload/YourApp.app"

    where YourApp.app is the actual name of your app bundle.

Here are the steps to do the same thing for a Mac app:

  1. In the Xcode Organizer, instead of Submit to the Mac App Store, do Export as Mac Installer Package. That will create a local copy of the .pkg file that would be submitted to the Mac App Store.

  2. When asked to choose an identity to sign with, select the same distribution identity you use when submitting to the Mac App Store.

  3. Save the .pkg file when prompted.

  4. Use the pkgutil tool to expand the package into its components:

    $ pkgutil --expand "YourApp.pkg" Expanded_pkg

    where YourApp.pkg is the actual name of the package you created in the previous step.

  5. Expand the compressed payload inside the package using the open tool or by double-clicking it:

    $ open Expanded_pkg/com.yourcompany.yourapp/Payload

    where com.yourcompany.yourapp is the actual bundle ID of your app.

  6. Use the codesign tool to check the entitlements on the app bundle like this:

    $ codesign -d --entitlements - "Expanded_pkg/com.yourcompany.yourapp/YourApp.app"

    where YourApp.app is the actual name of your app bundle.

Check if there is an aps-environment or com.apple.developer.aps-environment code signing entitlement listed under Internal requirements. It would normally appear after the application-identifier (iOS) or com.apple.application-identifier (OS X) entitlement.

If the aps-environment or com.apple.developer.aps-environment entitlement is missing, check which item you selected on the "Choose an identity to sign with" sheet. The name of the provisioning profile is shown in gray above the name of the signing certificate. Then go to Devices > Library > Provisioning Profiles in the Organizer, control-click the provisioning profile, and select Reveal Profile in Finder.

Drag the profile to TextEdit so you can look at its contents, and search for Entitlements. If you don't see aps-environment or com.apple.developer.aps-environment in the entitlements, you've either not configured your app ID for production push notifications, or you haven't updated your distribution provisioning profile since configuring your app ID.

You probably created your distribution provisioning profile before you configured your App ID for push notifications in Member Center. Or. it might just be that the distribution provisioning profile installed on your build machine is older than the one in Member Center. In that case, delete the one on the build machine and click Refresh at the bottom right of the Organizer window.

Once you have a distribution provisioning profile installed on your build system with the correct push notifications entitlement, you will need to rebuild your app, and re-submit it to the store as an update. You may be able to request an expedited review using this form.

Registration Succeeded But No Notifications Received

If your app has registered with the push service but it is not receiving notifications, the problem can be either on the device/computer side or on the provider side. Here are a few things to check on the device/computer side.

The device or computer may have lost its persistent connection to the push service and can't reconnect. Try quitting the app and relaunching it to see if registration completes the next time. (On iOS 4 and later on devices that support multitasking, you will need to force quit the app from the recents list.) If the registration does not complete, iOS or OS X has been unable to re-establish the persistent connection. You can troubleshoot this as described in the previous two sections.

Your app may have sent an incorrect device token to your provider. Your app should always ask for the device token by registering with the push service each time it is launched. Don't store a device token from your app and try to reuse it, because the token can change. Your provider should then pass that same token on to the push service.

Some Notifications Received, but Not All

If you are sending multiple notifications to the same device or computer within a short period of time, the push service will send only the last one.

Here's why. The device or computer acknowledges receipt of each notification. Until the push service receives that acknowledgment, it can only assume that the device or computer has gone off-line for some reason and stores the notification in the quality of service (QoS) queue for future redelivery. The round-trip network latency here is of course a major factor.

As described in the Local and Push Notification Programming Guide, the QoS queue holds a single notification per app per device or computer. If the service receives another notification before the one in the queue is sent, the new notification overwrites the previous one.

All of this points out that the intent is that a notification indicates to an app that something of interest has changed on the provider, and the app should check in with the provider to get the details. Notifications should not contain data which isn't also available elsewhere, and they should also not be stateful.

Any push notification that isn't delivered immediately was queued for future redelivery because your device was not connected to the service. "Immediately" of course needs to take latency for your connection into account. Outlying cases would be beyond 60 seconds as APNs will time out at that point.

Observing Push Status Messages

If these tips don't resolve the issue, you can enable additional messages from the APNs daemon on the device or computer.

Enabling Push Status Messages on iOS

To enable logging on iOS, install the configuration profile PersistentConnectionLogging.mobileconfig on your device by either putting the file on a web server and downloading it using Safari on your device, or by sending it as an email attachment and opening the attachment in Mail on your device. You'll be prompted to install "APS/PC Logging". Download the configuration profile by clicking Companion File at the upper right corner of this technote.

Once the configuration profile has been installed, turn the device off completely and then turn it back on. Then exercise your app again while watching the console using the Xcode Organizer. Or, sync your device with iTunes and the log will be saved in ~/Library/Logs/CrashReporter/MobileDevice/<DEVICE_NAME>/PersistentConnection/.

Look for messages from the apsd process. Ideally you'll see Connected to courier x-courier.sandbox.push.apple.com where x is a small integer. That indicates that the device has successfully established its persistent connection to the push service.

You might on the other hand see Disconnecting in response to connection failure. That means that the persistent connection failed. In that case, the goal is to figure out what's going on with your network that's causing the connection failure. Check that no firewalls are blocking TCP traffic on port 5223.

The message connection set ignored topics means that the user chose to turn off notifications for the apps listed in the message. That will be followed by Sending filter message for enabled hashes which is where iOS actually sends the enabled and ignored topics to APNs.

The message Failed to parse JSON message payload indicates that the JSON dictionary in the notification payload is not in valid JSON format.

The message Received message for enabled topic <your app's CFBundleIdentifier> means that the device received a notification from the push service.

On OS X, some log messages refer to the hash of your app's CFBundleIdentifier. Here's how to compute this SHA-1 hash:

$ echo -n "<your CFBundleIdentifier>" | openssl dgst -sha1

If things appear to look normal in the log but you're still not receiving notifications, try turning off the Notifications switch in Settings, and then turn it back on. That will try to re-establish the device's persistent connection with APNs.

To remove the configuration profile, go to Settings > General > Profiles, tap "APS/PC Logging", then tap Remove. Turn the device off completely and turn it back on.

Enabling Push Status Messages on OS X

To enable logging on OS X, use the following commands:

$ sudo defaults write /Library/Preferences/com.apple.apsd APSWriteLogs -bool TRUE
$ sudo defaults write /Library/Preferences/com.apple.apsd APSLogLevel -int 7
$ sudo killall apsd

The logs are stored in /Library/Logs/apsd.log.

Issues with Sending Push Notifications

If your app is not receiving push notifications, and everything looks correct on the device or computer, here are some things to check on the server side.

Problems Connecting to the Push Service

One possibility is that your server is unable to connect to the push service. This can mean that you don't have the certificate chain needed for TLS/SSL to validate the connection to the service. In addition to the SSL identity (certificate and associated private key) created by Member Center, you should also install the Entrust CA (2048) root certificate on your provider. This allows TLS/SSL to verify the full APNs server cert chain. If you need to get this root certificate, you can download it from Entrust's site. Also verify that these identities are installed in the correct location for your provider and that your provider has permission to read them.

You can test the TLS/SSL handshake using the OpenSSL s_client command, like this:

$ openssl s_client -connect gateway.sandbox.push.apple.com:2195 -cert YourSSLCertAndPrivateKey.pem -debug -showcerts -CAfile server-ca-cert.pem

where server-ca-cert.pem is the Entrust CA (2048) root certificate.

Be sure the SSL identity and the hostname are the correct ones for the push environment you're testing. You can configure your App ID in Member Center separately for the sandbox and production environment, and you will be issued a separate identity for each environment.

Using the sandbox SSL identity to try to connect to the production environment will return an error like this:

CRITICAL | 14:48:40.304061 | Exception creating ssl connection to Apple: [Errno 1] _ssl.c:480: error:14094414:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate revoked

One common OpenSSL error is verify error:num=20:unable to get local issuer certificate.

That message means that one or more of the certificates from the server could not be verified because the issuer (root or intermediate) certificates were not found.

The CAfile argument to s_client specifies the trusted root certificates to use to verify the server certificate. The trusted root certificate for the push servers is the Entrust CA (2048) root certificate mentioned previously.

If you can't even open a connection to APNs, perhaps your APNs TLS/SSL certificate has expired. These certificates are valid for one year but production APNs certificates can be renewed at any time.

Another possibility is that you've connected too many times to APNs and further connections have been temporarily blocked. As is documented in the Local and Push Notification Programming Guide, developers are expected to open a connection and leave it open. If a connection is opened and closed repeatedly, APNs will treat it as a denial of service attack and block connections for a period of time.

This temporary block will expire if no connection attempts are made for about one hour.

Yet another possibility is that there is a firewall blocking access to the ports used by APNs. Please see IP Address Range Used by the Push Service for details. Try running a telnet command on your server to see if the server can reach APNs, like this:

$ telnet 1-courier.push.apple.com 5223
$ telnet gateway.sandbox.push.apple.com 2195
$ telnet gateway.push.apple.com 2195

Handling Malformed Notifications

The simple notification format drops the connection if the push service receives a notification that is incorrect in some way. Your provider may see this as an EPIPE or broken pipe error in response to sending a notification. On the other hand, the enhanced notification format will send an error response with more detailed information about what was wrong with the notification before dropping the connection. Be sure your provider catches and handles these conditions properly.

Many push notification servers do not handle error responses or dropped connections robustly. An easy way to check this is to intentionally send a notification to a sandbox environment device token, assuming your server is communicating with the production push environment. Doing that should return an invalid token response and drop the connection. To learn more about checking error responses from the push service, please see Push Notification Throughput and Error Checking.

The most common problem is an invalid device token. If the token came from the sandbox environment, such as when you are testing a development build in house, you can't send it to the production push service. Each push environment will issue a different token for the same device or computer. If you do send a device token to the wrong environment, the push service will see that as an invalid token and discard the notification.

The user may have deleted your app from their device or computer. You should check the feedback service at least once a day for device tokens that are no longer active.

Other possible issues might be sending a payload longer than 256 bytes, your payload might not be formatted correctly, or perhaps your JSON dictionary has incorrect syntax.

An occasional disconnect while your provider is idle is nothing to be concerned about; just re-establish the connection and carry on. If one of the push servers is down, the load balancing mechanism will transparently direct your new connection to another server assuming you connect by hostname and not by static IP address.

Using the Enhanced Notification Format

If your provider is using the original simple notification, consider migrating to the enhanced notification format. That interface provides more control over the expiration of notifications and also returns more detailed error responses if a problem occurs with a particular notification.

You can read more about the enhanced notification format in the Local and Push Notification Programming Guide.

Issues with Using the Feedback Service

If you remove your app from your device or computer and then send a push notification to it, you would expect to have the device token rejected, and the invalidated device token should appear on the feedback service. However, if this was the last push-enabled app on the device or computer, it will not show up in the feedback service. This is because deleting the last app tears down the persistent connection to the push service before the notice of the deletion can be sent.

You can work around this by leaving at least one push-enabled app on the device or computer in order to keep the persistent connection up. To keep the persistent connection to the production environment up, just install any free push-enabled app from the App Store and you should then be able to delete your app and see it appear in the feedback service.

Recall that each push environment has its own persistent connection. So to keep the persistent connection to the sandbox environment up, install another development push-enabled app.

Other Tips and Tricks

Push Notification Throughput and Error Checking

There are no caps or batch size limits for using APNs. The iOS 6.1 press release stated that APNs has sent over 4 trillion push notifications since it was established. It was announced at WWDC 2012 that APNs is sending 7 billion notifications daily.

If you're seeing throughput lower than 9,000 notifications per second, your server might benefit from improved error handling logic.

Here's how to check for errors when using the enhanced notification format. Keep writing until a write fails. If the stream is ready for writing again, resend the notification and keep going. If the stream isn't ready for writing, see if the stream is available for reading.

If it is, read everything available from the stream. If you get zero bytes back, the connection was closed because of an error such as an invalid command byte or other parsing error. If you get six bytes back, that's an error response that you can check for the response code and the ID of the notification that caused the error. You'll need to send every notification following that one again.

Once everything has been sent, do one last check for an error response.

It can take a while for the dropped connection to make its way from APNs back to your server just because of normal latency. It's possible to send over 500 notifications before a write fails because of the connection being dropped. Around 1,700 notifications writes can fail just because the pipe is full, so just retry in that case once the stream is ready for writing again.

Now, here's where the tradeoffs get interesting. You can check for an error response after every write, and you'll catch the error right away. But this causes a huge increase in the time it takes to send a batch of notifications.

Device tokens should almost all be valid if you've captured them correctly and you're sending them to the correct environment. So it makes sense to optimize assuming failures will be rare. You'll get way better performance if you wait for write to fail or the batch to complete before checking for an error response, even counting the time to send the dropped notifications again.

None of this is really specific to APNs, it applies to most socket-level programming.

If your development tool of choice supports multiple threads or interprocess communication, you could have a thread or process waiting for an error response all the time and let the main sending thread or process know when it should give up and retry.

IP Address Range Used by the Push Service

Push providers, iOS devices, and Mac computers are often behind firewalls. To send notifications, you will need to allow inbound and outbound TCP packets over port 2195. To reach the feedback service, you will need to allow inbound and outbound TCP packets over port 2196. Devices and computers connecting to the push service over Wi-Fi will need to allow inbound and outbound TCP packets over port 5223.

The IP address range for the push service is subject to change; the expectation is that providers will connect by hostname rather than IP address. The push service uses a load balancing scheme that yields a different IP address for the same hostname. However, the entire 17.0.0.0/8 address block is assigned to Apple, so you can specify that range in your firewall rules.

Resetting the Push Notifications Permissions Alert on iOS

The first time a push-enabled app registers for push notifications, iOS asks the user if they wish to receive notifications for that app. Once the user has responded to this alert it is not presented again unless the device is restored or the app has been uninstalled for at least a day.

If you want to simulate a first-time run of your app, you can leave the app uninstalled for a day. You can achieve the latter without actually waiting a day by following these steps:

  1. Delete your app from the device.

  2. Turn the device off completely and turn it back on.

  3. Go to Settings > General > Date & Time and set the date ahead a day or more.

  4. Turn the device off completely again and turn it back on.

Viewing a Certificate Signing Request (CSR)

Part of configuring your App ID for push notifications is creating a certificate signing request, or CSR. Occasionally it's useful to look at the contents of a CSR, which you can do with the OpenSSL req command:

$ openssl req -noout -text -in server.csr

References

Local and Push Notification Programming Guide

App Distribution Guide

Understanding Notifications

Downloadables



Document Revision History


DateNotes
2013-09-24

Minor editorial changes.

2013-09-17

Updated the PersistentConnectionLogging configuration profile and instructions for resetting the push permissions alert. Other minor changes.

2013-03-29

Fixed link to PersistentConnectionLogging configuration profile.

2013-03-27

Expanded coverage of checking code signing entitlements and error responses.

2011-09-26

Update link to Entrust root certificate. Replace apsd logging configuration profile with a signed version.

2011-06-08

Added information about push notifications in Mac OS X Lion.

2010-09-29

New document that describes steps developers can take to troubleshoot sending and receiving of push notifications.