Technical Note TN2265

Troubleshooting Push Notifications

Describes techniques you can use to resolve issues with sending and receiving push (remote) notifications in iOS and OS X.

Introduction
Issues with Receiving Push Notifications
Issues with Sending Push Notifications
Other Tips and Tricks
References
Downloadables
Document Revision History

Introduction

Many iOS and Mac applications present dynamic content delivered over the Internet. Push notifications (also known as remote notifications) are a way to let users or apps know that new or updated content is available.

This technote describes techniques you can use to resolve issues with sending and receiving push notifications.

Issues with Receiving Push Notifications

Registering for Push Notifications

In order to receive push notifications, an application must first register with the Apple Push Notification service (APNs or "push service"). Registration has four stages:

  1. On iOS, the application asks the user for permission to receive push notifications by calling the registerUserNotificationSettings: method of UIApplication.

  2. The application calls the registerForRemoteNotifications: method of UIApplication (iOS) or the method registerForRemoteNotificationTypes: of NSApplication (OS X).

  3. The application implements the application:didRegisterForRemoteNotificationsWithDeviceToken: method of UIApplicationDelegate (iOS) or NSApplicationDelegate (OS X) to receive the unique device token generated by the push service.

  4. The application implements the application:didFailToRegisterForRemoteNotificationsWithError: method of UIApplicationDelegate (iOS) or NSApplicationDelegate (OS X) to receive an error if the registration failed.

The application passes the device token to your provider as a non-object, binary value. The provider is your server that delivers dynamic content to your application.

Registering for push notifications is straightforward, but there are a few important details you need to be aware of.

No Delegate Callbacks

When the first push-capable app is installed, iOS or OS X attempts to establish a persistent network connection to the push service that will be shared by all push-capable apps on the system. If neither delegate callback application:didRegisterForRemoteNotificationsWithDeviceToken: nor application:didFailToRegisterForRemoteNotificationsWithError: is called, that means that this connection has not yet been established.

This is not necessarily an error condition. The system may not have Internet connectivity at all because it is out of range of any cell towers or Wi-Fi access points, or it may be in airplane mode. Instead of treating this as an error, your app should continue normally, disabling only that functionality that relies on push notifications.

Keep in mind that network availability can change frequently. Once the persistent connection to the push service succeeds, one of the previously-mentioned application delegate methods will be called.

On iOS, push notifications use the cellular data network whenever possible, even if the device is currently using Wi-Fi for other network activity such as web browsing or email. However, the push service will fall back to Wi-Fi if cellular data service isn't available.

If your iOS device is capable of using the cellular data network, check that it has an active cellular data plan. Turn off Wi-Fi in Settings and see if you can still browse the web with Safari, for example. On the other hand, if the push service is using Wi-Fi, any firewalls between your device or computer and the Internet must allow TCP traffic to and from port 5223.

Error Delegate Callback

The first time a push-capable application is run, the user is asked if they want to receive push notifications for this app. If the app isn't even offering the option of receiving push notifications, the chances are that it is missing its aps-environment (on iOS) or com.apple.developer.aps-environment (on OS X) code signing entitlement. Xcode takes this entitlement from the provisioning profile used when building the app, and it controls which push environment the app will connect to in order to receive remote notifications.

If an iOS app is running on a device, or if the app is a Mac app, the error method application:didFailToRegisterForRemoteNotificationsWithError: will be called if the code signing entitlements for accessing the push service are invalid. If you have not implemented this method in your application, you should do that as a way of checking that your aps-environment (iOS) or com.apple.developer.aps-environment (OS X) entitlement is present at runtime.

When using Xcode to submit an app, Xcode will re-sign it using a distribution code signing identity and associated provisioning profile. So the signature of the submitted app and its contents might be different than what's in the Xcode archive.

To check the signature of an app being submitted to the App Store:

  1. In the Xcode Organizer, instead of Upload to App Store..., do Export... followed by Save for iOS or Mac App Store Deployment. That will create a local copy of the app that would be submitted to the App Store.

  2. When asked to select a development team, choose the team you use when uploading to the App Store.

  3. When the Summary sheet appears, open the disclosure triangle under Binary and Entitlements to see the entitlements.

Check if there is an aps-environment (iOS) or com.apple.developer.aps-environment (OS X) entitlement shown.

If the aps-environment or com.apple.developer.aps-environment entitlement is missing, check which provisioning profile is listed on the Summary sheet. Click the gray arrow to the right of the profile name to show it in Finder.

Drag the profile to TextEdit so you can look at its contents, and search for Entitlements. If you don't see aps-environment or com.apple.developer.aps-environment in the entitlements, you've either not configured your app ID for production push notifications, or you haven't updated your distribution provisioning profile since configuring your app ID.

You probably created your distribution provisioning profile before you configured your App ID for push notifications in your developer account. Or. it might just be that the distribution provisioning profile installed on your build machine is older than the one in your account. In that case, delete the one on the build machine, go to Xcode > Preferences... > Accounts, select your team, and click View Details.... In the sheet that appears, click Download All.

For more details about how to configure your app for push notifications, please see Configuring Push Notifications. The key points are that you must have an explicit App ID and a distribution provisioning profile that references that App ID. If you have an explicit App ID already, enable that for push notifications by turning on the Push Notifications capability in Xcode.

Once you have a distribution provisioning profile installed on your build system with the correct push notifications entitlement, you will need to rebuild your app and re-submit it to the store as an update. You may be able to request an expedited review using this form.

Registration Succeeded But No Notifications Received

If your app has registered with the push service but it is not receiving notifications, the problem can be either on the device/computer side or on the provider side. Here are a few things to check on the device/computer side.

The device or computer may have lost its persistent connection to the push service and can't reconnect. Try quitting the app and relaunching it to see if registration completes the next time. (On iOS 4 and later on devices that support multitasking, you will need to force quit the app from the recents list.) If the registration does not complete, iOS or OS X has been unable to re-establish the persistent connection. You can troubleshoot this as described in the previous two sections.

Your app may have sent an incorrect device token to your provider. Your app should always ask for the device token by registering with the push service each time it is launched. Don't store a device token from your app and try to reuse it, because the token can change. Your provider should then pass that same token on to the push service.

Some Notifications Received, but Not All

If you are sending multiple notifications to the same device or computer within a short period of time, the push service will send only the last one.

Here's why. The device or computer acknowledges receipt of each notification. Until the push service receives that acknowledgment, it can only assume that the device or computer has gone off-line for some reason and stores the notification in the quality of service (QoS) queue for future redelivery. The round-trip network latency here is of course a major factor.

As described in the Local and Remote Notification Programming Guide, the QoS queue holds a single notification per app per device or computer. If the service receives another notification before the one in the queue is sent, the new notification overwrites the previous one.

If your notification payload contains the content-available key, your app will receive the notification if iOS or OS X determines it is energy-efficient to do so. If the energy or data budget for the device has been exceeded, your app will not receive any more notifications with the content-available key until the budget has been reset. This occurs once a day and cannot be changed by user or developer action. This throttle is disabled if the app is run from Xcode, so be sure to test your app by running it from the device to have the same user experience your customers will have.

All of this points out that the intent is that a notification indicates to an app that something of interest has changed on the provider, and the app should check in with the provider to get the details. Notifications should not contain data which isn't also available elsewhere, and they should also not be stateful.

Any push notification that isn't delivered immediately was queued for future redelivery because your device was not connected to the service. "Immediately" of course needs to take latency for your connection into account. Outlying cases would be beyond 60 seconds as APNs will time out at that point.

Observing Push Status Messages

If these tips don't resolve the issue, you can enable additional messages from the APNs daemon on the device or computer.

Enabling Push Status Messages on iOS

To enable logging on iOS, install the configuration profile PersistentConnectionLogging.mobileconfig on your device by either putting the file on a web server and downloading it using Safari on your device, or by sending it as an email attachment and opening the attachment in Mail on your device. You'll be prompted to install "APS/PC Logging". Download the configuration profile by clicking Companion File at the upper right corner of this technote.

Once the configuration profile has been installed, turn the device off completely and then turn it back on. Then exercise your app again while watching the console using the Xcode Organizer. Or, sync your device with iTunes and the log will be saved in ~/Library/Logs/CrashReporter/MobileDevice/<DEVICE_NAME>/PersistentConnection/.

Look for messages from the apsd process. Ideally you'll see Connected to courier x-courier.sandbox.push.apple.com where x is a small integer. That indicates that the device has successfully established its persistent connection to the push service.

You might on the other hand see Disconnecting in response to connection failure. That means that the persistent connection failed. In that case, the goal is to figure out what's going on with your network that's causing the connection failure. Check that no firewalls are blocking TCP traffic on port 5223.

The message connection set ignored topics means that the user chose to turn off notifications for the apps listed in the message. That will be followed by Sending filter message for enabled hashes which is where iOS actually sends the enabled and ignored topics to APNs.

The message Failed to parse JSON message payload indicates that the JSON dictionary in the notification payload is not in valid JSON format.

The message Received message for enabled topic <your app's CFBundleIdentifier> means that the device received a notification from the push service.

On OS X, some log messages refer to the hash of your app's CFBundleIdentifier. Here's how to compute this SHA-1 hash:

$ echo -n "<your CFBundleIdentifier>" | openssl dgst -sha1

If things appear to look normal in the log but you're still not receiving notifications, try turning off the Notifications switch in Settings, and then turn it back on. That will try to re-establish the device's persistent connection with APNs.

To remove the configuration profile, go to Settings > General > Profiles, tap "APS/PC Logging", then tap Remove. Turn the device off completely and turn it back on.

Enabling Push Status Messages on OS X

To obtain the current state of the APNs daemon on OS X, use this command:

$ /System/Library/PrivateFrameworks/ApplePushService.framework/apsctl status

To enable logging on OS X, use the following commands:

$ sudo touch /Library/Logs/apsd.log
$ sudo defaults write /Library/Preferences/com.apple.apsd APSWriteLogs -bool TRUE
$ sudo defaults write /Library/Preferences/com.apple.apsd APSLogLevel -int 7
$ sudo killall apsd

The logs are stored in /Library/Logs/apsd.log.

Issues with Sending Push Notifications

If your app is not receiving push notifications, and everything looks correct on the device or computer, here are some things to check on the server side.

Problems Connecting to the Push Service

One possibility is that your server is unable to connect to the push service. This can mean that you don't have the certificate chain needed for TLS/SSL to validate the connection to the service. In addition to the SSL identity (certificate and associated private key) created by Member Center, you should also install the root certificate on your provider. This allows TLS/SSL to verify the full APNs server cert chain.

The root certificate depends on which APNs API you’re using. If you’re using the HTTP/2 APNs provider API, the root certificate is the GeoTrust Global CA root certificate. You can download this from GeoTrust’s site.

If you’re using the binary provider API, the root certificate is the Entrust.net Certification Authority (2048) root certificate. You can download it from Entrust's site.

Also verify that these identities are installed in the correct location for your provider and that your provider has permission to read them.

You can test the TLS/SSL handshake using the OpenSSL s_client command, like this for the HTTP/2 APNs provider API:

$ openssl s_client -connect api.development.push.apple.com:443 -cert YourSSLCertAndPrivateKey.pem -debug -showcerts -CAfile server-ca-cert.pem

Or, for the binary provider API:

$ openssl s_client -connect gateway.sandbox.push.apple.com:2195 -cert YourSSLCertAndPrivateKey.pem -debug -showcerts -CAfile server-ca-cert.pem

where server-ca-cert.pem is the Entrust CA (2048) root certificate.

Be sure the SSL identity and the hostname are the correct ones for the push environment you're testing. You can configure your App ID in your developer account separately for the sandbox and production environment. If you don’t request a universal push certificate, you will be issued a separate identity for each environment.

Using the sandbox SSL identity to try to connect to the production environment will return an error like this:

CRITICAL | 14:48:40.304061 | Exception creating ssl connection to Apple: [Errno 1] _ssl.c:480: error:14094414:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate revoked

One common OpenSSL error is verify error:num=20:unable to get local issuer certificate.

That message means that one or more of the certificates from the server could not be verified because the issuer (root or intermediate) certificates were not found.

The CAfile argument to s_client specifies the trusted root certificates to use to verify the server certificate. The trusted root certificate for the push servers is the GeoTrust or Entrust root certificate mentioned previously.

If you can't even open a connection to APNs, perhaps your APNs TLS/SSL certificate has expired. These certificates are valid for one year but production APNs certificates can be renewed at any time.

Another possibility is that you've connected too many times to APNs and further connections have been temporarily blocked. As is documented in the Local and Remote Notification Programming Guide, developers are expected to open a connection and leave it open. If a connection is opened and closed repeatedly, APNs will treat it as a denial of service attack and block connections for a period of time.

This temporary block will expire if no connection attempts are made for about one hour.

Yet another possibility is that there is a firewall blocking access to the ports used by APNs. Please see IP Address Range Used by the Push Service for details. Try running a telnet command on your server to see if the server can reach APNs, like this:

$ telnet 1-courier.push.apple.com 5223
$ telnet gateway.sandbox.push.apple.com 2195
$ telnet gateway.push.apple.com 2195

Handling Malformed Notifications

The simple notification format drops the connection if the push service receives a notification that is incorrect in some way. Your provider may see this as an EPIPE or broken pipe error in response to sending a notification. On the other hand, the enhanced notification format will send an error response with more detailed information about what was wrong with the notification before dropping the connection. Be sure your provider catches and handles these conditions properly.

Many push notification servers do not handle error responses or dropped connections robustly. An easy way to check this is to intentionally send a notification to a sandbox environment device token, assuming your server is communicating with the production push environment. Doing that should return an invalid token response and drop the connection. To learn more about checking error responses from the push service, please see Push Notification Throughput and Error Checking.

The most common problem is an invalid device token. If the token came from the sandbox environment, such as when you are testing a development build in house, you can't send it to the production push service. Each push environment will issue a different token for the same device or computer. If you do send a device token to the wrong environment, the push service will see that as an invalid token and discard the notification.

A device token is also valid only for the app that received it during registration. If you attempt to use that token to send a notification to a different app (different bundle identifier) it will be rejected as invalid.

The user may have deleted your app from their device or computer. You should check the feedback service at least once a day for device tokens that are no longer active.

Other possible issues might be sending a payload longer than 2048 bytes, your payload might not be formatted correctly, or perhaps your JSON dictionary has incorrect syntax.

An occasional disconnect while your provider is idle is nothing to be concerned about; just re-establish the connection and carry on. If one of the push servers is down, the load balancing mechanism will transparently direct your new connection to another server assuming you connect by hostname and not by static IP address.

Using the Enhanced Notification Format

If your provider is using the original simple notification format, consider migrating a newer API. The newer APIs provide more control over the priority and expiration of notifications and also returns more detailed error responses if a problem occurs with a particular notification.

You can read more about the HTTP/2 APNs provider API in the Local and Remote Notification Programming Guide.

Issues with Using the Feedback Service

If you remove your app from your device or computer and then send a push notification to it, you would expect to have the device token rejected, and the invalidated device token should appear on the feedback service. However, if this was the last push-enabled app on the device or computer, it will not show up in the feedback service. This is because deleting the last app tears down the persistent connection to the push service before the notice of the deletion can be sent.

You can work around this by leaving at least one push-enabled app on the device or computer in order to keep the persistent connection up. To keep the persistent connection to the production environment up, just install any free push-enabled app from the App Store and you should then be able to delete your app and see it appear in the feedback service.

Recall that each push environment has its own persistent connection. So to keep the persistent connection to the sandbox environment up, install another development push-enabled app.

Other Tips and Tricks

Push Notification Throughput and Error Checking

There are no caps or batch size limits for using APNs. The iOS 6.1 press release stated that APNs has sent over 4 trillion push notifications since it was established. It was announced at WWDC 2012 that APNs is sending 7 billion notifications daily.

If you're seeing throughput lower than 9,000 notifications per second, your server might benefit from improved error handling logic.

Here's how to check for errors when using the enhanced notification format. Keep writing until a write fails. If the stream is ready for writing again, resend the notification and keep going. If the stream isn't ready for writing, see if the stream is available for reading.

If it is, read everything available from the stream. If you get zero bytes back, the connection was closed because of an error such as an invalid command byte or other parsing error. If you get six bytes back, that's an error response that you can check for the response code and the ID of the notification that caused the error. You'll need to send every notification following that one again.

Once everything has been sent, do one last check for an error response.

It can take a while for the dropped connection to make its way from APNs back to your server just because of normal latency. It's possible to send over 500 notifications before a write fails because of the connection being dropped. Around 1,700 notifications writes can fail just because the pipe is full, so just retry in that case once the stream is ready for writing again.

Now, here's where the tradeoffs get interesting. You can check for an error response after every write, and you'll catch the error right away. But this causes a huge increase in the time it takes to send a batch of notifications.

Device tokens should almost all be valid if you've captured them correctly and you're sending them to the correct environment. So it makes sense to optimize assuming failures will be rare. You'll get way better performance if you wait for write to fail or the batch to complete before checking for an error response, even counting the time to send the dropped notifications again.

None of this is really specific to APNs, it applies to most socket-level programming.

If your development tool of choice supports multiple threads or interprocess communication, you could have a thread or process waiting for an error response all the time and let the main sending thread or process know when it should give up and retry.

IP Address Range Used by the Push Service

Push providers, iOS devices, and Mac computers are often behind firewalls. To send notifications, you will need to allow inbound and outbound TCP packets over port 443 for the HTTP/2 provider API or port 2195 for the binary provider API.

To reach the feedback service, you will need to allow inbound and outbound TCP packets over port 2196.

Devices and computers connecting to the push service over Wi-Fi will need to allow inbound and outbound TCP packets over port 5223, or port 443 for a fallback when devices can’t reach APNs on port 5223.

OS X systems will also need to allow inbound and outbound TCP traffic over port 80.

The IP address range for the push service is subject to change; the expectation is that providers will connect by hostname rather than IP address. The push service uses a load balancing scheme that yields a different IP address for the same hostname. However, the entire 17.0.0.0/8 address block is assigned to Apple, so you can specify that range in your firewall rules.

Resetting the Push Notifications Permissions Alert on iOS

The first time a push-enabled app registers for push notifications, iOS asks the user if they wish to receive notifications for that app. Once the user has responded to this alert it is not presented again unless the device is restored or the app has been uninstalled for at least a day.

If you want to simulate a first-time run of your app, you can leave the app uninstalled for a day. You can achieve the latter without actually waiting a day by following these steps:

  1. Delete your app from the device.

  2. Turn the device off completely and turn it back on.

  3. Go to Settings > General > Date & Time and set the date ahead a day or more.

  4. Turn the device off completely again and turn it back on.

Viewing a Certificate Signing Request (CSR)

Part of configuring your App ID for push notifications is creating a certificate signing request, or CSR. Occasionally it's useful to look at the contents of a CSR, which you can do with the OpenSSL req command:

$ openssl req -noout -text -in server.csr

References

Local and Remote Notification Programming Guide

App Distribution Guide

Understanding Notifications

Downloadables



Document Revision History


DateNotes
2016-07-28

Updated links to Local and Remote Notification Programming Guide.

2016-07-25

Updated for iOS 8 and later. Refreshed logging profile.

2016-07-20

Updated for iOS 8 and later. Refreshed logging profile.

 

Updated for iOS 8 and later. Refreshed logging profile.

2014-10-01

Updated maximum payload size.

2013-09-24

Minor editorial changes.

2013-09-17

Updated the PersistentConnectionLogging configuration profile and instructions for resetting the push permissions alert. Other minor changes.

2013-03-29

Fixed link to PersistentConnectionLogging configuration profile.

2013-03-27

Expanded coverage of checking code signing entitlements and error responses.

2011-09-26

Update link to Entrust root certificate. Replace apsd logging configuration profile with a signed version.

2011-06-08

Added information about push notifications in Mac OS X Lion.

2010-09-29

New document that describes steps developers can take to troubleshoot sending and receiving of push notifications.