Background uploads with NSURLSession

Do you recommend me to use NSURLSessionUploadTask or another class?

NSURLSessionUploadTask

What do I have to do to continue all these uploads in background?

What do I have to do to continue all these uploads when app is not running?

Dump all the uploads into an NSURLSession background session.

For all of the first 3 points, do you recommend me to wrap NSURLSession objects in NSOperation ones in order to use an NSOperationQueue? Or iOS would use its own internal queue to properly handle NSURLSession objects?

That's hard to say. In general I'm a big fan of NSOperation but it's a tricky fit for NSURLSession background sessions because of the way that the background session interacts with your app's lifecycle. Specifically, a common sequence is this:

1. your app starts a bunch of background transfers

2. the user moves it into the background

3. the system suspends your app

4. the transfers continue in the background

5. the system terminates your app

6. the transfers continue in the background

7. the transfers complete

8. the system relaunches your app

If you're using an NSOperation to track each transfer, you have to reconstitute those operations when you're relaunched at step 8. Of course, you have to reconstitute something at step 8, it's just that reconstituting an NSOperation may be trickier than reconstituting some less complex object.

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks Quinn!

I understand your point concerning NSOperationQueue. I guess iOS has its own queuing system for network requests...

But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?

If yes, is there any limit in term of number of tasks?

Thank you

Best

But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?

Yes. The NSURLSession background download system will happily deal with a few hundred requests. It serialises them internally, so only a few of those requests hit the 'wire' at any time.

If yes, is there any limit in term of number of tasks?

There is no hard limit but I generally recommend that you avoid pushing things too far. IMO hundreds of requests are fine, thousands of requests are pushing things, tens of thousands of requests would be silly.

I generally recommend that, if you have to deal with thousands of individual items, you zip them up and transfer them as one resumable transfer. There are, however, some gotcha there:

o Doing this sort of thing requires sophisticated server-side support.

o NSURLSession background sessions automatically handle resumable downloads (if the server supports it). That's not true for uploads. If you want to implement a resumable upload, you'll have to get involved each time the connection 'tears'. For that reason, it makes sense to chunk your uploads into reasonably sized chunks, so the upload get make a bunch of progress, even in the presence of connection failures, before your app has to resume again.

Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hey Quinn -- I've been reading a lot of your posts for the past hour trying to understand some gotcha's I'm facing with my app. Basically I'm trying to upload ~30 large files in the background, when the user's phone is plugged in and connected to wifi. I wrote a file manager that is dispatching these one-by-one, but from what I'm reading it looks like the better approach would be to send them all to NSURLSession at once.

Given this, I want to run a completion handler when all the files are finished uploading. I read your post about the resume rate limiter. Since the resume time backs off exponentially, I'm worried that by the time the 30th file finishes uploading, I won't be able to resume my app to handle the upload's completion handler. Since I have no clue which file is going to finish first, I can't pass nil as the completion handler for 29 and an actual completion handler for the 30th. Can you provide any insight?

In general an NSURLSession background session won’t resume (or relaunch) an app in the background until all of the requests in that session have completed. So, if you dump all 30 requests into a session and then get put in the background by the user, you’ll resume exactly once, when all 30 requests are complete.

When that happens you’ll see a sequence of events like this:

1. App resumes (or relaunches)

   In the relaunch case you must remember to re-create your background session with the same identifier that you used originally. A common mistake is not re-creating the background session in the relaunch case. The best approach in most cases is to create that session in code called from

   -application:willFinishLaunchingWithOptions:

   .

2. System calls

   -application:handleEventsForBackgroundURLSession:completionHandler:

   , passing it a completion handler

3. NSURLSession calls

   -URLSession:task:didCompleteWithError:

   for each task that’s complete

4. NSURLSession calls

   -URLSessionDidFinishEventsForBackgroundURLSession:

   , at which point it’s appropriate to call the completion handler from step 2

IMPORTANT You can’t use NSURLSession’s convenience APIs, things like

-uploadTaskWithRequest:fromFile:completionHandler:

-uploadTaskWithRequest:fromFile:

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the help. I'm currently using the S3 transfer utility library (https://cocoapods.org/pods/AWSS3) so I'll have to figure out what that's doing under the hood. I may need to write my own implementation based on signed URLs. Will report back when I find out.

Hi,

I'm working on a similar iOS project, where I need to upload a very large video file (2 GB) to S3.

I tried setting up a single upload request to handle the entire thing, but as you pointed out, when there is a connection failure or when the user terminates the app, the upload transfer fails and I'd have to start over again.

If I were to split the 2 GB file into smaller 25 mb portions, I could add them all to my NSURLSession object to upload.

However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests? Does it have to upload all of the 25 mb files again or can it tell which ones have already been uploaded and only do the ones that haven't?

Thanks,

-Sreeni

However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests?

By “terminated”, do you mean:

• “Removed from the multitasking UI”, so that all of your uploads get cancelled?

• Terminated by the OS due to memory pressure, so that all of your uploads continue in the background?

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hi,

By terminated, I mean “Removed from the multitasking UI”. Given that the app resumes (or relaunches) only when all tasks in the session complete, is there no way of keeping track of which files have been become uploaded and which are still pending? If all of the uploads become cancelled when a user force quits, then I'm guessing I can't retrieve the tasks that haven't completed (and were cancelled) and only resume those.

Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed?

Thanks,

-Sreeni

Given that the app resumes (or relaunches) only when all tasks in the session complete …

To be clear, if the user removes your app from the multitasking UI then you won’t be resumed or relaunched in the background. The user will have to manually relaunch your app.

The behaviour for tasks that haven’t completed when the user removes the app from the multitasking UI varies by OS:

• Originally the entire session would just disappear; it would be like you’d never issued those tasks

• On current systems the tasks fail with

  NSURLErrorCancelled

I believe the cutover was with iOS 8, that is, iOS 7 have the first behaviour and iOS 8 and later have the second, although I’ve never sat down to check that.

I’ve also never looked at the behaviour for tasks that have completed. Based on my understanding of how things fit together I suspect they’ll also get

NSURLErrorCancelled

Regardless, the best way to recover from this situation is to talk to the server to see which of the uploads completed successfully. This is generally pretty easy to do via the

HEAD

Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed?

That’s true for downloads but not for uploads. There’s actually a couple of problems doing a single, resumable upload:

• With downloads, NSURLSession can automatically resume a failed download using HTTP standard technology (QA1761 gives an outline of how it works). For uploads there’s no way for NSURLSession to automatically restart the transfer, so it has to relaunch or resume your app to do the job. So, if you upload as a single file and the upload fails a bunch of times, you eventually provoke the ire of the resume rate limiter.

• There’s no way to tell NSURLSession to start a resume from some offset in a file. So if you have a large upload that fails, you have to make a copy of your upload file to remove the bytes at the front that have successfully been uploaded.

The alternative is to segment the file into relatively large chunks and upload those. This helps with both of these problems, at the cost of some code complexity.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the help! I think I know how to tackle the issue now.

Hi,

I have two more questions, if you can offer any advice:

1. If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean? Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi? What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)?

2. I setup and initiate around a 100 upload tasks to a NSURLSession object. If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks? I'm only allowing uploads over wifi. I've been seeing the following strange behavior:

1. I initate 100 upload tasks in a single NSURLSession

2. 10 uploads were successfully completed

3. I lose wifi connection

4. I gain wifi connection an hour later

5. The session would complete another 15 upload tasks

6. The session stops uploading anymore data (the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow).

Any help would be greatly appreciated,

Thanks,

-Sreeni

If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean?

To start, understand that the “discretionary property” isn’t an actual property, at least not one that you can see. Rather, it’s a value computer by NSURLSession based on a number of criteria, including:

• The type of session (standard vs background)

• The

  discretionary

  property of the configuration used to create that session

• Whether the app was in the foreground when the task was created

• Whether the app is in the foreground right now

Moreover, the specific effects of this ‘property’ are not guaranteed; they have changed in the past and I fully expect them to change in the future.

Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi?

In general, yes. However, it’s really up to the OS as to when it schedules discretionary tasks, and it’s possible that, for example, if the device ‘learns’ that it’s never on Wi-Fi, it may chose to run discretionary tasks over WWAN.

What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)?

Again, no property values change here but, yes, modern versions of iOS include the foreground state of the originating app when determining whether to run a discretionary request.

I setup and initiate around a 100 upload tasks to a NSURLSession object. If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks?

Again, the exact behaviour is not specified and depends on a bunch of things. But, yes, I would expect these tasks to eventually complete.

(the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow)

You shouldn’t need to guess here. If a task gets cancelled you should be told about it (via the standard task completion mechanism). You can also confirm whether a task is still around using

-getAllTasksWithCompletionHandler:

I expect what you’ll find is that the remaining tasks are still around, they’re just not running right now for some reason (lack of Wi-Fi, lack of power, resource budgets, and so on).

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hi,

This thread has helped me a lot but i am still seeing some strange behavior when I try to do background upload tasks The problem I am seeing occurs only for large file uploads (300-500 MBs). What I am observing is that iOS sometimes does NOT call didCompleteWithError even after my server sends a response. In the client logs I can see that the didSendBodyData method gets called several times. Which is expected, however it does not seem to stop even after sending all the bytes. I have also noticed that sometimes the taskId changes or the totalBytesSent gets reset. Which does not make any sense since I have not queued up another upload task. I only called resume once.

Also note: This behavior happens when the app is in the foreground. I have not attempted testing background uploads yet.

Everythin works as expected the file is small.(~50 Mbs). The didRecieveData method gets called along with the didCompleteWithError method.

Below are some code snippets of the background task:

//FileOperationsDelegate.mm

- (void)URLSession:(NSURLSession *)session
              task:(NSURLSessionTask *)task
   didSendBodyData:(int64_t)bytesSent
    totalBytesSent:(int64_t)totalBytesSent
totalBytesExpectedToSend:(int64_t)totalBytesExpectedToSend
{
    NSNumber* taskId = [NSNumber numberWithUnsignedInteger:task.taskIdentifier];

    AwLog(@"DidSendBodyData for taskID: %@  BytesSent: %lld TotalBytesSent: %lld ExpectedToSend: %lld\n", taskId, bytesSent, totalBytesSent, totalBytesExpectedToSend);

    if (updateFileProgressCallback != NULL)
    {
        updateFileProgressCallback(bytesSent, totalBytesSent);
    }
}

Here is by backgroundSession.

//FileOperationsDelegate.mm

- (NSURLSession *)backgroundSession
{

    static NSURLSession *session = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
       NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier: KEY_BACKGROUND_SESSSION_ID];

        configuration.discretionary = YES;
        configuration.sessionSendsLaunchEvents = YES;
        configuration.URLCache = nil;

        bool wifiOnly = [ClientConfiguration.sharedInstance isWifiOnlyEnabled];

        if (wifiOnly)
        {
            configuration.allowsCellularAccess = false;
        }

        NSOperationQueue* delegateMainQueue = [NSOperationQueue mainQueue];
        delegateMainQueue.maxConcurrentOperationCount = 1;

        session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue: delegateMainQueue];
    });
    return session;
}

Creating the FileUpload task

//FileUploader.mm

NSURLSession* backgroundSession = [FileOperationsSessionDelegate.sharedInstance backgroundSession];

                       NSMutableString *file = [[NSMutableString alloc] initWithString:absLocalPath];
                       [file insertString:@"file://" atIndex:0];

                       NSURL* fileToBeUploaded = [NSURL URLWithString:file];
                       NSURLSessionUploadTask* uploadTask = [backgroundSession uploadTaskWithRequest:urlRequest fromFile:fileToBeUploaded];

                       uploadTask.countOfBytesClientExpectsToSend = self.uploadSizeBytes;
                       uploadTask.countOfBytesClientExpectsToReceive = 450;


 ConnectionCallback uploadCompleteCallback = ^(NSURLResponse *  response, NSData * data,  NSError *  error) {
// A callback to complete the file upload. It will update the UI and notify the user 
// on how the upload went. 
}

  BytesUploadedCallback updateFileProgressCallback = ^(long bytesWritten, long totalBytesWritten)
// A callback that will update the progress bar.
                       };


                       [FileOperationsSessionDelegate.sharedInstance setUploadCallback:uploadCompleteCallback];
                       [FileOperationsSessionDelegate.sharedInstance setConnectionCallback:updateFileProgressCallback];;


                       [uploadTask resume];

Any help with this will be greatly apperiated.

Edit: I have attempted using a non background session and I observe the same behavior.

Thanks : )

Specs:

iOS version 12.3

Device: iPad Pro 11

Xcode: 10.1