Explore the integration of media technologies within your app. Discuss working with audio, video, camera, and other media functionalities.

All subtopics
Posts under Media Technologies topic

Post

Replies

Boosts

Views

Activity

How to upload large videos with PHAssetResourceUploadJobChangeRequest?
I'm implementing a PHBackgroundResourceUploadExtension to back up photos and videos from the user's library to our cloud storage service. Our existing upload infrastructure uses chunked uploads for large files (splitting videos into smaller byte ranges and uploading each chunk separately). This approach: Allows resumable uploads if interrupted Stays within server-side request size limits Provides granular progress tracking Looking at the PHAssetResourceUploadJobChangeRequest.createJob(destination:resource:) API, I don't see a way to specify byte ranges or create multiple jobs for chunks of the same resource. Questions: Does the system handle large files (1GB+) automatically under the hood, or is there a recommended maximum file size for a single upload job? Is there a supported pattern for chunked/resumable uploads, or should the destination URL endpoint handle the entire file in one request? If our server requires chunked uploads (e.g., BITS protocol with CreateSession → Fragment → CloseSession), is this extension the right mechanism, or should we use a different approach for large videos? Any guidance on best practices for large asset uploads would be greatly appreciated. Environment: iOS 26, Xcode 18
5
1
1.5k
Jun ’26
MusicKit.js / WKWebView: Are audible hitches during seek a known WebKit or Apple Media Stack limitation?
We are building a musician practice application that uses MusicKit.js inside a WKWebView on iOS and macOS. One of our core use cases is musical looping, where playback repeatedly seeks back to a loop start position while a song is playing. A typical loop length is 1-4 seconds, making interruptions that might be acceptable for general media playback very noticeable in our use case. For looping we currently use MusicKit’s seekToTime() API. We have also experimented with audioElement.fastSeek() and directly setting audioElement.currentTime. In all cases, seeking to a nearby position during playback produces an audible hitch or pause, even when the destination appears to already be buffered. We have observed similar interruptions when changing the playback rate during playback. For comparison, the same JavaScript bridge architecture running in a Chromium-based WebView on Android exhibits significantly smoother seek behavior and supports near-seamless looping. At this point, the WebView media stack appears to be the primary platform difference we have identified. We are currently using MusicKit.js v2. We are aware that v3 is available, but have not found anything suggesting this behavior has changed. As part of our investigation, we also experimented with Apple’s native MusicKit APIs. We observed that seeking and playback-rate changes can still produce audible interruptions, although playback-rate changes behave noticeably better than our MusicKit.js / HTML audio implementation. This led us to wonder whether the behavior originates from the underlying Apple media pipeline rather than MusicKit.js itself. Our questions are: Is it expected that seeking during playback, whether through seekToTime(), fastSeek(), or currentTime, causes the playback pipeline to be interrupted or rebuilt on Apple platforms? Are there known differences in WebKit’s media pipeline, buffering, or seek handling that would explain why the same implementation exhibits smoother seek behavior in a Chromium-based WebView? If so, are there recommended WebKit, WKWebView, or MusicKit.js techniques to achieve smoother, near-seamless looping? Given that we observe similar, though less pronounced, behavior when using native MusicKit APIs, is this a known characteristic or limitation of the underlying Apple media stack? If so, are there any APIs or implementation patterns recommended for applications that require tight musical looping and low-latency seeks? Any guidance would be greatly appreciated. Our goal is to provide musicians with seamless looping for practice and transcription, where even very small interruptions are noticeable.
1
0
368
Jun ’26
CIContext memoryLimit tips?
I would like to use the memory limit during interactive editing on iOS to decrease jetsam risk. However, I'm unsure of what to set it to. Do you have a recommendation as a fraction of physical memory? I have set the extended virtual memory entitlement, but I don't know what fraction of physical memory (or more) that entitlement enables - perhaps I should use a larger memory limit for M1-based iPads?
6
1
440
Jun ’26
List of RAW 9 bugs
Thanks for adding CoreML into the Apple RAW decoder. I'm excited with what it can bring. But I'm having a number of issues with it. My code base is written in Objective-C, and I'm using my custom MTKView. Now when I enabled RAW 9 support, the render is extremely slow (instant vs seconds). The UI would appear to hang with a spinning color wheel. It feels like CoreML processing is being delegated in the rendering thread, which runs on the main queue. How can I improve performance? As someone has reported, there is a black horizontal line in the middle of the image. The image that I was testing came from Fuji X-T5. If I enable EDR (Extended Dynamic Range), RAW 9 stops rendering, and will return ANE error. My app does HDR rendering of RAW files. I cannot get something like Sony A7R IV ARW image to load. Same ANE error. Is there a memory setting I need to tweak? When I render the X-T5 image, the colors appear to be overly saturated, which is different from the result of RAW 8. I cannot speak for the color rendering of other types of RAW files. I was told code base with Catalyst (and perhaps Swift) works without issue, other than being slow (except for large RAW files), but there seems to be an issue with Objective-C?
2
0
415
Jun ’26
CIContext memoryTarget defaults and Extended Virtual Addressing on macOS
Is the memoryLimit option mentioned in WWDC26 session 305 the same option as CIContextOption.memoryTarget? The session says the default memory target on iOS is 256 MB. What is the default memoryTarget on macOS? Also, the session recommends the Extended Virtual Addressing entitlement for interactive RAW editing. That entitlement does not appear to be available for macOS, so does Core Image automatically use a larger intermediate cache budget on macOS? Should memoryTarget be used only for export contexts, or can/should it also be tuned for interactive editing contexts with cacheIntermediates enabled? Thanks!
9
0
386
Jun ’26
List of RAW 9 bugs
Thanks for adding CoreML into the Apple RAW decoder. I'm excited with what it can bring. But I'm having a number of issues with it. My code base is written in Objective-C, and I'm using my custom MTKView. Now when I enabled RAW 9 support, the render is extremely slow (instant vs seconds). The UI would appear to hang with a spinning color wheel. It feels like CoreML processing is being delegated in the rendering thread, which runs on the main queue. How can I improve performance? As someone has reported, there is a black horizontal line in the middle of the image. The image that I was testing came from Fuji X-T5. If I enable EDR (Extended Dynamic Range), RAW 9 stops rendering, and will return ANE error. My app does HDR rendering of RAW files. I cannot get something like Sony A7R IV ARW image to load. Same ANE error. Is there a memory setting I need to tweak? When I render the X-T5 image, the colors appear to be overly saturated, which is different from the result of RAW 8. I cannot speak for the color rendering of other types of RAW files. I was told code base with Catalyst (and perhaps Swift) works without issue, other than being slow (except for large RAW files), but there seems to be an issue with Objective-C?
0
0
248
Jun ’26
Interactive Performance Tips?
Aside from setting inputScaleFactor and using cacheIntermediates, what are the most important things to do to get great interactive performance with Core Image (both when constructing a CIImage from URL and using CIRAWFilter). Perhaps CIImage lifetimes or how they are mutated? Alternatively, what patterns (again other than inputScaleFactor / cacheIntermediates) most hurt performance?
2
0
305
Jun ’26
Core Image intermediate rendering for full-extent lower-resolution kernel inputs
I render images using a tiled/LOD pipeline. The final render usually requests only the visible ROI at the current display scale, which works well for local filters. The difficult case is filters that need a lower-resolution full-extent or larger-extent input, such as guided filters, blurs, histograms/statistics, and reduce passes. If that dependency remains inside the same CI graph, ROI propagation can force larger-extent evaluation of the upstream image, defeating small-ROI rendering. My current approach is to render those intermediates in a separate lower-resolution CI render pass, cache the result, and feed the rendered intermediate back into the final graph. Is this the recommended architecture, or is there a better Core Image structure for expressing this kind of intermediate render/cache barrier without manually creating and rendering a separate intermediate graph? Thanks!
2
0
389
Jun ’26
CIRAWFilter isHighlightRecoveryEnabled
Could you provide a more detailed description of how isHighlightRecoveryEnabled affects the image, and how it differs from isGamutMappingEnabled? I’d like to explain these controls accurately to my users. Does highlight recovery run before or after linearSpaceFilter, and should it be disabled when generating scene-referred linear output? Thanks!
4
0
382
Jun ’26
CI_PRINT_TREE: debugging intermediate cache hits
In WWDC20 it was mentioned that missing compileTime/renderTime (or no intermediate image with the dump-intermediates option) can indicate a cached intermediate. I have also seen renderTime=cacheHit on passes that are clearly rendering intermediates, and an actual renderTime is only properly shown on the final pass which always seems to equal the final time render time. In compileTime it says cacheMiss. Which fields should be considered authoritative for tracking intermediate cache hits versus actual rendering? Thanks!
3
0
264
Jun ’26
CIRAWFilter isDraftModeEnabled
In my testing, enabling isDraftModeEnabled improves decode performance for very large non-RAW images. Does it also affect decoding performance or quality for RAW images? Are there non-RAW formats that benefit more than others, or not at all, with draft mode? For non-RAW images, is it generally recommended to load them through CIRAWFilter or CIImage? If CIImage is preferred, is there an equivalent to isDraftModeEnabled for CIImage initialization? Conversely, if CIRAWFilter is acceptable for non-RAW images, how can I get the expandToHDR behavior that is available through CIImage options? Thanks!
3
0
305
Jun ’26
RAW Boost filter / EDR
When processing RAW images with CIRAWFilter's extendedDynamicRange, does the boost filter’s behavior change? How does boost handle extended range values and does boost have a limit before clamping? Also, in the RAW EDR case, should we still use the linearSpace filter or is it now fine to run filters post-boost (assuming the filters are linear-dependent and can run post-boost)
2
0
192
Jun ’26
CIImage compression to JPEG
I am trying to compress a CIImage below a certain file size threshold when creating a JPEG. Currently, I am handling this in, potentially, multiple passes. In the first pass, I am essentially calling:
encodeJPEG(CIImage, quality: 0.85, colorSpace: CIImage.colorSpace ?? CGColorSpace(name: CGColorSpace.sRGB)!)
I am then checking the image size. If the image is still too big, I try to estimate how much I need to reduce it and try again until the image size is small enough. My question: Is there a more efficient way of doing this?
3
0
216
Jun ’26
Core Image processing for a photo overlaid text effect
The iPhone Lock Screen lets you display a small time and date at the top of the display that has a neat image processing effect where the text takes on the underlying colors with a blurred gradient while still maintaining legibility appearing largely white. In this screenshot you can see the white text features blue colors at the top left and transitions to a purple color on the right due to the underlying vibrant blue and purple blocks. Can you share how this text effect is achieved using Core Image?
1
0
209
Jun ’26
CIRAWFilter decoderVersion scope and RAW 9
I’m really excited about RAW 9, great work! Since RAW 9 is not enabled by default, are there any recommended use cases where it makes sense to select it by default, if supported? For compatibility handling, should I treat decoderVersion as covering all processing and adjustments performed by CIRAWFilter, or does it mostly affect the RAW decoding stage? When switching decoder versions, should I re-evaluate which CIRAWFilter adjustments are supported? Also, the macOS 26 headers appear to expose version9 and version9DNG. Is RAW 9 actually supported on macOS 26 runtimes? Thanks!
2
0
259
Jun ’26
CIRAWFilter despeckleAmount
CIRAWFilter documentation lists despeckleAmount as available on macOS 12.0+, but I cannot find this property in the macOS 26 headers. Could you provide a detailed description of how despeckleAmount affects the image so I can explain this adjustment accurately to my users? Does it run before or after linearSpaceFilter, and what should it be set to when generating scene-referred linear output? Thanks!
2
1
253
Jun ’26
RAW 9: High watermark for memory?
What is the high water mark for rendering an image in RAW 9 vs. RAW 8? I realize it depends on the megapixels of the render, but what can you share memory wise, especially compared to RAW 8, perhaps as a multiple? I'm mostly concerned about iOS in case that's relevant.
1
1
220
Jun ’26
How to upload large videos with PHAssetResourceUploadJobChangeRequest?
I'm implementing a PHBackgroundResourceUploadExtension to back up photos and videos from the user's library to our cloud storage service. Our existing upload infrastructure uses chunked uploads for large files (splitting videos into smaller byte ranges and uploading each chunk separately). This approach: Allows resumable uploads if interrupted Stays within server-side request size limits Provides granular progress tracking Looking at the PHAssetResourceUploadJobChangeRequest.createJob(destination:resource:) API, I don't see a way to specify byte ranges or create multiple jobs for chunks of the same resource. Questions: Does the system handle large files (1GB+) automatically under the hood, or is there a recommended maximum file size for a single upload job? Is there a supported pattern for chunked/resumable uploads, or should the destination URL endpoint handle the entire file in one request? If our server requires chunked uploads (e.g., BITS protocol with CreateSession → Fragment → CloseSession), is this extension the right mechanism, or should we use a different approach for large videos? Any guidance on best practices for large asset uploads would be greatly appreciated. Environment: iOS 26, Xcode 18
Replies
5
Boosts
1
Views
1.5k
Activity
Jun ’26
MusicKit.js / WKWebView: Are audible hitches during seek a known WebKit or Apple Media Stack limitation?
We are building a musician practice application that uses MusicKit.js inside a WKWebView on iOS and macOS. One of our core use cases is musical looping, where playback repeatedly seeks back to a loop start position while a song is playing. A typical loop length is 1-4 seconds, making interruptions that might be acceptable for general media playback very noticeable in our use case. For looping we currently use MusicKit’s seekToTime() API. We have also experimented with audioElement.fastSeek() and directly setting audioElement.currentTime. In all cases, seeking to a nearby position during playback produces an audible hitch or pause, even when the destination appears to already be buffered. We have observed similar interruptions when changing the playback rate during playback. For comparison, the same JavaScript bridge architecture running in a Chromium-based WebView on Android exhibits significantly smoother seek behavior and supports near-seamless looping. At this point, the WebView media stack appears to be the primary platform difference we have identified. We are currently using MusicKit.js v2. We are aware that v3 is available, but have not found anything suggesting this behavior has changed. As part of our investigation, we also experimented with Apple’s native MusicKit APIs. We observed that seeking and playback-rate changes can still produce audible interruptions, although playback-rate changes behave noticeably better than our MusicKit.js / HTML audio implementation. This led us to wonder whether the behavior originates from the underlying Apple media pipeline rather than MusicKit.js itself. Our questions are: Is it expected that seeking during playback, whether through seekToTime(), fastSeek(), or currentTime, causes the playback pipeline to be interrupted or rebuilt on Apple platforms? Are there known differences in WebKit’s media pipeline, buffering, or seek handling that would explain why the same implementation exhibits smoother seek behavior in a Chromium-based WebView? If so, are there recommended WebKit, WKWebView, or MusicKit.js techniques to achieve smoother, near-seamless looping? Given that we observe similar, though less pronounced, behavior when using native MusicKit APIs, is this a known characteristic or limitation of the underlying Apple media stack? If so, are there any APIs or implementation patterns recommended for applications that require tight musical looping and low-latency seeks? Any guidance would be greatly appreciated. Our goal is to provide musicians with seamless looping for practice and transcription, where even very small interruptions are noticeable.
Replies
1
Boosts
0
Views
368
Activity
Jun ’26
CIContext memoryLimit tips?
I would like to use the memory limit during interactive editing on iOS to decrease jetsam risk. However, I'm unsure of what to set it to. Do you have a recommendation as a fraction of physical memory? I have set the extended virtual memory entitlement, but I don't know what fraction of physical memory (or more) that entitlement enables - perhaps I should use a larger memory limit for M1-based iPads?
Replies
6
Boosts
1
Views
440
Activity
Jun ’26
List of RAW 9 bugs
Thanks for adding CoreML into the Apple RAW decoder. I'm excited with what it can bring. But I'm having a number of issues with it. My code base is written in Objective-C, and I'm using my custom MTKView. Now when I enabled RAW 9 support, the render is extremely slow (instant vs seconds). The UI would appear to hang with a spinning color wheel. It feels like CoreML processing is being delegated in the rendering thread, which runs on the main queue. How can I improve performance? As someone has reported, there is a black horizontal line in the middle of the image. The image that I was testing came from Fuji X-T5. If I enable EDR (Extended Dynamic Range), RAW 9 stops rendering, and will return ANE error. My app does HDR rendering of RAW files. I cannot get something like Sony A7R IV ARW image to load. Same ANE error. Is there a memory setting I need to tweak? When I render the X-T5 image, the colors appear to be overly saturated, which is different from the result of RAW 8. I cannot speak for the color rendering of other types of RAW files. I was told code base with Catalyst (and perhaps Swift) works without issue, other than being slow (except for large RAW files), but there seems to be an issue with Objective-C?
Replies
2
Boosts
0
Views
415
Activity
Jun ’26
RAW 9 / RAW 9.dng
What is the difference (if any) between RAW 9 and RAW 9.DNG in terms of quality, noise reduction, and color?
Replies
2
Boosts
0
Views
418
Activity
Jun ’26
CIContext memoryTarget defaults and Extended Virtual Addressing on macOS
Is the memoryLimit option mentioned in WWDC26 session 305 the same option as CIContextOption.memoryTarget? The session says the default memory target on iOS is 256 MB. What is the default memoryTarget on macOS? Also, the session recommends the Extended Virtual Addressing entitlement for interactive RAW editing. That entitlement does not appear to be available for macOS, so does Core Image automatically use a larger intermediate cache budget on macOS? Should memoryTarget be used only for export contexts, or can/should it also be tuned for interactive editing contexts with cacheIntermediates enabled? Thanks!
Replies
9
Boosts
0
Views
386
Activity
Jun ’26
List of RAW 9 bugs
Thanks for adding CoreML into the Apple RAW decoder. I'm excited with what it can bring. But I'm having a number of issues with it. My code base is written in Objective-C, and I'm using my custom MTKView. Now when I enabled RAW 9 support, the render is extremely slow (instant vs seconds). The UI would appear to hang with a spinning color wheel. It feels like CoreML processing is being delegated in the rendering thread, which runs on the main queue. How can I improve performance? As someone has reported, there is a black horizontal line in the middle of the image. The image that I was testing came from Fuji X-T5. If I enable EDR (Extended Dynamic Range), RAW 9 stops rendering, and will return ANE error. My app does HDR rendering of RAW files. I cannot get something like Sony A7R IV ARW image to load. Same ANE error. Is there a memory setting I need to tweak? When I render the X-T5 image, the colors appear to be overly saturated, which is different from the result of RAW 8. I cannot speak for the color rendering of other types of RAW files. I was told code base with Catalyst (and perhaps Swift) works without issue, other than being slow (except for large RAW files), but there seems to be an issue with Objective-C?
Replies
0
Boosts
0
Views
248
Activity
Jun ’26
Interactive Performance Tips?
Aside from setting inputScaleFactor and using cacheIntermediates, what are the most important things to do to get great interactive performance with Core Image (both when constructing a CIImage from URL and using CIRAWFilter). Perhaps CIImage lifetimes or how they are mutated? Alternatively, what patterns (again other than inputScaleFactor / cacheIntermediates) most hurt performance?
Replies
2
Boosts
0
Views
305
Activity
Jun ’26
Core Image intermediate rendering for full-extent lower-resolution kernel inputs
I render images using a tiled/LOD pipeline. The final render usually requests only the visible ROI at the current display scale, which works well for local filters. The difficult case is filters that need a lower-resolution full-extent or larger-extent input, such as guided filters, blurs, histograms/statistics, and reduce passes. If that dependency remains inside the same CI graph, ROI propagation can force larger-extent evaluation of the upstream image, defeating small-ROI rendering. My current approach is to render those intermediates in a separate lower-resolution CI render pass, cache the result, and feed the rendered intermediate back into the final graph. Is this the recommended architecture, or is there a better Core Image structure for expressing this kind of intermediate render/cache barrier without manually creating and rendering a separate intermediate graph? Thanks!
Replies
2
Boosts
0
Views
389
Activity
Jun ’26
CIRAWFilter isHighlightRecoveryEnabled
Could you provide a more detailed description of how isHighlightRecoveryEnabled affects the image, and how it differs from isGamutMappingEnabled? I’d like to explain these controls accurately to my users. Does highlight recovery run before or after linearSpaceFilter, and should it be disabled when generating scene-referred linear output? Thanks!
Replies
4
Boosts
0
Views
382
Activity
Jun ’26
CI_PRINT_TREE: debugging intermediate cache hits
In WWDC20 it was mentioned that missing compileTime/renderTime (or no intermediate image with the dump-intermediates option) can indicate a cached intermediate. I have also seen renderTime=cacheHit on passes that are clearly rendering intermediates, and an actual renderTime is only properly shown on the final pass which always seems to equal the final time render time. In compileTime it says cacheMiss. Which fields should be considered authoritative for tracking intermediate cache hits versus actual rendering? Thanks!
Replies
3
Boosts
0
Views
264
Activity
Jun ’26
RAW 9 bugs
Here are two additional RAW 9 bugs: Horizontal Line in RAW 9 decode: FB23076983 CIRAWFilter.scaleFactor not respected in RAW 9 for some RAW formats: FB23076984
Replies
1
Boosts
1
Views
280
Activity
Jun ’26
CIRAWFilter isDraftModeEnabled
In my testing, enabling isDraftModeEnabled improves decode performance for very large non-RAW images. Does it also affect decoding performance or quality for RAW images? Are there non-RAW formats that benefit more than others, or not at all, with draft mode? For non-RAW images, is it generally recommended to load them through CIRAWFilter or CIImage? If CIImage is preferred, is there an equivalent to isDraftModeEnabled for CIImage initialization? Conversely, if CIRAWFilter is acceptable for non-RAW images, how can I get the expandToHDR behavior that is available through CIImage options? Thanks!
Replies
3
Boosts
0
Views
305
Activity
Jun ’26
RAW Boost filter / EDR
When processing RAW images with CIRAWFilter's extendedDynamicRange, does the boost filter’s behavior change? How does boost handle extended range values and does boost have a limit before clamping? Also, in the RAW EDR case, should we still use the linearSpace filter or is it now fine to run filters post-boost (assuming the filters are linear-dependent and can run post-boost)
Replies
2
Boosts
0
Views
192
Activity
Jun ’26
Grounding and bounding boxes and segmentation
Which of the Vision framework‘s support for grounding/bounding boxes/segmentation carry over to the iOS 27 on-device Foundation Model image support?
Replies
1
Boosts
0
Views
209
Activity
Jun ’26
CIImage compression to JPEG
I am trying to compress a CIImage below a certain file size threshold when creating a JPEG. Currently, I am handling this in, potentially, multiple passes. In the first pass, I am essentially calling:
encodeJPEG(CIImage, quality: 0.85, colorSpace: CIImage.colorSpace ?? CGColorSpace(name: CGColorSpace.sRGB)!)
I am then checking the image size. If the image is still too big, I try to estimate how much I need to reduce it and try again until the image size is small enough. My question: Is there a more efficient way of doing this?
Replies
3
Boosts
0
Views
216
Activity
Jun ’26
Core Image processing for a photo overlaid text effect
The iPhone Lock Screen lets you display a small time and date at the top of the display that has a neat image processing effect where the text takes on the underlying colors with a blurred gradient while still maintaining legibility appearing largely white. In this screenshot you can see the white text features blue colors at the top left and transitions to a purple color on the right due to the underlying vibrant blue and purple blocks. Can you share how this text effect is achieved using Core Image?
Replies
1
Boosts
0
Views
209
Activity
Jun ’26
CIRAWFilter decoderVersion scope and RAW 9
I’m really excited about RAW 9, great work! Since RAW 9 is not enabled by default, are there any recommended use cases where it makes sense to select it by default, if supported? For compatibility handling, should I treat decoderVersion as covering all processing and adjustments performed by CIRAWFilter, or does it mostly affect the RAW decoding stage? When switching decoder versions, should I re-evaluate which CIRAWFilter adjustments are supported? Also, the macOS 26 headers appear to expose version9 and version9DNG. Is RAW 9 actually supported on macOS 26 runtimes? Thanks!
Replies
2
Boosts
0
Views
259
Activity
Jun ’26
CIRAWFilter despeckleAmount
CIRAWFilter documentation lists despeckleAmount as available on macOS 12.0+, but I cannot find this property in the macOS 26 headers. Could you provide a detailed description of how despeckleAmount affects the image so I can explain this adjustment accurately to my users? Does it run before or after linearSpaceFilter, and what should it be set to when generating scene-referred linear output? Thanks!
Replies
2
Boosts
1
Views
253
Activity
Jun ’26
RAW 9: High watermark for memory?
What is the high water mark for rendering an image in RAW 9 vs. RAW 8? I realize it depends on the megapixels of the render, but what can you share memory wise, especially compared to RAW 8, perhaps as a multiple? I'm mostly concerned about iOS in case that's relevant.
Replies
1
Boosts
1
Views
220
Activity
Jun ’26