Detect issues like logic failures, UI problems, and performance regressions by running tests on your app.

Posts under Testing tag

83 Posts

Post

Replies

Boosts

Views

Activity

Enable local network access during iOS UI test in iOS14
We are building an iOS app that connects to a device using Bluetooth. To test unhappy flow scenarios for this app, we'd like to power cycle the device we are connecting to by using an IoT power switch that connects to the local network using WiFi (a Shelly Plug-S). In my test code on iOS13, I was able to do a local HTTP call to the IP address of the power switch and trigger a power cycle using its REST interface. In iOS 14 this is no longer possible, probably due to new restrictions regarding local network usage without permissions (see: https://developer.apple.com/videos/play/wwdc2020/10110 ). When running the test and trying a local network call to the power switch in iOS14, I get the following error: Task <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2> finished with error [-1009] Error Domain=NSURLErrorDomain Code=-1009 "The internet connection appears to be offline." UserInfo={_kCFStreamErrorCodeKey=50, NSUnderlyingError=0x2833f34b0 {Error Domain=kCFErrorDomainCFNetwork Code=-1009 "(null)" UserInfo={_kCFStreamErrorCodeKey=50, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2>, _NSURLErrorRelatedURLSessionTaskErrorKey=("LocalDataTask <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2>"), NSLocalizedDescription=The internet connection appears to be offline., NSErrorFailingURLStringKey=http://192.168.22.57/relay/0?turn=on, NSErrorFailingURLKey=http://192.168.22.57/relay/0?turn=on, _kCFStreamErrorDomainKey=1} An external network call (to google.com) works just fine in the test. I have tried fixing this by adding the following entries to the Info.plist of my UI test target: <key>NSLocalNetworkUsageDescription</key> <string>Local network access is needed for tests</string> <key>NSBonjourServices</key> <array> <string>_http._tcp</string> </array> <key>NSAppTransportSecurity</key> <dict> <key>NSAllowsArbitraryLoads</key> <true/> </dict> However, this has no effect. I have also tried adding these entries to the Info.plist of my app target to see if that makes a difference, but it doesn't. I'd also rather not add these entries to my app's Info.plist, because the app does not need local network access. Only the test does. Does anyone know how to enable local network access during an iOS UI test in iOS14?
13
1
19k
Nov ’24
Xcode 15 performance tests ignore baseline
I'm trying to write performance tests with Xcode 15, but it's not working: even though setting a baseline seems to work (baseline data files are created), Xcode seems to ignore that baseline altogether whenever the test is run. As a result, performance tests never fail, no matter how bad performance gets. The source editor persistently displays a “No baseline average for Time” message. Has anyone else encountered this? Figured out how to sidestep the issue? Of note, the issue is easily reproduced: create a new (macOS in my case) app project with tests, add a dummy performance test; run the test once and save its baseline, and despite this subsequent tests will always succeed, even if you significantly slow the dummy test down. (I've filed this as FB13330139)
11
10
2.7k
2w
Step-by-step instructions on how to send an app to external testers with TestFlight in 2024?
New Apple Developer here, I've built my first iOS app in Xcode and wants to get it onto some friends' iPhones to do some external testing and give me some feedback. I've read the Apple Developer documentation on how to use TestFlight, but it's not understandable since step 1 is entering information about the app in App Store Connect and step 2 is uploading the app to App Store Connect? Can someone please write some easy step-by-step directions on how to get an iOS app from Xcode onto external testers' iPhones for a first-timer in 2024? Thank you so much!
1
1
744
4w
Unable to Launch Tests/UITests in Xcode 16 with iOS 18 Simulator
Hello, I'm unable to run any test cases in Xcode 16 with the iOS 18 simulator. Steps to Reproduce: Create a new iOS project in Xcode 16 using the default iOS app template with the testing option checked. Run any test or UI test case by clicking the "Play" icon in the Test navigator. Xcode gets stuck at the "Testing" state indefinitely. Environment: macOS 15 beta 8 Xcode 16 beta 6 iOS 18 beta 7 simulator
11
6
3.2k
Sep ’25
Do I need to down load iOS 18 for testing on iPhone 11 having iOS version 17.6.1 ?
I am using two iPhone11 devices, having iOS v17.6.1 for automation testing using Seleniumv4.X-Appium v2.x. Today I updated the mac to Sequoia15 version which updated the XCode to v16. Just after this update, I could not able to run the previously running automation script. The console message is "org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Response code 500. Message: Unable to launch WebDriverAgent. Original error: xcodebuild failed with code 70. This usually indicates an issue with the local Xcode setup or WebDriverAgent project configuration or the driver-to-platform version mismatch."
3
0
1.2k
Nov ’24
Running local package's tests in main app's test plan
Hi, I work on a iOS application which includes several local swift packages containing modularised code. Each of those local swift packages has a test target defined in their respective Package.swift, and when opening a local package folder standalone in Xcode, then the tests run without issues. However I would like in the main app's test plan to add the local package test targets and have them all run together when the main app's default test plan is run. Xcode allows me to select local package test targets within the main app's test plan (also within Test portion of the main app's scheme), however attempting to build for testing throws a "Module '…' was not compiled for testing". Any ideas on how to achieve this goal (run local package tests in conjunction with main app's tests) and avoid that error? Thanks Peter
2
5
922
Nov ’24
Xcode not generating code coverage.
No coverage is generated for private xxxx.framework, but tests are running under this framework. This was perfectly working before and started giving this error after updating to Xcode 16. Failed to generate coverage for target xxxx.framework at paths ( "/Users/xxxxxx/Desktop/xcode/Debug-iphonesimulator/Common/xxxxxx.framework/xxxx" ): No object file for requested architecture The operation couldn’t be completed. (XCTHCoverageReportGenerationErrorDomain error 2.). (Underlying Error: No object file for requested architecture)
2
1
938
Nov ’24
xcrun xcresulttool Failed to create a new result bundle reader
I need to read results of a test in Xcode post-action of Test. So the script that runs after tests contains SUMMARY=$(xcrun xcresulttool get test-results summary --path $RESULT_FILE 2>> "$LOGS") If I run my script from terminal, everything is ok but if I launch tests and the post-action is executed, I got Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_FORMAT Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_FORMAT Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION Error: Failed to create a new result bundle reader, underlying error: failed to read metadata with underlying error (type: FileSystemError: 3 - [:] - The operation couldn’t be completed. (MinimalTSCBasic.FileSystemError error 3.) Usage: xcresulttool <subcommand> See 'xcresulttool --help' for more information. It might be that the environment set by Xcode is missing something but I cannot figure out what, any idea?
2
2
1.1k
Nov ’24
Unable to run test in devices with MDM Profiles
Requirement : We wanted to run UI Automation using xcuitest in devices with MDM profiles. Steps : We have created a test target for our app and signed with MDM profile (in-house) Generated app and testbundle-Runner along with xctestrun files using “xcodebuild build-for-testing -scheme “App-TestRelease" -configuration “Release” -derivedDataPath "${CURDIR}/derivedData" -workspace "${PROJECT_NAME}.xcworkspace" -sdk iphoneos” Tried running “xcodebuild test-without-building -destination platform=iOS,id=000842883-**** -xctestrun App-TestRelease_iphoneos17.5-arm64.xctestrun -derivedDataPath /derived/“ Facing below issues xcodebuild[35481:3588352] IDELaunchReport: e2934a094f300d40:e2934a094f300d40: Finished with error: Unable to launch com.org.mdm.XCTest.xctrunner Domain: com.apple.platform.iphoneos Code: -12 User Info: { IDERunOperationFailingWorker = IDELaunchiPhoneLauncher; } -- Request to launch com.org.mdm.XCTest.xctrunner failed. Domain: com.apple.dt.deviceprocesscontrolservice Code: 2 Failure Reason: The request to open "com.org.mdm.XCTest.xctrunner" failed. : Failed to launch process with bundle identifier 'com.org.mdm.XCTest.xctrunner'. -- The request to open "com.org.mdm.XCTest.xctrunner" failed. Domain: FBSOpenApplicationServiceErrorDomain Code: 1 Failure Reason: The request was denied by service delegate (SBMainWorkspace) for reason: Security ("Unable to launch com.org.mdm.XCTest.xctrunner because it has an invalid code signature, inadequate entitlements or its profile has not been explicitly trusted by the user"). User Info: { BSErrorCodeDescription = RequestDenied; FBSOpenApplicationRequestID = 0x9f9; } -- The operation couldn’t be completed. Unable to launch com.org.mdm.XCTest.xctrunner because it has an invalid code signature, inadequate entitlements or its profile has not been explicitly trusted by the user. Domain: FBSOpenApplicationErrorDomain Code: 3 Failure Reason: Unable to launch com.org.mdm.XCTest.xctrunner because it has an invalid code signature, inadequate entitlements or its profile has not been explicitly trusted by the user. User Info: { BSErrorCodeDescription = Security; }
1
0
700
Oct ’24
How to Test FinanceKit Transaction History in iOS?
Hello, I am currently developing an iOS application that leverages FinanceKit to access transaction history. I have already enabled FinanceKit in my developer account, but I am encountering challenges when trying to test the transaction history feature. Could anyone guide me on the best practices or specific steps needed to effectively test FinanceKit’s transaction history in a development environment? Any insights on mock data handling, API responses, or potential limitations during testing would be greatly appreciated. Thank you in advance for your help!
0
2
341
Nov ’24
Testing Right-to-Left localization in a macOS app
I’m currently trying to localize one of my Mac apps for Right-to-Left languages. In Xcode, to test this, I set the language to ‘Arabic’ in the scheme, which then displays the language and mirrors the layout accordingly. I have two questions about this: macOS titlebar buttons: In the title bar at the top, shouldn’t the Close and Minimize buttons also be on the right side (mirrored)? How can I test this? Numerals: My UI still displays Latin numerals (123). I know this is correct in some Arabic-speaking regions, but I’d also like to test cases where Arabic-Indic numerals (١٢٣) are used. Is there a way to configure the scheme to test this?
2
0
651
Nov ’24
Run test specific setup logic on different target
Our use case is that we need to run a CLI script (on Mac) to set up backend configuration before each test case (UI test on real iOS device). The setup logic is test specific, so using pre-build scripts is not an option. Is this support in any way or what are best practices around such a setup? Ideally, we would still be able to execute the tests from within Xcode.
0
0
348
Nov ’24
NEPacketTunnelProvider stopped connecting on arm macOS 14 during tests
I have tests where I connect to NEPacketTunnelProvider. I run tests with circleci and fastlane, on self hosted intel and arm macs. I updated macs from macOS 13 to macOS 14 and the tests on arm stopped connecting, while the same tests on intel kept working as usual. Moreover, I noticed the tests don't work when run from circleci and fastlane. If I cancel the job and click "connect" myself on the app that stayed hanging from the cancelled tests, the connection will succeed. But if the tests are running, the connection will fails. Running the tests from xcode succeeds too. These are the logs from the tunnel. Could you suggest me where to dig? Or maybe you can see the issue from the logs? Tunnel logs when they fail
1
0
425
Dec ’24
I get this Issue : ContentView.swift:3016:25 The compiler is unable to type-check this expression in reasonable time; try breaking up the expression into distinct sub-expressions. But I don´t know what I have to do. I am a noob. Can anybody help me.
// Füge OrderRow hinzu struct OrderRow: View { let order: Order var body: some View { VStack(alignment: .leading, spacing: 8) { HStack { Text("Tisch \(order.tableNumber)") .font(.headline) Spacer() Text(order.status.rawValue) .font(.caption) .padding(.horizontal, 8) .padding(.vertical, 4) .background(order.status.color.opacity(0.2)) .foregroundColor(order.status.color) .cornerRadius(8) } Text(order.timestamp, style: .time) .font(.caption) .foregroundColor(.gray) ForEach(order.items) { item in Text("\(item.quantity)x \(item.item.name)") .font(.subheadline) } HStack { Spacer() Text(String(format: "%.2f €", order.totalAmount)) .bold() } } .padding() .background(Color.white) .cornerRadius(12) .shadow(radius: 2) } } struct MenuItemRow: View { let item: MenuItem var body: some View { HStack { Text(item.name) .font(.headline) Spacer() Text(String(format: "%.2f €", item.price)) .font(.subheadline) } .padding() } } struct MenuItemsView: View { @Binding var selectedItems: [OrderItem] @Environment(.dismiss) private var dismiss func filteredItems(for category: MenuCategory) -&gt; [MenuItem] { return sampleMenuItems.filter { $0.category == category } } func addItemToOrder(_ item: MenuItem) { if let index = selectedItems.firstIndex(where: { $0.item.id == item.id }) { selectedItems[index].quantity += 1 } else { selectedItems.append(OrderItem(item: item, quantity: 1)) } } var body: some View { NavigationView { List { ForEach(MenuCategory) { category in Section(header: Text(category.rawValue)) { let filteredItems = sampleMenuItems.filter ForEach(filteredItems) { item in } MenuItemRow(item: sampleMenuItems.filter) { addItemToOrder } } } } } .navigationTitle("Menü") .toolbar { ToolbarItem(placement: .navigationBarTrailing) { Button("Fertig") { dismiss() } } } } } } // Sample Menü-Items let sampleMenuItems: [MenuItem] = [ MenuItem(name: "Cola", description: "0,33l", price: 3.50, category: .drinks, image: "cola"), MenuItem(name: "Sprite", description: "0,33l", price: 3.50, category: .drinks, image: "sprite"), MenuItem(name: "Bier", description: "0,5l", price: 4.00, category: .drinks, image: "beer"), MenuItem(name: "Nachos", description: "mit Käsesauce", price: 5.50, category: .snacks, image: "nachos"), MenuItem(name: "Pommes", description: "mit Ketchup", price: 4.50, category: .food, image: "fries"), MenuItem(name: "********", description: "alkoholfrei", price: 6.50, category: .specials, image: "********") ] struct NewEventView: View { var body: some View { Text("New Event") } } struct EventManagementView: View { var body: some View { Text("Event Management") } } struct OrderRow: View { var order: String var body: some View { Text("Order: (order)") } } #Preview { ContentView() }
4
0
535
Dec ’24
My iOS app that I created since 2011, crashes on M4 machine (M1 is fine!)
A user gave me the crash log on his M4 machine when he first launches the app. I have M1 and my app works fine, which begs the question, why M4 has the crash! This is my app the I developed since 2011 until now. Any hints will be appreciated or anyone has similar issues like to share? ------------------------------------- Translated Report (Full Report Below) ------------------------------------- Incident Identifier: DA1CEECB-28D4-40A9-8F0C-7125812084F4 CrashReporter Key: FB8A5DBB-7140-D370-AC72-EC158266383C Hardware Model: Mac16,3 Process: Stocks Live [5506] Path: /private/var/folders/*/Stocks Live.app/Stocks Live Identifier: com.cinnamonmobile.StocksLive Version: 34.2 (34.2.0) AppStoreTools: 16B39 AppVariant: 1:MacFamily20,1:18 Code Type: ARM-64 (Native) Role: Foreground Parent Process: launchd [1] Coalition: com.cinnamonmobile.StocksLive [2018] Date/Time: 2024-11-27 12:41:48.0456 -0500 Launch Time: 2024-11-27 12:41:47.5710 -0500 OS Version: macOS 15.1.1 (24B2091) Release Type: User Report Version: 104 Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Subtype: KERN_MEMORY_ERROR at 0x0000000120034000 Exception Codes: 0x000000000000000a, 0x0000000120034000 VM Region Info: 0x120034000 is in 0x120034000-0x120038000; bytes after start: 0 bytes before end: 16383 REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL CoreAnimation 120030000-120034000 [ 16K] rw-/rwx SM=PRV ---> mapped file 120034000-120038000 [ 16K] r--/rw- SM=COW Object_id=14630bfe CoreAnimation 120038000-12003c000 [ 16K] r--/r-- SM=PRV Termination Reason: SIGNAL 10 Bus error: 10 Terminating Process: exc handler [5506] Triggered by Thread: 0 Kernel Triage: decmpfs - (arg = 0x80000001) compressor is not registered APFS - (arg = 0x200040) Pagein of compressed inode failed VM - (arg = 0x190000002d) Filesystem pagein returned an error in vnode_pagein VM - (arg = 0x0) Page has error bit set Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 CoreFoundation 0x18803f48c __CFStringCreateImmutableFunnel3 + 716 1 FincancialCoreFrameWork 0x10358d3c4 -[MOStringDocument loadFromContents:ofType:error:] + 156 2 UIKitCore 0x1ba34a67c __32-[UIDocument readFromURL:error:]_block_invoke + 104 3 libdispatch.dylib 0x187e20658 _dispatch_client_callout + 20 4 libdispatch.dylib 0x187e2fdb0 _dispatch_async_and_wait_invoke + 92 5 libdispatch.dylib 0x187e20658 _dispatch_client_callout + 20 6 libdispatch.dylib 0x187e2ef68 _dispatch_main_queue_drain + 980 7 libdispatch.dylib 0x187e2eb84 _dispatch_main_queue_callback_4CF + 44 8 CoreFoundation 0x1880f8e60 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16 9 CoreFoundation 0x1880b8a4c __CFRunLoopRun + 1996 10 CoreFoundation 0x1880b7bc4 CFRunLoopRunSpecific + 588 11 HIToolbox 0x193529f64 RunCurrentEventLoopInMode + 292 12 HIToolbox 0x19352fd54 ReceiveNextEventCommon + 636 13 HIToolbox 0x19352feb8 _BlockUntilNextEventMatchingListInModeWithFilter + 76 14 AppKit 0x18bbe398c _DPSNextEvent + 660 15 AppKit 0x18c523ddc -[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 688 16 AppKit 0x18bbd6a64 -[NSApplication run] + 480 17 AppKit 0x18bbad2e8 NSApplicationMain + 888 18 AppKit 0x18bdfb7f4 _NSApplicationMainWithInfoDictionary + 24 19 UIKitMacHelper 0x1a28f8b38 UINSApplicationMain + 972 20 UIKitCore 0x1b99de3a8 UIApplicationMain + 148 21 Stocks Live 0x102bcc058 0x102bc4000 + 32856 22 dyld 0x187c50274 start + 2840 Thread 1:: Dispatch queue: UIDocument File Access 0 libsystem_kernel.dylib 0x187f93b8c __ulock_wait + 8 1 libdispatch.dylib 0x187e21048 _dlock_wait + 56 2 libdispatch.dylib 0x187e20dfc _dispatch_thread_event_wait_slow + 56 3 libdispatch.dylib 0x187e2ffc4 __DISPATCH_WAIT_FOR_QUEUE__ + 368 4 libdispatch.dylib 0x187e2fb70 _dispatch_sync_f_slow + 148 5 UIKitCore 0x1ba351384 -[UIDocument(UIDocumentInternal) _performBlock:synchronouslyOnQueue:] + 76 6 UIKitCore 0x1ba347ff0 __40-[UIDocument openWithCompletionHandler:]_block_invoke + 588 7 libdispatch.dylib 0x187e1e8f8 _dispatch_call_block_and_release + 32 8 libdispatch.dylib 0x187e20658 _dispatch_client_callout + 20 9 libdispatch.dylib 0x187e27c60 _dispatch_lane_serial_drain + 744 10 libdispatch.dylib 0x187e28768 _dispatch_lane_invoke + 380 11 libdispatch.dylib 0x187e337e8 _dispatch_root_queue_drain_deferred_wlh + 288 12 libdispatch.dylib 0x187e33034 _dispatch_workloop_worker_thread + 540 13 libsystem_pthread.dylib 0x187fcf3d8 _pthread_wqthread + 288 14 libsystem_pthread.dylib 0x187fce0f0 start_wqthread + 8 Thread 2: 0 libsystem_pthread.dylib 0x187fce0e8 start_wqthread + 0 Thread 3: 0 libsystem_pthread.dylib 0x187fce0e8 start_wqthread + 0
3
0
581
Dec ’24
Swift Testing tests in enum (namespace) extensions are acting strange
Using Xcode 16.1 (16B40). When adding a test suite with tests in an extension to an enum: The tests can be compiled and executed as expected (by clicking the diamond), and e.g. "Reveal in Test Navigator" works as expected. But after the tests have been executed, there is no check or cross diamond icons in the Test Navigator, If I right click the empty diamond in the margin for one of the tests and click "Jump to report", Xcode shows an alert saying "No test found matching the identifier MyEnumNameSpace/SomeTests/example()", which is very strange. Steps to reproduce: Create a new iOS app project using SwiftUI and Swift Testing. Add a file in the app target defining an enum namespace: public enum NamespaceDefinedInSomeApp {} Replace the content of the already created …Tests.swift file with: import Testing @testable import SomeApp // This works as expected: @Suite struct SomeTests { @Test func example() async throws { #expect(true) } } // This compiles and executes but acts strange: extension NamespaceDefinedInSomeApp { @Suite struct SomeTests { @Test func example() async throws { #expect(true) } } } // This compiles and executes but acts strange: enum AnotherNamespace {} extension AnotherNamespace { @Suite struct SomeTests { @Test func example() async throws { #expect(true) } } }
1
1
637
Dec ’24