Detect issues like logic failures, UI problems, and performance regressions by running tests on your app.

Posts under Testing tag

124 Posts
Sort by:
Post not yet marked as solved
10 Replies
14k Views
Where can I find the older versions of iOS to download for testing purposes?I am looking for 8.0, 8.1, 8.2Thanks in advance!!!
Posted
by
Post not yet marked as solved
1 Replies
631 Views
HiThe app under test is localized and internationalized. So when device language is changed, strings displayed on the app are changed.I am trying to write test cases that work independent of the current language, in order to achieve this some of the static text/button labels have to be localized for the element query to work correctly.How can we achieve this? If I try to have all the strings in an external file, the bundle created for UITesting for some reason does NOT include this strings file.Can someone please shed light on how to include non-code files into UI Testing bundle?I tried having the file named as .json, .strings. It does not work.ThanksNaveen
Posted
by
Post not yet marked as solved
3 Replies
884 Views
Sometimes when running UI tests, the tests fail on keyboard input because the "Speed up your typing" keyboard tutorial overlay obscures the keys. This happens randomly, but sometimes repeatedly. Is there a way to turn this off? Checking for this tutorial before any keyboard input would significantly complicate tests just to check for one edge case...
Posted
by
Post not yet marked as solved
0 Replies
336 Views
Hi, I found problem when my app run UITests using Xcode 12 beta 12A6159. It always give this error Failed to get matching snapshot: Error getting main window kAXErrorAPIDisabled I tried to po app.otherElements or app.collectionViews but always give me empty result. Here is my code snippet: let app = XCUIApplication() app.launch() let homeCollection = app.collectionViews.firstMatch homeCollection.waitForExistence(timeout: 10) measure(metrics: [XCTOSSignpostMetric.scrollDecelerationMetric]) { homeCollection.swipeUp(velocity: .fast) }
Posted
by
Post not yet marked as solved
11 Replies
4.0k Views
We have noticed a 4X increase in app launch time for UI tests in Xcode 12 compared with Xcode 11. App launch times were typically ~15 seconds and are now approximately ~75 seconds. When I pause the debugger, I can see that it is hanging on XCUIApplication.launch() Is this a known issue? Are others experiencing this issue? Should we be expecting to see this increase? Are there mitigations we can implement to improve the launch time?
Posted
by
Post not yet marked as solved
13 Replies
26k Views
Anyone able to get an Android Emulator up and running on Silicon? I've tried the AVD of Android Studio as well as Genymotion; neither work.
Posted
by
Post not yet marked as solved
2 Replies
472 Views
This looks like xcodebuild bug, but I just wanted to bring it up here as well. I've been having issues with this in our project and seen other people mentioning similar issue https://github.com/realm/realm-cocoa/issues/6814, so I've put together an example project that suffers from this issue. https://github.com/petrpavlik/CodeCoverageBugExample/tree/master This is unmodified app template generated by Xcode, where I've 2 swift packages that contain ObjC code - CocoaAsyncSocket and SDWebImage. I was able to replicate this with Realm package for instance, so it's not about these 2 particular packages. Running xcodebuild to run tests and collect code coverage xcodebuild "-workspace" "CodeCoverageBugExample.xcworkspace" "-scheme" "CodeCoverageBugExample" "build" "COMPILER_INDEX_STORE_ENABLE=NO" "test" "-destination" "id=2E465F79-A48A-4C32-BBE0-F1104A27E4C7" "-resultBundlePath" "Test.xcresult" "GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES" "GCC_GENERATE_TEST_COVERAGE_FILES=YES" fails with following error. duplicate symbol '___gcov_flush' in: 		/Users/petr/Library/Developer/Xcode/DerivedData/CodeCoverageBugExample-gictjvuwzgnjolaocdbqaotvidpx/Build/Products/Debug-iphonesimulator/SDWebImage.o 		/Users/petr/Library/Developer/Xcode/DerivedData/CodeCoverageBugExample-gictjvuwzgnjolaocdbqaotvidpx/Build/Products/Debug-iphonesimulator/CocoaAsyncSocket.o ld: 1 duplicate symbol for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation)
Posted
by
Post not yet marked as solved
1 Replies
931 Views
Hello I have a iOS app that we use in our organization. We are trying to create some training material with it, using a screen-recording (keystroke) recorder on the PC. I am wondering what emulator options exist for iOS (majority of employees have iPhone 6s Plus). I've been scanning the internet for days and seem to be limited to a few and not sure how the price points line up to what else is out there. Essentially I need to be able to install our app to the emulator and then run the application, supporting navigation using mouse/keyboard on the PC. Is there inexpensive options, free options that are out there? Any suggestions would be greatly appreciated. Thank you bean-tbay
Posted
by
Post not yet marked as solved
3 Replies
852 Views
I've got an iOS app with an add-on non-consumable. In sandbox testing, the purchase attempt *always* shows an extra Buy prompt, and then fails if the sandbox account is new. Then, it always succeeds if you just try again. Is this normal? Details: Test begins with a brand new Sandbox tester account, in USA region and verified via email invitation. Device iTunes & App Store SANDBOX ACCOUNT is logged in with this new account. App is completely deleted from device, then build/deployed with Xcode. Note that launching the app initially always prompts to enter the password if this is the first time I've used this sandbox test account. This prompt doesn't say anything about sandbox, but it uses the sandbox email as Apple ID and sandbox login works. Receipt retrieval and local validation works in the app, as does initial retrieval of the add-on product information with SKProductsRequest Requesting purchase of add-on with SKPaymentQueue.add() correctly puts up the "Confirm Your In-App Purchase" modal, showing [Environment: Sandbox]. A paymentQueue:upatedTransactions invocation happens, with transactionState .purchasing. After entering correct password and clicking Buy, there's a brief pause, then a second purchase confirmation modal appears exactly the same way, again with [Environment: Sandbox] and again requiring password When I enter the password and click Buy again, the transaction fails. I get a .failed transaction paymentQueue update and the failure alert comes up saying try again later. At this point, however, all I have to do is try again - immediately, or after quitting the app and restarting - and everything works perfectly and the purchase succeeds. The success in the second case makes me think that the code is correct, and the problem with the first attempt is just some known sandbox or configuration issue. The double prompt seems especially weird since there's no transaction update between them. Do others see this behavior? Is there any way to make a sandbox purchase attempt succeed on the first attempt in this kind of environment? Xcode 12.2, running on Intel 11.0.1; device running iOS 13.7.
Posted
by
Post not yet marked as solved
8 Replies
3.8k Views
We are building an iOS app that connects to a device using Bluetooth. To test unhappy flow scenarios for this app, we'd like to power cycle the device we are connecting to by using an IoT power switch that connects to the local network using WiFi (a Shelly Plug-S). In my test code on iOS13, I was able to do a local HTTP call to the IP address of the power switch and trigger a power cycle using its REST interface. In iOS 14 this is no longer possible, probably due to new restrictions regarding local network usage without permissions (see: https://developer.apple.com/videos/play/wwdc2020/10110 ). When running the test and trying a local network call to the power switch in iOS14, I get the following error: Task <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2> finished with error [-1009] Error Domain=NSURLErrorDomain Code=-1009 "The internet connection appears to be offline." UserInfo={_kCFStreamErrorCodeKey=50, NSUnderlyingError=0x2833f34b0 {Error Domain=kCFErrorDomainCFNetwork Code=-1009 "(null)" UserInfo={_kCFStreamErrorCodeKey=50, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2>, _NSURLErrorRelatedURLSessionTaskErrorKey=("LocalDataTask <D206B326-1820-43CA-A54C-5B470B4F1A79>.<2>"), NSLocalizedDescription=The internet connection appears to be offline., NSErrorFailingURLStringKey=http://192.168.22.57/relay/0?turn=on, NSErrorFailingURLKey=http://192.168.22.57/relay/0?turn=on, _kCFStreamErrorDomainKey=1} An external network call (to google.com) works just fine in the test. I have tried fixing this by adding the following entries to the Info.plist of my UI test target: <key>NSLocalNetworkUsageDescription</key> <string>Local network access is needed for tests</string> <key>NSBonjourServices</key> <array> <string>_http._tcp</string> </array> <key>NSAppTransportSecurity</key> <dict> <key>NSAllowsArbitraryLoads</key> <true/> </dict> However, this has no effect. I have also tried adding these entries to the Info.plist of my app target to see if that makes a difference, but it doesn't. I'd also rather not add these entries to my app's Info.plist, because the app does not need local network access. Only the test does. Does anyone know how to enable local network access during an iOS UI test in iOS14?
Posted
by
Post not yet marked as solved
1 Replies
647 Views
I am trying to test the In-App purchases by adding my test email to"Testers" in the Developer console. When I use any of the sandbox emails added there, I get the following message: Ask Permission A request to buy ____ will be sent to your parent or guardian [Environment: sandbox] if I click on "Ask", nothing happens. I've rechecked that this email is not a child account. it has a DOB of 1980. In fact, it's not enrolled in any family sharing program. I've tested multiple accounts after adding them to sandbox. Same issue occurs in all the accounts. One of those accounts is: testkinderpass123@gmail.com I also tried adding these accounts to family sharing plan so that even if a parental request is sent, I can accept that from another device. Unfortunately, "Try again" pop up errors started coming while doing so.
Posted
by
Post marked as solved
1 Replies
971 Views
I have recently started Beta Testing my SwiftUI App on my iPhone 6s, previously I was testing it on the iPhone 12 Simulator. While Testing on the Real Device I encountered issues like: CPU going past 98% Really Laggy ScrollView Tab Bar Freezes Randomly Since My app is going to go in Production State in a few days, Since this is my first iOS App as well, I am not too sure if these issues are because of SwiftUI, my Phone or what, I have asked Apple Support but, haven't received a response so far. My App Stats: Lines of Code: ~1,500 CPU Usage on Simulator: Max. 10% Bundle Size: 2MB RAM Max. on Simulator: 32.3mb
Posted
by
Post not yet marked as solved
8 Replies
4.1k Views
Hello, I'm in a big project app with unit testing and ui testing running well. Recently, one member of team, buy a MacBook Pro M1 with Big Sur 11.1 and try to execute the ui testing with the new Xcode 12.4 (available today 27/01/21 from Mac AppStore). When he selects any simulator with iOS 13.7 it's ok, but if he selects iOS 14.4 then fail. It doesn't matter if it's an iPad or an iPhone. The error talks about architectures but the settings haven't any custom config depends iOS version. Showing All Messages MYAPPUITests-Runner.app (72942) encountered an error (Failed to load the test bundle. If you believe this error represents a bug, please attach the result bundle at /Users/user/Library/Developer/Xcode/DerivedData/MYAPP-gefifpwdxlqqbidolncfjzimagfq/Logs/Test/Test-MYAPP-2021.01.27_23-13-22-+0100.xcresult. (Underlying Error: The bundle “MYAPPUITests” couldn’t be loaded because it doesn’t contain a version for the current architecture. The bundle doesn’t contain a version for the current architecture. Try installing a universal version of the bundle. dlopen_preflight(/Users/user/Library/Developer/Xcode/DerivedData/MYAPP-gefifpwdxlqqbidolncfjzimagfq/Build/Products/Debug-iphonesimulator/MYAPPUITests-Runner.app/PlugIns/MYAPPUITests.xctest/MYAPPUITests): no suitable image found.&#9;Did find: /Users/user/Library/Developer/Xcode/DerivedData/MYAPP-gefifpwdxlqqbidolncfjzimagfq/Build/Products/Debug-iphonesimulator/MYAPPUITests-Runner.app/PlugIns/MYAPPUITests.xctest/MYAPPUITests: mach-o, but wrong architecture)) Why it's ok with iOS 13.7 and why not with iOS 14.4 simulator?? Can you think something ? Regards.
Posted
by
Post not yet marked as solved
1 Replies
1.3k Views
Hi! I've run into some problems with running unit tests for iOS framework project recently. It had worked, but it stopped since one of the command line tool's updates. When I try to run unit tests with a command line (xcodebuild) I get 'Failed to create a bundle instance representing' error. But only for an iOS framework project and when -derivedDataPath is set with a custom path. xcodebuild -project TestFramework.xcodeproj -scheme TestFramework -derivedDataPath ./output  -destination 'platform=iOS Simulator,id=***' clean build test In the Xcode I can run tests without any problems. When I remove -derivedDataPath parameter, it works like a charm as well. On the other hand, unit tests for an iOS application project works from command line and Xcode, whenever I set -derivedDataPath flag or not. Has any of you been dealing with the same problem? Or do you know any solution? Or is it a known issue? Cheers! PS. $ xcodebuild -version  Xcode 12.4 Build version 12D4e $ sw_vers  ProductName: Mac OS X ProductVersion: 10.15.7 BuildVersion: 19H2
Posted
by
Post not yet marked as solved
3 Replies
859 Views
I've noticed that XCTMemoryMetric &amp; XCTCPUMetric seem to record empty or nonsensical data when running a UI test flow for an XCUIApplication. I'm attempting to test the memory and CPU footprint for a SwiftUI iOS app using the following code: Swift func testBasicFlowMemory() throws{ let app = XCUIApplication() app.launch() var metrics:[XCTMetric] = [] metrics.append( XCTClockMetric() ) metrics.append( XCTMemoryMetric(application: app) ) metrics.append( XCTCPUMetric(application: app) ) self.measure(metrics: metrics){ /*Method which uses XCUI API to test the app instance*/ self.runBasicFlowTest(app: app) } } When I run the test above, I notice runBasicFlowTest is executed 6 times (even though the metics only record 5 values. Of the three metrics I wanted to track only XCTClockMetric returned meaningful data: [Clock Monotonic Time, s] values: [114.728229, 114.944770, 121.813337, 116.394432, 117.491242] XCTMemoryMetric mostly recorded 0.0 or nonsense data: [Memory Physical, kB] values: [3596.288000, 0.000000, 0.000000, 0.000000, 0.000000] [Memory Peak Physical, kB] values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000] XCTCPUMetric likewise recorded 0.0 or nonsense data: [CPU Instructions Retired, kI] values: [0.000000, 206223944.266000, 0.000000, 0.000000, 211895544.471000] [CPU Cycles, kC] values: [0.000000, 252096240.472000, 0.000000, 0.000000, 257352232.305000], [CPU Time, s] values: [0.000000, 86.585296, 0.000000, 0.000000, 0.000000] I'm on Xcode Version 12.4 (12D4e), and my app is targeting iOS 14.4 on a simulated iPhone 11 Pro. Has anyone had any luck using XCTMetrics with UI Tests?
Posted
by
Post not yet marked as solved
1 Replies
794 Views
Here is how we are setup while facing the issue. We have a project which has both UI and Unit tests. Uses fastlane. Has CI/CD implementation using Jenkins which runs on an aws ec2 instance. Quite recently we moved to Xcode12 (I know!!! However, better late than never) Problem statement: After moving to xcode12, the jenkins pipeline is unable to generate the Coverage.profdata file which in turn will be used by Slather to derive code coverage metrics. Yes, did not happen when running xcode11. We are using the xcodebuild command after pointing it to the appropriate command line tool version (in this case it is 12.4). First we build the app using (after getting the simulator ID) xcodebuild build-for-testing -workspace Example.xcworkspace -scheme ExampleUITests -derivedDataPath ./derivedData -destination 'platform=iOS Simulator,id=simulator_id' Then we run the test using xcodebuild test-without-building -workspace Example.xcworkspace -scheme ExampleUITests -enableCodeCoverage YES -destination 'platform=iOS Simulator,id=simulator_id' -derivedDataPath ./derivedData All the UI tests run successfully but end up with the following statements when the test suite completes. warning: Path_to_derived_data/Build/ProfileData/Some_UUID/Filename.profraw: Invalid instrumentation profile data (file header is corrupt) error: No profiles could be merged. P.S: This is not happening on my local machine. Only on the mac in the Jenkins ec2 instance.
Posted
by
Post not yet marked as solved
3 Replies
768 Views
Hello, Since Xcode 12, I can't see the test report popup when running performance tests. Therefore, I am not able to set the baseline anymore. This used to work fine in Xcode 11. Here's an example: swift func testStoragePerformance() throws {     let application = XCUIApplication()     application.launch()     let storageMetric = XCTStorageMetric(application: application)     self.measure(metrics: [storageMetric]) {       self.runUIFlow(application: application)     }   } When running the test above, Xcode shows the test as successful and that's it... Same thing happens with XCTCPUMetric and XCTMemoryMetric. It seems to only work with XCTClockMetric (baseline can be set then). I am able to reproduce this bug on a new project created in Xcode 12.4. I have reached out to people at Apple and also created a ticket in Feedback Assistant but I've never got any response. Is anyone else experiencing the same issue? Is this a known bug? Thanks
Posted
by
Post not yet marked as solved
12 Replies
3.4k Views
Hi Everyone, our project has a few vendor libraries that don't support the M1 architecture when building for the simulator, so we have to exclude the arm64 arch in order to compile. However, when we do this, generating code coverage will fail for all targets with the following message: Failed to generate coverage for target 'UnitTests.app' at paths ("/Users/xxxx/Library/Developer/Xcode/DerivedData/UnitTests-dzyobvtkdsjkmlbzpenutqaqgbbl/Build/Products/Debug-iphonesimulator/UnitTests.app/UnitTests"): No object file for requested architecture Resulting in coverage data missing for our project. Has anyone else encountered this? Known issue? Any workarounds? Thanks in advance for any help.
Posted
by
Post not yet marked as solved
3 Replies
1.3k Views
MKLocalSearch.start - https://developer.apple.com/documentation/mapkit/mklocalsearch/1452652-start is giving me MKErrorGEOError=-8 which is not documented in MKError - https://developer.apple.com/documentation/mapkit/mkerror. This only happens in my CI machine! This only happens when running the tests, the app code works fine I tried enabling location services on macOS just in case, but it didn't work. That machine shouldn't have any network requests/URLs blocked What fixed for me was to add request.naturalLanguageQuery = "Venue", but I don't understand why, nor why this would fix an error that is happening in only 1 machine. Does anyone have any information about this? Here's some sample code: let request = MKLocalSearch.Request() request.naturalLanguageQuery = "Venue" // this fixes it, but why?! let ibereCamargoMuseumCoordinates = CLLocationCoordinate2D(latitude: -30.0777596, longitude: -51.2477212) let span = MKCoordinateSpan(latitudeDelta: 5, longitudeDelta: 5) request.region = MKCoordinateRegion(center: ibereCamargoMuseumCoordinates, span: span) if #available(iOS 13.0, *) { request.resultTypes = .pointOfInterest } let localSearch = MKLocalSearch(request: request) localSearch.start { response, error in ... } Here's some debugging output: (lldb) po error Error Domain=MKErrorDomain Code=4 "(null)" UserInfo={MKErrorGEOError=-8} (lldb) po error.userInfo ▿ 1 element ▿ 0 : 2 elements ▿ key : AnyHashable("MKErrorGEOError") - value : "MKErrorGEOError" value : -8 If you're interested I asked this on Stack Overflow - https://stackoverflow.com/questions/66976777/mklocalsearch-start-returning-undocumented-mkerrorgeoerror too.
Posted
by