For RoomAnchors there's different mesh classifications for mesh anchors, but only walls and floors are supported by geometries() function.
So given this how can I get information about other mesh classifications?
ARKit
RSS for tagIntegrate iOS device camera and motion features to produce augmented reality experiences in your app or game using ARKit.
Posts under ARKit tag
200 Posts
Sort by:
Post
Replies
Boosts
Views
Activity
I'm developing an augmented images app using ARKit. The images themselves are sourced online. The app is mostly done and working fine. However, I download the images the app will be tracking every time the app starts up. I'd like to avoid this by perhaps downloading the images and storing them to the device.
My concern is that as the number of images grow, the app would download too many images to the device. I'd like some thoughts on how to best approach this. For example, should I download and store some of the images in CoreData, or perhaps not store them at all?
We can use the CreateML App to build object tracking model in Xcode 16, but is it possible to use CreateML framework as well?
No documentation of Create ML object tracking is found yet. The latest documentation I can found is Xcode 15.
https://developer.apple.com/documentation/CreateML?changes=latest_minor
Really apricated the new feature of object tracking, thank you Apple Team.
I am trying to attach a button to user's left hand. the position is tracked.
the button stays above the user's left hand. but it doesn't face the user. or it doesn't even face where the wrist is pointing. this is the main code snippet:
if (model.editWindowAdded) {
let originalMatrix = model.originFromWristLeft
let theattachment = attachments.entity(for: "sample")!
entityDummy.addChild(theattachment)
let testrotvalue = simd_quatf(real: 0.9906431, imag: SIMD3<Float>(-0.028681312, entityDummy.orientation.imag.y, 0.025926698))
entityDummy.orientation = testrotvalue
theattachment.position = [0, 0.1, 0]
var timer = Timer.scheduledTimer(withTimeInterval: 0.1, repeats: true) {_ in
let originalMatrix = model.originFromWristLeft
print(originalMatrix.columns.0.y)
let testrotvalue = simd_quatf(real: 0.9906431, imag: SIMD3<Float>(-0.028681312,0.1, 0.025926698))
entityDummy.orientation = testrotvalue
}
}
The 3D object capture feature doesn’t seem to work on my iphone 12 pro. The circle that is supposed to show up when you begin to begin to move around the object doesnt show up so object capture doesn’t even begin. It says ‘more light..’ or ‘move closer’ but this doesnt happen on my iphone 14 pro. Works perfectly fine on that even with the same lighting. How can this be fixed?
Hey Everyone, this is like my first post here in the apple forum.
I need your help to understand better Reality Kit and file exports, but let me explain.
I'm trying to create a little 3D Object editor, and it looks like to work pretty well using RealityViews and managing materials on the Entity.
I'm currently working with all the Beta Apis and I would like to export my entity into an .usdz or a .obj file.
I've found a method that allows me to create a .Reality File
let path = FileManager.default.urls(for: .documentDirectory,
in: .userDomainMask)[0].appendingPathComponent("model.reality")
try await self.appState.parentEntity.write(to: path)
but I now I don't know how to convert it into a .usdz or a .obj file, or otherwise any standard 3d format.
Do you have any idea on how could I do?
Thankyou so much!
Have a nice day ^^
When I wanted to call the Reality Composer Pro scene containing Object Tracking, I tried the following code:
RealityView { content in
if let model = try? await Entity(named: "Scene", in: realityKitContentBundle) {
content.add(model)
}
}
Obviously, this is wrong. We need to add some configurations that can enable Object Tracking to Reality View. What do we need to add?
Hi All,
I am using RealityKit along with ARKit and Swift UI to develop an app where I am augmenting a usdz model of a complex geometry like that of a car.
I have some other usdz files with a simple plane geometry having the material properties embedded within them which also i am loading as model entities.
I want to traverse through my car usdz file such that i can pick the material from simple usdz file and apply it to the car as car paint. To do this i know the name of the mesh holding the car paint as well as the name of the material applied.
I have tried to traverse through the usdz files using both RealityKit and SceneKit but I am not successful to reach to the lowest mesh and copy the material properties to it.
With RealityKit, I have tried to get the instance data using modelEntity as follows :-
"sourceModel?.model?.mesh.contents.instances". But this returns instance id, model name and transform only.
Any help will be highly appreciated.
Thank You
I have an app on the App Store for many years enabling users to post text into clouds in augmented reality. Yet last week abruptly upon installing the app on the iPhone the screen started going totally dark and a list of little comprehensible logs came up of the kind:
ARSCNCompositor <0x300ad0e00>: ARSCNCompositor (0, 0) initialization failed. Matting is not set up properly.
many times, then
RWorldTrackingTechnique <0x106235180>: Unable to update pose [PredictorFailure] for timestamp 870.392108
ARWorldTrackingTechnique <0x106235180>: Unable to predict pose [1] for timestamp 870.392108
again several times and then:
ARWorldTrackingTechnique <0x106235180>: SLAM error callback: Error Domain=Slam Error Code=7 "Non fatal error occurred due to significant drop in a IMU data" UserInfo={NSDescription=Non fatal error occurred due to significant drop in a IMU data, NSLocalizedFailureReason=SlamEngineNodeGroup Failure: IMU issue: gyro data stream verification failed [Significant data drop]. Failed on timestamp: 870.413247, Last known timestamp: 865.350198, Delta: 5.063049, System timestamp: 870.415781, Delta between system and frame: 0.002534. }
and then again the pose issues several times.
I hoped the new beta version would have solved the issue, but it was not the case. Unfortunately I do not know if that depends on the beta version or some other issue, given the app may be not installed on the Mac simulator.
Hi, I have a problem in the new version of visionOS (2.0). When I call a magnify gesture for an attachment some times it works fine but some times the gesture doesn't call onEnded and onChanged is called just one time. Is it something known
it is my code
let portal = Entity()
portal.components[ModelComponent.self] = .init(mesh: .generatePlane(width: Float(size.width),
height: Float(size.height),
cornerRadius: 0.02),
materials: [PortalMaterial()])
portal.components[PortalComponent.self] = .init(target: world)
portal.components[PortalComponent.self]?.clippingPlane = .init(position: SIMD3(x: 0, y: 0, z: 0), normal: SIMD3(x: 0, y: 0, z: 0))
portal.components.set(HoverEffectComponent())
I added RealityView to multiple HStacks and implemented the portal effect. I found that the portal effect would cause confusion in the rendering level on some machines, as shown in the figure
Hi everyone,
I'm curious about the capabilities of ARKit's object tracking feature. Specifically, I'd like to know:
Is there a size limit for the objects that can be tracked?
Can ARKit differentiate between two objects with the same shape but different models (e.g., different colors)?
Are objects with single colors and generic shapes (like squares or circles) effectively trackable?
Any insights or examples from your experiences would be greatly appreciated!
Thanks in advance.
We are building an app that uses ARKit occasionally, but not always.
We would like to test the non-ARKit parts in the simulator, since it offers more debugging features (e.g. SwiftUI previews or the Thread Sanitizer).
However, we can't even build the app for the simulator, since the simulator SDK does not know about certain classes (e.g. "AnchorEntity"). This also means that none of the SwiftUI previews work, even if the views are not using ARKit.
What is the best approach to test such an app in the simulator, without using any ARKit features?
I am using ArKit to create an augmented reality application in Unity. Following the addition of an object reference object Because it tracks the object in front of it slowly and inaccurately, the application does not update the screen quickly.
How can I track objects more quickly?
FYI.
The source code of the FindSurface demo app for Apple Vision Pro (visionOS) is available now.
The Swift package of FindSurface™ library is required to build the source code into the demo app.
https://github.com/CurvSurf/FindSurface-visionOS
After starting the app, the floating panels (below) will appear on your right side, and you will see wireframe meshes that approximately describe your environments. Performing a spatial tap (pinching with your thumb and index finger) with staring at a location on the meshes will invoke FindSurface, with an indicator (blue disk) appearing on the surface you've gazed.
Voice commands:
“Tap” – Spatial tap (gazing & pinching). Invoke FindSurface.
“Tap plane” – Plane selection.
“Tap sphere” or “Tap ball” – Sphere selection.
“Tap cylinder” – Cylinder selection.
“Tap cone” – Cone selection.
“Tap torus” or “Tap donut” – Torus selection.
“Tap accuracy” or “Tap measurement accuracy” – Accuracy selection.
“Tap mean distance”, “Tap average distance”, or “Tap distance” – Avg. Distance selection.
“Tap touch radius” or “Tap seed radius” – Touch Radius selection.
“Tap Inlier” – “Show inlier points” toggle.
“Tap outline” – “Show geometry outline” toggle.
“Tap clear” – “Clear Scene” click.
Since moving up to IOS18 lat week I am getting an indication that there was a significant drop in IMU data being sent. Using the search capability, I can find very little information in the Developer Documentation that will tell me what the cause is and how to remedy it. Is there some documentation repository like Tech Notes that will tell me what I need to know to get going again? What additional sources of documentation are available for developers? The Search engine used for Developer documentation just does not cut it because it delivered a lot of useless entries that have no obvious relevance to my search terms.
"2024-06-20 19:27:00.669334-0500 RoomPlanExampleApp[902:299709] [Technique] ARWorldTrackingTechnique <0x104bf6d80>: SLAM error callback: Error Domain=Slam Error Code=7 "Non fatal error occurred due to significant drop in a IMU data" UserInfo={NSDescription=Non fatal error occurred due to significant drop in a IMU data, NSLocalizedFailureReason=SlamEngineNodeGroup Failure: IMU issue: gyro data stream verification failed [Significant data drop]. Failed on timestamp: 53902.785827, Last known timestamp: 53901.416828, Delta: 1.368999, System timestamp: 53902.786251, Delta between system and frame: 0.000423. }"
While using the iOS 18 Beta, I encountered the following bugs and have some feedback:
Bugs:
Notifications: Notifications sometimes appear very small, making them difficult to read and manage. Additionally, notifications occasionally disappear completely or get placed at the bottom of the screen, where they are hard to notice. This causes issues in keeping track of important information.
Feedback
Passwords: It is not possible to view the personal hotspot password and QR code in the passwords application. This is particularly problematic when someone wants to share their hotspot with others, as the password must be entered manually.
Please investigate these issues so they do not persist in the final release.
Trouble with Core ML Object Tracking for Spherical Objects Using WWDC Sample Code and Object Capture
Hi everyone,
I'm working with Core ML for object tracking and have successfully trained a couple of models. However, when I try to use the reference object file in the object tracking sample code from the WWDC video, it doesn't work.
I'm training the model on a single-color plastic spherical object. Could this be the reason for the issue? I also attempted to use USDZ 3D assets that resemble the real object. Do these need to be trained with the Object Capture app to work properly?
Speaking of the Object Capture app, my experience hasn't been great. When I scanned my spherical object, the result was far from a sphere—it looked more like a mushy dough.
Any insights or suggestions would be greatly appreciated!
I have indoor map and I want to draw path between two route location ex. from A to B I want to draw ARKit based Arrow path in ios Application. Currently I am using ARAnchor to achieve this but challenges is if A to B is 10 meter and I am adding Nodes on each one meter so instead of 10 different nodes i am getting single Arrow nodes showing all 10 in it. I am using below code.
// Below Code from where I am calling addArpath function
if let lat1 = mCurrentPosition?.latitude, let long1 = mCurrentPosition?.longitude {
let latEnd = steplocation.latitude
let longEnd = steplocation.longitude
// if let lastLat = arrpath.last?.latitude,let lastLong = arrpath.last?.longitude,let lastAltitude = arrpath.last?.altitude{
let userLocation = CLLocation(latitude: lat1, longitude: long1)
let endLocation = CLLocation(coordinate: CLLocationCoordinate2DMake(CLLocationDegrees(latEnd), CLLocationDegrees(longEnd)), altitude: CLLocationDistance(steplocation.altitude), horizontalAccuracy: CLLocationAccuracy(5), verticalAccuracy: CLLocationAccuracy(0), course: CLLocationDirection(-1), speed: CLLocationSpeed(5), timestamp: Date())
let heading = getHeadingForDirectionFromCoordinate(from: userLocation, to: endLocation)
let lon1 = degreesToRadians(long1) //DegreesToRadians(long1)
let lon2 = degreesToRadians(longEnd); //DegreesToRadians(longEnd);
let lat2 = degreesToRadians(latEnd);
let dLon = lon2 - lon1
let y = sin(dLon) * cos(lat2);
yVal = yVal + y
// let distanceToendpoint = calculateDistance(lat: endLocation.coordinate.latitude, long: endLocation.coordinate.longitude)
let distvalue = Int(distance) + Int(pathlength)
distance += CGFloat(distvalue)
for i in stride(from: 0, to: distance, by:1) {
print("current loop iteration is:" ,i)
let headingValue = heading - self.heading
zValue = zValue + headingValue
distanceVal = CGFloat(i) + distanceVal
zGlobal = zValue
// Calling addARPathtoLocation
addARPathtoLocation(stepLocation:endLocation,zvalue: zValue, yvalue: yVal, distance:Float(i), direction: manuoverType)
}
// }
}
// MARK: - ARSessionDelegate
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard !(anchor is ARPlaneAnchor) else { return }
let sphereNode = generateArrowNodes(anchor: anchor)
DispatchQueue.main.async {
node.addChildNode(sphereNode)
}
}
//create ARAnchor to add to nodes
func generateArrowNodes(anchor: ARAnchor) -> SCNNode {
let imageMaterial = SCNMaterial()
imageMaterial.isDoubleSided = true
imageMaterial.diffuse.contents = UIImage(named: "blueArrow")
let plane = SCNPlane(width:0.5, height:0.5)
plane.materials = [imageMaterial]
plane.firstMaterial?.isDoubleSided = true
let blueNode = SCNNode(geometry: plane)
blueNode.name = "blueNode"
blueNode.position = SCNVector3(x:Float(zGlobal), y:0, z:Float(distanceVal))
blueNode.eulerAngles.x = -.pi / 2
blueNode.eulerAngles.y -= Float(CGFloat(CGFloat.pi/4*6))
return blueNode
}
func addARPathtoLocation(stepLocation: CLLocation, zvalue: CGFloat, yvalue: CGFloat, distance:Float, direction:VMEManeuverType) {
// give you the depth of anything ARKit has detected
guard let query = sceneView.raycastQuery(from: sceneView.center , allowing: .estimatedPlane, alignment: .any) else {
return
}
let results = sceneView.session.raycast(query)
guard let hitResult = results.first else {
print("No surface found")
return
}
// Add ARAnchor to Scene
let anchor = ARAnchor(transform: hitResult.worldTransform)
sceneView.session.add(anchor: anchor)
}
func radiansToDegrees(_ radians: Double) -> Double {
return (radians) * (180.0 / Double.pi)
}
func degreesToRadians(_ degrees: Double) -> Double {
return (degrees) * (Double.pi / 180.0)
}
func getHeadingForDirectionFromCoordinate(from: CLLocation, to: CLLocation) -> Double {
let fLat = degreesToRadians(from.coordinate.latitude)
let fLng = degreesToRadians(from.coordinate.longitude)
let tLat = degreesToRadians(to.coordinate.latitude)
let tLng = degreesToRadians(to.coordinate.longitude)
let deltaL = tLng - fLng
let x = sin(deltaL) * cos(tLng) //cos(tLat) * sin(deltaL)
let y = cos(fLat) * sin(tLat) - sin(fLat) * cos(tLat) * cos(deltaL)
let bearing = atan2(x,y)
let bearingInDegrees = bearing.toDegrees
print("Bearing Degrees :",bearingInDegrees) // sanity check
// let degree = radiansToDegrees(atan2(sin(tLng-fLng)*cos(tLat), cos(fLat)*sin(tLat)-sin(fLat)*cos(tLat)*cos(tLng-fLng)))
if bearingInDegrees >= 0 {
return bearingInDegrees
} else {
return bearingInDegrees + 360
}
}
In our VisionOS project we want to apply wall panel on walls in a room , we are occluding furniture kept in front of walls by creating a mesh of occlusion material on everything except walls but this way is not able to occlude objects perfectly , edges are not smooth , panel coming over tv, table etc.
Is there any other way to achieve this.