How can i show UIView on a ModelEntity?
On android, i use ViewRenderable to load a view , and attach it to a Node, to show a View in the AR scene.
But i not found any ways to show view in the AR scene of the ARView used the RealityKit.
Anyone can help me ?
Thanks very much!
Hi,
is your UIView static or does it change a lot? If it's just a static view you could create a snapshot, extract it's CGImage and convert that into a TextureResource (https://developer.apple.com/documentation/realitykit/textureresource/3768851-generate) that could be mapped on whatever model you want.
In case it is animated I assume you could render that view with your desired update interval into a Metal Texture and then use the DrawableQueue API to update your material with the new texture: https://developer.apple.com/documentation/realitykit/textureresource/drawablequeue
Maybe this GitHub sample I've done could help you with the latter option: https://github.com/arthurschiller/realitykit-drawable-queue
Thanks arthurfromberlin.
My UIView is animated, and is a interactive view, maybe contain a web, a table, a list, some buttons, and so on.
In ARKit+SceneKit project, i see someone loading view in the SCNMaterial.diffuse.contents, it work well, as following:
private func show(viewController: BillboardViewController) {
let material = SCNMaterial()
material.isDoubleSided = true
material.cullMode = .front
material.diffuse.contents = viewController.view
billboard?.viewController = viewController
billboard?.billboardNode?.geometry?.materials = [material]
}
But, if i use RealityKit to build a AR scene, it can't use SCNNode and SCNMaterial to create a entity, so , i don't know how to show my UIView in the AR scene.
I read the document of the DrawableQueue carefully, but i don't think it can achieve the functions I need, it lack the ui events loop.
Hi, I fear adding interactions could prove troublesome. Even with the SceneKit option afaik you don't get support for touch gestures etc as it just renders the visual representation of the underlying CALayer. Potentially you could try rerouting touch events from your ARView to the UIView you want to display. Beforehand you could do a hittest and see if the geometry for the custom view intersects with the users touch. To reroute touch events you could like into the UIKit hitTest API: https://khanlou.com/2018/09/hacking-hit-tests/
I had tested the diffuse.contents property, attach a view to it, it work well many times, the ui touch gestures also work well , but the UIEditField can't get the focus, and the input method can't be ejected.
And I need to handle the gestures of the SCNNode, rotations, scales, transforming, these are troubles!
On android platform, in the SceneForm (ARKit ) scene , it use ViewRenderable to load a view.xml, it had same problems with ARKit.
Hi,
is your UIView static or does it change a lot? If it's just a static view you could create a snapshot, extract it's CGImage and convert that into a TextureResource (https://developer.apple.com/documentation/realitykit/textureresource/3768851-generate) that could be mapped on whatever model you want.
In case it is animated I assume you could render that view with your desired update interval into a Metal Texture and then use the DrawableQueue API to update your material with the new texture: https://developer.apple.com/documentation/realitykit/textureresource/drawablequeue
Maybe this GitHub sample I've done could help you with the latter option: https://github.com/arthurschiller/realitykit-drawable-queue
Thanks arthurfromberlin.
My UIView is animated, and is a interactive view, maybe contain a web, a table, a list, some buttons, and so on.
In ARKit+SceneKit project, i see someone loading view in the SCNMaterial.diffuse.contents, it work well, as following:
private func show(viewController: BillboardViewController) {
let material = SCNMaterial()
material.isDoubleSided = true
material.cullMode = .front
material.diffuse.contents = viewController.view
billboard?.viewController = viewController
billboard?.billboardNode?.geometry?.materials = [material]
}
But, if i use RealityKit to build a AR scene, it can't use SCNNode and SCNMaterial to create a entity, so , i don't know how to show my UIView in the AR scene.
I read the document of the DrawableQueue carefully, but i don't think it can achieve the functions I need, it lack the ui events loop.
Hi, I fear adding interactions could prove troublesome. Even with the SceneKit option afaik you don't get support for touch gestures etc as it just renders the visual representation of the underlying CALayer. Potentially you could try rerouting touch events from your ARView to the UIView you want to display. Beforehand you could do a hittest and see if the geometry for the custom view intersects with the users touch. To reroute touch events you could like into the UIKit hitTest API: https://khanlou.com/2018/09/hacking-hit-tests/
I had tested the diffuse.contents property, attach a view to it, it work well many times, the ui touch gestures also work well , but the UIEditField can't get the focus, and the input method can't be ejected.
And I need to handle the gestures of the SCNNode, rotations, scales, transforming, these are troubles!
On android platform, in the SceneForm (ARKit ) scene , it use ViewRenderable to load a view.xml, it had same problems with ARKit.
-
—
Fattie
Add a CommentOne thing, regarding interactions, you'd have to handle that completely manually in a custom way yourself. There is no meaningful relationship between the 3D world and UX interactions. Youi'd have to program it by hand on a case-by-case basis.
If you're familiart with the typical touch handling in 3D engines where you cast through the screen to see what you're touching, that would be the way to begin the process.