Following the post on
https://developer.apple.com/documentation/realitykit/custommaterial it's simple to use shader for materials and get uniforms and params from each vertex. However it's not available for visionOS. Any alternative to use in this case? I want to write shader to fill material by myself. (I have shader experience from web, familiar with fragment shader)
Delve into the world of graphics and game development. Discuss creating stunning visuals, optimizing game mechanics, and share resources for game developers.
Post
Replies
Boosts
Views
Activity
Hi all im having a variety of issues with gamekit matchmaking. On the simulator the matchmaking ui pops up and I can click Quick Match, then immediately "Failed to find Players" this is the same with a real Apple ID and a sandbox account.
If I use real devices the app at least discovers a match, but then the match none of the delegate methods for the match ever get called and the logs are filled with socket not connected and various errors.
My questions are:
Should match making via quick match work in the simulator, I have seen tutorial videos etc of this working, but I can't seem to get it to work.
How do people debug issues with GameCenter / Gamekit to find out why its not able to connect?
Many thanks in advance
I am trying to simulate a paste command and it seems to not want to paste. It worked at one point with the same code and now is causing issues.
My code looks like this:
` func simulatePaste() {
guard let source = CGEventSource(stateID: .hidSystemState) else {
print("Failed to create event source")
return
}
let keyDown = CGEvent(keyboardEventSource: source, virtualKey: CGKeyCode(9), keyDown: true)
let keyUp = CGEvent(keyboardEventSource: source, virtualKey: CGKeyCode(9), keyDown: false)
keyDown?.flags = .maskCommand
keyUp?.flags = .maskCommand
keyDown?.post(tap: .cgAnnotatedSessionEventTap)
keyUp?.post(tap: .cgAnnotatedSessionEventTap)
print("Simulated Cmd + V")
}
I know that there is some issues around permissions and so in my Info.plist I have this:
<string>NSApplication</string>
<key>NSAppleEventsUsageDescription</key>
<string>This app requires permission to send keyboard input for pasting from the clipboard.</string>
I have also disabled sandbox. It does ask me if I want to give the app permissions but after approving it, it still doesn't paste.
Hey there,
I tried to install GPTK again, since I had to reinstall the OS for irrelevant reasons. But every time I try to install the tool kit, it gives me theError: apple/apple/game-porting-toolkit 1.1 did not build error. Before that error occired, I had the Openssl error, which I fixed with the rbenv version of openssl. Is there any way to fix this error? Down bellow you'll find the full error message it gave me. The specs for my Mac are (if they are helpful in any way): M1 Pro MBP 14" with 16GB Ram and 512GB SSD.
Thanks!
``Error: apple/apple/game-porting-toolkit 1.1 did not build
Logs:
/Users/myuser/Library/Logs/Homebrew/game-porting-toolkit/00.options.out
/Users/myuser/Library/Logs/Homebrew/game-porting-toolkit/01.configure
/Users/myuser/Library/Logs/Homebrew/game-porting-toolkit/01.configure.cc
/Users/myuser/Library/Logs/Homebrew/game-porting-toolkit/wine64-build
If reporting this issue please do so at (not Homebrew/brew or Homebrew/homebrew-core):
https://github.com/apple/homebrew-apple/issues```
Helle there
Currently, I’m attempting to create an interactive learning application with a 3D view. I’ve discovered the framework SceneKit, but I lack the necessary knowledge to animate, load and moving objects. Could someone kindly suggest some good articles or tutorials on this topic?
When inspecting the geometry in Xcode's metal debugger, I noticed that the shown "frustrum box" didn't make sense. Since Metal uses depth range 0,1 in NDC space, I would expect a vertex that is projected to z:0 to be on the front clipping plane of the frustrum shown in the geometry inspector. This is however not the case. A vertex with ndc z:0 is shown halfway inside the frustrum. Vertices with ndc z less than 0 are correctly culled during rendering, while the geometry inspector's frustrum shows that the vertex is stil inside the frustrum.
The image shows vertices that are visually in the middle of the frustrum on z axis, but at the same time the out position shows that they are projected to z:0. How is this possible, unless there's a bug in the geometry inspector?
Hi,
I wanted to do something quite simple: Put a box on a wall or on the floor.
My box:
let myBox = ModelEntity(
mesh: .generateBox(size: SIMD3<Float>(0.1, 0.1, 0.01)),
materials: [SimpleMaterial(color: .systemRed, isMetallic: false)],
collisionShape: .generateBox(size: SIMD3<Float>(0.1, 0.1, 0.01)),
mass: 0.0)
For that I used Plane Detection to identify the walls and floor in the room. Then with SpatialTapGesture I was able to retrieve the position where the user is looking and tap.
let position = value.convert(value.location3D, from: .local, to: .scene)
And then positioned my box
myBox.setPosition(position, relativeTo: nil)
When I then tested it I realized that the box was not parallel to the wall but had a slightly inclined angle.
I also realized if I tried to put my box on the wall to my left the box was placed perpendicular to this wall and not placed on it.
After various searches and several attempts I ended up playing with transform.matrix to identify if the plane is wall or a floor, if it was in front of me or on the side and set up a rotation on the box to "place" it on the wall or a floor.
let surfaceTransform = surface.transform.matrix
let surfaceNormal = normalize(surfaceTransform.columns.2.xyz)
let baseRotation = simd_quatf(angle: .pi, axis: SIMD3<Float>(0, 1, 0))
var finalRotation: simd_quatf
if acos(abs(dot(surfaceNormal, SIMD3<Float>(0, 1, 0)))) < 0.3 {
logger.info("Surface: ceiling/floor")
finalRotation = simd_quatf(angle: surfaceNormal.y > 0 ? 0 : .pi, axis: SIMD3<Float>(1, 0, 0))
} else if abs(surfaceNormal.x) > abs(surfaceNormal.z) {
logger.info("Surface: left/right")
finalRotation = simd_quatf(angle: surfaceNormal.x > 0 ? .pi/2 : -.pi/2, axis: SIMD3<Float>(0, 1, 0))
} else {
logger.info("Surface: front/back")
finalRotation = baseRotation
}
Playing with matrices is not really my thing so I don't know if I'm doing it right.
Could you tell me if my tests for the orientation of the walls are correct? During my tests I don't always correctly identify whether the wall is in front or on the side.
Is this generally the right way to do it?
Is there an easier way to do this?
Regards
Tof
I want to use SwiftUI and RealityView to get AR scene understanding data (ARMeshAnchor) on iOS devices with LiDAR. The only way we can do that is by using ARSession (unless there is another way).
However in previous iOS 18 builds there was this function:
https://developer.apple.com/documentation/realitykit/spatialtrackingsession/run(_:session:arconfiguration:)
, which worked with SpatialTrackingSession and a custom ARSession together. This function in the the latest iOS and Xcode has since been removed in the RealityKit framework but still there on documentation.
I also wanted to get ARFaceAnchor data which I still cannot get without ARSession, the closest I can get is by using:
let target = AnchoringComponent.Target.face
let anchoringComponent = AnchoringComponent(target, trackingMode: .predicted)
entity = Entity()
entity!.components.set(anchoringComponent)
But I still can't find a way to get the current frame (ARFrame) or the anchors ([ARAnchor]) in the view.
Alternatively if I use if I use this function: https://developer.apple.com/documentation/realitykit/spatialtrackingsession/run(_:) and start the ARSession separately. The session (didUpdate and didAdd) only runs for a few frames before getting interrupted.
And if I completely remove SpatialTrackingConfiguration and just run the ARSession. There still is a valid tracked entity for the AnchoringComponent.Target.face component. IF in the configuration for the ARSession I use the ARWorldTrackingConfiguration with face tracking. And I still get updated facial data each frame. But the ARSession didUpdate or didAdd functions don't get called passed the first few frames.
Interestingly if I switch the RealityViewCameraContent.RealityViewCamera to .virtual. I get ARMeshAnchor and ARFaceAnchor data, but no camera feed (as expected). This with or without SpatialTrackingConfiguration.
My overarching question is what is the proper way to access ARMeshAnchors and other ARAnchors created by the system and track them live while also using SwiftUI.
GitHub Repo with sample project can be found here: https://github.com/bpate75/RealityViewTesting
I'm trying to build a Shader in "Reality Composer Pro" that updates from a start time. Initially I tried the following:
The idea was that when the startTime was 0, the output would be 0, but then I would set startTime from within code and this would be compared with the current GPU time, and difference used to drive another part of the shader graph:
if
let testEntity = root.findEntity(named: "Test"),
var shaderGraphMaterial = testEntity.components[ModelComponent.self]?.materials.first as? ShaderGraphMaterial
{
let time = CFAbsoluteTimeGetCurrent()
try! shaderGraphMaterial.setParameter(name: "StartTime", value: .float(Float(time)))
testEntity.components[ModelComponent.self]?.materials[0] = shaderGraphMaterial
}
However, I haven't found a reference to the time the shader would be using.
So now I am trying to write an EntityAction to achieve the same effect. Instead of comparing a start time to the GPU's time I'm trying to animate one of the shader's uniform input. However, I'm not sure how to specify the bind target. Here's my attempt so far:
import RealityKit
struct ShaderAction: EntityAction {
let startValue: Float
let targetValue: Float
var animatedValueType: (any AnimatableData.Type)? { Float.self }
static func registerEntityAction() {
ShaderAction.subscribe(to: .updated) { event in
guard let animationState = event.animationState else { return }
let value = simd_mix(event.action.startValue, event.action.targetValue, Float(animationState.normalizedTime))
animationState.storeAnimatedValue(value)
}
}
}
extension Entity {
func updateShader(from startValue: Float, to targetValue: Float, duration: Double) {
let fadeAction = ShaderAction(startValue: startValue, targetValue: targetValue)
if let shaderAnimation = try? AnimationResource.makeActionAnimation(for: fadeAction, duration: duration, bindTarget: .material(0).customValue) {
playAnimation(shaderAnimation)
}
}
}
'''
Currently when I run this I get an assertion failure: 'Index out of range (operator[]:line 797) index = 260, max = 8'
Furthermore, even if it didn't crash I don't understand how to pass a binding to the custom shader value "startValue".
Any clues of how to achieve this effect - even if it's a completely different way.
Hello!
I need to "draw" a set of particles into the texture. It would be trivial in render encoder of course. However, I would like to implement the task in compute kernel. Every particle draw operation is expected to set 5 texels - "center" one and left/right/upper/lower. Particles can and will overlap, so concurrent draws are to be expected.
I tried using texture atomics - atomic_store() to be more precise. This worked, albeit pretty slowly - too slow for my purpose.
Just to test what would happen, I tried using normal texture write(). I was expecting to see some kind of visual artefacts, but to my surprise, it worked very well (and much faster).
My question: is it safe? I understand that calling write() doesn't guarantee any ordering of the operations, so if multiple threads write to the same texel, the final value may come from any of those threads. But suppose all the threads were to write the very same color? Can I assume that the texel in question will have said color after the compute kernel finishes?
I am using M2 Pro MacBook, but ideally I would love to get the answer for the all Apple Silicon devices. My texture format is R32Int (so as to be able to use atomics), but I could do with any single-channel format, the purpose of the texture is to be binary mask of sorts.
Thanks!
Description:
I'm developing an AR effect using SceneKit and applying a transparent material to a face mesh. However, I'm facing an issue where the front faces of the mesh overlap each other, causing incorrect rendering.
Problem:
The front faces of the mesh overlap with each other when transparency is applied.
This causes areas like the cheeks to be visible through the nose, even though they should be occluded.
Expected Behavior: The material should behave as if it were opaque to itself—that is, overlapping front faces should be occluded properly, while still allowing transparency for background elements.
Actual Behavior: The mesh renders its own front faces incorrectly, making parts of the face visible through others when they should be blocked.
What I Have Tried:
testMaterial.writesToDepthBuffer = true
testMaterial.readsFromDepthBuffer = true
Question:
👉 How can I prevent SceneKit's transparent material from rendering overlapping front faces?
👉 Is there a way to force SceneKit to treat its own mesh as opaque for itself while still being transparent to the background?
👉 Does SceneKit support a proper depth pre-pass or an equivalent to Unity’s ZWrite shaders to solve this issue?
Attached screenshots demonstrate the problem visually. Any help would be greatly appreciated! 🚀
The solo Leveling:arise is a game but the game mode is not switching on and game crashing everything time while playing
Hi everyone,
I’m experiencing a critical issue with USDZ files created in Reality Composer on an iPad 9th Generation (iPadOS 18.3). The files work perfectly on iPads from the 10th Generation onwards and on iPad Pros. However, on older devices like the iPad 9th Generation and older iPhones, QuickLook (file preview) crashes when opening them.
This is a major issue because these USDZ files are part of an exhibition where artworks are extended with AR elements via a web page. If some visitors cannot view the 3D content, it significantly impacts the experience.
What’s puzzling is that two years ago, we exported USDZ files from Reality Composer, made them available via a website, and they worked flawlessly on all devices, including older iPads and iPhones. Now, with the latest iPadOS, they consistently crash on older devices.
Has anyone encountered a similar issue? Are there known limitations with QuickLook on older devices, or is there a way to optimize the USDZ files to prevent crashes? Could this be related to changes in iPadOS or RealityKit? Any advice or workaround would be greatly appreciated!
Thanks in advance!
Hey! I'm facing an issue with Equipment collision when adding and moving TabletopKit equipment with different pose rotations.
Let me share a very simple TabletopKit setup as an example:
Table
struct Table: Tabletop {
var shape: TabletopShape = .rectangular(width: 1, height: 1, thickness: 0.01)
var id: EquipmentIdentifier = .tableID
}
Board
struct Board: Equipment {
let id: EquipmentIdentifier = .boardID
var initialState: BaseEquipmentState {
.init(
parentID: .tableID,
seatControl: .restricted([]),
pose: .init(position: .init(), rotation: .zero),
boundingBox: .init(center: .zero, size: .init(1.0, 0, 1.0))
)
}
}
Equipment
struct Object: EntityEquipment {
var id: ID
var size: SIMD2<Float>
var position: SIMD2<Double>
var rotation: Float
var entity: Entity
var initialState: BaseEquipmentState
init(id: Int, size: SIMD2<Float>, position: SIMD2<Double>, rotation: Float) {
self.id = EquipmentIdentifier(id)
self.size = size
self.position = position
self.rotation = rotation
self.entity = objectEntity
self.initialState = .init(
parentID: .boardID,
seatControl: .any,
pose: .init(
position: .init(x: position.x, z: position.y),
rotation: .degrees(Double(rotation))
),
entity: entity
)
}
}
Setup
class GameSetup {
var setup: TableSetup
init(root: Entity) {
setup = TableSetup(tabletop: Table())
setup.add(equipment: Board())
setup.add(seat: PlayerSeat())
let object1 = Object(
id: 2,
size: .init(x: 0.1, y: 0.1),
position: .init(x: 0.1, y: -0.1),
rotation: 0
)
let object2 = Object(
id: 3,
size: .init(x: 0.2, y: 0.1),
position: .init(x: -0.1, y: -0.1),
rotation: 90
)
setup.add(equipment: object1)
setup.add(equipment: object2)
}
}
The issue
When I add two equipment entities with different rotation poses, the collisions between them behave oddly. If one is 90º and the other 0º, for example, the former will intersect with the latter as if its bounding box was not rotated as you can see below:
But if both equipment have the example rotation (e.g. 0 or 90º), though, then there's no collision issue at all, which seems to indicate their bounding box were correctly rotated:
I'd really appreciate some help understanding if this is a bug or if I'm just missing something.
Thanks in advance!
I am interested in learning the Metal framework for rendering development. However, most of Apple’s official documentation uses Objective-C code. Therefore, I am seeking guidance on whether it is more advantageous for me to focus solely on learning Swift to gain proficiency in Metal.
I have a bare-bones Metal app setup where I attach a CAMetalLayer to a window that inherits from a NSWindow with a custom delegate. Everything else is vanilla. I'm also using metal-cpp and metal shader converter.
I'm running into a issue where the application runs fine in the beginning, but once I resize the window, it starts hitching. It turns out that [CAMetalLayer nextDrawable:] frequently (but not always) takes around a full second (plus or minus a few milliseconds) to return once drawableSize has been updated.
I've tried setting allowsNextDrawableTimeout to false which doesn't work; it returns a valid drawable after a second instead of nil. Setting displaySyncEnabled to false reduces the likelihood of this happening to around 50% from 90%+ but does not eliminate it. Setting maximumDrawableCount to 2 or 3 does not seem to make a difference.
By dumping the resource IDs of the returned textures I've noticed something interesting: Before resizing, the layer seems to shuffle between 2 textures or at least 2 resource IDs, but after resizing it starts to create new textures for each returned drawable. Occasionally it seems to reuse a previous resource ID, but it does not seem to have anything to do with whether the method returns quickly or not.
Why does this happen, and how can I fix it? Should I create a new CAMetalLayer when resizing the window instead of updating drawableSize?
I've just started working on my first SpriteKit game that will eventually run on both tvOS and iOS and am looking at how to build a "button". So far, I've got a custom node that looks like:
class MyButton: SKSpriteNode {
...
#if os(tvOS)
override var canBecomeFocused: Bool {
true
}
override func didUpdateFocus(...) {
...
}
#endif
}
The above let me nicely handle focus changes in tvOS and now I'm looking at reacting to selecting the button.
Searching around, all the articles/questions/posts are from 2015-2016 - which is a LOOOONG time ago. Most of the guidance appears to be to add a tap gesture recognizer in the owning scene and getting the scene to hand it off to the button. That seems pretty brittle and I'd much prefer if the button itself is responsible for its own tap management.
So, I guess my question is whether I should just add a gesture recognizer to my custom button class? Is this inefficient if I end up having 7-8 buttons on the screen and each one has its own gesture recognizer?
Somewhat related, all of the 10-year-old advice is that if we add recognizers to scenes, then they need to be removed from the view controller... however, in the modern day world with SwiftUI, my project doesn't even have a view controller (yet, anyway)... what gesture recognizer lifecycle management do I need in a SpriteKit scene that is presented within a SpriteKitView?
Or, is there a better way? I was kind of hoping that overriding pressesBegan() (or something similar) in my custom button might have been triggered on tvOS (like touchesBegan() lets me manage touches for the iOS variant of my app)
Any pointers or suggestions would be gladly received. Thanks.
Hola. At this point, I'm at my wit's end because I've tried EVERYTHING just to be able to play a single game on my Mac, but the new update makes it impossible.
So I'm just gonna ask for one game: I've been trying to play Stardew Valley, modded with SMAPI, for a week now. Despite playing the game with mods for almost 2 years, OS is refusing to open the game because SMAPI "contains malware". I tried reinstalling the mod, but no dice. It just automatically deletes the terminal and blocks the game from opening.
You can imagine my frustration because the mod has been 100% safe for 2 years. No option to "Open Anyway" in Security Settings, either. I have no say in this. I've tried code signing it in Terminal (three times). Also no dice. Followed these two forums.
https://www.reddit.com/r/StardewValley/comments/1h071jl/mac_deleted_stardew_modding_api_because_of_malware/
https://www.reddit.com/r/SMAPI/comments/1h0fgv9/solution_for_mac_malware_issue_with_smapi_417/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
NOTHING. Please tell me there's a way to override this??? Just let me install malware on my computer!😭
Hi,
I'm working with a very simple app that tries to read a coordinates card and past the data into diferent fields. The card's layout is COLUMNS from 1-10, ROWs from A-J and a two digit number for each cell. In my app, I have field for each of those cells (A1, A2...). I want that OCR to read that card and paste the info but I just cant. I have two problems. The camera won't close. It remains open until I press the button SAVE (this is not good because a user could take 3, 4, 5... pictures of the same card with, maybe, different results, and then? Which is the good one?). Then, after I press save, I can see the OCR kinda works ( the console prints all the date read) but the info is not pasted at all.
Any idea? I know is hard to know what's wrong but I've tried chatgpt and all it does... just doesn't work
This is the code from the scanview
import SwiftUI
import Vision
import VisionKit
struct ScanCardView: UIViewControllerRepresentable {
@Binding var scannedCoordinates: [String: String]
var useLettersForColumns: Bool
var numberOfColumns: Int
var numberOfRows: Int
@Environment(.presentationMode) var presentationMode
func makeUIViewController(context: Context) -> VNDocumentCameraViewController {
let scannerVC = VNDocumentCameraViewController()
scannerVC.delegate = context.coordinator
return scannerVC
}
func updateUIViewController(_ uiViewController: VNDocumentCameraViewController, context: Context) {}
func makeCoordinator() -> Coordinator {
return Coordinator(self)
}
class Coordinator: NSObject, VNDocumentCameraViewControllerDelegate {
let parent: ScanCardView
init(_ parent: ScanCardView) {
self.parent = parent
}
func documentCameraViewController(_ controller: VNDocumentCameraViewController, didFinishWith scan: VNDocumentCameraScan) {
print("Escaneo completado, procesando imagen...")
guard scan.pageCount > 0, let image = scan.imageOfPage(at: 0).cgImage else {
print("No se pudo obtener la imagen del escaneo.")
controller.dismiss(animated: true, completion: nil)
return
}
recognizeText(from: image)
DispatchQueue.main.async {
print("Finalizando proceso OCR y cerrando la cámara.")
controller.dismiss(animated: true, completion: nil)
}
}
func documentCameraViewControllerDidCancel(_ controller: VNDocumentCameraViewController) {
print("Escaneo cancelado por el usuario.")
controller.dismiss(animated: true, completion: nil)
}
func documentCameraViewController(_ controller: VNDocumentCameraViewController, didFailWithError error: Error) {
print("Error en el escaneo: \(error.localizedDescription)")
controller.dismiss(animated: true, completion: nil)
}
private func recognizeText(from image: CGImage) {
let request = VNRecognizeTextRequest { (request, error) in
guard let observations = request.results as? [VNRecognizedTextObservation], error == nil else {
print("Error en el reconocimiento de texto: \(String(describing: error?.localizedDescription))")
DispatchQueue.main.async {
self.parent.presentationMode.wrappedValue.dismiss()
}
return
}
let recognizedStrings = observations.compactMap { observation in
observation.topCandidates(1).first?.string
}
print("Texto reconocido: \(recognizedStrings)")
let filteredCoordinates = self.filterValidCoordinates(from: recognizedStrings)
DispatchQueue.main.async {
print("Coordenadas detectadas después de filtrar: \(filteredCoordinates)")
self.parent.scannedCoordinates = filteredCoordinates
}
}
request.recognitionLevel = .accurate
let handler = VNImageRequestHandler(cgImage: image, options: [:])
DispatchQueue.global(qos: .userInitiated).async {
do {
try handler.perform([request])
print("OCR completado y datos procesados.")
} catch {
print("Error al realizar la solicitud de OCR: \(error.localizedDescription)")
}
}
}
private func filterValidCoordinates(from strings: [String]) -> [String: String] {
var result: [String: String] = [:]
print("Texto antes de filtrar: \(strings)")
for string in strings {
let trimmedString = string.replacingOccurrences(of: " ", with: "")
if parent.useLettersForColumns {
let pattern = "^[A-J]\\d{1,2}$" // Letras de A-J seguidas de 1 o 2 dígitos
if trimmedString.range(of: pattern, options: .regularExpression) != nil {
print("Coordenada válida detectada (letras): \(trimmedString)")
result[trimmedString] = "Valor" // Asignación de prueba
}
} else {
let pattern = "^[1-9]\\d{0,1}$" // Solo números, de 1 a 99
if trimmedString.range(of: pattern, options: .regularExpression) != nil {
print("Coordenada válida detectada (números): \(trimmedString)")
result[trimmedString] = "Valor"
}
}
}
print("Coordenadas finales después de filtrar: \(result)")
return result
}
}
}
I have used the Mac M1 and M4.
Developing OpenGL projects on machines running macOS 15.2 and 13.6.
Call the OpenGL library functions of Mac.
glTexImage2D
If you use GL_LUMINANCE, GL_LUMINANCE_ALPHA, GL_ALPHA these three textures, you will get an error gl 500.
It makes me unable to draw normally on Mac.
What's the reason for this? Don't they support it?