hey if i wanted to create an app that takes screenshots from an apple device (and any app within) to give context to an ai so the ai can then respond. Then the app parses the response then executes commands on behalf of the ai/user,
how would I do so with the rule that "screenshots/captures are not allowed within other apps"? Want to stay within bounds of the rules in place.
Possibilities: Ai assistant, Ai pals, passive automation
basically ai-controlled devices/apps, that can better assist people. The parser acts as the Ai's eyes and hands sending it images and executing commands on its behalf. :3
hope I got the tags right on this post lol. hello? :x
If a different app doesn't allow screenshots, there's really no way to override that setting. One option you might try is instead of having an app with system level permissions, allow the user to bring the screenshots they have into your app (by asking the user to give your app access to Photos) and then process them from inside your app.
ok i think i found a way around this. You record in the app ask the USER for permission(s). Then use replay framework to capture screen, processes it live. Send to ai endpoint (needs to be a singular text-image chat endpoint), parse response implement touch controls and execute them. phew. I already have the bones of the app for another project. So might well figure out how to do this in iphone lingo (swift/ios ?). And I wouldn't technically be breaking any Apple rules.
eh we still need the singular text-image chat ai endpoint ofcourse.
but then there is a new problem, even with Replaykit and the magic endpoint we need a way for the ai to interact with the device. Apparently that's also tricky hmm as Apple doesn't have any way to programmatically control or send touch/tap commands or something similar to a virtual keyboard or virtual mouse commands akin to pyautogui/py module:keyboard. hmmm the ai would there but heavily locked down, eh. :x
is there any kind of special Apple's Remote Management Service like remote service to dox or kyc myself to get that kind of programmatic functionality for the project? :x
