Getting selected text from ImageAnalysisInteraction on UIImageView

I work on an iOS app that displays images that often contain text, and I'm adding support for ImageAnalysisInteraction as described in this WWDC 2022 session. I have gotten as far as making the interaction show up and being able to select text and get the system selection menu, and even add my own action to the menu via the buildMenuWithBuilder API. But what I really want to do with my custom action is get the selected text and do a custom lookup-like thing to check the text against other content in my app.

So how do I get the selected text from an ImageAnalysisInteraction on a UIImageView? The docs show methods to check if there is selected text, but I want to know what the text is.

Up vote post of Mr. Jefferson Down vote post of Mr. Jefferson
2.1k views
Posted by
Mr. Jefferson
Reply

  • I've filed this as Feedback FB10794147.

    —
    Mr. Jefferson
Add a Comment

Accepted Reply

There's an API for this in iOS 17 and later! ImageAnalysisInteraction.selectedText. https://developer.apple.com/documentation/visionkit/imageanalysisinteraction/4176689-selectedtext

Posted by
Mr. Jefferson
Up vote reply of Mr. Jefferson Down vote reply of Mr. Jefferson
Post marked as solved
Add a Comment

Replies

To clarify my use case: my app is an e-reader app focused on Biblical materials, with a cross-referencing system that can surface content from any downloaded books related to any verse or entity (people, places, things, ideas). There are lots of atlas-like resources with maps of the ancient near east in Biblical times, and it would be amazing to be able to select the name of a city on a map image and feed it into my existing functionality to find content related to that city name - verses, encyclopedia articles, etc.

[edit: this should have been a comment, but now I can't find a way to delete this "reply"]

Posted by
Mr. Jefferson
Up vote reply of Mr. Jefferson Down vote reply of Mr. Jefferson
Add a Comment

I am currently facing the same problem, enabled Live Text and couldn't get selected text so had to use VNRecognizeTextRequest instead. Have you found a solution to this?

Posted by
josessolmaximo
Up vote reply of josessolmaximo Down vote reply of josessolmaximo
Add a Comment

There's an API for this in iOS 17 and later! ImageAnalysisInteraction.selectedText. https://developer.apple.com/documentation/visionkit/imageanalysisinteraction/4176689-selectedtext

Posted by
Mr. Jefferson
Up vote reply of Mr. Jefferson Down vote reply of Mr. Jefferson
Post marked as solved
Add a Comment

How to change the system UIMenuController in VisionKit? I tried a few things but couldn't get it to work!

ex:


    func configMenuItems() {
        self.becomeFirstResponder()
        
        let menu       = UIMenuController.shared
        let paste      = UIMenuItem(title: "***".localized(), action: #selector(myPaste))
        let selectAll  = UIMenuItem(title: "zzz".localized(), action: #selector(mySelectAll))
        menu.menuItems = [paste, selectAll]
        menu.setMenuVisible(true, animated: true)
    }
Posted by
Jxiang
Up vote reply of Jxiang Down vote reply of Jxiang
Add a Comment