Guided Vision launches in Gemini Live, and Google says it is not a white cane replacement

Guided Vision launches in Gemini Live, and Google says it is not a white cane replacement

Google launched Guided Vision in Gemini Live on October 1, a mode that describes what an Android phone’s camera is pointed at and tells the user how to move the phone to get a better look. Google says it brings "voice-forward, real-time visual interpretation to people who are blind, have low vision, or want intuitive visual assistance". The same post draws the line around it: Guided Vision is "not a medical device, mobility aid, or white cane replacement".

Android Drop artwork with Google app icons

Google previewed Guided Vision in the September Android Drop on September 1. Image: Google

That disclaimer runs further. "As with any generative AI technology, Guided Vision is an assistive utility and can make mistakes," Google writes, and it is "not intended for navigation, safe-travel guidance, or obstacle detection. Users should continue relying on established mobility aids and safe-travel practices in physical environments."

Gemini tells you how to move the camera

The feature a user notices first is the coaching. "If your camera is pointed too high, too close, or slightly off to the side, Gemini provides natural verbal prompts to help you reframe," Google says, "asking you to pan slowly to the right, tilt downward, or step back so it can get the clear visual context needed to answer your questions."

Camera sharing in Gemini Live is not new; the spoken reframing cues and the follow-up questions around them are what Guided Vision adds. Google lists the jobs it has in mind: reading small nutrition labels, appliance dials and washing machine cycles, or a printed menu in a dimly lit restaurant; finding a dropped earbud on the floor or the black pepper in a crowded spice cabinet; asking about colours, patterns and shapes when picking an outfit; and describing an unfamiliar room’s layout.

Google adds that the same help reaches further than it was built for, naming older adults, people with low literacy and anyone reading fine print in poor light.

Aira trained it, with more than 1,000 testers

Google built the model with Aira, the visual interpreting service. "To train Gemini Live for conversational, real-world context, we partnered with Aira to visually interpret data for a total of tens of thousands of hours," Google says. "More than 1,000 members of Aira’s Trusted Tester network helped stress-test and refine the model across daily routines, while Aira specialists worked directly with our teams as subject-matter experts to help establish and evaluate our safety guardrails."

Google says the work drew on testing and feedback from blind and low-vision communities in India, Brazil, Singapore, Indonesia and Japan among others, and that Guided Vision handles conversations in several languages.

Three ways to start it on Android

The first route is the Gemini app: open the profile settings, switch on Use Guided Vision in Live, then open Gemini Live and share the camera. The second is an Android accessibility shortcut, set under Settings, Accessibility, Vision assistance, Guided Vision, which can be triggered by the floating accessibility button, a two-finger swipe or a press of both volume keys.

The third is for TalkBack users, who can open the TalkBack menu with a three-finger tap and pick Guided Vision from there. Google says the feature is available on Android devices running Android 9 and above, in the regions and languages where Gemini Live is supported, and that both the Gemini app and the system software need to be up to date.

Google previewed Guided Vision in the September Android Drop, which went up on September 1 and described the feature as coming soon. This is a phone feature, with no accuracy or latency figure attached to it, which leaves Google’s own framing as the measure of it: an assistive utility that can make mistakes, and not something to navigate by. Gemini Live’s other recent addition, an avatar that switches between 97 languages, went to Gemini Enterprise rather than the consumer app.

1 Comment

  1. Mel Colton says:

    That explains why Google declined to assist my Wolpz app that is specifically designed to help individuals with sight issues read smallprint and packaging I guess 😁

Leave a Comment

Your email address will not be published. Required fields are marked *