Google’s AI Mode can now see and search with images

Google is adding multimodal capabilities to its search-centric AI Mode chatbot that enable it to “see” and answer questions about images, as it expands access to AI Mode to “millions more” users.

The update combines a custom version of Gemini AI with the company’s Lens image recognition tech, allowing AI Mode Search users to take or upload a picture and receive a “rich, comprehensive response with links” about its contents. The multimodal update for AI Mode is available starting today and can be accessed in the Google app on Android and iOS.

“AI Mode builds on our years of work on visual search and takes it a step further,” says Robby Stein, VP of product for Google Search. “With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes, and arrangements.”

Google says the update uses a “fan-out technique” that issues multiple queries about the image it sees, and any objects within it, to provide responses that are “incredibly nuanced and contextually relevant.” That allows it to do things like identify books that are displayed within an image, issue suggestions for similar titles with positive ratings, and answer questions to further curate recommendations.

AI Mode for Search serves as Google’s answer to Perplexity and ChatGPT Search, a chatbot-like experience that responds to inquiries with AI-generated summaries pulled from everything in Google’s search index.

AI Mode launched exclusively for Google One AI Premium subscribers last month, though only within Labs. Now, Google says it has started to make AI Mode available to “millions more” Labs users in the US, beyond just paying AI Premium subscribers.