Google is enhancing its visual search app, Lens, by adding the capability to provide near-real-time answers to questions about your surroundings.
Now, English-speaking Android and iOS users who have the Google app installed can begin recording a video through Lens and inquire about objects of interest seen in the video.
Lou Wang, the director of product management for Lens, explained that the feature leverages a “customized” Gemini model to analyze the video and relevant questions. Gemini is Google’s AI model family, which powers various products within the company.
To access the new video analysis feature in Lens, users need to sign up for Google’s Search Labs program and opt-in to the “AI Overviews and more” experimental features in Labs. In the Google app, holding the smartphone’s shutter button activates Lens’ video-capturing mode.
While recording a video, users can ask a question, and Lens will provide a linked answer supplied by AI Overviews, a feature in Google Search that summarizes information from the web using AI.
According to Wang, Lens uses AI to identify the most “interesting” and relevant frames in a video related to the asked question to ground the answer from AI Overviews.
Alongside video analysis, Lens has also introduced the ability to conduct searches using images and text at the same time. English-speaking users can use the Google app to take a photo by holding the shutter button and ask a question verbally.
Moreover, Lens now includes e-commerce-specific functionality. When Lens on Android or iOS identifies a product, it will display details such as price, deals, brand, reviews, and stock in select countries and shopping categories. There’s also an advertising component, with the Lens-identified products showing relevant shopping ads with options and prices.
Google sees this as a significant opportunity, given that approximately 4 billion Lens searches each month are shopping related. The addition of ads in Lens aligns with the company’s advertising-focused business model.
