Google kicked off its I/O 2024 presentation by highlighting the latest advancements in artificial intelligence. One of the standout projects is Project Astra, described by Google DeepMind as “the future of assistants.”

Demis Hassabis, co-founder of Google DeepMind, explained that their engineers have always aimed to create a universal AI agent that can be helpful in all aspects of daily life. “An agent needs to understand and respond to the complex and dynamic world just like people do.” Hassabis said. “And take in and remember what it sees and hears to understand context and take action.”

This AI assistant would possess all the qualities we seek in a personal assistant. It would be able to watch, learn, and communicate with us in real-time, without any delays. Overcoming these delays has been one of the biggest challenges for Google DeepMind, but they have made significant progress.

Project Astra is an innovative future project based on AI agents. The first prototype can recognize objects using a mobile camera. Users can ask it to identify objects that produce sound, point out parts of a speaker, and even give creative responses about crayons. Perhaps most impressively, the assistant can also recognize code on a computer screen, explain how it works, and determine your location in the city just by looking out the window.

The Project Astra demonstration goes even further by showing interactions with other devices. At one point, the user switches from using a mobile phone to smart glasses, which have the assistant powered by Gemini integrated. These glasses resemble the Google Glass with AR that were showcased in 2022.

What Powers Project Astra

At I/O 2024, the Google DeepMind co-founder revealed that Project Astra is powered by Gemini artificial intelligence.

Google engineers have developed agents that can process information more quickly by continuously encoding video frames, integrating video and voice input into a timeline of events, and storing this information for efficient retrieval.

“These agents can better understand the context they’re being used in, and respond quickly, in conversation.” Hassabis said. “With technology like this, it’s easy to envision a future where people could have an expert AI assistant by their side, through a phone or glasses.”

Project Astra is still in its early stages, but some of these advancements will be available in the Gemini app and web version by the end of the year.