Google DeepMind recently unveiled SIMA (Scalable Instructable Multiworld Agent), an AI agent with the ability to play video games in a manner strikingly similar to human players.
SIMA’s inception is the result of extensive research and development by Google DeepMind, aiming to bridge the gap between AI’s capabilities and human-like intuition and adaptability within 3D virtual settings. Drawing inspiration from its predecessor, AlphaStar, which made headlines for mastering the complex strategy game StarCraft II, SIMA has been trained across a broad spectrum of video games.
This training included collaborations with eight game studios, testing the AI in nine different video games such as “No Man’s Sky” by Hello Games and “Teardown” by Tuxedo Labs. These games, known for their intricate worlds and mechanics, provided the perfect training ground for SIMA to develop a diverse skill set — from simple navigation and menu use to more complex tasks like mining resources and crafting items.
The core of SIMA’s learning process is its ability to understand and execute tasks based on natural-language instructions. This marks a significant departure from previous AIs, focusing on a generalized approach to AI gaming research. SIMA’s technology is built on pre-trained vision models and a central model equipped with memory. This sophisticated setup allows SIMA to interpret both visual and language inputs, translating them into actions within the game using keyboard and mouse inputs, akin to a human player.
One of the most notable aspects of SIMA’s development is its training in a new environment built with Unity called the Construction Lab. Here, SIMA honed its skills in object manipulation and intuitive understanding of physical world concepts by building sculptures from blocks. This aspect of training highlights the AI’s adaptability and potential for applications beyond gaming.
SIMA’s performance has been evaluated across 600 basic skills, demonstrating proficiency in tasks such as navigation, object interaction, and menu navigation — all within short durations. What sets SIMA apart is its ability to generalize knowledge across multiple environments, showing competence even in games it was not explicitly trained on. This generalization is largely due to the AI’s reliance on language proficiency, with controlled tests showing that language inputs significantly influence SIMA’s behavior.
