Google DeepMind’s SIMA Is an AI Agent That Can Play 3D Video Games

Google DeepMind introduced a unique artificial intelligence (AI) model on Wednesday that can play 3D video games just like a human would. The AI model is called Scalable Instructable Multiworld Agent or SIMA, and it is learning how to interact with various gaming environments and carry out different tasks. The model is currently in research and is still being trained to build expertise in performing more complex movements. Google says that once perfected, the AI model can have a wide range of use cases in both the online and real world.

In a blog post, DeepMind’s SIMA team explained that the AI model is not being developed to be a super-intelligent gamer that can beat any game. Instead, the aim is to teach it how to move around in a 3D game in open-world situations and understand how a human would react and interact with it using natural-language instructions. Google highlights that it is a significantly difficult task and one that should be a focus.

“This is an important goal for AI in general, because while Large Language Models have given rise to powerful systems that can capture knowledge about the world and generate plans, they currently lack the ability to take actions on our behalf,” the SIMA team stated.

Tasks performed by AI model SIMA
Photo Credit: Google DeepMind

To create a learning environment for the AI model, Google DeepMind partnered with eight game studios and trained SIMA on nine different video games. Some of them include No Man’s Sky by Hello Games, Teardown by Tuxedo Labs, Goat Simulator 3 and Valheim by Coffee Stain Studios, and more. The AI model was exposed to new interactive worlds in each game and had to learn navigating through the world, interaction with objects, menu use, and more.

The company also created four research environments including a new environment built with Unity called the Construction Lab. In this particular environment, the AI model was tasked to build sculptures from building blocks to test its object manipulation and intuitive understanding of the physical world.

Google highlighted that the current version of SIMA was evaluated across 600 basic skills including navigation such as turning left and driving a car, object interaction such as climbing a ladder and crafting a helmet, and more. The tasks were largely simple and could be completed within 10 seconds.

According to the company, training an AI model on a diverse set of 3D video games where it follows human instructions and acts accordingly can have a major impact in future. The company is now focusing on training SIMA with more complex instructions “that require high-level strategic planning and multiple sub-tasks to complete, such as find resources and build a camp.” Google hints that these tasks can translate to the real world and the company is working on finding ways it can be helpful to humans.