AI can beat chess grandmasters, but it can’t adapt to modern video games

Despite all the buzz surrounding AI’s victories in chess, go, and coding, there remains a significant weakness that is often overlooked. AI struggles when faced with a new video game that it has never encountered before.

A recent paper by NYU highlights how these high-profile achievements may give a misleading impression of how close machines are to achieving true general intelligence.

The distinction is crucial. Games like chess and go have fixed rules and structured environments, making it easier for AI to excel. However, when it comes to more complex modern video games, AI falls short as it struggles to adapt like humans do.

Challenges in AI Gaming

Researchers point out that many of AI’s gaming successes are based on systems that are specifically tailored to one game. Within those constraints, AI can outperform humans. However, even minor changes to the game’s rules or environment can cause AI’s performance to plummet.

This is where video games serve as a true test of AI intelligence. Unlike traditional games, modern video games require a wide range of skills such as spatial reasoning, long-term planning, trial-and-error learning, and social intuition. According to the report, gaming provides a more accurate measure of flexible intelligence compared to isolated tasks.

Challenges with Reinforcement Learning and LLMs

The paper also highlights that while reinforcement learning can yield impressive results, it often requires millions or billions of simulated runs to achieve acceptable performance. AI becomes an expert in a specific scenario but struggles when faced with any deviations. Even minor changes in the game environment can disrupt its performance.

Similarly, Large Language Models (LLMs) also face difficulties when dealing with unfamiliar games. These models tend to rely on custom game-specific structures to interpret game states, manage memory, and execute actions. Without this scaffolding, their performance deteriorates rapidly.

The True Benchmark

The researchers argue that a truly intelligent game-playing AI should be able to learn a new game in a similar timeframe to a skilled human player, without extensive simulations or prior exposure. Current AI systems fall short of this benchmark.

These challenges extend beyond gaming and have implications for AI’s ability to adapt to real-world unpredictability. While AI’s victories in chess make headlines, the limitations exposed by modern video games reveal the distance AI still has to cover in achieving human-like adaptability.