Minecraft as a Turing Test: Evaluating Open-Ended Agentic AI

ABSTRACT

Evaluating the long-horizon planning and adaptability of Agentic AI in the real world is fraught with safety and cost limitations. We establish Minecraft as the premier sandbox for open-ended agentic evaluation. We introduce MineBench-25, a multi-agent framework where autonomous swarms must collaborate to build complex redstone machinery, survive dynamic threats, and manage scarce resources over millions of simulation ticks. Our findings reveal that current state-of-the-art agents struggle with task prioritization in highly unstructured environments, frequently succumbing to 'goal drift' without rigid hierarchical orchestration.