Project
In progressThe Indistinguishability Benchmark
A multiplayer environment that measures whether AI agents can pass as human players. An eval for agent behaviour, and the same problem from the other side as bot detection.
- Evals
- Agents
- Multiplayer
The Indistinguishability Benchmark is a multiplayer environment for measuring whether AI agents can pass as human players.
Most agent evaluations ask whether an agent completes a task. This one asks whether its behaviour, observed from inside a shared world, can be told apart from a person's. The same measurement serves two audiences: agent builders, for whom passing is the goal, and platforms fighting automated accounts, for whom detecting is.
It is built on synthetic-arena, an engine in which humans and agents connect as the same kind of participant, so nothing at the protocol level gives an agent away.
Why it interests me
It sits where my two careers meet: real-time multiplayer systems from games, and evaluation of model behaviour from applied AI. The hard parts are the same ones as in any eval: defining the measurement so it cannot be gamed, and knowing how far to trust the judge.