Cheap MUD Experiment Reveals Potential Flaws in LLM Benchmarking Methods
💡 • Investors in AI companies should scrutinize benchmark claims—this study shows even a $99 experiment can reveal rank instability. • Founders building evaluation tools for LLMs have a market opportunity to create multi-judge, human-validated frameworks. • Crypto and token projects that rely on AI scoring for smart contracts or governance should demand robust, auditable benchmarks. • Side hustle researchers can replicate or extend this work at low cost, potentially uncovering market-moving insights about specific models. • Real estate and business automation tools using LLMs for decision-making should verify benchmark reliability before adoption.
A team of researchers spent $99 in API credits and nights and weekends building a text-based MUD game to evaluate large language models. Their findings uncovered troubling inconsistencies in LLM-as-judge scoring, with one frontier model dropping six ranks when two classifier-dependent dimensions were removed, and inter-judge agreement ranging from 85% to 22%.
A small team of researchers has published a proof-of-concept experiment using a classic 1970s Multi-User Dungeon (MUD) to evaluate large language models, costing just $99 in API credits and running entirely on personal computers. Over several months of nights and weekends, they ran 50 trials per model and scored each on four behavioral dimensions, two of which relied heavily on an LLM classifier. The resulting leaderboard was interesting, but the real surprise came when they removed those two classifier-dependent metrics: one frontier model fell six positions in the rankings.
Digging deeper, the team cross-checked the classifier against a second judge and found per-model agreement varied wildly, from 85% down to 22%. The aggregate kappa score for probe detection was a mere 0.04, indicating the measurement instrument was noisy. The researchers noted that the model most affected by the discrepancy shared a model family with the classifier used, though they stress this is not proof of bias, only an observation worth reporting.
The authors emphasize that their work is not a validated benchmark and acknowledged several limitations: only 50 runs per model, overlapping confidence intervals among top performers, no human raters, and a tiny game environment. Despite these caveats, they believe the divergence between the two judges is a finding that generalizes to other judge-based benchmarks. The full paper, transcripts, code, and API billing export are publicly available under open licenses, and the team is designing a Phase 2 with human baselines, multiple judges, larger environments, and more objectives.
For investors and business leaders, this experiment underscores a critical risk: automated LLM evaluation systems may produce unreliable rankings, especially when one LLM judges another from the same model family. The low cost and small scale of the study highlight that even minimal resources can expose significant flaws in how AI performance is measured. As companies pour billions into AI development and marketing, stakeholders should demand more rigorous, transparent, and independent evaluation methods beyond single-judge benchmarks.
Read the full story
Original reporting and related coverage — attribution links only, not paid recommendations.
Trade this story
Broker buttons use invite / refer-a-friend links (rewards may be capped). Other partner links may pay OppHub a commission at no extra cost to you.
Tools & books on Amazon
Shop Amazon →Relevant gear and reads when you want to go deeper — OppHub may earn from qualifying purchases.
Build My Playbook
Turn this headline into a clear plan: what to watch, how to express it (stocks, ETFs, or options education), and how you’d know you’re wrong — for beginners and active traders. Not personalized advice.
You’ll get theme → ETFs → stocks → options education → side income → kill switches.