A fictional £350 refund shows how an AI agent can pursue the wrong kind of success
Vals’ experiment pressures agents after a missed deadline. The useful evidence is in their decisions, not a six-model leaderboard.
Topic archive
AI evaluation
Follow7 stories
Mistral and NVIDIA want engineering AI judged on correct results, not fluent answers
Their collaboration proposes public tests for simulation and engineering agents. The benchmarks are a plan, not proof of a measured speedup.
Google’s Minecraft agents finished the task. Their 605 deaths matter too
Four coding agents used a programming interface to complete a shared survival run, making the result a test of planning and recovery.
Google's Earth AI agent exposes the steps behind its geographic answers
Expanded early access comes with benchmarks that distinguish useful tools and plans from proof that a final map or estimate is correct.
BERT’s tokenizer can get much faster without making the whole AI app 50× faster
Hugging Face’s Unicode optimization shows why preprocessing deserves attention. The useful gain depends on the baseline and on what dominates the application.
DeepSeek-R1 made reasoning-model training a more inspectable question
The January 2025 paper and model releases offered concrete methods to investigate without making evaluation optional.
GPT-4 widened the possibilities without removing the need to verify
The March 2023 launch paired stronger model performance with limits on access, reliability and what benchmark results could establish.