The Benchmark Is Not the Territory

A very good demo is not a very good model. The leaderboard has already changed since you started reading this.

People keep asking the Bureau which model is best. It is the wrong question, asked with great confidence — which is itself a benchmark result.

Here is what we have learned running the numbers: a benchmark measures whatever the benchmark measures, and then everyone agrees it measures intelligence. This is not science. This is a group project that got out of hand.

In Timeline B, Google ended July on top of the leaderboard — a 1.1% outcome our universe priced like a rounding error. The winners immediately stopped questioning the methodology. The losers immediately started. That is the most reliable signal in the entire field, and it appears on no chart.

We do publish scores. We publish them the way weather stations publish humidity: precisely, and in full knowledge that no one will dress for it correctly. A very good demo is not a very good model. A number that went up is not a promise that it will stay up.

Evaluate everything. Believe the vibes. Trust neither.