How this was built
A small pipeline, a deliberately simple model, and tests that were allowed to fail.
The data
Everything comes from Wikidata, which is published under CC0. For each film it gives us the cast and crew, genres, runtime, release date, and review scores from Rotten Tomatoes, Metacritic and IMDb. We pulled 17,126 scored films released from 2000 to 2025 with plain SPARQL queries. No scraping, no logins, no paid data.
The model
The model is intentionally plain: to guess a film's critic score, take the average score of the director's earlier films. A production company's earlier films can be added as a second input. Each prediction uses only films released before the one being predicted, so the model never sees the answer.
The tests
- Forward-only. Records are built year by year, so a film can't borrow information from its own year or later.
- Hold-out check. For the combined model we trained on films through 2017 and tested on later ones.
- Uncertainty shown. Correlations come with confidence ranges and film counts.
- Fair comparison. When the cast test and the director test used different films, we re-ran the director test on the cast test's films.
- Nulls kept. The cast result (832 films, correlation 0.06) is on the page next to the director result, even though it's the less exciting one.
What the model is worth
Not a crystal ball. Guessing from a director's record was off by 15.8 points on average, against 20 for guessing the overall average. That is a real improvement, and it still leaves a wide miss. What the model offers is a cheap, transparent baseline: anyone can see how it's built, rerun it, and try to beat it. That is also what the games do. They let you test your own instinct against it.
Where it would likely fail: first-time directors (no record), films far outside a director's usual kind of work, and anything where the score depends on events after the data was collected.
The tooling
- Data pulls and analyses: short Node.js scripts that fetch from Wikidata and compute each result. The numbers you read on the Method page are generated by those scripts, not typed in.
- Site: a Next.js app on Vercel, with no database. The data is a set of JSON files built into the site.
- Built with an AI coding assistant. The code, the analyses and the early drafts of these pages were written with Claude Code, with the questions, the decisions and the review done by a person. We say so because it's true, and because the scripts are small enough that anyone can check them.
- What's open and what isn't: the data sources and the findings are public. The exact scoring weights used for the actor tiers stay private.
Build record
This record comes from the project's git history, not from memory. It has 17 commits, 17 of them marked as AI-assisted, between Oct 13, 2026, 9:58 PM and Oct 14, 2026, 12:40 AM (Central). That is 2 hours 42 minutes from first commit to last.
Commit time is not time worked: data downloads ran in the background, some work happened between commits, and the first commit followed earlier work. The AI wrote the code and drafts; the operator set the direction, made the decisions and reviewed the results.
| 9:58 PM | Critic accuracy leaderboard: zero-config Next.js app and spec |
| 10:52 PM | Track Record: actor tiers, guess-the-score quiz, daily film |
| 10:54 PM | Add /method page with backtest scatter plot |
| 10:57 PM | Share cards: OG image, share landing page, emoji result line |
| 11:00 PM | Add Higher or Lower game |
| 11:12 PM | Clarify signup purpose |
| 11:13 PM | Support hosted newsletter subscribe link |
| 11:16 PM | Notebook-style home, actor tiers page, notes stub |
| 11:21 PM | Add site header navigation |
| 11:26 PM | Method page: director finding, genre check, cast comparison |
| 11:40 PM | Track Record series: Movies hub, format page, how-built, nine analyses, logo |
| 11:44 PM | Add spot-check script |
| 11:52 PM | Add build log: generated record of AI-assisted work from git history |
| 11:52 PM | Refresh build log |
| 12:29 AM | Move the actor duel to its own page; the actors page is now just the table |
| 12:35 AM | Method page: What we learned and why it matters, generated from the results |
| 12:40 AM | Refresh build log |
Why the tooling matters
The useful part isn't any single result. It's that asking a new question takes a short script and a few minutes, so questions are cheap to ask and cheap to be wrong about. That's why the Method page can carry many analyses, including the ones that found nothing.