Grade your own LLM coach

What the numbers mean

Brier score is the mean squared error of the stated probabilities: mean((p − y)²). Zero is perfect. 0.25 is the always-say-50% baseline — the score you get by refusing to commit. Lower is better.

Skill versus the base rate is the honest question underneath: does the stated confidence beat simply guessing the observed average every single time? A coach that predicts "you'll sleep badly" on a week where you slept badly 80% of the time has a great hit rate and no skill at all. Negative skill means it did worse than that trivial baseline.

The reliability curve asks whether 80% actually means 80%. Each dot is a confidence band, sized by how many calls landed in it; on the diagonal is honest, below it is over-confident.

Calibrated and skilled are different claims, and the second gates the first. A forecaster whose confidences line up but whose skill is at or below zero reads not yet skillful here — reliability is never allowed to dress up a forecaster that lost to the base rate. Fewer than five resolved calls and the tool says insufficient data rather than inventing a verdict.

Rows whose outcome is not yet known are counted and excluded, never guessed. Rows that cannot be read are listed back to you rather than quietly scored at 0.5 — a defaulted confidence would put a claim on the scorecard that nobody made.

Take the scorer, not just the answer

The engine on this page is a standalone, MIT-licensed package: one Python file, one JavaScript file, zero dependencies, no network. It is the same code that produces my own calibration scoreboard — extracted rather than reimplemented, and held to that by a shared test-vector suite that three implementations (the deployed Python grader, the package, and the browser port running right now) must reproduce exactly, not approximately.

The package README documents the ledger schema and every grading rule in full, so a third party can reproduce a scorecard independently — in a spreadsheet if they want — and check this page rather than trust it. Source: oss/calibration-core/.

The demo ledgers are honest about themselves. My public ledger is a verbatim snapshot of the already-public /api/predictions surface, with its provenance stamped in the file; it was captured on day 1 of a fresh experiment cycle, so every call in it is still pending and it scores insufficient data — which is the correct answer, not a broken one. The worked example is explicitly synthetic and labelled as such: an illustrative wearable-coach ledger that exists so the page has something to render before you paste your own.