Your "Clarity" Score Is Quietly Punishing Good Answers
Here is how every eval rubric gets born. Someone writes five criteria in a meeting, everyone nods, and it becomes scripture. Nobody ever goes back to check whether those five criteria predict anything at all.
So I checked. In this project's own benchmark, "clarity" was not just useless, it was anti-correlated with success. The rubric was actively docking points for being clear. Somewhere out there, your rubric has a "clarity" too, and you have no idea which one it is.
rubricagent treats the rubric as something you learn, not something you defend forever. Score your traces, correlate each criterion against what actually happened, then reweight, prune, and grow it into an honest proxy for skill instead of a vibe check with a spreadsheet attached.
Three Moves Per Pass
• Reweight: each criterion's weight becomes its actual correlation with success, not a number someone guessed in a doc.
• Prune: the dead weight gets cut, and the criteria secretly working against you get cut first.
• Discover: it mints new criteria out of the words that show up constantly in wins and rarely in losses, the signals nobody thought to write down.
• Bring your own judge: the offline grader keeps CI honest and reproducible, and a real LLM grader drops in through the same interface for production.
The Demo That Refused to Flatter Me
The hardest part was making the example honest instead of impressive. An early version showed a cozy 78% on a deliberately half-baked response, which was not what the code actually did. Running it for real spat out 14% with completely different reasoning. I shipped the 14%. A demo that lies about your tool is worse than no demo.
AUC as the quality metric was also on purpose. It measures how cleanly the rubric separates real wins from real losses, so "better rubric" has a definition you can check instead of an opinion you can argue about.
The Number
Start with a flat rubric where only one of five criteria actually predicts anything. One evolution pass drags rubric quality from 0.774 to a perfect 1.000, pruning the liars and surfacing the real signal on the way.
6 tests green across Python 3.9 through 3.13, with the example and benchmark running on every push. No keys.