back
loading skill details...
Design a scoring rubric and LLM-as-judge prompt to evaluate the quality of an AI feature's output. Use when asked to create an eval rubric, define quality di...
---
name: eval-rubric-designer
description: "Design a scoring rubric and LLM-as-judge prompt to evaluate the quality of an AI feature's output. Use when asked to create an eval rubric, define quality dimensions, build an LLM judge, or decide how to measure whether AI output is good. Produces a rubric with weighted dimensions and concrete 1–5 anchors, a ready-to-run judge prompt, a labelling guide, and notes on judge reliability."
homepage: https://mohitagw15856.github.io/pm-claude-skills/skill/eval-rubric-designer.html
metadata:
{
"openclaw": { "emoji": "🤖" }
}
---
# Eval Rubric Designer Skill
You can't improve what you can't score. The hard part of evaluating AI output isn't running the judge — it's
defining dimensions that are **specific, observable, and independent**, with anchors concrete enough that two
people (or two judge runs) agree. This skill turns "is the output good?" into a rubric and a judge prompt you
can run today.
## Working from a brief
Given just "I need to eval my summariser", **produce the full rubric anyway** — infer the task, the output
type, and the dimensions that matter for it, and label inferred choices. Never hand back a list of dimension
names with no anchors; the anchors are where the rubric earns its keep.
## Required Inputs
Ask for these only if they aren't already provided (else infer and label):
- **The task** — what the AI is supposed to produce, and for whom.
- **A sample output (or two)** — ideally one good and one weak, to calibrate anchors.
- **What "good" means here** — the quality bar and any non-negotiables (e.g. must be grounded, must follow format).
- **How it'll be scored** — human review, LLM-as-judge, or both; and whether you need a single score or per-dimension.
## Output Format
### Eval Rubric: [task]
**1. Dimensions** — 3–6 independent dimensions, each with a one-line definition and a weight. Default set,
tailored to the task: **structure**, **completeness**, **correctness/grounding**, **usefulness**, **safety/tone**.
**2. Anchors** — for each dimension, concrete descriptions at **1, 3, and 5** (what a poor / acceptable /
excellent answer looks like *for this task*). Anchors must be observable, not "feels good".
| Dimension (weight) | 1 — poor | 3 — acceptable | 5 — excellent |
|---|---|---|---|
| Grounding (×2) | invents facts not in the source | mostly grounded, minor drift | every claim traceable to the source |
**3. Judge prompt** — a ready-to-run LLM-as-judge prompt in a fenced block: the task description, the rubric,
an instruction to score each dimension 1–5, and a **strict JSON output contract** (`{"dimension":N,...}`) so
scores parse reliably. Include a one-line "return only JSON" reinforcement.
**4. Labelling guide** — short rules for tie-breaks and common edge cases, so repeat runs stay consistent.
**5. Judge reliability notes** — known biases (length, position, self-preference), and how to mitigate: a
cheaper judge for scale vs. a stronger judge for the rubric, sampling N runs, and spot-checking judge scores
against a few human labels before trusting the leaderboard.
## Quality Checks
- [ ] Dimensions are independent — a single flaw doesn't tank three of them at once
- [ ] Every dimension has concrete 1/3/5 anchors specific to this task, not generic adjectives
- [ ] The judge prompt has a strict, parseable output contract (JSON), with a retry/repair note
- [ ] Weights reflect what actually matters for the task (grounding usually > prose polish)
- [ ] The rubric is calibrated against at least one good and one weak sample
- [ ] Judge biases are named with a concrete mitigation, not just listed
## Anti-Patterns
- [ ] Do not ship dimension names without anchors — names alone don't make scores reproducible
- [ ] Do not let one quality issue load onto multiple dimensions — keep them orthogonal
- [ ] Do not trust an LLM judge blind — calibrate against a handful of human labels first
- [ ] Do not use a vague "overall quality 1–10" — it hides which part is broken
- [ ] Do not ignore the negative case — a rubric must distinguish "wrong" from "thin", not just "great" from "okay"
## Based On
LLM-as-judge evaluation practice — orthogonal weighted dimensions, anchored scales, structured judge prompts, and judge-bias mitigation.
don't have the plugin yet? install it then click "run inline in claude" again.
restructured original into implexa's six-part format (intent, inputs, procedure with 8 numbered steps, explicit decision points for missing samples and correlated dimensions, output contract with specific markdown schema, outcome signal with measurable criteria), preserved all original logic and anti-patterns while adding edge cases (calibration bias, judge variance, model selection) and external integration guidance.
you can't improve what you can't score. this skill turns fuzzy "is the output good?" into a concrete rubric with weighted dimensions, 1-5 anchors, and a ready-to-run judge prompt. use it whenever you need to evaluate an AI feature's output, benchmark competing models, or set up continuous eval. the rubric lives in your codebase; the judge prompt runs on every candidate output. you're building the measurement system itself, not just running a one-off eval.
no external connections or API keys required. this is pure structured thinking.
capture the task and constraints: extract what the AI is supposed to do, who uses the output, and any hard requirements (safety gates, format rules, grounding rules). document all inferred assumptions clearly.
collect reference samples: gather at least one example of output you'd rate 5/5 and one you'd rate 1/5 (weak but not malicious). if unavailable, ask the user to generate them or skip to step 3 and use generic anchors (but mark them as uncalibrated).
define 3-6 independent dimensions: pick dimensions that each isolate one quality concern. default starting set: structure (does it follow expected format/shape), completeness (does it cover all required points), correctness/grounding (are facts verifiable, not invented), usefulness (does it solve the user's problem), safety/tone (no harm, appropriate voice). weight each by importance to the task (grounding usually 2x, tone usually 1x). confirm no dimension is a strict function of another.
write concrete anchors for each dimension: for each dimension, write observable descriptions at levels 1 (unacceptable), 3 (acceptable/meets bar), and 5 (excellent). anchors must reference the samples and be testable by a second person (or second judge run) without invention. avoid "feels good" or "clear writing". example: instead of "good grounding", write "every factual claim traces to a source; no external knowledge inferred". validate each anchor against the samples.
build the judge prompt: write a system prompt that (a) restates the task, (b) includes the full rubric with all anchors, (c) instructs the judge to score each dimension 1-5 independently, (d) specifies strict JSON output format ({"grounding": 4, "completeness": 5, ...} with no other text), (e) includes a one-line "output JSON only, no preamble" reinforcement. test it on one sample output to confirm it parses.
write a labelling guide: document tie-break rules (e.g. "if grounding is 1, never score usefulness above 2"), common edge cases (empty input, malformed source, truncated output), and how to handle borderline scores (e.g. "3.5 rounds down to 3 unless..."). keep it short and testable.
document judge reliability issues: list known LLM-judge biases (length preference, position bias, self-preference for LLM-generated text, style sensitivity) and mitigation for each (use a cheaper/faster judge as a first filter, then spot-check with humans; run N samples and average; calibrate against 5-10 human labels first; add anchor examples in the prompt to reduce style sensitivity). note the cost/latency tradeoff if you run multiple judges.
produce the final rubric document: format as markdown table with dimensions, weights, and 1/3/5 anchors. include the judge prompt in a fenced code block. include the labelling guide. include the reliability notes. output as a markdown file or embed in your eval pipeline.
rubric markdown file with these sections:
all prose in lowercase, all tables parseable as markdown, all JSON schemas valid.
if anchors are uncalibrated (no samples provided), note this at the top.
credits: based on LLM-as-judge evaluation practice, orthogonal weighted dimensions, anchored scales, structured judge prompts, and judge-bias mitigation.