langsmith-online-eval-engineering

SKILL.md

Online Eval Engineering

Build online evaluators iteratively:

inspect traces and interview user -> propose directions -> user chooses
-> build evaluator -> test, attach, verify -> review and repeat

Read references/langsmith-api.md before creating or modifying evaluators.

1. Inspect traces

Ask the user for their LangSmith project name. Fetch recent root-level traces and print their structure. Read references/trace-inspection.md. Find:

run name and type;

available input and output field names;

the shape and content of the data (truncated samples);

which fields carry the data an evaluator would need.