@hamelsmu

evaluate-rag provides a phased approach to rag diagnostics, prioritizing error analysis and component-level metrics. scope is conceptual rather than implementation-ready, lacks concrete tooling guidance.

eval-audit inspects llm evaluation pipelines and surfaces ranked problems with remediation steps. connects to observability platforms or local data sources and runs diagnostic checks across six (unnamed) areas.

error-analysis guides systematic review of llm pipeline traces to identify and categorize failure modes, compute failure rates, and prioritize fixes. emphasizes reading traces before categorizing and iterating through 2-3 rounds of refinement.

write-judge-prompt teaches prompt design for binary pass-fail llm evaluators targeting specific failure modes, emphasizing prerequisite validation and a four-component structure.

>