ml-pipeline

SKILL.md

Production-grade ML pipeline infrastructure with experiment tracking, orchestration, feature stores, and automated model lifecycle management.

Covers end-to-end pipeline design: data validation, feature engineering, distributed training orchestration, experiment tracking, and model evaluation gates

Supports multiple orchestration frameworks (Kubeflow, Airflow, Prefect) and experiment tracking systems (MLflow, Weights & Biases) with code templates and reference guides

Enforces reproducibility through versioning (DVC, Git tags, model registry), pinned dependencies, logged hyperparameters, and containerized environments

Includes data validation checkpoints, hyperparameter tuning configuration, A/B testing patterns, and deployment strategies with rollback support

ML Pipeline Expert

Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.

Core Workflow

Design pipeline architecture — Map data flow, identify stages, define interfaces between components

Validate data schema — Run schema checks and distribution validation before any training begins; halt and report on failures

Implement feature engineering — Build transformation pipelines, feature stores, and validation checks

Orchestrate training — Configure distributed training, hyperparameter tuning, and resource allocation

Track experiments — Log metrics, parameters, and artifacts; enable comparison and reproducibility

Validate and deploy — Run model evaluation gates; implement A/B testing or shadow deployment before promotion

Reference Guide

Load detailed guidance based on context: