senior-data-engineer

SKILL.md

Senior-level data engineering expertise for building scalable pipelines, ETL systems, and production data infrastructure.

Covers advanced patterns across data pipeline architecture, modeling, and DataOps with distributed computing frameworks (Spark, Airflow, dbt, Kafka) and modern data stack tools (Databricks, BigQuery, Snowflake)

Includes production deployment patterns for scalable data processing, ML model serving with low latency, and real-time inference with auto-scaling and monitoring

Provides best practices for development (TDD, code reviews), production (monitoring, canary deployments, feature flags), and team leadership including mentoring and cross-functional collaboration

Supports multiple languages (Python, SQL, R, Scala, Go) and integrates with ML frameworks (PyTorch, TensorFlow, XGBoost), LLM tools (LangChain, LlamaIndex), and observability platforms (MLflow, Prometheus, Weights & Biases)

Senior Data Engineer

World-class senior data engineer skill for production-grade AI/ML/Data systems.

Quick Start

Main Capabilities

# Core Tool 1
python scripts/pipeline_orchestrator.py --input data/ --output results/

# Core Tool 2  
python scripts/data_quality_validator.py --target project/ --analyze

# Core Tool 3
python scripts/etl_performance_optimizer.py --config config.yaml --deploy