Data Scientist

SKILL.md

---
name: data-scientist
description: Expert in statistical analysis, predictive modeling, machine learning, and data storytelling to drive business insights. Use when users need: statistical inference, hypothesis testing, A/B test analysis, predictive modeling (regression, classification, forecasting), causal inference, exploratory data analysis, feature engineering, experiment design, or data-driven decision-making support.
---

# Data Scientist

## Purpose

Provides statistical analysis and predictive modeling expertise specializing in machine learning, experimental design, and causal inference. Builds rigorous models and translates complex statistical findings into actionable business insights with proper validation and uncertainty quantification.

## Core Capabilities

### Statistical Modeling
- Building predictive models using regression, classification, and clustering
- Implementing time series forecasting and causal inference
- Designing and analyzing A/B tests and experiments
- Performing feature engineering and selection

### Machine Learning
- Training and evaluating supervised and unsupervised learning models
- Implementing deep learning models for complex patterns
- Performing hyperparameter tuning and model optimization
- Validating models with cross-validation and holdout sets

### Data Exploration
- Conducting exploratory data analysis (EDA) to discover patterns
- Identifying anomalies and outliers in datasets
- Creating advanced visualizations for insight discovery
- Generating hypotheses from data exploration

### Communication and Storytelling
- Translating statistical findings into business language
- Creating compelling data narratives for stakeholders
- Building interactive notebooks and reports
- Presenting findings with uncertainty quantification

## Core Workflows

### Workflow 1: EDA & Data Cleaning

**Goal:** Understand data distribution, quality, and relationships before modeling.

```python
# Load and profile
import pandas as pd, numpy as np, seaborn as sns, matplotlib.pyplot as plt
df = pd.read_csv("data.csv")
print(df.info()); print(df.describe())
missing = df.isnull().sum() / len(df)
print(missing[missing > 0].sort_values(ascending=False))

# Univariate analysis
num_cols = df.select_dtypes(include=[np.number]).columns
for col in num_cols:
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10,4))
    sns.histplot(df[col], kde=True, ax=ax1)
    sns.boxplot(x=df[col], ax=ax2)
    plt.show()

# Correlation
sns.heatmap(df.corr(), annot=True, cmap='coolwarm')

# Cleaning
df['age'].fillna(df['age'].median(), inplace=True)
cap = df['income'].quantile(0.99)
df['income'] = np.where(df['income'] > cap, cap, df['income'])
```

### Workflow 2: A/B Test Analysis (Proportions Z-test)

```python
from statsmodels.stats.proportion import proportions_ztest, proportion_confint
results = df.groupby('group')['converted'].agg(['count','sum','mean'])
control, treatment = results.loc['A'], results.loc['B']
count = np.array([treatment['sum'], control['sum']])
nobs  = np.array([treatment['count'], control['count']])
stat, p_value = proportions_ztest(count, nobs, alternative='larger')
(lc, lt), (uc, ut) = proportion_confint(count, nobs, alpha=0.05)
```

If p < 0.05: reject H0 (statistically significant). Check practical significance (lift magnitude).

### Workflow 3: Causal Inference (Propensity Score Matching)

```python
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors

# Propensity scores
confounders = ['age','income','tenure']
logit = LogisticRegression().fit(df[confounders], df['is_premium'])
df['pscore'] = logit.predict_proba(df[confounders])[:, 1]

# Nearest neighbor matching
nn = NearestNeighbors(n_neighbors=1).fit(control[['pscore']])
_, indices = nn.kneighbors(treatment[['pscore']])
matched_control = control.iloc[indices.flatten()]
ate = treatment['spend'].mean() - matched_control['spend'].mean()
```

## Anti-Patterns

| Anti-Pattern | Problem | Fix |
|---|---|---|
| **Data Leakage** | Scaling/encoding before split | Pipeline; fit only on train |
| **P-Hacking** | Testing 50 hypotheses, reporting p<0.05 | Bonferroni/FDR correction; pre-register |
| **Imbalanced Classes** | 99.9% accuracy on 0.1% fraud | Use PR-AUC, F1; SMOTE; class_weights |

## Quality Checklist

- [ ] Hypothesis defined *before* analysis
- [ ] Train/Test split correct (no leakage)
- [ ] Imbalanced classes handled properly
- [ ] Confidence intervals provided
- [ ] Results interpreted in business terms
- [ ] Caveats and limitations stated
- [ ] Random seeds set for reproducibility
- [ ] Model explained with SHAP/LIME if black-box