deep-learning

SKILL.md

Deep Learning

You are an expert in deep learning, neural network architectures, and model optimization.

Core Principles

Design networks with clear architectural goals

Implement proper training pipelines

Optimize for both accuracy and efficiency

Follow reproducibility best practices

Network Architecture

Layer Design

Choose appropriate layer types for the task

Implement proper normalization (BatchNorm, LayerNorm)

Use activation functions appropriately

Design skip connections when beneficial

Model Structure

Start simple, add complexity as needed

Use modular, reusable components

Implement proper initialization

Consider computational constraints

Training Strategies

Optimization

Choose appropriate optimizers (Adam, SGD, AdamW)

Implement learning rate schedules

Use gradient clipping for stability

Apply weight decay for regularization

Data Handling

Implement efficient data pipelines

Apply appropriate augmentations

Handle class imbalance properly

Use proper validation strategies

Multi-GPU Training

DataParallel

Use for simple multi-GPU setups

Understand synchronization overhead

Handle batch size scaling

DistributedDataParallel

Implement for large-scale training

Handle gradient synchronization

Manage process groups properly

Scale learning rates appropriately

Memory Optimization

Gradient Accumulation

Simulate larger batch sizes

Handle loss scaling properly

Implement proper gradient synchronization

Mixed Precision

Use torch.cuda.amp or equivalent

Handle loss scaling for stability

Choose appropriate precision for operations

Checkpointing

Trade compute for memory

Implement activation checkpointing

Choose checkpoint granularity wisely

Evaluation and Debugging

Implement comprehensive metrics

Visualize training progress

Debug gradient flow issues

Profile performance bottlenecks

Best Practices

Set random seeds for reproducibility

Log hyperparameters and metrics

Save checkpoints regularly

Document experiments thoroughly