python-performance-optimization

SKILL.md

Python Performance Optimization

Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.

When to Use This Skill

Identifying performance bottlenecks in Python applications

Reducing application latency and response times

Optimizing CPU-intensive operations

Reducing memory consumption and memory leaks

Improving database query performance

Optimizing I/O operations

Speeding up data processing pipelines

Implementing high-performance algorithms

Profiling production applications

Core Concepts

1. Profiling Types

CPU Profiling: Identify time-consuming functions

Memory Profiling: Track memory allocation and leaks

Line Profiling: Profile at line-by-line granularity

Call Graph: Visualize function call relationships

2. Performance Metrics

Execution Time: How long operations take

Memory Usage: Peak and average memory consumption

CPU Utilization: Processor usage patterns

I/O Wait: Time spent on I/O operations

3. Optimization Strategies

Algorithmic: Better algorithms and data structures

Implementation: More efficient code patterns

Parallelization: Multi-threading/processing

Caching: Avoid redundant computation

Native Extensions: C/Rust for critical paths

Quick Start

Basic Timing

import time

def measure_time():
    """Simple timing measurement."""
    start = time.time()

    # Your code here
    result = sum(range(1000000))

    elapsed = time.time() - start
    print(f"Execution time: {elapsed:.4f} seconds")
    return result

# Better: use timeit for accurate measurements
import timeit

execution_time = timeit.timeit(
    "sum(range(1000000))",
    number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Profile before optimizing - Measure to find real bottlenecks

Focus on hot paths - Optimize code that runs most frequently

Use appropriate data structures - Dict for lookups, set for membership

Avoid premature optimization - Clarity first, then optimize

Use built-in functions - They're implemented in C

Cache expensive computations - Use lru_cache

Batch I/O operations - Reduce system calls

Use generators for large datasets

Consider NumPy for numerical operations

Profile production code - Use py-spy for live systems

Common Pitfalls

Optimizing without profiling

Using global variables unnecessarily

Not using appropriate data structures

Creating unnecessary copies of data

Not using connection pooling for databases

Ignoring algorithmic complexity

Over-optimizing rare code paths

Not considering memory usage