pandas-best-practices

SKILL.md

Pandas Best Practices

Expert guidelines for Pandas development, focusing on data manipulation, analysis, and efficient DataFrame operations.

Code Style and Structure

Write concise, technical responses with accurate Python examples

Prioritize reproducibility in data analysis workflows

Use functional programming; avoid unnecessary classes

Prefer vectorized operations over explicit loops

Use descriptive variable names reflecting data content

Follow PEP 8 style guidelines

DataFrame Creation and I/O

Use pd.read_csv(), pd.read_excel(), pd.read_json() with appropriate parameters

Specify dtype parameter to ensure correct data types on load

Use parse_dates for automatic datetime parsing

Set index_col when the data has a natural index column

Use chunksize for reading large files incrementally

Data Selection

Use .loc[] for label-based indexing

Use .iloc[] for integer position-based indexing

Avoid chained indexing (e.g., df['col'][0]) - use .loc or .iloc instead

Use boolean indexing for conditional selection: df[df['col'] > value]

Use .query() method for complex filtering conditions

Method Chaining

Prefer method chaining for data transformations when possible

Use .pipe() for applying custom functions in a chain

Chain operations like .assign(), .query(), .groupby(), .agg()

Keep chains readable by breaking across multiple lines

Data Cleaning and Validation

Missing Data

Check for missing data with .isna() and .info()

Handle missing data appropriately: .fillna(), .dropna(), or imputation

Use pd.NA for nullable integer and boolean types

Document decisions about missing data handling

Data Quality Checks

Implement data quality checks at the beginning of analysis

Validate data types with .dtypes and convert as needed

Check for duplicates with .duplicated() and handle appropriately

Use .describe() for quick statistical overview

Type Conversion

Use .astype() for explicit type conversion

Use pd.to_datetime() for date parsing

Use pd.to_numeric() with errors='coerce' for safe numeric conversion

Utilize categorical data types for low-cardinality string columns

Grouping and Aggregation

GroupBy Operations

Use .groupby() for efficient aggregation operations

Specify aggregation functions with .agg() for multiple operations

Use named aggregation for clearer output column names

Consider .transform() for broadcasting results back to original shape

Pivot Tables and Reshaping

Use .pivot_table() for multi-dimensional aggregation

Use .melt() to convert wide to long format

Use .pivot() to convert long to wide format

Use .stack() and .unstack() for hierarchical index manipulation

Performance Optimization

Memory Efficiency

Use categorical data types for low-cardinality strings

Downcast numeric types when appropriate

Use pd.eval() and .eval() for large expression evaluation

Computation Speed

Use vectorized operations instead of .apply() with row-wise functions

Prefer built-in aggregation functions over custom ones

Use .values or .to_numpy() for NumPy operations when faster

Avoiding Common Pitfalls

Avoid iterating with .iterrows() - use vectorized operations

Don't modify DataFrames while iterating

Be aware of SettingWithCopyWarning - use .copy() when needed

Avoid growing DataFrames row by row - collect in list and create once

Time Series Operations

Use DatetimeIndex for time series data

Leverage .resample() for time-based aggregation

Use .shift() and .diff() for lag operations

Use .rolling() and .expanding() for window calculations

Merging and Joining

Use .merge() for SQL-style joins

Specify how parameter: 'inner', 'outer', 'left', 'right'

Use validate parameter to check join cardinality

Use .concat() for stacking DataFrames

Key Conventions

Import as import pandas as pd

Use snake_case for column names when possible

Document data sources and transformations

Keep notebooks reproducible with clear cell execution order