peft-fine-tuning

intent

fine-tune large language models (7B-70B) by training less than 1% of parameters using parameter-efficient methods like LoRA, QLoRA, and 25+ adapter techniques. use this skill when you need to adapt a base model to your domain or task but lack the gpu memory or budget for full fine-tuning, or when you want to deploy multiple task-specific variants from one base model without storing redundant copies.

inputs

base model & tokenizer

• huggingface model identifier (e.g. "meta-llama/Llama-3.1-8B")
• tokenizer must support padding; set pad_token = eos_token if needed

training data

• huggingface dataset or local dataset with instruction/response pairs (or equivalent text format)
• minimum recommended: 1k examples; 10k+ for best results
• format: list of dicts with fields like "instruction" and "response"

hardware & environment

• gpu with 6-24GB vram (varies by method; LoRA typical 16-18GB for 8B model, QLoRA ~6GB)
• pytorch + transformers >= 4.30, peft >= 0.4, accelerate, bitsandbytes (if using QLoRA)
• install via: pip install peft transformers accelerate bitsandbytes datasets

hyperparameters (user-supplied or tuned)

• lora_r (rank): 8-64, default 16
• lora_alpha (scaling): typically 2x rank, default 32
• learning_rate: 1e-4 to 5e-4 for LoRA, 1e-4 to 2e-4 for QLoRA
• num_train_epochs: 1-5 depending on data size
• per_device_train_batch_size: 1-8 depending on memory

external connections

• huggingface hub (optional): used to pull models/datasets and push merged models; requires HF_TOKEN env var for gated models or private repos
• quantization backend (bitsandbytes) if using QLoRA: requires cuda-compatible bitsandbytes; set CUDA_VISIBLE_DEVICES if multi-gpu

procedure

step 1: install dependencies and verify gpu

inputs: system environment, pip outputs: confirmed peft + torch + transformers installed; gpu detected

run pip install peft transformers accelerate bitsandbytes datasets (full stack recommended). verify torch sees your gpu via python -c "import torch; print(torch.cuda.is_available())". if false, debug cuda installation before proceeding.

step 2: load base model and tokenizer

inputs: model name (string), device_map strategy outputs: model object loaded in gpu memory, tokenizer object with pad_token set

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "meta-llama/Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    torch_dtype="auto",  # Use bfloat16 or float16 if available
    device_map="auto"    # Shard across available gpus
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

note: if loading a gated model, ensure HF_TOKEN env var is set with your hub token.

step 3: prepare dataset via tokenization

inputs: raw dataset (huggingface or local), tokenizer, max_length (e.g. 512) outputs: tokenized dataset with input_ids, attention_mask, labels

from datasets import load_dataset

dataset = load_dataset("databricks/databricks-dolly-15k", split="train")

def tokenize_fn(example):
    text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['response']}"
    tokens = tokenizer(
        text,
        truncation=True,
        max_length=512,
        padding="max_length"
    )
    tokens["labels"] = tokens["input_ids"].copy()
    return tokens

tokenized_dataset = dataset.map(
    tokenize_fn,
    remove_columns=dataset.column_names,
    batched=False
)

edge case: if dataset is empty or has missing fields, tokenize will fail; validate dataset schema before mapping.

step 4a: configure LoRA adapter (standard fine-tuning)

inputs: task type, rank (r), alpha, target_modules (architecture-specific) outputs: LoraConfig object

from peft import LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],  # Llama/Mistral
    bias="none"
)

adjust target_modules per model architecture:

• llama/mistral/qwen: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
• gpt-2/neo: ["c_attn", "c_proj", "c_fc"]
• falcon/bloom: ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]
• auto-detect: use target_modules="all-linear" (peft 0.6.0+)

step 4b: configure QLoRA adapter (memory-constrained)

inputs: quantization config (load_in_4bit, compute dtype), LoRA config outputs: quantized model + LoraConfig

from transformers import BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=bnb_config,
    device_map="auto"
)

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=64,
    lora_alpha=128,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    bias="none"
)

step 5: wrap model with peft adapter

inputs: model, lora_config (from step 4a or 4b) outputs: peft-wrapped model with trainable parameters reduced to <1%

from peft import get_peft_model

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 13,631,488 || all params: 8,043,307,008 || trainable%: 0.17%

verify trainable % is <1%. if significantly higher, review target_modules.

step 6: configure training arguments

inputs: output_dir, batch_size, learning_rate, num_epochs, fp16/bf16, logging frequency outputs: TrainingArguments object

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./lora-llama",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,  # Use bfloat16 if available on your gpu
    logging_steps=10,
    save_strategy="epoch",
    eval_strategy="no"  # Set to "epoch" if you have eval_dataset
)

tuning notes:

• if oom: reduce batch_size, increase gradient_accumulation_steps, or enable gradient_checkpointing
• for qLora: learning_rate typically 1e-4 to 2e-4; try 1e-4 first

step 7: create trainer and run training

inputs: model, training_args, train_dataset, data_collator outputs: trained model with weights saved to output_dir

from transformers import Trainer
import torch

def data_collator_fn(batch):
    return {
        "input_ids": torch.stack([f["input_ids"] for f in batch]),
        "attention_mask": torch.stack([f["attention_mask"] for f in batch]),
        "labels": torch.stack([f["input_ids"] for f in batch])
    }

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=data_collator_fn
)

trainer.train()

edge case: if training crashes mid-epoch, resume from last checkpoint via trainer.train(resume_from_checkpoint=True) or specify checkpoint path.

step 8: save adapter (lightweight checkpoint)

inputs: trained model, save path outputs: adapter files (~6-50MB depending on rank) in output_dir

model.save_pretrained("./lora-llama-adapter")
tokenizer.save_pretrained("./lora-llama-adapter")

adapter is now separable from base model; deploy without storing redundant base weights.

step 9: load and test adapter

inputs: adapter path, base model name outputs: model with adapter loaded, ready for inference

from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained(
    "./lora-llama-adapter",
    device_map="auto"
)

inputs = tokenizer("### Instruction:\nWhat is 2+2?\n\n### Response:", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

step 10: merge adapter into base model (optional, for deployment)

inputs: peft model + loaded adapter outputs: single merged model file suitable for vllm, ollama, or quantization tools

merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama-merged")
tokenizer.save_pretrained("./llama-merged")

# Push to huggingface hub (optional)
merged_model.push_to_hub("your-username/llama-finetuned")

tradeoff: merged model is larger (~8-16GB for llama 8B) but requires no runtime adapter loading; adapter-only approach keeps deployment size small but requires base model + adapter at inference.

step 11 (multi-adapter serving): load multiple adapters and switch at runtime

inputs: base model, multiple adapter paths outputs: model with multiple adapters loaded, switchable via set_adapter()

from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", device_map="auto")
model = PeftModel.from_pretrained(base_model, "./adapter-task1")

model.load_adapter("./adapter-task2", adapter_name="task2")
model.load_adapter("./adapter-task3", adapter_name="task3")

# Inference
model.set_adapter("task1")
out1 = model.generate(**inputs1)

model.set_adapter("task2")
out2 = model.generate(**inputs2)

# Disable adapters (use base model)
with model.disable_adapter():
    base_out = model.generate(**inputs)

decision points

Q: How much gpu memory do i have?

• 6-8GB (e.g. rtx 4060): use QLoRA with r=8-16, per_device_train_batch_size=1, gradient_accumulation_steps=8-16
• 12-16GB (e.g. rtx 3060 ti, a10): use LoRA with r=16, per_device_train_batch_size=2-4, gradient_accumulation_steps=4-8
• 24GB+ (e.g. rtx 4090, a100 sxm4): use LoRA with r=32-64, or full fine-tuning if justified
• out of memory even with these settings: enable gradient_checkpointing (model.gradient_checkpointing_enable()), reduce max_length, or use IA3 instead

Q: Do i need maximum quality or can i trade off?

• maximum quality (e.g. production llm evaluation): use full fine-tuning if compute budget allows; else LoRA with r=32-64, lora_alpha=64-128, target all linear layers
• good enough quality with efficiency priority: LoRA r=16, alpha=32, target attention+mlp layers only
• minimal quality requirement, speed critical: IA3 or prefix tuning

Q: Am i fine-tuning a 70B+ model?

• yes, on single gpu: use QLoRA (4-bit quantization) with r=64, lora_alpha=128, batch_size=1
• yes, on multi-gpu cluster: use LoRA with distributed training, r=32-64, batch_size=4-8 per gpu
• no, <13B: standard LoRA is fine; r=8-16 sufficient

Q: Do i need to deploy multiple task-specific models?

• yes: save only adapters (step 8), load multiple adapters at inference (step 11), switch via set_adapter()
• no: merge adapter into base model (step 10) for single deployable artifact

Q: Is my base model already quantized (4-bit or 8-bit)?

• yes: skip step 4b quantization; apply LoRA directly, but use prepare_model_for_kbit_training() before wrapping
• no: proceed with step 4a (standard LoRA) or step 4b (QLoRA if memory-constrained)

Q: Do i have eval data to measure overfitting?

• yes: set eval_strategy="epoch" in TrainingArguments, provide eval_dataset to Trainer, monitor loss divergence
• no: use early stopping heuristics (e.g. save checkpoint every N steps, eyeball loss curve); default to save_strategy="epoch"

Q: Is training crashing with cuda oom?

• immediate action: reduce per_device_train_batch_size to 1, increase gradient_accumulation_steps to 16
• next: enable gradient_checkpointing (trades compute for memory)
• last resort: switch to QLoRA or reduce max_length from 512 to 256

Q: After training, does the adapter output look wrong?

• verify adapter is active: print(model.active_adapters) should show your adapter name
• check model is in train mode: model.eval() disables dropout, model.train() re-enables; inference usually needs eval mode
• verify tokenizer consistency: same tokenizer used for training must be used for inference

output contract

successful training produces:

• ./lora-llama-adapter/adapter_config.json: yaml-like config with r, alpha, target_modules, etc. (~1KB)
• ./lora-llama-adapter/adapter_model.bin: pytorch checkpoint with adapter weights (~6-50MB depending on rank)
• ./lora-llama-adapter/tokenizer.json: tokenizer state (required for inference)
• ./lora-llama/ (output_dir): huggingface trainer checkpoints at each epoch, containing full model state (no separate adapter needed here)

if merged (step 10):

• ./llama-merged/pytorch_model.bin: full model weights (~8-16GB for 8B model)
• ./llama-merged/config.json: model config
• ./llama-merged/tokenizer.json: tokenizer

data format for adapter_config.json:

{
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": false,
  "init_lora_weights": true,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 16,
  "target_modules": ["q_proj", "v_proj", "k_proj", "o_proj"],
  "task_type": "CAUSAL_LM"
}

expected trainable parameter ratio: <1% of total model params (typically 0.17% for LoRA r=16 on 8B model)

training logs (stdout):

• loss per logging_steps (default 10 steps): should decrease over time, flatten at end
• tokens/sec throughput: benchmarks ~2000-3500 tokens/sec depending on rank and batch size

outcome signal

training succeeded if:

1. no oom errors and training completes without crashing
2. final loss is lower than initial loss (e.g. 2.5 -> 1.8 over 3 epochs)
3. model.print_trainable_parameters() shows <1% (e.g. "0.17%")
4. checkpoint files exist in output_dir with non-zero filesize (adapter .bin >1