Skip to content

[TP] Keep optimizer state aligned after parameter swaps - #4141

Open
sdjasj wants to merge 1 commit into
huggingface:mainfrom
sdjasj:agent/fix-tp-adagrad-state
Open

[TP] Keep optimizer state aligned after parameter swaps#4141
sdjasj wants to merge 1 commit into
huggingface:mainfrom
sdjasj:agent/fix-tp-adagrad-state

Conversation

@sdjasj

@sdjasj sdjasj commented Jul 29, 2026

Copy link
Copy Markdown

What does this PR do?

Keeps optimizer state keyed by the active parameters when tensor parallel preparation replaces model parameters with DTensors.

Optimizers such as Adagrad initialize per-parameter state in their constructor. TP preparation previously updated only param_groups, leaving the state dictionary keyed by the original parameters. PyTorch then raised a KeyError when serializing the inconsistent optimizer.

AcceleratedOptimizer._switch_parameters now moves any existing state to the replacement parameter, and TP preparation uses that state-aware path. The regression test verifies both the state keys and values before calling state_dict().

Fixes #3855

Validation

CUDA_VISIBLE_DEVICES='' python -m pytest -q tests/test_optimizer.py
# 2 passed, 1 skipped

# Four-GPU TP smoke test with a partially DTensor-backed model and Adagrad
accelerate launch --multi_gpu --num_processes 4 --main_process_port 0 /tmp/repro-accelerate-3855.py
# Adagrad optimizer state follows TP parameters

ruff check src/accelerate/accelerator.py src/accelerate/optimizer.py tests/test_optimizer.py
ruff format --check src/accelerate/accelerator.py src/accelerate/optimizer.py tests/test_optimizer.py

@sdjasj
sdjasj marked this pull request as ready for review August 1, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KeyError in optimizer.state_dict() under FSDP2 when using Adagrad optimizer

1 participant