RuntimeError: Tensor on device meta is not on the expected device cuda:0!

[nitinmukesh]

System Info

- `Accelerate` version: 1.7.0.dev0
- Platform: Windows-10-10.0.26100-SP0
- `accelerate` bash location: C:\Users\nitin\miniconda3\envs\sddw-dev\Scripts\accelerate.exe
- Python version: 3.11.9
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- PyTorch SDAA available: False
- PyTorch MUSA available: False
- System RAM: 15.73 GB
- GPU type: NVIDIA GeForce RTX 4060 Laptop GPU
- `Accelerate` default config:
        Not found

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

Install diffusers
pip install git+https://github.com/huggingface/diffusers.git@refs/pull/11428/head

import torch
from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel
from diffusers.utils import export_to_video, load_image
from transformers import SiglipImageProcessor, SiglipVisionModel

transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained("lllyasviel/FramePackI2V_HY", torch_dtype=torch.bfloat16)
feature_extractor = SiglipImageProcessor.from_pretrained("lllyasviel/flux_redux_bfl", subfolder="feature_extractor")
image_encoder = SiglipVisionModel.from_pretrained("lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16)
pipe = HunyuanVideoFramepackPipeline.from_pretrained("hunyuanvideo-community/HunyuanVideo", transformer=transformer, feature_extractor=feature_extractor, image_encoder=image_encoder, torch_dtype=torch.float16)
pipe.enable_sequential_cpu_offload()

image = load_image("inputs/penguin.png")
output = pipe(
    image=image,
    prompt="A penguin dancing in the snow",
    height=832,
    width=480,
    num_frames=31,
    num_inference_steps=2,
    guidance_scale=9.0,
    generator=torch.Generator().manual_seed(0),
).frames[0]
export_to_video(output, "output.mp4", fps=30)

Error info on Linux

Traceback (most recent call last):
  File "/home/aryan/work/diffusers/workflows/integrations/framepack_hunyuan_video/test_pipeline.py", line 41, in <module>
    output = pipe(
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/aryan/work/diffusers/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py", line 736, in __call__
    image_embeds = self.encode_image(image, device=device).to(transformer_dtype)
  File "/home/aryan/work/diffusers/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py", line 394, in encode_image
    image_embeds = self.image_encoder(**image).last_hidden_state
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/raid/aryan/accelerate-dev/src/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/transformers/utils/generic.py", line 965, in wrapper
    output = func(self, *args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/transformers/models/siglip/modeling_siglip.py", line 1030, in forward
    return self.vision_model(
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/transformers/utils/generic.py", line 965, in wrapper
    output = func(self, *args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/transformers/models/siglip/modeling_siglip.py", line 944, in forward
    pooler_output = self.head(last_hidden_state) if self.use_head else None
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/raid/aryan/accelerate-dev/src/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/transformers/models/siglip/modeling_siglip.py", line 969, in forward
    hidden_state = self.attention(probe, hidden_state, hidden_state)[0]
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/raid/aryan/accelerate-dev/src/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1373, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/nn/functional.py", line 6383, in multi_head_attention_forward
    attn_output = linear(attn_output, out_proj_weight, out_proj_bias)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 291, in _fn
    result = fn(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py", line 83, in inner
    r = f(*tree_map(increase_prec, args), **tree_map(increase_prec, kwargs))
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py", line 1460, in addmm
    return out + beta * self
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 291, in _fn
    result = fn(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 143, in _fn
    result = fn(**bound.arguments)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_refs/__init__.py", line 1145, in add
    output = prims.add(a, b)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_ops.py", line 723, in __call__
    return self._op(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_library/fake_impl.py", line 95, in meta_kernel
    return fake_impl_holder.kernel(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_library/utils.py", line 31, in __call__
    return self.func(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/library.py", line 1193, in inner
    return func(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_library/custom_ops.py", line 592, in fake_impl
    return self._abstract_fn(*args, **kwargs)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_prims/__init__.py", line 403, in _prim_elementwise_meta
    utils.check_same_device(*args_, allow_cpu_scalar_tensors=True)
  File "/raid/aryan/nightly-venv/lib/python3.10/site-packages/torch/_prims_common/__init__.py", line 764, in check_same_device
    raise RuntimeError(msg)
RuntimeError: Tensor on device meta is not on the expected device cuda:0!

Error info on Windows

(sddw-dev) C:\aiOWN\diffuser_webui>python framepack3.py
Fetching 3 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  5.99it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 15.53it/s]
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████| 7/7 [00:03<00:00,  1.81it/s]
Traceback (most recent call last):
  File "C:\aiOWN\diffuser_webui\framepack3.py", line 27, in <module>
    output = pipe(
             ^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\pipelines\hunyuan_video\pipeline_hunyuan_video_framepack.py", line 755, in __call__
    image_latents = self.prepare_image_latents(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\pipelines\hunyuan_video\pipeline_hunyuan_video_framepack.py", line 482, in prepare_image_latents
    latents = self.vae.encode(image).latent_dist.sample(generator=generator)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\utils\accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_hunyuan_video.py", line 821, in encode
    h = self._encode(x)
        ^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_hunyuan_video.py", line 795, in _encode
    return self.tiled_encode(x)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_hunyuan_video.py", line 928, in tiled_encode
    tile = self.encoder(tile)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_hunyuan_video.py", line 501, in forward
    hidden_states = self.conv_in(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_hunyuan_video.py", line 79, in forward
    return self.conv(hidden_states)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\conv.py", line 725, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\nitin\miniconda3\envs\sddw-dev\Lib\site-packages\torch\nn\modules\conv.py", line 720, in _conv_forward
    return F.conv3d(
           ^^^^^^^^^
NotImplementedError: Could not run 'aten::slow_conv3d_forward' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::slow_conv3d_forward' is only available for these backends: [CPU, Meta, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastXPU, AutocastMPS, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

Expected behavior

enable_sequential_cpu_offload() should work

Logging this based on the information here
huggingface/diffusers#11428 (comment)

@SunMarc @BenjaminBossan

[SunMarc]

Left it as a first good issue so the community can help ! Not sure I will have the bandwidth this week

[maximizemaxwell]

Does this error persist after the PR had merged upward?

[nitinmukesh]

Does this error persist after the PR had merged upward?

Please clarify, not able to understand.
I installed from source
pip install git+https://github.com/huggingface/accelerate.git

[southernbear]

Let's break it down:

1. out_proj of torch.nn.MultiheadAttention is a module, not a parameter. See Doesn't work when register hook to torch.nn.MultiheadAttention.out_proj pytorch/pytorch#78109
2. Submodules won't be loaded by accelerate.hooks.AlignDevicesHook.pre_forward if self.place_submodules is False.
3. accelerate.big_modeling.cpu_offload needs the preload_module_classes argument to set place_submodules to True.
4. diffusers.pipelines.pipeline_utils.DiffusionPipeline.enable_sequential_cpu_offload calls cpu_offload without the preload_module_classes argument.

You can try to add preload_module_classes=["MultiheadAttention"] to cpu_offload in diffusers/pipelines/pipeline_utils.py.

[LuciferDono]

Opened a fix in #3941.

The root cause is that attach_align_device_hook sets place_submodules=False when preload_module_classes is not provided. For nn.MultiheadAttention, out_proj is a child submodule whose forward() is never called -- instead MHA accesses out_proj.weight/out_proj.bias directly. So the child hook never fires and those weights stay on meta.

The fix auto-detects modules that have both direct parameters and child submodules with their own parameters, and sets place_submodules=True for them. This keeps full_offload semantics unchanged.