Skip to content

fix: unset CI in ray.sub and virtual_cluster to prevent vLLM aliasing errors#3293

Draft
kajalj22 wants to merge 1 commit into
mainfrom
fix-ci-env-ray-main
Draft

fix: unset CI in ray.sub and virtual_cluster to prevent vLLM aliasing errors#3293
kajalj22 wants to merge 1 commit into
mainfrom
fix-ci-env-ray-main

Conversation

@kajalj22

Copy link
Copy Markdown
Contributor

Summary

  • PyTorch sets `error_on_custom_op_aliasing=True` when `CI=true` (via `bool(os.getenv("CI"))` in `torch/_functorch/config.py`), causing `RuntimeError` in vLLM custom ops such as `sequence_parallel_chunk_impl`
  • `CI` propagates to Ray workers via the Ray daemon environment (started by sbatch before any test script runs), so per-recipe `vllm_cfg.env_vars` or `unset CI` in test scripts alone are insufficient
  • Fix at two levels so all launchers are covered:
    • `ray.sub`: `unset CI` in both `head_cmd` and `worker_cmd` before `ray start`, ensuring Ray daemons and all workers they spawn never see `CI=true`
    • `virtual_cluster.py`: strip `CI` from the `runtime_env` env_vars passed to `ray.init` and emit `UserWarning` so non-ray.sub launchers (e.g. k8s, local) also get the fix with visibility

Test plan

  • Nightly audiomcq tests pass without per-recipe `vllm_cfg.env_vars: {CI: ""}` workaround
  • `UserWarning` appears in logs when `CI` is set at launch time

🤖 Generated with Claude Code

PyTorch sets error_on_custom_op_aliasing=True when CI=true (see
torch/_functorch/config.py:99), causing RuntimeError in vLLM custom
ops such as sequence_parallel_chunk_impl. The CI env var propagates
to Ray workers through the Ray daemon environment, so unsetting it
in test scripts or per-recipe yaml (vllm_cfg.env_vars) is insufficient
for general use.

Fix at two levels:
- ray.sub: unset CI in both head_cmd and worker_cmd before ray start,
  so all Ray daemons and the workers they spawn never see CI=true
- virtual_cluster.py: strip CI from the runtime_env env_vars passed
  to ray.init + emit UserWarning so non-ray.sub launchers also get
  the fix with visibility

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant