[GOLD] GOLDTrainer VLM support

[Strongich]

What does this PR do?

Adds VLM support to GOLDTrainer:

• Supports same-family VLM distillation with JSD loss when student and teacher share compatible tokenization/image-token semantics.
• Supports cross-family VLM distillation with ULD loss by processing images separately through the student and teacher processors.
• Preserves raw PIL images through the dataloader with an identity collator, then materializes VLM batches only for the current accumulation slice.
• Adds VLM on-policy generation paths for both local generation and vLLM-backed generation where the vLLM backend supports the selected model.
• Adds teacher-side VLM input construction for ULD, including a fix for a silent alignment bug: teacher completions are now rendered with the teacher chat template instead of approximating them from generated text plus a manually appended EOS.
• Adds examples/scripts/gold_vlm.py with documented same-family JSD and cross-family ULD examples.
• Adds tests covering VLM collation, label masking, raw image preservation, cross-architecture validation, teacher processor setup, vLLM behavior, lazy slice materialization, and ULD alignment regressions.

Motivation

The GOLD algorithm has no theoretical constraints against VLM-to-VLM distillation -- the barriers were purely engineering (incompatible image token formats, different tokenizers, raw image handling through the dataloader).

Key changes

• GOLDTrainer detects VLM datasets and uses an identity collator to preserve raw PIL images through the dataloader
• For cross-architecture pairs, a _teacher_processor is stored and used in compute_loss to build teacher-compatible vision tensors from raw images
• Auto-resolves teacher_tokenizer_name_or_path
• Added examples/scripts/gold_vlm.py with two documented usage examples (same-family JSD + vLLM, cross-family ULD)
• Added tests for VLM collator (label masking, completion preservation), cross-architecture detection (rejects JSD, stores teacher processor for different archs, skips it for same arch), VLM + vLLM init (copied from the LLM example), rejects LLM teacher with vision dataset
• VLM handling (identity collator, raw image storage, vLLM multimodal path) is borrowed (where it was possible) from SFTTrainer and GRPOTrainer

Note

Looking for feedback:

• The main design point I would like feedback on is the buffering/generation structure for VLM batches: raw images are kept through the dataloader and VLM slices are collated lazily only when consumed, to avoid keeping all accumulated pixel_values on GPU at once.
• I didn't add VLM usage examples to docs/source/gold_trainer.md -- will add if that's desirable, just let me know.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@qgallouedec @kashif

---

Note

High Risk
Large changes to GOLDTrainer training, buffering, loss, and eval paths plus multimodal vLLM generation; mistakes could cause wrong distillation alignment or GPU/memory issues.

Overview
GOLDTrainer gains end-to-end vision-language model (VLM) distillation: same-family pairs can use JSD; different model_type pairs require ULD and get a separate teacher processor so images and completions are tokenized correctly on the teacher side.

For vision datasets the dataloader uses an identity collator so raw PIL images stay intact; a new DataCollatorForVisionLanguageChatML builds student batches lazily per gradient-accumulation slice (train, eval, on-policy local gen, and vLLM multimodal gen). Eval and prediction_step collate raw examples before loss instead of indexing them as tensors. JSD on VLMs uses a safer completion start from labels; ULD teacher completions are rendered with the teacher chat template (fixing silent misalignment). Liger and text-only teachers on image data are rejected at init. GOLDConfig defaults remove_unused_columns=False. Adds examples/scripts/gold_vlm.py and a large VLM regression test suite.

^{Reviewed by Cursor Bugbot for commit 3b04456. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Strongich]

@kashif @qgallouedec I think you guys might be interested in this PR, looking forward to hearing from u

[Strongich]

In 325f897, I added off-policy eval support for VLM distillation (GRPO-style prediction_step, so eval runs the distillation loss over the held-out split without generation). I validated all three configurations with a 5% eval split:
1. Qwen3-VL-8B -> Qwen3-VL-2B, JSD, lmbda=0.5 (same-family)
Results:

2. Qwen3-VL-8B -> LFM2.5-VL-1.6B, ULD, lmbda=0.5, no-vllm (cross-family)
Results:

3. Qwen3-VL-8B -> Qwen3-VL-2B, ULD, lmbda=0.5
Results:

LGTM

[cursor]

Non-vLLM text path overwrites ULD original_prompt_text with special tokens

Medium Severity

In _generate_non_vllm_for_slices, original_prompt_text is unconditionally set to prompt_texts which is decoded with skip_special_tokens=False, including student chat-template markers. In contrast, _generate_on_policy_vlm_slice correctly preserves the collator's untemplated text. When ULD loss re-tokenizes these with the teacher tokenizer, student-specific tokens like <|im_end|> are treated as literal text, producing spurious teacher tokens and misaligned logits.

Additional Locations (1)

• trl/experimental/gold/gold_trainer.py#L1302-L1304

 

^{Reviewed by Cursor Bugbot for commit c187966. Configure here.}

[kashif]

thanks @Strongich I am working on a refactoring of the tokenization byte-offset issue in this branch: https://github.com/kashif/trl/tree/fix-issue-4393-byte-offsets for issue #4393 so perhaps your feature can also be added to my fix? what do you think?

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 030a831. Configure here.}

[Strongich]

Hi @kashif, sounds good. I'm currently updating the readability of my code, and once I have the time, I will take a look at this issue (maybe something from my imp needs to be addressed too)

[Strongich]

@kashif took a look, pretty clear: since the to_canonical_pieces are gone, and ULDLoss signature is changed, let me know when ur branch will be stable enough for me to port VLM features onto it

[kashif]

@Strongich i haved added you to my fork so you can try to add VLM support to the refactor if you like?

[Strongich]

This PR will be closed & a clean one will be opened once the #5885 is merged into main

[kashif]

thanks @Strongich looking forward!

[Strongich]

Closed in favor of #5969