[Feature] support deepseek-vl2 for pytorch engine

[CUHKSZzxy]

Motivation

Issue 3042 support DeepSeek-VL2 for the PyTorch engine.

Modification

1. Add chat template, builders...
2. Add vision model preprocessing, model forward related file etc.
3. Modify several configuration check conditions, since deepseek-vl2 has different huggingface configuration formats, which leads to errors in the original implementations.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

Single image inference

import os
from lmdeploy import pipeline, PytorchEngineConfig, GenerationConfig
from lmdeploy.vl import load_image

os.environ['CUDA_VISIBLE_DEVICES'] = '0'


model_path = '/mnt/141/deepseek-vl2'

pt_backend_config = PytorchEngineConfig(
                    tp=1,
                    max_batch_size=32,
                    cache_max_entry_count=0.8,
                    session_len=8192,
                )

gen_config = GenerationConfig(
                    max_new_tokens=512,
                    do_sample=False,
                )


def main():
    pipe = pipeline(
                    model_path,
                    backend_config=pt_backend_config,
                    )

    image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')

    response = pipe(('describe this image.', image), gen_config=gen_config)
    print(response)

if __name__ == '__main__':
    main()

Expected outputs:

Multi-round conversations

import os
from lmdeploy import pipeline, PytorchEngineConfig, GenerationConfig
from lmdeploy.vl import load_image

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'


model_path = '/mnt/141/deepseek-vl2'

pt_backend_config = PytorchEngineConfig(
                    tp=2,
                    max_batch_size=32,
                    cache_max_entry_count=0.8,
                    session_len=8192,
                )

gen_config = GenerationConfig(
                    max_new_tokens=512,
                    do_sample=False,
                )


def main():
    pipe = pipeline(
                    model_path,
                    backend_config=pt_backend_config,
                    )

    image_woman = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg')
    image_tiger = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')

    sess = pipe.chat(('<IMAGE_TOKEN>Describe this image', image_woman))
    print(sess.response.text)
    print('-'*50)
    sess = pipe.chat('What is the woman doing?', session=sess)
    print(sess.response.text)
    print('-'*50)
    sess = pipe.chat(('<IMAGE_TOKEN>Can you describe this image as well?', image_tiger), session=sess)
    print(sess.response.text)
    print('-'*50)
    sess = pipe.chat(('Tell me a story using those images.'), session=sess)
    print(sess.response.text)
    print('-'*50)


if __name__ == '__main__':
    main()

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[lvhan028]

may update supported models in RADME and supported_models.md

[RunningLeon]

instead of using timm, we may need to add vision transformers using kernels of lmdeploy just like

lmdeploy/lmdeploy/pytorch/models/chatglm2.py

Line 330 in a1b86d9

class EVA2CLIPAttention(nn.Module):

[CUHKSZzxy]

I have tried to implement the VisionTransformer part using the upstream repository code, replacing the operators inside with LMDeploy kernels. While upstream code depends on the xformers memory_efficient_attention here, which I replaced with torch scaled_dot_product_attention but yields different tensor outputs.

Therefore, I refer to vllm deepseek-vl2 vision part for simplicity.

I may spare more time and try to accomplish previous attempts to reuse LMDeploy kernels instead of timm.

[RunningLeon]

using pure timm without xformers is show and if it can be repalced with sdpa, it can be replaced by kernels in lmdeploy as well.

[lvhan028]

Let's make another PR to replace timm.

[RunningLeon]

LGTM