[fix] fix vl gradio, use pipeline api and remove interactive chat

[irexyc]

Motivation

Use pipeline api and remove interactive chat to support vl models like qwen2-vl

Currently with pytoch backend, it seems that the generator cannot be interrupted when sending a stop request. @grimoire

# reproduction code

import asyncio
from lmdeploy import pipeline, PytorchEngineConfig
pipe = pipeline('/mnt/141/vicuna-7b-v1.5/', backend_config=PytorchEngineConfig())

event = asyncio.Event()
messages = '写一篇高考作文'
session_id = 0

async def inference():
    step = 0
    async for out in pipe.generate(messages=messages, session_id=session_id):
        print(step, out)
        step += 1
        if (step == 5):
            event.set()
    print('inference done')


async def stop():
    await event.wait()
    await pipe.stop_session(session_id=session_id)


async def main():
    await asyncio.gather(inference(), stop())


if __name__ == '__main__':
    asyncio.run(main())

[grimoire]

The asyncio.Event should be created in a coroutine to prevent different even loop error.

Pytorch engine assumes user would break the generate loop by themself. If the automatic break is required, you can change _on_stop_session:

lmdeploy/lmdeploy/pytorch/engine/engine.py

Line 268 in e91ccf0

def _on_stop_session(self, reqs: Request, **kwargs):

to

    def _on_stop_session(self, reqs: Request, **kwargs):
        """on stop session callback."""
        for req in reqs:
            session_id = req.data['session_id']
            resp = req.data.get('response', True)
            resp_type = ResponseType.SESSION_NOT_EXIST
            if session_id in self.scheduler.sessions:
                self.scheduler.stop_session(session_id)
                session =  self.scheduler.sessions[session_id]
                for seq in session.sequences.values():
                    resp: Response = getattr(seq, 'resp', None)
                    if resp is not None:
                        resp.type = ResponseType.FINISH
                        self.req_manager.response(resp)
                resp_type = ResponseType.SUCCESS
            if resp:
                self._response(req.resp, resp_type)

[irexyc]

@grimoire
After applying this modification, there is still a probability that the program may hang

The following code is easier to reproduce

import asyncio
import numpy as np
from lmdeploy import pipeline, PytorchEngineConfig
import random

messages = '写一篇高考作文'
pipe = pipeline('/mnt/141/vicuna-7b-v1.5/', backend_config=PytorchEngineConfig())


async def inference(session_id):
    async for out in pipe.generate(messages=messages, session_id=session_id):
        pass
    print('inference done')


async def timer(event):
    dur = 1 + random.random() * 2
    await asyncio.sleep(dur)
    event.set()


async def stop(event, session_id):
    await event.wait()
    await pipe.stop_session(session_id=session_id)
    print('stop session')


async def main():
    session_id = 0
    event = asyncio.Event()
    while True:
        await asyncio.gather(inference(session_id), timer(event), stop(event, session_id))
        print(f'session={session_id} done\n')
        session_id += 1
        event.clear()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

[grimoire]

lmdeploy/lmdeploy/pytorch/engine/engine.py

Line 236 in 188f069

def _response(self, resp: Response, resp_type: ResponseType, data: Any = None, err_msg: str = ''):

    def _response(self, resp: Response, resp_type: ResponseType, data: Any = None, err_msg: str = ''):
        """response."""
        if resp.type == ResponseType.FINISH:
            return
        resp.type = resp_type
        resp.data = data
        resp.err_msg = err_msg
        self.req_manager.response(resp)

[zhulinJulia24]

It works on gradio！

[zhulinJulia24]

lgtm