Add video-llama LVM microservice under lvms #495

BaoHuiling · 2024-08-15T11:02:23Z

Description

Add support for video-llama LVM, which allows us to chat based on video file with LVM. The first usecase is VideoRAGQnA use case from Multimodal RAG proposal.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is still under review. opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

python:3.9-slim git git-lfs faiss-cpu sentencepiece pandas Pillow prometheus_fastapi_instrumentator ftfy decord einops iopath omegaconf opencv-python-headless timm torch torchaudio langchain-core transformers webdataset pytorchvideo DAMO-NLP-SG/Video-LLaMA(github repo) DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned (huggingface repo)

Tests

Describe the tests that you ran to verify your changes.
https://github.com/siddhivelankar23/GenAIComps/blob/huiling-lvm/tests/test_lvms_video-llama.sh: build, start, validate, clean the microservice

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

codecov · 2024-08-16T02:45:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
comps/cores/proto/docarray.py	`99.00% <100.00%> (+0.06%)`	⬆️

... and 1 file with indirect coverage changes

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

BaoHuiling · 2024-08-16T10:14:19Z

@chensuyue Hello Suyue, one question here, does the server that runs the CI test have access to the huggingface repo? I need to download files under https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned/tree/main, but it seems that this step is bypassed or failed.

chensuyue · 2024-08-16T12:44:16Z

@chensuyue Hello Suyue, one question here, does the server that runs the CI test have access to the huggingface repo? I need to download files under https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned/tree/main, but it seems that this step is bypassed or failed.

Yes, generally it should have. Use $HF_TOKEN in your test scripts.

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

BaoHuiling · 2024-08-17T04:06:19Z

@chensuyue Thank you for your comment, the image is download successfully.

However another problem occured, the port that i'm using 9000 is already allocated. The old containers are already cleaned before running the test, could you help to check environment and remove the conflict?

chensuyue · 2024-08-17T16:02:34Z

@chensuyue Thank you for your comment, the image is download successfully.

However another problem occured, the port that i'm using 9000 is already allocated. The old containers are already cleaned before running the test, could you help to check environment and remove the conflict?

port 9000 is widely used, also in GenAIExamples CI test, and they share the test machine. So please change the port into 5xxx which is suggested to be used in GenAIComps CI.

BaoHuiling · 2024-08-19T02:04:21Z

@Spycsh @lvliang-intel Please help to review the code, thanks!

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

BaoHuiling · 2024-08-20T01:49:03Z

Hello, @chensuyue @Spycsh @lvliang-intel could anyone help to review the code? The DDL is tomorrow and we may need some time to fix required changes.🙂

Spycsh · 2024-08-20T03:15:24Z

Hi @BaoHuiling , thanks for this exciting feature! It looks good to me. I just have a few questions: This only support Xeon CPU/CUDA, right? Have you tried video-llama LVM on Gaudi? Do you have a plan on optimizing it on Gaudi? Don't worry, we will still merge it for now if the HPU support is not clear for this component.

BaoHuiling · 2024-08-20T03:17:56Z

Hi @BaoHuiling , thanks for this exciting feature! It looks good to me. I just have a few questions: This only support Xeon CPU/CUDA, right? Have you tried video-llama LVM on Gaudi? Do you have a plan on optimizing it on Gaudi? Don't worry, we will still merge it for now if the HPU support is not clear for this component.

Thanks for the comment. Yes only Xeon is supported now. We haven't planned optimization on Gaudi yet, but I think it's good to have in the future.

Signed-off-by: BaoHuiling <huiling.bao@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit db8c893)

Signed-off-by: BaoHuiling <huiling.bao@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: BaoHuiling <huiling.bao@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

Signed-off-by: BaoHuiling <huiling.bao@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…e for v0.9 (#538) * clip embedding support Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * test script for embedding Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * fix freeze workflow (#522) Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Fix Dataprep Potential Error in get_file (#540) * fix get file error & refine logs Signed-off-by: letonghan <letong.han@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <letong.han@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Support SearchedDoc input type in LLM for No Rerank Pipeline (#541) Signed-off-by: letonghan <letong.han@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Add dependency for pdf2image and OCR processing (#421) Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Add local_embedding return 768 length to align with chatqna example (#313) Signed-off-by: Chendi.Xue <chendi.xue@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * add telemetry doc (#536) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Add video-llama LVM microservice under lvms (#495) Signed-off-by: BaoHuiling <huiling.bao@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Fix the data load issue for structured files (#505) Signed-off-by: XuhuiRen <xuhui.ren@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Add finetuning component (#502) Signed-off-by: Xinyu Ye <xinyu.ye@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com> Co-authored-by: test <test@intel.com> Co-authored-by: root <root@idc708073.jf.intel.com> Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * add torchvision into requirements (#546) Signed-off-by: chensuyue <suyue.chen@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Use Gaudi base images from Dockerhub (#526) * Use Gaudi base images from Dockerhub Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> * Fixing the malformed tag Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> * fix another malformed tag Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> --------- Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Add toxicity detection microservice (#338) * Add toxicity detection microservice Signed-off-by: Qun Gao <qun.gao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Modification to toxicity plugin PR (#432) * changed microservice to use Service.GUARDRAILS and input/output to TextDoc Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> * simplify dockerfile to use langchain Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> * sort requirements Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Minor SPDX header update (#434) Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> * Remove 'langsmith' per code review (#534) Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> * Add toxicity detection microservices with E2E testing Signed-off-by: Qun Gao <qun.gao@intel.com> --------- Signed-off-by: Qun Gao <qun.gao@intel.com> Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> Co-authored-by: Tyler W <tyler.wilbers@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * rename script and use 5xxx Signed-off-by: BaoHuiling <huiling.bao@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * add proxy for build Signed-off-by: BaoHuiling <huiling.bao@intel.com> Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * fixed commit issues Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * Fix docarray constraint Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * updated docarray Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm telemetry which cause error in mega Signed-off-by: BaoHuiling <huiling.bao@intel.com> * renamed dirs Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> * renamed test Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> --------- Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com> Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com> Signed-off-by: letonghan <letong.han@intel.com> Signed-off-by: Chendi.Xue <chendi.xue@intel.com> Signed-off-by: BaoHuiling <huiling.bao@intel.com> Signed-off-by: XuhuiRen <xuhui.ren@intel.com> Signed-off-by: Xinyu Ye <xinyu.ye@intel.com> Signed-off-by: chensuyue <suyue.chen@intel.com> Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> Signed-off-by: Qun Gao <qun.gao@intel.com> Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sun, Xuehao <xuehao.sun@intel.com> Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com> Co-authored-by: Zaili Wang <109502517+ZailiWang@users.noreply.github.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com> Co-authored-by: Huiling Bao <huiling.bao@intel.com> Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com> Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com> Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com> Co-authored-by: test <test@intel.com> Co-authored-by: root <root@idc708073.jf.intel.com> Co-authored-by: chen, suyue <suyue.chen@intel.com> Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com> Co-authored-by: qgao007 <108324932+qgao007@users.noreply.github.com> Co-authored-by: Tyler W <tyler.wilbers@intel.com>

BaoHuiling requested a review from lvliang-intel as a code owner August 15, 2024 11:02

BaoHuiling and others added 16 commits August 15, 2024 19:04

initial commit

8ebf205

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

update video llama, model download

bcffdff

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

add test script

2cd5e55

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Add LVM container, update video_path to URL

354166c

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

update for prod

c8aed32

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

save

275bb3d

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

test script passed

d49969a

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

add readme

25d643e

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

add clean in readme

140f041

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

change input param name to "input"

d4144e0

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

fix typo

57be2e6

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Add support for UI

80a8366

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

fix typo in test

149b6c9

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

584ddec

for more information, see https://pre-commit.ci

fix check

90cfea0

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

fix typo in patch

c2a3e89

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

BaoHuiling force-pushed the lvm branch 2 times, most recently from 7476487 to ff6110c Compare August 16, 2024 02:39

fix bug caused by bot

ff6110c

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

BaoHuiling force-pushed the lvm branch from 1d82d50 to 6f367c8 Compare August 16, 2024 03:06

BaoHuiling and others added 5 commits August 16, 2024 11:17

do not modify patch

88545b7

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

extract when build

6f367c8

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

support local file incase no internet

439174a

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

40a0605

for more information, see https://pre-commit.ci

update script to use local file

000e2a5

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

kevinintel requested a review from Spycsh August 16, 2024 06:38

use localhost in test

a59766b

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Merge branch 'main' into lvm

45bf663

BaoHuiling mentioned this pull request Aug 16, 2024

Add local Rerank microservice for VideoRAGQnA #496

Merged

4 tasks

BaoHuiling added 4 commits August 16, 2024 21:44

add debug steps

5869d9f

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Merge branch 'main' into lvm

db805e8

asdd Pillow for lvm

aabd962

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Merge branch 'main' into lvm

eb0437e

chensuyue added this to the v0.9 milestone Aug 17, 2024

Merge branch 'main' into lvm

c72482c

BaoHuiling added 3 commits August 19, 2024 10:16

use 5xxx for CI port

3bda667

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Merge branch 'main' into lvm

2b7cb4d

update volume name

1ec2e48

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

Merge branch 'main' into lvm

db1d78f

Spycsh approved these changes Aug 20, 2024

View reviewed changes

lvliang-intel approved these changes Aug 21, 2024

View reviewed changes

chensuyue merged commit db8c893 into opea-project:main Aug 21, 2024
14 checks passed

BaoHuiling mentioned this pull request Aug 29, 2024

Multimodal dataprep #575

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add video-llama LVM microservice under lvms #495

Add video-llama LVM microservice under lvms #495

BaoHuiling commented Aug 15, 2024

codecov bot commented Aug 16, 2024 •

edited

Loading

BaoHuiling commented Aug 16, 2024

chensuyue commented Aug 16, 2024

BaoHuiling commented Aug 17, 2024

chensuyue commented Aug 17, 2024

BaoHuiling commented Aug 19, 2024

BaoHuiling commented Aug 20, 2024

Spycsh commented Aug 20, 2024

BaoHuiling commented Aug 20, 2024

Add video-llama LVM microservice under lvms #495

Add video-llama LVM microservice under lvms #495

Conversation

BaoHuiling commented Aug 15, 2024

Description

Issues

Type of change

Dependencies

Tests

codecov bot commented Aug 16, 2024 • edited Loading

Codecov Report

BaoHuiling commented Aug 16, 2024

chensuyue commented Aug 16, 2024

BaoHuiling commented Aug 17, 2024

chensuyue commented Aug 17, 2024

BaoHuiling commented Aug 19, 2024

BaoHuiling commented Aug 20, 2024

Spycsh commented Aug 20, 2024

BaoHuiling commented Aug 20, 2024

codecov bot commented Aug 16, 2024 •

edited

Loading