Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add video-llama LVM microservice under lvms #494

Closed
wants to merge 76 commits into from
Closed

Add video-llama LVM microservice under lvms #494

wants to merge 76 commits into from

Conversation

BaoHuiling
Copy link
Collaborator

Description

Add support for video-llama LVM, which allows us to chat based on video file with LVM. The first usecase is VideoRAGQnA use case from Multimodal RAG proposal.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is still under review. opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)
  • Others (enhancement, documentation, validation, etc.)

Dependencies

python:3.9-slim git git-lfs faiss-cpu sentencepiece pandas Pillow prometheus_fastapi_instrumentator ftfy decord einops iopath omegaconf opencv-python-headless timm torch torchaudio langchain-core transformers webdataset pytorchvideo DAMO-NLP-SG/Video-LLaMA(github repo) DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned (huggingface repo)

Tests

Describe the tests that you ran to verify your changes.
https://github.com/siddhivelankar23/GenAIComps/blob/huiling-lvm/tests/test_lvms_video-llama.sh: build, start, validate, clean the microservice

@BaoHuiling BaoHuiling force-pushed the huiling-lvm branch 2 times, most recently from 461cfa8 to 0574132 Compare August 15, 2024 10:16
Spycsh and others added 22 commits August 15, 2024 18:25
@Spycsh @BaoHuiling
fix wrong indent (opea-project#364)
bf8c09e 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
initial commit
8d6427b 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
update video llama, model download
3117dd1 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
add test script
467d91b 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
Add LVM container, update video_path to URL
ec278d9 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
update for prod
f3a79ad 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
save
ee18b10 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
test script passed
565d6bf 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
add readme
f71ea37 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
add clean in readme
5ff6630 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
change input param name to "input"
a1f1257 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
fix typo
2856073 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
Add support for UI
4a4de0c 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
fix typo in test
2b35c01 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@bjzhjing @Spycsh @BaoHuiling
Redis-dataprep: Make Redis connection consistent (opea-project#359)
6875d48 
There are both 'from langchain_community.vectorstores import Redis' and
'import redis' in prepare_doc_redis.py. Redis.from_texts_return_keys()
is with REDIS_URL as connection, while redis.Redis() takes the
combination of REDIS_HOST and REDIS_PORT as connection. When the former
and the latter are not the same, it will fail to get data from vector
database. Change to use REDIS_URL as the unified way.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @pre-commit-ci @BaoHuiling
Update Dataprep with Parameter Settings (opea-project#351)
cf4a43b 
* update dataprep with parameter settings

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update port

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @pre-commit-ci @BaoHuiling
Support delete for Milvus vector db in Dataprep (opea-project#368)
8fe43ea 
* support delete for milvus

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Fix dockerfile format (opea-project#363)
82a3d10 
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@chensuyue @pre-commit-ci @BaoHuiling
refactor build image build and push workflow (opea-project#385)
c3698b0 
* refactor build image build and push workflow

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@minmin-intel @pre-commit-ci
@chensuyue @BaoHuiling
Add ML detection strategy to PII detection guardrail (opea-project#292)
7df37e0 
* first code of business safety classifier

* allow strategy options and update readme

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* disable ray for ml strategy and update test script

* add log to pii test

Signed-off-by: minmin-intel <minmin.hou@intel.com>

* update logging in test gaurdrail

* rm llm strategy and change url in test

Signed-off-by: minmin-intel <minmin.hou@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* delete file check in test and update readme

Signed-off-by: minmin-intel <minmin.hou@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: minmin-intel <minmin.hou@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @pre-commit-ci @BaoHuiling
Refine Dataprep Code & UT (opea-project#404)
60d2850 
* refine dataprep upload_link & ut

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add requirement

Signed-off-by: letonghan <letong.han@intel.com>

* modify container name

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@lvliang-intel @BaoHuiling
Add Uvicorn dependency (opea-project#408)
ab51e7e 
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
hteeyeoh and others added 29 commits August 15, 2024 18:28
@hteeyeoh @pre-commit-ci @BaoHuiling
prompt_registry: Unifying API endpoint port (opea-project#443)
7ae1b85 
* prompt_registry: Unifying API endpoint port

Unifying all port for API endpoints for prompt registry services with
recent register_microservice wrapper function update.

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@Spycsh @BaoHuiling
fix ffmpeg build on hpu (opea-project#441)
1dbcf83 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Add path check for dockerfiles in compose.yaml and change workflow na…
5830f07 
…me. (opea-project#412)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@zahidulhaque @BaoHuiling
Fix OpenVINO vLLM build scripts and update unit test case (opea-proje…
74d6476 
…ct#446)

Signed-off-by: Zahidul Haque <zahidul.haque@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@chensuyue @BaoHuiling
Optional container build instructions (opea-project#457)
a497345 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@xiguiw @BaoHuiling
LLM micro service extract input model name (opea-project#447)
cc5b741 
No input model from UI.
vLLM and OLLAMA get the model name from the envrionment
TGI mode model form ModelID

Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@lkk12014402 @sys-lpot-val
@lvliang-intel @BaoHuiling
refact embedding/ranking/llm request/response by referring to openai …
dacb69b 
…format (opea-project#405)

Co-authored-by: sys-lpot-val <sys_lpot_val@intel.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@pallavijaini0525 @BaoHuiling
Pinecone update to OPEA (opea-project#420)
ca545cf 
Signed-off-by: pallavi jaini <pallavi.jaini@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@rbrugaro @BaoHuiling
remove deprecated langchain imports and switch to langchain-huggingfa…
7546b23 
…ce (opea-project#417)

* remrove deprecated imports and switch to langchain-huggingface

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* fix typo

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* set requirements to langchain latest release

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* remove version for langchain in requirements

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

---------

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@yogeshmpandey @BaoHuiling
Added support for Unified Port, GET/DELETE endpoints in pgvector (ope…
ad8317f 
…a-project#444)

Signed-off-by: Yogesh <yogeshpandey@intel.com>
Co-authored-by: Yogesh <yogeshpandey@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @pre-commit-ci @BaoHuiling
Fix validation failure without exit. (opea-project#460)
aac314a 
* Fix validation failure without exit.

Signed-off-by: zepan <ze.pan@intel.com>

* Change validate log.

Signed-off-by: zepan <ze.pan@intel.com>

* Fix validation failure without exit.

Signed-off-by: zepan <ze.pan@intel.com>

* Change validate log.

Signed-off-by: zepan <ze.pan@intel.com>

* Change validate log.

Signed-off-by: zepan <ze.pan@intel.com>

* Fix issue for reranking-langchain-mosec.

Signed-off-by: zepan <ze.pan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix script issue

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@lianhao @BaoHuiling
whisper: Fix container build failure (opea-project#470)
c354a04 
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @BaoHuiling
Update Milvus docker-compose.yaml (opea-project#459)
e72ec0f 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @BaoHuiling
Refine robustness of Dataprep Redis (opea-project#463)
c8e347a 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@chensuyue @BaoHuiling
clean local images before test (opea-project#471)
446aff0 
* clean test images before test

Signed-off-by: chensuyue <suyue.chen@intel.com>

* add --no-cache for image build

Signed-off-by: chensuyue <suyue.chen@intel.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Fix embeddings_langchain-mosec issue. (opea-project#473)
3262aee 
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ganesanintel @BaoHuiling
Address test case failure (opea-project#462)
62360c1 
Signed-off-by: V, Ganesan <ganesan.v@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @BaoHuiling
Fix Dataprep qdrant issues and add Test Script (opea-project#474)
1e342c4 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@wangkl2 @BaoHuiling
Refine the instructions to run the retriever example with qdrant (ope…
c7570ec 
…a-project#442)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@rbrugaro @pre-commit-ci @BaoHuiling
fix HuggingFaceEmbedding deprecated in favor of HuggingFaceInferenceA…
1fcc547 
…PIEmbedding (opea-project#466)

* Fix HuggingFaceEmbedding deprecated in favor of HuggingFaceInferenceAPIEmbedding

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
update test scripts and bug fix (opea-project#464)
1acda9e 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling
Add output evaluation for guardrails (opea-project#332)
2a38ac4 
* add single query input/output guardrails

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* removed comment

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

---------

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@lvliang-intel @BaoHuiling
Revert max_new_tokens to 1024 (opea-project#479)
e11d0c9 
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@letonghan @BaoHuiling
fix empty content issue (opea-project#476)
0b647c8 
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@Spycsh @pre-commit-ci
@chensuyue @BaoHuiling
support whisper long-form generation (opea-project#469)
e59594b 
* fix long form asr accuracy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test input pad issue

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Fix issue (opea-project#483)
b749da8 
* Standardize image names and labels.

Signed-off-by: zepan <ze.pan@intel.com>

* Remove test cases that are no longer in use.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Change /root/ to /home/user/. (opea-project#475)
0862d2b 
* Change /root/ to /hmoe/user/.

Signed-off-by: zepan <ze.pan@intel.com>

* Fix issue.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@ZePan110 @BaoHuiling
Check build fail and change port 8008 to 5025/5026. (opea-project#487)
91278e0 
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@XuehaoSun @BaoHuiling
Freeze requirements workflow for release tag (opea-project#485)
0574132 
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: NeuralChatBot <grp_neural_chat_bot@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@BaoHuiling BaoHuiling closed this Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lvliang-intel lvliang-intel Awaiting requested review from lvliang-intel lvliang-intel is a code owner

@Spycsh Spycsh Awaiting requested review from Spycsh Spycsh is a code owner

@XuhuiRen XuhuiRen Awaiting requested review from XuhuiRen

@letonghan letonghan Awaiting requested review from letonghan letonghan is a code owner

@XinyuYe-Intel XinyuYe-Intel Awaiting requested review from XinyuYe-Intel XinyuYe-Intel is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.