Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support rerank models provided by vLLM, Xinference, Bedrock #572

Merged
merged 6 commits into from
Jan 10, 2025
Merged

feat: support rerank models provided by vLLM, Xinference, Bedrock #572

merged 6 commits into from
Jan 10, 2025

Conversation

jrj5423
Copy link
Collaborator

@jrj5423 jrj5423 commented Jan 6, 2025

Close #537.

  • Support for the rerank models provided by vLLM, Xinference and Amazon Bedrock.
  • Since Ollama does not support rerank models officially for the time being, these changes do not include support for Ollama.

@vercel Vercel
Copy link

vercel bot commented Jan 6, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
tidb-ai-preview ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 10, 2025 2:12am
tidb-ai-storybook ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 10, 2025 2:12am

@vercel Vercel
Copy link

vercel bot commented Jan 6, 2025

@jrj5423 is attempting to deploy a commit to the pingcap Team on Vercel.

A member of the Team first needs to authorize it.

@jrj5423
Copy link
Collaborator Author

jrj5423 commented Jan 6, 2025

By the way, I found that we have different names for the API endpoint URL of the model providers in different code files, for example, users need to fill in the "api_base" field in the config of the llm and embedding models, but in the config of the rerank model, users need to fill in the "api_url" field. Does it make sense to unify them as "api_base"?

@jrj5423 jrj5423 requested a review from sykp241095 January 6, 2025 21:04
@sykp241095 sykp241095 requested review from Mini256 and removed request for sykp241095 January 7, 2025 01:38
@sykp241095
Copy link
Member

By the way, I found that we have different names for the API endpoint URL of the model providers in different code files, for example, users need to fill in the "api_base" field in the config of the llm and embedding models, but in the config of the rerank model, users need to fill in the "api_url" field. Does it make sense to unify them as "api_base"?

Agree to combine them into one var name. cc @Mini256

Mini256
Mini256 requested changes Jan 7, 2025
View reviewed changes
backend/app/rag/chat_config.py Outdated Show resolved Hide resolved
backend/app/rag/chat_config.py Show resolved Hide resolved
backend/requirements-dev.lock Outdated Show resolved Hide resolved
backend/app/rag/reranker_model_option.py Outdated Show resolved Hide resolved
backend/app/rag/reranker_model_option.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/reranker_model_option.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
@Mini256 Mini256 changed the title More rerankers support Support rerank models provided by vLLM, Xinference, Bedrock Jan 7, 2025
@Mini256 Mini256 changed the title Support rerank models provided by vLLM, Xinference, Bedrock feat: support rerank models provided by vLLM, Xinference, Bedrock Jan 7, 2025
@jrj5423
Copy link
Collaborator Author

jrj5423 commented Jan 9, 2025

Changes requested were implemented.
For the upgrade of llama-index-core and other dependencies, creating llm/embedding model/rerank model, and generating KG were tested. @Mini256

@jrj5423 jrj5423 requested a review from Mini256 January 9, 2025 13:01
Mini256
Mini256 reviewed Jan 10, 2025
View reviewed changes
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/node_postprocessor/vllm_reranker.py Outdated Show resolved Hide resolved
backend/app/rag/reranker_model_option.py Outdated Show resolved Hide resolved
backend/app/rag/reranker_model_option.py Outdated Show resolved Hide resolved
frontend/app/src/pages/docs/reranker-model.mdx Outdated Show resolved Hide resolved
frontend/app/src/pages/docs/reranker-model.mdx Outdated Show resolved Hide resolved
frontend/app/src/pages/docs/reranker-model.mdx Outdated Show resolved Hide resolved
frontend/app/src/pages/docs/reranker-model.mdx Outdated Show resolved Hide resolved
Mini256
Mini256 approved these changes Jan 10, 2025
View reviewed changes
Copy link
Member

@Mini256 Mini256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Mini256 Mini256 merged commit 39319ed into pingcap:main Jan 10, 2025
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mini256 Mini256 Mini256 approved these changes

Assignees

@Mini256 Mini256

Labels
require-build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

core(reranker): support more rerankers, including documentation
3 participants
@jrj5423 @sykp241095 @Mini256