Skip to content

Issues: embeddings-benchmark/mteb

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

124 Open 493 Closed
124 Open 493 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Leaderboard: Speed up get_scores()
#1740 opened Jan 9, 2025 by x-tabdeveloping
2
downsample MIRACLReranking enhancement New feature or request good first issue Good for newcomers
#1738 opened Jan 9, 2025 by KennethEnevoldsen
Downsample MIRACLReranking using MIRACLRetrievalHardNegatives enhancement New feature or request good first issue Good for newcomers
#1737 opened Jan 9, 2025 by KennethEnevoldsen
Combine different version of the Encoder interface [v2]
#1735 opened Jan 8, 2025 by KennethEnevoldsen
Potentially relevant multi turn rag dataset new dataset Issues related to adding a new task or dataset
#1732 opened Jan 8, 2025 by KennethEnevoldsen
Non Cosine-Sim Similarity functions for Sentence Transformer models are broken in v2.0.0
#1731 opened Jan 8, 2025 by orionw
5
Remove "memory_usage" from ModelMeta
#1729 opened Jan 8, 2025 by KennethEnevoldsen
How should we handle gated models when testing
#1727 opened Jan 8, 2025 by KennethEnevoldsen
2
Problem loading sentence-transformer/multi-qa-MiniLM-L6-cos-v1
#1726 opened Jan 8, 2025 by KennethEnevoldsen
Add support for STS evaluation for Cross-Encoders
#1725 opened Jan 8, 2025 by sam-hey
How to calculate MTEB score from individual files?
#1721 opened Jan 7, 2025 by orionw
2
Leaderboard: Add training data annotations for models good first issue Good for newcomers leaderboard issues related to the leaderboard
#1720 opened Jan 7, 2025 by x-tabdeveloping
1
Inconsistent casing for MIRACL metrics
#1718 opened Jan 6, 2025 by Muennighoff
1
malenia1/ternary-weight-embedding does not load weights
#1717 opened Jan 6, 2025 by Muennighoff
Add test to ensure that the leaderboard can build leaderboard issues related to the leaderboard
#1715 opened Jan 6, 2025 by KennethEnevoldsen
There are multiple 'mteb/arguana' configurations in the cache: default, corpus, queries with HF_HUB_OFFLINE=1
#1714 opened Jan 6, 2025 by Bhavya6187
3
integrate VSI-BENCH modality: image The image extension of MTEB new benchmark Issues related to adding a new benchmark
#1712 opened Jan 6, 2025 by Samoed
Add MMEB modality: image The image extension of MTEB new benchmark Issues related to adding a new benchmark
#1709 opened Jan 5, 2025 by Muennighoff
May I ask if MTEB supports evaluating downloaded data？
#1701 opened Jan 4, 2025 by BtlWolf
4
Add benchmark on word order enhancement New feature or request new benchmark Issues related to adding a new benchmark
#1699 opened Jan 3, 2025 by KennethEnevoldsen
BUCC fails
#1694 opened Jan 2, 2025 by Muennighoff
1
4
[mieb] SketchyI2IRetrieval hangs in the beginning modality: image The image extension of MTEB
#1686 opened Jan 2, 2025 by Muennighoff
2
voyage fails
#1681 opened Jan 2, 2025 by Muennighoff
2
LLM2Vec fails
#1679 opened Jan 2, 2025 by Muennighoff
BeastyZ/e5-R-mistral-7b fails
#1661 opened Jan 1, 2025 by Muennighoff
1
ProTip! Type g i on any issue or pull request to go back to the issue listing page.