Implemant semantic search

[iliatimofeev]

Implement APIs using semantic search.

Definition of done:

1. We can add to our collection 10 test documents including the Asq Quran.
2. When we search with query “Вопросы о боге” reference to Ask Quran returned in top 3 documents.
3. That criteria is implemented as unit test and passes
4. Search the exact part in a large document implemented as tests.

Scope

• Add Document:

  1. Split document into blocks (paragraphs) using lxml or pyquery
  2. Encode block using Cohere Embed
  3. Save it into index using Redis

• Search

  1. Encode query using cohere
  2. Request documents from redis
  3. Send results

Semantic search tutorials:

For the embedding part, look into Cohere semantic search; we will use Redis instead of AnnoyIndex.

For vector databases, look into Redis semantic search; we will use cohere instead of sentence_transformers.
Impement API

Cohere

cohere multilingual language models
Co.Embed API
cohere-python

Redis

https://github.com/RedisAI/vecsim-demo