⚡️ evento

Joining forces with innovators and AI enthusiasts, this project is a dynamic collaboration aimed at crafting a cutting-edge event-retrieval system, proudly participating in the Ho Chi Minh AI Challenge 2024.

Table of Contents

• 📍 Overview
• 🎯 Features
• 🤖 Tech Stack
• 🚀 Usage
• 📋 API
• 🎬 Demo
• 👣 Workflow
• 📐 App Structure
• 🧑‍💻 Contributors

📍 Overview

Welcome to ⚡️ evento, an ambitious collaborative project aimed at revolutionizing event retrieval through the innovative use of visual data. Our team, AIO_TOP10, is honored to participate in the prestigious Ho Chi Minh AI Challenge 2024, where we strive to showcase our expertise in artificial intelligence. We are committed to developing a cutting-edge, robust, and efficient event-retrieval system, leveraging the immense potential of AI to enhance information retrieval processes.

More details about the challenge refers to this link.

🎯 Features

• Multimodal search.
• Synthetic data with a Multimodal Model.
• Share similarity search.

Note: We are happy to share our trip while developing this app.

🤖 Tech Stack

Server building

• Back-end: FastAPI.
• Front-end: Streamlit.

Core technology

• Keyframe-extraction: TransNetV2 + Adaptive clustering.
• LLM: Gemini.
• Embedding: CLIP, BLIP2.

🚀 Usage

1. Clone the repository

git clone https://github.com/MinLee0210/evento.git
cd /evento

2. Setup Environment

Download dataset from Kaggle

We store our dataset on Kaggle. Please, download it from here and compress it in db directory, /backend/db. Additionally, we have 2 other appoaches to download. You can read the detail from here.

Note: Currently, we do not offer an automated solution for transferring our local dataset to a database. However, we highly recommend considering MySQL for efficient data reading, MongoDB for seamless data writing, and Redis for caching high-similarity queries, a feature we refer to as 'share search'. For more insights into our share search mechanism, please refer to our comprehensive documentation (docs).

Set API's key in backend directory

We use Gemini's API for extracting keywords and refining queries. As a result, setting Gemini's API key is essential to run the app. We also provide a env.template as an example in the backend.

3. Run the application

Run the backend

# /evento

cd /backend
bash start_be.sh 

In case you can not run bash:

1. Install requirements with pip install -r requirements.txt
2. Move to the /app and run uvicorn main:app --host=0.0.0.0 --port=8000 --reload.

Run the frontend

# /evento

cd /frontend
bash start_fe.sh 

In case you can not run bash:

1. Install requirements with pip install -r requirements.txt
2. Run streamilit run app.py.

📋 API

Method	Type	Description
/	GET	Get a random quote. Just for checking basic connection between frontend and backend.
/search	POST	Search by text.
/search/ocr	POST	Search by fuzzy matching between extracted keywords and OCR-based extraced data.
/search/image/{image_idx}	GET	Get image by image_idx.
/search/video/{vid_idx}	GET	Get video metadata by vid_idx.

Note: Detail about how to get response after running the app successfully is in notebook

🎬 Demo

• Galleries

• Videos: here

👣 Workflow

📐 App Structure

.
├── backend
│   ├── app
│   │   ├── api
│   │   │   └── v1
│   │   │       └── query_refine
│   │   ├── components
│   │   │   ├── embedding
│   │   │   ├── fuzzymatching
│   │   │   ├── kfe
│   │   │   ├── llms
│   │   │   └── translation
│   │   ├── core
│   │   ├── routes
│   │   ├── schema
│   │   ├── services
│   │   └── utils
│   ├── db
│   │   ├── features
│   │   ├── media-info
│   │   ├── objects
│   │   └── s_optimized_keyframes
│   ├── experimental
│   │   └── recommender
│   └── test
│       ├── api                               # Testing logic of defined business's API.
│       └── unit                              # Testing each components.
├── docs
│   ├── notebooks
│   └── test_query
├── frontend
│   ├── api
│   ├── assets
│   ├── components
│   └── views
└── scripts

🧑‍💻 Collaborators

A big thank you to the following amazing individuals for their valuable contributions to this project:

• Vũ Hoàng Phát - Lead Project Manager & AI Solutions Developer

  • Primarily responsible for researching and implementing AI solutions
  • Analyzed and processed data

• Lê Đức Minh - Backend Developer & AI Researcher

  • Deployed the backend
  • Conducted research to propose ideas for new features and improvements

• Trần Nguyễn Vân Anh - AI Researcher & Technical Writer

  • Contributed to AI research and technical documentation

• Phạm Thị Ngọc Huyền - AI Researcher, Technical Writer & UI/UX Designer

  • Focused on AI research, writing technical content, and designing user-friendly interfaces

• Phạm Nguyễn Quốc Huy - Frontend Developer

  • Developed and optimized the frontend

• Nguyễn Hải Đăng - Technical Writer Mentor

  • Mentored and guided the team in creating high-quality technical documentation