Skip to content

Commit

Permalink
v5.1
Browse files Browse the repository at this point in the history 
- Implemented video call support with image recognition using diffusion description
- Improved UI
- Updated UI to Kawai v5.1
- Fixed menus
- Improved mobile adaptivity
- Stabilized server works
- Full client solution
- Better dark mode
- Fixed overlaying CSS
- Project partial factoring
- Small directory renamings
- Uploaded datasets in the first beta
- Moved libraries in the "lib/" directory
- Fixed request bug
- Fixed overlaying topbar
- Improved voice recognition
  • Loading branch information
@yukiarimo
yukiarimo committed Nov 20, 2023
1 parent 8500945 commit 5cac182
Show file tree
Hide file tree
Showing 24 changed files with 52,827 additions and 184 deletions.
10 changes: 4 additions & 6 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
node_modules
__pycache__/
static/db/*.json
!static/db/history_template.json
static/audio/output.aiff
static/audio/output.mp3
lib/datasets/
lib/models/
lib/yunacpp/models/
lib/yunacpp/yuna.so
lib/yunacpp/build-yuna/
lib/datasets/yuna/
lib/yuna/models/
lib/yuna/build-yuna/
static/img/call/captured_image.jpg
16 changes: 10 additions & 6 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,18 @@ To install Yuna AI, follow these steps:
1. Clone the Yuna AI repository to your local machine using `git clone https://github.com/yukiarimo/yuna-ai.git`.
2. Make sure you have Python 3.10 or later installed.
3. Install the required dependencies by running `pip install -r requirements.txt`.
4. Run the `config.py` file to build external dependencies for Yuna in the `lib/yunacpp/` directory.
5. Put your AI model like `gguf_model.bin` into the `lib/yunacpp/models/` directory.
4. Open terminal and run `cd 'lib/yuna/'`.
5. Activate virtual environment (optional).
6. Run the `config.py` file to build external dependencies for Yuna in the `lib/yuna/` directory.
7. Put your AI model like `gguf_model.bin` into the `lib/yuna/models/` directory.

To start using AI, do the following:
1. Run the AI starter file with the command `sh start.sh` in the `libyunacpp/` directory to start the Yuna model.
2. Run `python index.py` command in the main directory to start the WebUI.
3. Double-click the `index.html` file in the directory or run it via Live Server.
4. Done!
1. Go to previous terminal
2. Run the AI starter file with the command `sh start.sh` in the `lib/yuna/` directory to start the Yuna model.
3. Open second terminal window
4. Run `python index.py` command in the main directory to start the WebUI.
5. Double-click the `index.html` file in the directory or run it via Live Server (better).
6. Done!

> Note: port and directory or file names can depend on your configuration.
Expand Down
67 changes: 16 additions & 51 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@
</head>

<body>
<div class="sidebar-o hideside" id="sidebar">
<div class="sidebar-o" id="sidebar">
<div class="coll-lo v-coll" style="background: none;">
<div class="block-text la">Menu</div>
<img src="static/img/yuna.png" alt="Yuna" class="block-image">
</div>

<div class="side-tab-block">
Expand All @@ -54,21 +55,14 @@
</div>
</div>

<div class="block-o center">
<div class="tab-o tab-o-default" id="1">
<div class="topbar-o">
<div class="block-button" id="toggleButtonSide" onclick="toggleSidebar();">Side</div>
<div class="top-tab-block">
<div class="top-tab-block-e block-text">
<img src="static/img/yuna.png" alt="Yuna" class="block-image">
</div>
<div class="top-tab-block-e block-text">
<h1 class="la block-text">Yuna</h1>
</div>
</div>
<div class="la block-text" id="settingsButton" onclick="openConfigParams();">⚙️</div>
</div>
<div class="topbar-o">
<div class="block-button" id="toggleButtonSide" onclick="toggleSidebar();">Side</div>
<div><div class="la block-text">Yuna</div></div>
<div class="la block-text" id="settingsButton" onclick="openConfigParams();">⚙️</div>
</div>

<div class="block-o">
<div class="tab-o tab-o-default" id="1">
<div class="coll-lo">
<div class="block-card el-9">
<div class="block-scroll" id="message-container">
Expand All @@ -87,19 +81,6 @@ <h1 class="la block-text">Yuna</h1>
</div>

<div class="tab-o" id="2">
<div class="topbar-o">
<div class="block-button" id="toggleButtonSide" onclick="toggleSidebar();">Side</div>
<div class="top-tab-block">
<div class="top-tab-block-e block-text">
<img src="static/img/yuna.png" alt="Yuna" class="block-image">
</div>
<div class="top-tab-block-e block-text">
<h1 class="la block-text">Yuna</h1>
</div>
</div>
<button class="block-button" id="startButton">Talk</button>
</div>

<div class="coll-lo">
<audio id="backgroundMusic">
<source src="static/audio/output.mp3" type="audio/mp3">
Expand All @@ -110,32 +91,19 @@ <h1 class="la block-text">Yuna</h1>
<img src="static/img/yuna-full.png" alt="Yuna Participant" style="border-radius: 20px;">
<video id="localVideo" playsinline autoplay muted class="draggable-video"></video>
<div class="controls">
<button class="block-button" id="camera-off">📷</button>
<button class="block-button" id="end-call">📞</button>
<button class="block-button" id="mute-microphone">🎤</button>
<button class="block-button" id="camera-off">Off</button>
<button class="block-button" id="end-call">End</button>
<button class="block-button" id="mute-microphone">Mute</button>

<canvas id="capture-canvas" style="display: none;"></canvas> <!-- Add this canvas element -->
<button class="block-button" id="capture-image">📷</button> <!-- Add this button -->
<button class="block-button" id="send-captured-image">Send</button>
<button class="block-button" id="capture-image">Capture</button> <!-- Add this button -->
<button class="block-button" id="startButton">Talk</button>
</div>
</div>
</div>
</div>

<div class="tab-o" id="3">
<div class="topbar-o">
<div class="block-button" id="toggleButtonSide" onclick="toggleSidebar();">Side</div>
<div class="top-tab-block">
<div class="top-tab-block-e block-text">
<img src="static/img/yuna.png" alt="Yuna" class="block-image">
</div>
<div class="top-tab-block-e block-text">
<h1 class="la block-text">Yuna</h1>
</div>
</div>
<div class="la block-text">⚙️</div>
</div>

<div class="coll-lo">
<div class="block-card el-9">
<h1>Modes</h1>
Expand All @@ -157,7 +125,7 @@ <h1>Mode</h1>
</div>

<div class="block-card el-9">
<div id="parameter-container">
<div class="block-scroll" id="parameter-container">
<!-- Dynamic HTML elements will be added here -->
</div>
<button id="save-button" onclick="saveConfigParams();">Save Parameters to Local Storage</button>
Expand All @@ -182,10 +150,7 @@ <h1>Mode</h1>
<div class="block-list-e" onclick="location.reload();">🔄 Refresh</div>
<div class="block-list-e" onclick="scrollMsg();">📜 Scroll</div>
<div class="block-list-e" onclick="muteAudio()">📢 Mute</div>
<label for="history-select">History</label>
<button class="block-button" aria-label="History">
<select id="history-select" onchange="loadSelectedHistory()">📁</select>
</button>
<select class="block-list-e block-button" id="history-select" onchange="loadSelectedHistory()">📁</select>
</div>
</div>
</div>
Expand Down
66 changes: 62 additions & 4 deletions index.py
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
import base64
from flask import Flask, request, jsonify
import shutil
import subprocess
from generate import generate # Import the generate function from generate.py
from lib.generate import generate # Import the generate function from generate.py
import os
import json
from flask_cors import CORS
from ctransformers import AutoModelForCausalLM
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration

processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

app = Flask(__name__)
CORS(app)
Expand Down Expand Up @@ -70,12 +77,24 @@ def upload_captured_image():

# Decode the base64 image data URL and process it as needed
# You can use a library like Pillow to save or manipulate the image
image_data = base64.b64decode(image_data_url.split(',')[1])
image_path = os.path.join('static', 'img', 'call', 'captured_image.jpg')
with open(image_path, 'wb') as image_file:
image_file.write(image_data)

img_path = 'static/img/call/captured_image.jpg'
raw_image = Image.open(img_path).convert('RGB')

# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")

out = model.generate(**inputs, max_length=150)
image_caption = str(processor.decode(out[0], skip_special_tokens=True))
print(image_caption)

# Respond with a success message
print('ok')
return jsonify({'message': 'Captured image received and processed successfully'})
return jsonify({'message': f'{image_caption}'})

print('shit')
# If something goes wrong or no image data is provided
return jsonify({'error': 'Image upload failed'}), 400

Expand All @@ -95,6 +114,45 @@ def load_history_file(filename):
return json.load(file)
else:
return []

def stableDiffusion(prompt):
import torch
from diffusers import StableDiffusionPipeline

device = "mps" if torch.backends.mps.is_available() else "cpu"
torch.device(device)

pipe = StableDiffusionPipeline.from_pretrained('./image-gen', torch_dtype=torch.float16)
pipe = pipe.to(device)

prompt = "Japanese girl with background of london bridge in color with text"
image = pipe(prompt).images[0]

image.save("result.png")

def clearGen(prompt):
llm = AutoModelForCausalLM.from_pretrained(
"lib/yuna/models/pygmalion-2-7b.Q5_K_M.gguf",
model_type='llama2',
top_k=40,
top_p=0.1,
temperature=0.7,
repetition_penalty=1.18,
last_n_tokens=64,
seed=123,
batch_size=64,
context_length=8192,
max_new_tokens=300,
gpu_layers=1
)

while True:
test = input("YUKI: ")

print("Yuna: ", end="")
for text in llm("Yuki: " + test + "\nYuna:", stream=True):
print(text, end="", flush=True)
print()

if __name__ == '__main__':
app.run(host='0.0.0.0', port=4848)
Loading

0 comments on commit 5cac182

Please sign in to comment.