Support OpenAI and Ollama #60
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
shavit
force-pushed
the
59-openai-prompt-templates
branch
from
0d61eaf to
a466526
Compare
|
For ollama would be good to support |
|
The remote server backends will need API key field, to be added as authorization header, and a selected model name from a separate list. Since the model ID is used to select model names but also the remote model, the settings need another option to choose a backend type. Then the model ID can be used for remote backends. |
|
This is cool, great work! Would it make sense to go more general and migrate OllamaBackend -> OpenAIBackend? Now that there's template support in llama.cpp server, we could migrate the default LlamaServer logic to llama.cpp server's openAI API and hopefully share all of the code. |
|
They are similar but not the same:
The Settings and Agent is where all the backends share the same behavior. An interface of chat completion can have context, user messages, and maybe options for the temperature etc. that can be shared across all backends. |
|
Would their openai /v1/chat/completions endpoint give what we need? https://github.com/ollama/ollama/blob/main/docs/openai.md#endpoints |
|
Yes, I don't remember why I used the other endpoint. |
|
There are few more changes to make, such as backend initialization to ensure it is not nil, and solve the conflicts. Currently the local version of llama.cpp doesn't work, but it could be outdated. Other notes:
|
|
Just played with this, very cool. I like the general approach of allowing you to switch backends (and having the 0-config localhost backend by default). Try merging A few other thoughts... Since this is used for multiple backends, switch copy to "Configure your backend based on the model you're using"
|
shavit
force-pushed
the
59-openai-prompt-templates
branch
from
3a6afe4 to
1faa518
Compare
Also the prompt is ignored now. |
* Add backend types with default URLs * Use llama to only run the local instance * Make submit(input:) async
* Create backends for local and remote servers * List models for each backend * Save backend type * Change backends during chat conversations
Each backend has its own config, model, token, and a default host value. More changes: * Importing a single file will open the app and set the model and backend. * Each backend has its own model list. * Choosing a model will not override other backends.
* Update model list when file is added or deleted * Update completion params * Match the picker selection to a model file
* Update the backend response * Select and use imported model file
* Remove context from backends * Provide a fallback baseURL to new backends * Pass config to create backends and add another fallback to ensure initialization
* Use localized strings with markdown. * Add system prompt to completion. * Add a default port 443 for OpenAI to ensure port value in settings. * Determine default value for `selectedModelId`.
Fetch the backend config and create the backend on each reboot.
shavit
force-pushed
the
59-openai-prompt-templates
branch
from
f8894fd to
d71f565
Compare
|
Just had some time to test and found a fatal bug when I send a message after switching to the default backend. Not quite sure what's going on. 
|
* Create a backend during agent initialization. * Start the local llama server in conversation view
|
Yes, the backend was an implicitly unwrapped optional to find those errors, rather than silence them and not respond at all. Now the backend is being initialized together with the agent, and uses the default local server. Maybe the initialization parameters of the agent and error handling can be improved. |
This change extends previous work on remote models, and adds OpenAI compatible backend #59
Tasks and discussions:
EventStreamfrom0.0.5, because it can crash if users misconfigure their server: https://github.com/Recouse/EventSource/blob/8c0af68bf3a4c93819d3fa5f549232f612324de2/Sources/EventSource/ServerMessage.swift#L55-L57Ideally the change will not affect what's already working right now with Llama, and have the minimum necessary change. Upgrades or refactoring can be added at the end.