JerryMungo
Honorary Master
- Joined
- Jul 18, 2008
- Messages
- 37,596
- Reaction score
- 6,316
.
Last edited:
-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: Which is the most hair-raising South African mountain pass you've driven?
Ollama
- Thread starter JerryMungo
- Start date
crypticgoose
New Member
@Willie Trombone Using it on desktop at home and in AWS. I have a RTX 3090 I got second hand with and AMD Ryzen 6800X.
Use it for embedding & inference. Embedding: https://ollama.com/library/nomic-embed-text. Inference: mistral-7b-instruct, llama3.1-7b, etc.
I build RAG demos with a focus on spinning it out into a startup some day. Basically ingest a lot of data, summarize and RAG it, then build chat experiences.
But Ive also used more gpt4-o-mini recently as its cheap and fast. A single rtx3090 can still be slow depending on the model.
Use it for embedding & inference. Embedding: https://ollama.com/library/nomic-embed-text. Inference: mistral-7b-instruct, llama3.1-7b, etc.
I build RAG demos with a focus on spinning it out into a startup some day. Basically ingest a lot of data, summarize and RAG it, then build chat experiences.
But Ive also used more gpt4-o-mini recently as its cheap and fast. A single rtx3090 can still be slow depending on the model.
crypticgoose
New Member
You can also hook it up to: https://www.litellm.ai.
Then you can load balance in case ollama fails to gpt-4 or another service like groq.
Then you can load balance in case ollama fails to gpt-4 or another service like groq.