Building an Ollama Environment on Fly GPUs

• I tried using Fly.io because it can be configured to start automatically only when needed, which seemed like a good way to keep GPU instance costs down for experimental environments.
• Ollama is convenient for comparisons as it has various LLMs registered in its library and can launch Web APIs or interactive shells.

Documentation

https://fly.io/docs/gpus/
https://fly.io/blog/scaling-llm-ollama/

Environment Setup

Since Google's Gemma is in the library, I'll make it available for use.

https://ai.google.dev/gemma?hl=en

Just prepare a fly.toml like the following and run fly launch.

• Currently, it is only available in the ord (Chicago) region.
• The price is A100 40G PCIe: $2.50/hr per GPU.
• You can use ollama/ollama | Docker Hub without building the image yourself.
• Download the model during deployment with release_command = "ollama pull gemma".

API Access

❯ curl -sX POST https://XXX.fly.dev/api/generate -d '{
  "model": "gemma",
  "prompt":"日本で一番高い山は"
 }' | jq .response;

"日本で"
"一番"
"高い"
"山"
"は"
"富士山"
"です"
"。"
"富士山"
"は"
"海"
"抜"
"高度"
"3"
","
"7"
"7"
"6"
"メートル"
"に"
"達"
"し"
"、"
"日本の"
"最高"
"峰"
"です"
"。"
""

Response Speed

Compare the execution speed on my own Windows (WSL) and macOS.

Fly GPUs(a100-pcie-40gb)

> for i in {1..10}; do
  curl -sX POST https://XXX.fly.dev/api/generate -d '{
    "model": "gemma",
    "prompt":"日本で一番高い山は"
  }' -o /dev/null -w '%{time_total}\n' | xargs
done | awk '{ total += $1; count++ } END { print total/count }'

0.635101

WSL

• AMD Ryzen 5 5600X 6-Core Processor
• RAM 8GB
• NVIDIA GeForce RTX 4070
• CUDA Toolkit 11.5

$ for i in {1..10}; do
  curl -sX POST http://localhost:11434/api/generate -d '{
    "model": "gemma",
    "prompt":"日本で一番高い山は"
  }' -o /dev/null -w '%{time_total}\n' | xargs
done | awk '{ total += $1; count++ } END { print total/count }'

1.00433

MacBook Pro

• Apple M1 Pro(2021)
• Ollama-darwin

❯ for i in {1..10}; do
  curl -sX POST http://localhost:11434/api/generate -d '{
    "model": "gemma",
    "prompt":"日本で一番高い山は"
  }' -o /dev/null -w '%{time_total}\n' | xargs
done | awk '{ total += $1; count++ } END { print total/count }'
1.26624

I'm not sure if it's using Metal, but I predicted it would be a bit slower if it were just the CPU.

Interactive Shell

Run fly ssh console while it is active.

# ollama run gemma

>>> Write a poem,
My heart moves your heart,
And you move my heart.

This poem explores the theme of love, focusing on moving the heart of a loved one.

Tips

Don't want to expose the endpoint externally

Follow the instructions in fly ips allocate-v6 --private.

Insufficient storage in the model directory

Extend it like fly vol extend vol_XXX -s '32gb'.

Want to use a model not in the library

Ollama can import models converted to work with Llama.cpp, and the documentation is available here:

ollama/docs/import.md at main · ollama/ollama

Want a Web UI

There seems to be something called Open WebUI (Formerly Ollama WebUI) (unverified):

https://github.com/open-webui/open-webui