llama-stackを試す

$ mkdir llama-stack-test && cd llama-stack-test
$ python -m venv .venv
$ source .venv/bin/activate

$ git clone https://github.com/meta-llama/llama-stack llama-stack-repo && cd llama-stack-repo
$ pip install -e .

$ llama -h
usage: llama [-h] {download,model,stack} ...

Welcome to the Llama CLI

options:
  -h, --help            show this help message and exit

subcommands:
  {download,model,stack}

$ cd ..

$ llama stack build

> Enter a name for your Llama Stack (e.g. my-local-stack): my-local-stack
> Enter the image type you want your Llama Stack to be built as (docker or conda): docker
> Enter provider for the inference API: (default=meta-reference): meta-reference
> Enter provider for the safety API: (default=meta-reference): meta-reference
> Enter provider for the agents API: (default=meta-reference): meta-reference
> Enter provider for the memory API: (default=meta-reference): meta-reference
> Enter provider for the telemetry API: (default=meta-reference): meta-reference
 > (Optional) Enter a short description for your Llama Stack: my first stack

Dockerfile created successfully in /tmp/tmp.CpsrrGxSIg/DockerfileFROM python:3.10-slim
WORKDIR /app
(snip)
[+] Building 131.6s (9/13)                                       docker:default
 => => extracting sha256:a2318d6c47ec9cac5acc500c47c79602bcf953cec711a18b  0.5s
 => => extracting sha256:ddee35dfe011992ed9d752dc61e7dc4377fda83f63e03c0e  0.1s
 => => extracting sha256:8a7a13810b7d60799d98f4a85010c5343a11ce82bf99a2a9  0.3s
 => => extracting sha256:cd4054fd866013877f7664cf997a9b812814d226ee5f3d83  0.0s
 => [2/8] WORKDIR /app                                                     0.5s
 => [3/8] RUN apt-get update && apt-get install -y        iputils-ping n  10.3s
 => [4/8] RUN pip install llama-stack                                     21.0s
 => [5/8] RUN pip install fastapi fire httpx uvicorn accelerate blobfile  90.3s
()

You can run it with: podman run -p 8000:8000 llamastack-my-local-stack
Checking image builds...
name: my-local-stack
distribution_spec:
  description: my first stack
  docker_image: null
  providers:
    inference: meta-reference
    safety: meta-reference
    agents: meta-reference
    memory: meta-reference
    telemetry: meta-reference
image_type: docker
Build spec configuration saved at /SOMEWHERE/llama-stack-test/llama-stack-repo/tmp/configs/my-local-stack-build.yaml
You can now run `llama stack configure llamastack-my-local-stack`

$ docker images
REPOSITORY                  TAG      IMAGE ID       CREATED         SIZE
llamastack-my-local-stack   latest   25b0bacb0f1e   3 minutes ago   11GB
(snip)

$ llama model list
+----------------------------------+------------------------------------------+----------------+
| Model Descriptor                 | Hugging Face Repo                        | Context Length |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-8B                      | meta-llama/Llama-3.1-8B                  | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-70B                     | meta-llama/Llama-3.1-70B                 | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B:bf16-mp8           | meta-llama/Llama-3.1-405B                | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B                    | meta-llama/Llama-3.1-405B-FP8            | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B:bf16-mp16          | meta-llama/Llama-3.1-405B                | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-8B-Instruct             | meta-llama/Llama-3.1-8B-Instruct         | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-70B-Instruct            | meta-llama/Llama-3.1-70B-Instruct        | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B-Instruct:bf16-mp8  | meta-llama/Llama-3.1-405B-Instruct       | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B-Instruct           | meta-llama/Llama-3.1-405B-Instruct-FP8   | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.1-405B-Instruct:bf16-mp16 | meta-llama/Llama-3.1-405B-Instruct       | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-1B                      | meta-llama/Llama-3.2-1B                  | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-3B                      | meta-llama/Llama-3.2-3B                  | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-11B-Vision              | meta-llama/Llama-3.2-11B-Vision          | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-90B-Vision              | meta-llama/Llama-3.2-90B-Vision          | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-1B-Instruct             | meta-llama/Llama-3.2-1B-Instruct         | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-3B-Instruct             | meta-llama/Llama-3.2-3B-Instruct         | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-11B-Vision-Instruct     | meta-llama/Llama-3.2-11B-Vision-Instruct | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama3.2-90B-Vision-Instruct     | meta-llama/Llama-3.2-90B-Vision-Instruct | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-3-11B-Vision         | meta-llama/Llama-Guard-3-11B-Vision      | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-3-1B:int4-mp1        | meta-llama/Llama-Guard-3-1B-INT4         | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-3-1B                 | meta-llama/Llama-Guard-3-1B              | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-3-8B                 | meta-llama/Llama-Guard-3-8B              | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-3-8B:int8-mp1        | meta-llama/Llama-Guard-3-8B-INT8         | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Prompt-Guard-86M                 | meta-llama/Prompt-Guard-86M              | 128K           |
+----------------------------------+------------------------------------------+----------------+
| Llama-Guard-2-8B                 | meta-llama/Llama-Guard-2-8B              | 4K             |
+----------------------------------+------------------------------------------+----------------+

$ llama stack configure llamastack-my-local-stack

Could not find llamastack-my-local-stack. Trying conda build name instead...
Could not find llamastack-my-local-stack. Trying docker image name instead...
+ docker run -it -v /home/kun432/.llama/builds/docker:/app/builds llamastack-my-local-stack llama stack configure ./llamastack-build.yaml --output-dir /app/builds
Configuration already exists at `/app/builds/my-local-stack-run.yaml`. Will overwrite...
Configuring API `inference`...
=== Configuring provider `meta-reference` for API inference...
Enter value for model (default: Llama3.1-8B-Instruct) (required): Llama3.2-3B-Instruct
Do you want to configure quantization? (y/n): n
Enter value for torch_seed (optional):
Enter value for max_seq_len (default: 4096) (required):
Enter value for max_batch_size (default: 1) (required):

Configuring API `safety`...
=== Configuring provider `meta-reference` for API safety...
Do you want to configure llama_guard_shield? (y/n): y
Entering sub-configuration for llama_guard_shield:
Enter value for model (default: Llama-Guard-3-8B) (required): Llama-Guard-3-8B
Enter value for excluded_categories (default: []) (required):
Enter value for disable_input_check (default: False) (required):
Enter value for disable_output_check (default: False) (required):
Do you want to configure prompt_guard_shield? (y/n): y
Entering sub-configuration for prompt_guard_shield:
Enter value for model (default: Prompt-Guard-86M) (required): Prompt-Guard-86M

Configuring API `agents`...
=== Configuring provider `meta-reference` for API agents...
Enter `type` for persistence_store (options: redis, sqlite, postgres) (default: sqlite):

Configuring SqliteKVStoreConfig:
Enter value for namespace (optional):
Enter value for db_path (existing: /root/.llama/runtime/kvstore.db) (required):

Configuring API `memory`...
=== Configuring provider `meta-reference` for API memory...
> Please enter the supported memory bank type your provider has for memory: vect
or

Configuring API `telemetry`...
=== Configuring provider `meta-reference` for API telemetry...

> YAML configuration has been written to `/app/builds/my-local-stack-run.yaml`.
You can now run `llama stack run my-local-stack --port PORT`
YAML configuration has been written to /home/kun432/.llama/builds/docker/my-local-stack-run.yaml. You can now run `llama stack run /home/kun432/.llama/builds/docker/my-local-stack-run.yaml`

built_at: '2024-09-26T17:46:01.915092'
image_name: my-local-stack
docker_image: my-local-stack
conda_env: null
apis_to_serve:
- memory
- agents
- models
- inference
- memory_banks
- shields
- safety
api_providers:
  inference:
    providers:
    - meta-reference
  safety:
    providers:
    - meta-reference
  agents:
    provider_id: meta-reference
    config:
      persistence_store:
        namespace: null
        type: sqlite
        db_path: /root/.llama/runtime/kvstore.db
  memory:
    providers:
    - meta-reference
  telemetry:
    provider_id: meta-reference
    config: {}
routing_table:
  inference:
  - provider_id: meta-reference
    config:
      model: Llama3.2-3B-Instruct
      quantization: null
      torch_seed: null
      max_seq_len: 4096
      max_batch_size: 1
    routing_key: Llama3.2-3B-Instruct
  safety:
  - provider_id: meta-reference
    config:
      llama_guard_shield:
        model: Llama-Guard-3-8B
        excluded_categories: []
        disable_input_check: false
        disable_output_check: false
      prompt_guard_shield:
        model: Prompt-Guard-86M
    routing_key: llama_guard
  - provider_id: meta-reference
    config:
      llama_guard_shield:
        model: Llama-Guard-3-8B
        excluded_categories: []
        disable_input_check: false
        disable_output_check: false
      prompt_guard_shield:
        model: Prompt-Guard-86M
    routing_key: code_scanner_guard
  - provider_id: meta-reference
    config:
      llama_guard_shield:
        model: Llama-Guard-3-8B
        excluded_categories: []
        disable_input_check: false
        disable_output_check: false
      prompt_guard_shield:
        model: Prompt-Guard-86M
    routing_key: injection_shield
  - provider_id: meta-reference
    config:
      llama_guard_shield:
        model: Llama-Guard-3-8B
        excluded_categories: []
        disable_input_check: false
        disable_output_check: false
      prompt_guard_shield:
        model: Prompt-Guard-86M
    routing_key: jailbreak_shield
  memory:
  - provider_id: meta-reference
    config: {}
    routing_key: vector

$ llama model download --source meta --model-id Llama3.2-3B-Instruct

Please provide the signed URL you received via email (e.g., https://llama3-1.llamameta.net/*?Policy...):

$ llama model download --source meta --model-id Llama-Guard-3-8B
$ llama model download --source meta --model-id Prompt-Guard-86M

$ llama stack run my-local-stack

AssertionError: Could not find checkpoint dir: /root/.llama/checkpoints/Llama3.2-3B-Instruct/original.Please download model using `llama download Llama3.2-3B-Instruct`
++ error_handler 55
++ echo 'Error occurred in script at line: 55'
Error occurred in script at line: 55
++ exit 1

$ ls -lt ~/.llama/checkpoints/Llama3.2-3B-Instruct
合計 6277136
-rw-rw-r-- 1 kun432 kun432 6425585114  9月 27 20:00 consolidated.00.pth
-rw-rw-r-- 1 kun432 kun432        220  9月 27 19:48 params.json
-rw-rw-r-- 1 kun432 kun432    2183982  9月 27 19:48 tokenizer.model
-rw-rw-r-- 1 kun432 kun432        209  9月 27 19:48 checklist.chk

$ llama stack run /home/kun432/.llama/builds/docker/my-local-stack-run.yaml

$ export LLAMA_CHECKPOINT_DIR=~/.llama

$ llama stack run my-local-stack

+ docker run -it -p 5000:5000 -v /home/kun432/.llama/builds/docker/my-local-stack-run.yaml:/app/config.yaml -v /home/kun432/.llama:/root/.llama --gpus=all llamastack-my-local-stack python -m llama_stack.distribution.server.server --yaml_config /app/config.yaml --port 5000

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB. GPU 0 has a total capacity of 23.41 GiB of which 44.44 MiB is free

$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS                                       NAMES
ab7d9fbd1f08   llamastack-my-local-stack    "python -m llama_sta…"   7 minutes ago   Up 7 minutes   0.0.0.0:5000->5000/tcp, :::5000->5000/tcp   vibrant_yonath
(snip)

$ docker ps | grep "my-local-stack" | awk '{ print $1 }' | xargs docker stop
$ docker ps -a | grep "my-local-stack" | awk '{ print $1 }' | xargs docker rm

  Value error, Invalid model: Llama-Guard-3-1B:int4-mp1. Must be one of ['Llama-Guard-3-11B-Vision', 'Llama-Guard-3-8B', 'Llama-Guard-3-8B:int8-mp1'] [type=value_error, input_value='Llama-Guard-3-1B:int4-mp1', input_type=str]

$ llama model download --source meta --model-id  Llama-Guard-3-8B:int8-mp1

$ llama stack configure llamastack-my-local-stack

Could not find llamastack-my-local-stack. Trying conda build name instead...
Could not find llamastack-my-local-stack. Trying docker image name instead...
+ docker run -it -v /home/kun432/.llama/builds/docker:/app/builds llamastack-my-local-stack llama stack configure ./llamastack-build.yaml --output-dir /app/builds
Configuration already exists at `/app/builds/my-local-stack-run.yaml`. Will overwrite...
Configuring API `inference`...
=== Configuring provider `meta-reference` for API inference...
Enter value for model (default: Llama3.1-8B-Instruct) (required): Llama3.2-3B-Instruct
Do you want to configure quantization? (y/n): n
Enter value for torch_seed (optional):
Enter value for max_seq_len (default: 4096) (required):
Enter value for max_batch_size (default: 1) (required):

Configuring API `safety`...
=== Configuring provider `meta-reference` for API safety...
Do you want to configure llama_guard_shield? (y/n): y
Entering sub-configuration for llama_guard_shield:
Enter value for model (default: Llama-Guard-3-8B) (required): Llama-Guard-3-8B:int8-mp1
Enter value for excluded_categories (default: []) (required):
Enter value for disable_input_check (default: False) (required):
Enter value for disable_output_check (default: False) (required):
Do you want to configure prompt_guard_shield? (y/n): n

Configuring API `agents`...
=== Configuring provider `meta-reference` for API agents...
Enter `type` for persistence_store (options: redis, sqlite, postgres) (default: sqlite):

Configuring SqliteKVStoreConfig:
Enter value for namespace (optional):
Enter value for db_path (existing: /root/.llama/runtime/kvstore.db) (required):

Configuring API `memory`...
=== Configuring provider `meta-reference` for API memory...
> Please enter the supported memory bank type your provider has for memory: vect
or

Configuring API `telemetry`...
=== Configuring provider `meta-reference` for API telemetry...

> YAML configuration has been written to `/app/builds/my-local-stack-run.yaml`.
You can now run `llama stack run my-local-stack --port PORT`
YAML configuration has been written to /home/kun432/.llama/builds/docker/my-local-stack-run.yaml. You can now run `llama stack run /home/kun432/.llama/builds/docker/my-local-stack-run.yaml`

$ llama stack run my-local-stack --disable-ipv6

Serving GET /healthcheck
Serving GET /shields/get
Serving GET /shields/list
Serving GET /shields/get
Serving GET /shields/list
Serving GET /memory_banks/get
Serving GET /memory_banks/list
Serving POST /memory/create
Serving DELETE /memory/documents/delete
Serving DELETE /memory/drop
Serving GET /memory/documents/get
Serving GET /memory/get
Serving POST /memory/insert
Serving GET /memory/list
Serving POST /memory/query
Serving POST /memory/update
Serving POST /safety/run_shield
Serving POST /agents/create
Serving POST /agents/session/create
Serving POST /agents/turn/create
Serving POST /agents/delete
Serving POST /agents/session/delete
Serving POST /agents/session/get
Serving POST /agents/step/get
Serving POST /agents/turn/get
Serving GET /models/get
Serving GET /models/list
Serving POST /inference/chat_completion
Serving POST /inference/completion
Serving POST /inference/embeddings
Serving GET /models/get
Serving GET /models/list
Serving GET /memory_banks/get
Serving GET /memory_banks/list
Listening on 0.0.0.0:5000
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)

$ python -m llama_stack.apis.inference.client localhost 5000

User>hello world, write me a 2 sentence poem about the moon
{"error": {"message": "400: Invalid value: Could not find provider for Llama3.1-8B-Instruct"}}

INFO:     172.17.0.1:34618 - "POST /inference/chat_completion HTTP/1.1" 200 OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/server/server.py", line 231, in sse_generator
    async for item in event_gen:
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/routers/routers.py", line 117, in chat_completion
    async for chunk in self.routing_table.get_provider_impl(model).chat_completion(
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/routers/routing_tables.py", line 38, in get_provider_impl
    raise ValueError(f"Could not find provider for {routing_key}")
ValueError: Could not find provider for Llama3.1-8B-Instruct

$ python -m llama_stack.apis.inference.client localhost 5000
User>hello world, write me a 2 sentence poem about the moon
Assistant> Here's a 2-sentence poem about the moon:

The moon glows softly in the midnight sky,
A beacon of wonder, as it passes by.

$ python -m llama_stack.apis.safety.client localhost 5000
User>hello world, write me a 2 sentence poem about the moon
Error: HTTP 400 {"detail":"Invalid value: Trying to set a tensor of shape torch.Size([128256, 4096]) in \"weight\" (which has shape torch.Size([32000, 4096])), this looks incorrect."}