Zenn
Open8

langfuseとlitellmでLLMモニタリング

Langfuse
LiteLLM
もぶもぶもぶもぶ
  • DeploymentはCloud(Saas), Local, Self-hostedの3種類
    • Localはpostgre db含めdocker-composeで起動
    • Self-hostedはDBとしてAWS, GCP等を利用する手順のよう
    • ひとまず個人開発でお試しする分にはLocalでよさそう
    • Local Machine
もぶもぶもぶもぶ
  • ローカルでのlangfuse起動手順
    • Local Machine
    • 以下のコマンドでlocalhost:3000でlangfuseが起動する
# Clone repository
git clone https://github.com/langfuse/langfuse.git
cd langfuse
 
# Run server and db
docker compose up -d
 
# Apply db migrations
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres DIRECT_URL=postgresql://postgres:postgres@localhost:5432/postgres npx prisma migrate deploy
  • docker-compose.ymlの中身
version: "3.5"

services:
  langfuse-server:
    image: ghcr.io/langfuse/langfuse:latest
    depends_on:
      - db
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/postgres
      - DIRECT_URL=postgresql://postgres:postgres@db:5432/postgres
      - NEXTAUTH_SECRET=mysecret
      - NEXTAUTH_URL=http:localhost:3000
      - TELEMETRY_ENABLED=${TELEMETRY_ENABLED:-true}
      - NEXT_PUBLIC_SIGN_UP_DISABLED=false

  db:
    image: postgres
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - 5432:5432
    volumes:
      - database_data:/var/lib/postgresql/data

volumes:
  database_data:
    driver: local
もぶもぶもぶもぶ

起動するとこんな画面

sign upから適当にアカウント作成

Get started画面からNew project


Settingsのcreate new API KeysでAPIキー発行

もぶもぶもぶもぶ

litellmとlangfuseの連携機能お試し

.envに関連情報記載

LANGFUSE_SECRET_KEY=sk-xxx
LANGFUSE_PUBLIC_KEY=pk-xxx
LANGFUSE_HOST=http://localhost:3000

OPENAI_API_KEY=sk-xxx

litellmからChatGPT APIを叩く

import litellm
from litellm import completion

# load .env file
from dotenv import load_dotenv
load_dotenv()

litellm.success_callback = ["langfuse"] 
litellm.failure_callback = ["langfuse"]
 
# openai call
response = completion(
  model="gpt-4",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ]
)
 
print(response)

ダッシュボートからモニタリングできた！

もぶもぶもぶもぶ

積み残し

  • streaming時のモニタリング
  • APIエラー時のモニタリング
  • function callingのモニタリング
もぶもぶもぶもぶ

streamingのときの記録は残らなかった......

https://github.com/BerriAI/litellm/blob/6bd81346158ea2bd8216495570e3719d0d1aa9dd/litellm/integrations/langfuse.py#L40

litellm側のlangfuse integrationコードを見ると、streamingやfunction callingのモニタリングはできなさそう....

    def log_event(self, kwargs, response_obj, start_time, end_time, print_verbose):
        # Method definition
        from langfuse.model import InitialGeneration, Usage
        try:
            print_verbose(
                f"Langfuse Logging - Enters logging function for model {kwargs}"
            )
            # print(response_obj)
            # print(response_obj['choices'][0]['message']['content'])
            # print(response_obj['usage']['prompt_tokens'])
            # print(response_obj['usage']['completion_tokens'])

            self.Langfuse.generation(InitialGeneration(
                name="litellm-completion",
                startTime=start_time,
                endTime=end_time,
                model=kwargs['model'],
                # modelParameters= kwargs,
                prompt=[kwargs['messages']],
                completion=response_obj['choices'][0]['message']['content'],
                usage=Usage(
                    prompt_tokens=response_obj['usage']['prompt_tokens'],
                    completion_tokens=response_obj['usage']['completion_tokens']
                ),
            ))
            self.Langfuse.flush()

うーん、これだと実用上はつらいな