LLMのトレーシング・評価ツール「Arize Phoenix」を試してみる

$ pyenv virtualenv 3.10.13 phoenix
$ mkdir phoenix && cd phoenix
$ pyenv local phoenix

$ pip install jupyterlab ipywidgets
$ jupyter-lab --ip='0.0.0.0' --NotebookApp.token=''

$ mkdir phoenix-demo && cd phoenix-demo

$ docker run -p 6006:6006 arizephoenix/phoenix:latest

---------------------------
✅ Migrations complete.


██████╗ ██╗  ██╗ ██████╗ ███████╗███╗   ██╗██╗██╗  ██╗
██╔══██╗██║  ██║██╔═══██╗██╔════╝████╗  ██║██║╚██╗██╔╝
██████╔╝███████║██║   ██║█████╗  ██╔██╗ ██║██║ ╚███╔╝
██╔═══╝ ██╔══██║██║   ██║██╔══╝  ██║╚██╗██║██║ ██╔██╗
██║     ██║  ██║╚██████╔╝███████╗██║ ╚████║██║██╔╝ ██╗
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚══════╝╚═╝  ╚═══╝╚═╝╚═╝  ╚═╝ v4.14.1

|
|  🌎 Join our Community 🌎
|  https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q
|
|  ⭐️ Leave us a Star ⭐️
|  https://github.com/Arize-ai/phoenix
|
|  📚 Documentation 📚
|  https://docs.arize.com/phoenix
|
|  🚀 Phoenix Server 🚀
|  Phoenix UI: http:/0.0.0.0:6006
|  Log traces:
|    - gRPC: http://0.0.0.0:4317
|    - HTTP: http:/0.0.0.0:6006/v1/traces
|  Storage: sqlite:////root/.phoenix/phoenix.db

INFO:     Started server process [1]
INFO:     Waiting for application startup.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1721881468.895446       1 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:6006 (Press CTRL+C to quit)

!pip install openai python-dotenv 'arize-phoenix[evals]'

from dotenv import load_dotenv

load_dotenv(verbose=True)

import os

os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://X.X.X.X:6006"

from openai import OpenAI
from phoenix.trace.openai import OpenAIInstrumentor

# OpenAI用のInstrumentor
OpenAIInstrumentor().instrument()

client = OpenAI()

conversation = [
    {"role": "system", "content": "あなたは親切なアシスタントです。"},
    {"role": "user", "content": "日本の総理大臣は誰？"}
]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=conversation,
)

assistant_reply = response.choices[0].message.content
print(assistant_reply)

!pip install llama-index openinference-instrumentation-llama-index opentelemetry-proto opentelemetry-exporter-otlp opentelemetry-sdk

!pip freeze | egrep -i "llama-|openinference|opentelemetry"

llama-cloud==0.0.11
llama-index==0.10.58
llama-index-agent-openai==0.2.9
llama-index-cli==0.1.13
llama-index-core==0.10.58
llama-index-embeddings-openai==0.1.11
llama-index-indices-managed-llama-cloud==0.2.6
llama-index-legacy==0.9.48
llama-index-llms-openai==0.1.27
llama-index-multi-modal-llms-openai==0.1.8
llama-index-program-openai==0.1.7
llama-index-question-gen-openai==0.1.3
llama-index-readers-file==0.1.30
llama-index-readers-llama-parse==0.1.6
llama-parse==0.4.9
openinference-instrumentation==0.1.8
openinference-instrumentation-llama-index==2.1.1
openinference-semantic-conventions==0.1.9
opentelemetry-api==1.26.0
opentelemetry-exporter-otlp==1.26.0
opentelemetry-exporter-otlp-proto-common==1.26.0
opentelemetry-exporter-otlp-proto-grpc==1.26.0
opentelemetry-exporter-otlp-proto-http==1.26.0
opentelemetry-instrumentation==0.47b0
opentelemetry-proto==1.26.0
opentelemetry-sdk==1.26.0
opentelemetry-semantic-conventions==0.47b0

from pathlib import Path
import requests

# Wikipediaからのデータ読み込み
wiki_titles = ["イクイノックス", "ドウデュース"]
for title in wiki_titles:
    response = requests.get(
        "https://ja.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "extracts",
            # 'exintro': True,
            "explaintext": True,
        },
    ).json()
    page = next(iter(response["query"]["pages"].values()))
    wiki_text = page["extract"]

    data_path = Path("data")
    if not data_path.exists():
        Path.mkdir(data_path)

    with open(data_path / f"{title}.txt", "w") as fp:
        fp.write(wiki_text)

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

phoenix_endpoint = "http://X.X.X.X:6006/v1/traces"
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(phoenix_endpoint)))
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter()
Settings.chunk_size = 400
Settings.chunk_overlap = 100
Settings.context_window = 4096

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents, show_progress=True)
query_engine = index.as_query_engine(similarity_top_k=5)

response = query_engine.query("ドウデュースの主な勝ち鞍は？")
print(response)

import phoenix as px
import os

# トレースデータセットの出力先
directory = '/my_saved_traces'
os.makedirs(directory, exist_ok=True)

# トレースデータセットの保存
trace_id = px.Client().get_trace_dataset().save(directory=directory)

💾 Trace dataset saved to under ID: bf63b044-f45f-4b12-bb55-93f63ccf8433
📂 Trace dataset path: export_traces/trace_dataset-bf63b044-f45f-4b12-bb55-93f63ccf8433.parquet

$ tree export_traces
export_traces
└── trace_dataset-bf63b044-f45f-4b12-bb55-93f63ccf8433.parquet

1 directory, 1 file

import pandas as pd

trace_df = pd.read_parquet("export_traces/trace_dataset-bf63b044-f45f-4b12-bb55-93f63ccf8433.parquet")
px.Client().log_traces(trace_dataset=px.TraceDataset(trace_df))

!pip install datasets

from datasets import load_dataset

dataset = load_dataset("kun432/databricks-dolly-300-ja-for-rag", split="train")
df = dataset.to_pandas()
df

from llama_index.core.schema import TextNode

nodes = []
for idx, row in df.iterrows():
    id = row["index"]
    text = row["input"]
    node = TextNode(text=text, id_=id)
    nodes.append(node)

from llama_index.core import VectorStoreIndex
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

phoenix_endpoint = "http://X.X.X.X:6006/v1/traces"
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(phoenix_endpoint)))
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 8192
Settings.chunk_overlap = 0
Settings.context_window = 4096

index = VectorStoreIndex(nodes, show_progress=True)
query_engine = index.as_query_engine(similarity_top_k=5)

for idx, row in df.iterrows():
    query = row["instruction"]
    response = query_engine.query(query)
    print(f"Q{idx + 1}: {query}")
    print(f"A: {response}")
    print("----")

Q1: ヴァージン・オーストラリア航空はいつから運航を開始したのですか？
A: ヴァージン・オーストラリア航空は2000年8月31日に運航を開始しました。
----
Q2: 小森田友明はいつ生まれたの？
A: 小森田友明は1981年7月10日に生まれました。
----
Q3: カイル・ヴァンジルがチーム61得点のうち36得点を挙げたとき、誰と対戦していたのですか？
A: カイル・ヴァンジルが36得点を挙げたのは、ボランドU21との対戦時です。
----
(snip)
----
Q298: ウィリアム・ギブスンの最初のコミックシリーズの名前は？
A: ウィリアム・ギブスンの最初のコミックシリーズの名前は「Archangel」です。
----
Q299: COVID-19のさまざまな名称と、最も推奨される名称はどれですか？
A: COVID-19のさまざまな名称には、「コロナウイルス」「武漢コロナウイルス」「武漢肺炎」などがあります。最も推奨される名称は「COVID-19」で、これは「コロナウイルス病2019」の略語です。
----
Q300: ブラックシーってどこ？
A: ブラックシーについての情報は提供されていません。質問の内容に関連する情報がないため、具体的な回答はできません。

from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents
import phoenix as px

queries_df = get_qa_with_reference(px.Client())
retrieved_documents_df = get_retrieved_documents(px.Client())

queries_df

retrieved_documents_df

from phoenix.evals import OpenAIModel, HallucinationEvaluator, QAEvaluator, RelevanceEvaluator
from phoenix.evals import run_evals
from phoenix.trace import SpanEvaluations, DocumentEvaluations
import nest_asyncio

# notebookなのでイベントループのネストを有効化
nest_asyncio.apply()

# 評価で使用するLLMの定義
eval_model = OpenAIModel(model="gpt-4o-mini")

# 評価モジュールの定義
hallucination_evaluator = HallucinationEvaluator(eval_model)
qa_evaluator = QAEvaluator(eval_model)
relevance_evaluator = RelevanceEvaluator(eval_model)

# 評価の実行
hallucination_eval_df, qa_eval_df = run_evals(
    dataframe=queries_df,
    evaluators=[hallucination_evaluator, qa_evaluator],
    provide_explanation=True,
)
relevance_evals_df = run_evals(
    dataframe=retrieved_documents_df,
    evaluators=[relevance_evaluator],
    provide_explanation=True,
)[0]

# 評価結果をPhoenixサーバに送信
px.Client().log_evaluations(
    SpanEvaluations(eval_name="Hallucination", dataframe=hallucination_eval_df),
    SpanEvaluations(eval_name="QA Correctness", dataframe=qa_eval_df),
    DocumentEvaluations(eval_name="Relevance", dataframe=relevance_evals_df),
)

from datasets import load_dataset
import pandas as pd

dataset = load_dataset("kun432/databricks-dolly-300-ja-for-rag", split="train")
df = dataset.to_pandas()
# わかりやすいようにカラム名をリネーム
df.rename(columns={"instruction":"question", "input":"context", "output":"answer"}, inplace=True)

df

import phoenix as px

phoenix_client = px.Client()
dataset = phoenix_client.upload_dataset(
    dataframe=df[:50],
    dataset_name="sample-qa-dataset",
    input_keys=["question","context",],
    output_keys=["answer"],
    metadata_keys=["index","category"],
)

📤 Uploading dataset...
💾 Examples uploaded: http://X.X.X.X:6006/datasets/RGF0YXNldDox/examples
🗄️ Dataset version ID: RGF0YXNldFZlcnNpb246MQ==

from openai import OpenAI
from phoenix.experiments.types import Example

openai_client = OpenAI()

task_prompt_template = """\
事前知識を使わずに、以下のコンテキスト情報だけを使用して、質問に「簡潔」に答えてください。

コンテキスト情報: {context}
質問: {question}
回答: \
"""


def task(example: Example) -> str:
    question = example.input["question"]
    context = example.input["context"]
    message_content = task_prompt_template.format(question=question, context=context)
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": message_content}]
    )
    return response.choices[0].message.content

from phoenix.experiments.evaluators import ConcisenessEvaluator
from phoenix.evals.models import OpenAIModel
from phoenix.experiments.evaluators import create_evaluator
from typing import Any, Dict

# ビルトインの評価モジュールの定義
model = OpenAIModel(model="gpt-4o-mini")
# 回答の簡潔さをLLMを使って評価するモジュール
conciseness = ConcisenessEvaluator(model=model)

# カスタムな評価の定義:
# 出力された回答と正解データの回答をLLMに比較させて正確性を判定するカスタムな評価
eval_prompt_template = """
与えられた「質問」と「参考回答」を踏まえて、「回答」の「正確性」を判定してください。
判定結果は、accurate または inaccurate のどちらか1語だけを出力してください。

質問: {question}
参考回答: {reference_answer}
回答: {answer}

正確性 (accurate / inaccurate):
"""


@create_evaluator(kind="llm")  # デコレーターがないと、種類はデフォルトで "code "になる。
def accuracy(input: Dict[str, Any], output: str, expected: Dict[str, Any]) -> float:
    message_content = eval_prompt_template.format(
        # データセットのinputが"input"、outputが"expected"、タスクを実行して得られたレスポンスが"output"として渡される様子（ちょっとややこしい・・・）
        question=input["question"], reference_answer=expected["answer"], answer=output
    )
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": message_content}]
    )
    response_message_content = response.choices[0].message.content.lower().strip()
    return 1.0 if response_message_content == "accurate" else 0.0

from phoenix.experiments import run_experiment

experiment = run_experiment(
    dataset,
    task,
    experiment_name="initial-experiment",
    evaluators=[conciseness, accuracy],
)

🧪 Experiment started.
📺 View dataset experiments: http://X.X.X.X:6006/datasets/RGF0YXNldDox/experiments
🔗 View this experiment: http://X.X.X.X:6006/datasets/RGF0YXNldDox/compare?experimentId=RXhwZXJpbWVudDox
running tasks 
 11/50 (22.0%) | ⏳ 01:07

✅ Task runs completed.
🧠 Evaluation started.
running experiment evaluations 
 14/100 (14.0%) | ⏳ 00:35<03:55 |  2.74s/it

🔗 View this experiment: http://X.X.X.X:6006/datasets/RGF0YXNldDox/compare?experimentId=RXhwZXJpbWVudDox

Experiment Summary (07/26/24 07:09 PM +0900)
--------------------------------------------
     evaluator   n  n_scores  avg_score
0  Conciseness  50        50       0.86
1     accuracy  50        50       0.86

Tasks Summary (07/26/24 07:05 PM +0900)
---------------------------------------
   n_examples  n_runs  n_errors
0          50      50         0