brew install livekit-cli

brew info livekit-cli

==> livekit-cli: stable 2.3.1 (bottled), HEAD
Command-line interface to LiveKit
https://livekit.io
Installed
/opt/homebrew/Cellar/livekit-cli/2.3.1 (11 files, 45.8MB) *
  Poured from bottle using the formulae.brew.sh API on 2025-01-20 at 12:13:16
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/l/livekit-cli.rb
License: Apache-2.0

lk cloud auth

┃ What is the name of this device?
┃ > macmini

Device: macmini
Requesting verification token...
Please confirm access by visiting:

   https://cloud.livekit.io/cli/confirm-auth?t=XXXXXX
   
⢿ Awaiting confirmation...

┃ Make this project default?        [Yes]     No

Saved CLI config to /Users/kun432/.livekit/cli-config.yaml

default_project: test
projects:
    - name: test
      url: wss://test-xxxxxx.livekit.cloud
      api_key: xxxxxxxxxx
      api_secret: xxxxxxxxxxx

mkdir livekit-agents-work && cd livekit-agents-work

lk app create --template voice-pipeline-agent-python

┃ Application Name
┃ > my-app

┃ Enter OPENAI_API_KEY?
┃ > <To use other providers, press Enter for now and edit .env.local>

┃ Enter DEEPGRAM_API_KEY?
┃ > <To use other providers, press Enter for now and edit .env.local>

Cloning template...
Instantiating environment...
Cleaning up...
To setup and run the agent:

	cd /Users/kun432/work/livekit-agents-work/my-app
	python3 -m venv venv
	source venv/bin/activate
	pip install -r requirements.txt
	python3 agent.py dev

cd my-app
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

DEEPGRAM_API_KEY="<To use other providers, press Enter for now and edit .env.local>"
LIVEKIT_API_KEY="XXXXXXXXXX"
LIVEKIT_API_SECRET="XXXXXXXXXX"
LIVEKIT_URL="wss://test-XXXXXXXX.livekit.cloud"
OPENAI_API_KEY="XXXXXXXXXX"

import logging

from dotenv import load_dotenv
from livekit.agents import (
    AutoSubscribe,
    JobContext,
    JobProcess,
    WorkerOptions,
    cli,
    llm,
)
from livekit.agents.pipeline import VoicePipelineAgent
from livekit.plugins import openai, deepgram, silero


load_dotenv(dotenv_path=".env.local")
logger = logging.getLogger("voice-agent")


def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext):
    initial_ctx = llm.ChatContext().append(
        role="system",
        text=(
            "You are a voice assistant created by LiveKit. Your interface with users will be voice. "
            "You should use short and concise responses, and avoiding usage of unpronouncable punctuation. "
            "You were created as a demo to showcase the capabilities of LiveKit's agents framework."
        ),
    )

    logger.info(f"connecting to room {ctx.room.name}")
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    # Wait for the first participant to connect
    participant = await ctx.wait_for_participant()
    logger.info(f"starting voice assistant for participant {participant.identity}")

    # This project is configured to use Deepgram STT, OpenAI LLM and TTS plugins
    # Other great providers exist like Cartesia and ElevenLabs
    # Learn more and pick the best one for your app:
    # https://docs.livekit.io/agents/plugins
    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=deepgram.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

    agent.start(ctx.room, participant)

    # The agent should be polite and greet the user when it joins :)
    await agent.say("Hey, how can I help you today?", allow_interruptions=True)


if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        ),
    )

    initial_ctx = llm.ChatContext().append(
        role="system",
        text=(
            "あなたはLiveKit によって作成された音声アシスタントです。ユーザーとのインターフェースは音声になります。"
            "あなたは、短い簡潔な応答を使用し、発音できない句読点の使用は避けるべきです。"
            "あなたは、LiveKit のエージェントフレームワークの能力を紹介するデモとして作成されました。"
        ),
    )

    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=openai.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

    # エージェントは、ユーザーが参加した際に、礼儀正しく挨拶をするべきです。
    await agent.say("こんにちは、今日はどのようなご要件ですか？", allow_interruptions=True)

python agent.py dev

2025-01-20 12:46:07,059 - DEBUG asyncio - Using selector: KqueueSelector
2025-01-20 12:46:07,062 - DEV  livekit.agents - Watching /Users/kun432/work/livekit-agents-work/my-app
2025-01-20 12:46:07,554 - DEBUG asyncio - Using selector: KqueueSelector
2025-01-20 12:46:07,557 - INFO livekit.agents - starting worker {"version": "0.12.8", "rtc-version": "0.19.1"}
2025-01-20 12:46:08,264 - INFO livekit.agents - registered worker {"id": "AW_hkPbMWnMGp5C", "region": "Japan", "protocol": 15, "node_id": "NC_OTOKYO1B_p5THAfbv9dyd"}

cd livekit-agents-work

lk app create --template voice-assistant-frontend

┃ Application Name
┃ > frontend

Cloning template...
Instantiating environment...
Cleaning up...
To setup and run the frontend:

	cd /Users/kun432/work/livekit-agents-work/frontend
	pnpm install
	pnpm dev

cd frontend
mise use node@22
npm install -g pnpm
pnpm install
pnpm dev

> voice-assistant2@0.1.0 dev /Users/kun432/work/livekit-agents-work/frontend
> next dev

  ▲ Next.js 14.2.23
  - Local:        http://localhost:3000
  - Environments: .env.local

 ✓ Starting...
 ✓ Ready in 2.1s

from livekit.plugins import google
(snip)
    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        #stt=openai.STT(),
        stt=google.STT(languages="ja-JP"),
        llm=openai.LLM(model="gpt-4o-mini"),
        #tts=openai.TTS(),
        tts=google.TTS(language="ja-JP", voice_name="ja-JP-Standard-A", speaking_rate=1.2),
        chat_ctx=initial_ctx,
    )

    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=deepgram.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=deepgram.STT(),
        llm=openai.LLM(),
        tts=openai.TTS(),
        fnc_ctx=fnc_ctx,
        chat_ctx=initial_chat_ctx,
    )

class AssistantFnc(llm.FunctionContext):
    """
    音声アシスタントが実行できるLLM関数のセットを定義する。
    """

    @llm.ai_callable()
    async def get_weather(
        self,
        location: Annotated[
            str, llm.TypeInfo(description="天気を取得する場所")
        ],
    ):
        """ユーザーが天気について質問したときに呼び出され、指定された場所の天気情報を返す"""
        # 場所の文字列から特殊文字を削除
        location = re.sub(r"[^a-zA-Z0-9]+", " ", location).strip()

        # Function Calling実行中の場合、ユーザーに対して時間がかかることを通知するためのオプションがいくつかある
        # オプション1: Function Callingをトリガーした直後に.sayでフィラーメッセージを使用する
        # オプション2: Function Calling中に、エージェントにテキスト応答を返すように指示する
        agent = AgentCallContext.get_current().agent

        if (
            not agent.chat_ctx.messages
            or agent.chat_ctx.messages[-1].role != "assistant"
        ):
            # エージェントがすでに発話中の場合はスキップ
            filler_messages = [
                " {location} の天気を確認します。",
                " {location} の天気がどうなっているかをすぐに確認します。",
                # LLMにプリフィルでこの文章を補完させて、チャットコンテキストの最後に追加する
                "{location} の現在の天気は、",
            ]
            message = random.choice(filler_messages).format(location=location)
            logger.info(f"フィラーメッセージを発話: {message}")

            # NOTE: add_to_chat_ctx=True は、Function Callingのチャットコンテキストの末尾にメッセージを追加する
            speech_handle = await agent.say(message, add_to_chat_ctx=True)  # noqa: F841

        logger.info(f"天気を確認: {location}")
        url = f"https://wttr.in/{urllib.parse.quote(location)}?format=%C+%t"
        weather_data = ""
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                if response.status == 200:
                    # Function Callingの結果はLLMに返される
                    weather_data = (
                        f"{location} の天気は {await response.text()}."
                    )
                    logger.info(f"天気データ: {weather_data}")
                else:
                    raise Exception(
                        f"天気データを取得できませんでした, ステータスコード: {response.status}"
                    )

        # (オプション) Function Callingの結果を返す前に音声が終了するまで待つ
        # await speech_handle.join()
        return weather_data

fnc_ctx = AssistantFnc()

# エージェントに関数コンテキストを渡す
pipeline_agent = VoicePipelineAgent(
    ...
    fnc_ctx=fnc_ctx,
)

# マルチモーダルエージェントの場合も同じ
multimodal_agent = MultimodalAgent(
    ...
    fnc_ctx=fnc_ctx,
)

	(snip)
	# LlamaIndexのRAGインデックスを使って、会話型エンジンを生成
	chat_engine = index.as_chat_engine(chat_mode=ChatMode.CONTEXT)

    assistant = VoicePipelineAgent(
        vad=silero.VAD.load(),
        stt=deepgram.STT(),
        # LLMモジュールはLlamaIndexの会話型エンジンを使う
        llm=llama_index.LLM(chat_engine=chat_engine),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )
	(snip)

curl -sSL https://get.livekit.io | bash

brew install livekit

livekit-server --dev --bind 0.0.0.0

2025-01-20T16:10:09.708+0900	INFO	livekit	server/main.go:211	no keys provided, using placeholder keys	{"API Key": "devkey", "API Secret": "secret"}

2025-01-20T16:10:09.739+0900	INFO	livekit	service/server.go:258	starting LiveKit server	{"portHttp": 7880, "nodeID": "ND_yubH4Y536aZG", "nodeIP": "192.168.XX.XX", "version": "1.8.3", "bindAddresses": ["0.0.0.0"], "rtc.portTCP": 7881, "rtc.portUDP": {"Start":7882,"End":0}}

LIVEKIT_API_KEY="devkey"
LIVEKIT_API_SECRET="secret"
LIVEKIT_URL="ws://127.0.0.1:7880"

brew upgrade livekit livekit-cli

livekit-server -version

livekit-server version 1.8.4

lk --version

lk version 2.4.5

uv init -p 3.12.9 livekit-1.0-work && cd livekit-1.0-work

uv add  "livekit-agents[openai,silero,turn-detector]~=1.0" \
  "livekit-plugins-noise-cancellation~=0.2" \
  "python-dotenv"

(snip)
 + livekit==1.0.5
 + livekit-agents==1.0.11
 + livekit-api==1.0.2
 + livekit-plugins-noise-cancellation==0.2.1
 + livekit-plugins-openai==1.0.11
 + livekit-plugins-silero==1.0.11
 + livekit-plugins-turn-detector==1.0.11
 + livekit-protocol==1.0.1
(snip)
 + onnxruntime==1.21.0
(snip)

uv add "onnxruntime==1.20.1"

 - onnxruntime==1.21.0
 + onnxruntime==1.20.1

OPENAI_API_KEY=XXXXXXXXXX
LIVEKIT_API_KEY="devkey"
LIVEKIT_API_SECRET="secret"
LIVEKIT_URL="ws://127.0.0.1:7880"

from dotenv import load_dotenv

from livekit import agents
from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import (
    openai,
    noise_cancellation,
    silero,
)
from livekit.plugins.turn_detector.multilingual import MultilingualModel

load_dotenv()


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="あなたは親切な日本語のAI音声アシスタントです。")


async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        stt=openai.STT(model="whisper-1", language="ja"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(model="tts-1", voice="coral"),
        vad=silero.VAD.load(),
        turn_detection=MultilingualModel(),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        # ノイズキャンセルはLiveKitCloudを使用する場合のみ
        #room_input_options=RoomInputOptions(
        #    noise_cancellation=noise_cancellation.BVC(),
        #),
    )

    await session.generate_reply(
        instructions="ユーザーに挨拶し、支援を申し出てください。"
    )


if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

uv run main.py download-files

2025-04-11 23:35:12,485 - INFO livekit.agents - Downloading files for <livekit.plugins.openai.OpenAIPlugin object at 0x112e9a390> 
2025-04-11 23:35:12,485 - INFO livekit.agents - Finished downloading files for <livekit.plugins.openai.OpenAIPlugin object at 0x112e9a390> 
2025-04-11 23:35:12,485 - INFO livekit.agents - Downloading files for <livekit.plugins.silero.SileroPlugin object at 0x113dccf50> 
2025-04-11 23:35:12,485 - INFO livekit.agents - Finished downloading files for <livekit.plugins.silero.SileroPlugin object at 0x113dccf50> 
2025-04-11 23:35:12,486 - INFO livekit.agents - Downloading files for <livekit.plugins.turn_detector.EOUPlugin object at 0x114b2d3a0> 
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2025-04-11 23:35:15,476 - DEBUG urllib3.connectionpool - Starting new HTTPS connection (1): huggingface.co:443 
2025-04-11 23:35:15,979 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v1.2.2-en/tokenizer_config.json HTTP/1.1" 200 0 
2025-04-11 23:35:16,462 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v1.2.2-en/onnx/model_q8.onnx HTTP/1.1" 302 0 
2025-04-11 23:35:16,647 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v1.2.2-en/languages.json HTTP/1.1" 200 0 
2025-04-11 23:35:17,147 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v0.1.0-intl/tokenizer_config.json HTTP/1.1" 200 0 
2025-04-11 23:35:17,627 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v0.1.0-intl/onnx/model_q8.onnx HTTP/1.1" 302 0 
2025-04-11 23:35:17,814 - DEBUG urllib3.connectionpool - https://huggingface.co:443 "HEAD /livekit/turn-detector/resolve/v0.1.0-intl/languages.json HTTP/1.1" 200 0 
2025-04-11 23:35:17,817 - INFO livekit.agents - Finished downloading files for <livekit.plugins.turn_detector.EOUPlugin object at 0x114b2d3a0> 

uv add torch

 + torch==2.6.0

uv run main.py console