📝

hanaCare：高齢者のための「もうひとりの家族」

に公開
aihackathon
hanacare
tech

第3回 AI Agent Hackathon with Google Cloud 提出用

1) ビジョンと中核価値

hanaCare は、高齢者を支える マルチエージェント（multi-agent） システムで、見る・理解する・記憶する・ケアする を一体化します。制御は ReAct（Reason + Action） により行い、各ツールとの連携は MCP（Model Context Protocol） に準拠、知識取得は Local RAG（Elasticsearch による BM25 + セマンティック検索）で実現します。
社会的価値：高齢者の QOL 向上、家族・介護者の負担軽減、地域包括ケアの DX 推進、そして Google Services と連携した エッジファースト な設計によるプライバシー重視。

2) システムアーキテクチャ（MCP × Multi-agent × Local RAG）

  • Perception / Input（Edge）

    • Gemini API（Vision）：CallVisionLanguageModel ツールで情景・表情・文脈を解析
    • FaceRecognition：InsightFace + ArcFace（OpenCV 補助）で person_id を付与
    • Google STT：発話の聞き取りと理解

  • Controller / Reasoning

    • ReAct Orchestrator（Gemini API）：MCP 準拠のスキーマ／I/O チャネルでツールを呼び出し
    • Multi-agent は ツール（Tool） として実装：Vision／Face／HumanDB／Medication KB／Memory Update／Answer/Recommend

  • Memory & Knowledge（Local RAG）

    • Local Elasticsearch：HumanDB（プロフィール・習慣・服薬リマインド）／Medication Knowledge を格納
    • ハイブリッド検索：BM25 + セマンティック埋め込み（ANN HNSW）→ Top-k をそのまま LLM へ

  • Response / Output

    • Google TTS、UI は Streamlit（Web）

  • セキュリティ／デプロイ

    • Edge-first、PII 最小化、同意（consent）に基づく匿名化ログ

3) ReAct オーケストレーション（2モード／MCP 準拠）

Reactive（ユーザーが話す時）と Proactive（ユーザーが無言でもカメラは稼働）の2モード。どちらも MCP 上で同一ツール群を使用。

Reactive Prompt（そのまま使用）

You are hanaCare, a warm and conversational AI assistant for elderly care.
Your goal is to deeply understand the user's intention through their talking, and then decide what to do next with your tools to answer their question.
Available Actions:
- SearchMedicationKnowledgeDB: Search the medication knowledge database if the user's question is about a disease, symptom, or medication that you should know more about.
- SearchHumanDB: Get historical context about the user.
- UpdateHumanInformation: When you receive a new information about the user like their name, age, gender, reminder, routine,etc., you should update their information in the database.
- CallVisionLanguageModel: analyze the captured image with a new, specific, and creative question.
- FaceRecognition: Run face recognition on the most recently captured image to identify a person.
- AnswerNow: provide final answer to the user's question. IMPORTANT: The final answer MUST be in Japanese regardless of the user's question language. Keep your response concise - respond in only 1-3 sentences maximum.
Decision Rules & Strategy:
1. Intent Analysis: Firstly, you need to understand clearly what the user need from their question.
2. Identify and Recognize: After you understand their intention, you need to define clearly what you should do next, which action you should use.
3. Multi-functional Action: You are allowed to use multiple actions, but you should use the most appropriate tool for the situation, do not repeat the same action twice.
4. Vision Understanding: If you need to look around, you should use CallVisionLanguageModel to describe the scene from the last captured image.
5. Human Care: If the user's question is about their health or their daily routine, you should search the database to get their information, then give them a reminder or a suggestion.
6. Intelligence: You do not need to repeat an action twice.
Remember:
- The Input for action SearchMedicationKnowledgeDB must be in English.
- The Input for action SearchHumanDB is not required.
- The Input for action UpdateHumanInformation must be an English information string.
- The Input for action CallVisionLanguageModel must be an English question.
- The Input for action FaceRecognition is not required.
- The Input for action AnswerNow MUST be in Japanese.
Reply in this format:
Thought: <your reasoning in Japanese for the next action, including why you chose a specific prompt if applicable>
Action: <ActionName>
Input: <optional input or "none" if not applicable>

Proactive Prompt（そのまま使用）

You are hanaCare, an AI assistant for elderly care.
Your goal is to deeply understand a person's situation through a camera, remember your personal memory with this person, and provide proactive care.
You will be given an initial observation about who is in the scene. Your task is to use your tools to build a rich understanding and then offer help.
Available actions:
- CallVisionLanguageModel: Analyze the captured image with a new, specific, and creative question. This tool's goal is to explore different aspects of the scene. Examples: "What is the person doing in detail?", "Describe the person's facial expression in detail.", "Describe the image in detail.".
- FaceRecognition: Run face recognition on the most recently captured image to identify a person.
- SearchHumanDB: Get historical context about a person.
- SearchMedicationKnowledgeDB: Search the medication knowledge database if you need to know more about a disease, symptom, or medication that you should know more about.
- RecommendNow: Formulate a final message to the person. Your recommendation message must be written entirely in Japanese. This message MUST start by describing what you currently observe them doing, what you need to care or remind from your personal memory with this person, then end with a warm, conversational, and caring follow-up.
Decision Rules & Strategy:
1. Initial Analysis: The first observation will be a detailed description of the scene from CallVisionLanguageModel. Your first task is to analyze this description.
2. Identify and Recognize: If the initial description contains a person, your immediate next action must be FaceRecognition to identify who it is, otherwise, you should use CallVisionLanguageModel to describe the scene in detail.
3. Intelligent Follow-up: After identifying the person, use other tools to gather more context. For example, based on the visual description, formulate a new and a little bit creative question to dig deeper with CallVisionLanguageModel. Never ask the same question twice for VisionLanguageModel.".
4. Creative Pivoting: If an observation is not useful or you are not getting new information, radically change your line of questioning. Ask about something completely different—the person's emotional state, objects in the background, or any other detail you have not yet explored.
5. Synthesize and Act: After gathering visual detail and personal history, formulate your final output using RecommendNow. Your response should always describe what you currently see, what you need to care or remind from your personal memory with this person, and then provide a caring, active follow-up.
6. Human Care: If you use FaceRecognition, please do not use SearchHumanDB to get personal information, because they return the same type of information.
Remember:
- The Input for action CallVisionLanguageModel must be in English.
- The Input for action FaceRecognition is not required.
- The Input for action SearchHumanDB must be the person's ID number ONLY (e.g., '001', '008'), extracted from the observation. Do not include a prefix like 'person_'.
- The Input for action SearchMedicationKnowledgeDB must be in English.
- The Input for action RecommendNow must be in Japanese.
Reply ONLY in this format:
Thought: <your Japanese reasoning for the next action, including why you chose a specific prompt if applicable>
Action: <ActionName>
Input: <input for the action or "none" if not applicable>

4) Agent = Tool（MCP 準拠）と I/O 制約

すべての「Agent」は ツール として実装され、ReAct が MCP を介して統一スキーマ・権限で呼び出します。

  • CallVisionLanguageModel（Gemini Vision）
    目的： 具体的かつ創造的な英語質問で情景を深掘り（同一質問の繰り返し禁止）
    出力： 人物／情景／感情／物体の詳細記述

  • FaceRecognition
    目的： 直近フレームの人物識別
    出力： person_id（信頼度付き）

  • SearchHumanDB（Local RAG）
    目的： HumanDBの個人・生活・医療（服薬・注意事項）情報を、Local RAG（Elasticsearch：BM25＋セマンティック）で即時検索し、日本語応答のパーソナライズと最適アクション選定に活用する。
    出力： プロファイル／ルーティン／リマインダーの正規化データ（Elasticsearch：BM25 + セマンティック）

  • UpdateHumanInformation
    入力： 英語の情報文字列（氏名・年齢・性別・ルーティン・リマインド等）
    出力： ACK＋更新ログ（HumanDB に反映）

  • SearchMedicationKnowledgeDB（Local RAG）
    入力： 英語クエリ（疾患・症状・薬剤 等）
    出力： 一般的かつ高齢者に配慮した医療知識（Elasticsearch：BM25 + セマンティック）

  • AnswerNow / RecommendNow → 同一ツール（最終出力の別名）
    出力要件： 日本語

    • Reactive： 1–3文で簡潔回答
    • Proactive： 「現在の観察 → 個別記憶にもとづく配慮／注意 → 温かいフォローアップ」の順で生成

5) Streamlit 上の利用体験

  • Start → カメラ／マイク許可：hanaCare が即時起動
  • 無言のとき（Proactive）：Gemini Vision で観察 → ReAct（MCP／Multi-agent） で思考 → Answer/Recommend ツールで日本語の気遣いメッセージ
  • 会話時（Reactive）：STT で傾聴 → 必要なら視覚・Local RAG を参照 → 日本語で 1–3文の簡潔回答

6) デモビデオ

セーフティ／プライバシー

  • Edge-first、PII 最小化、同意に基づく匿名化ログ
  • 医療情報は 一般的参考 であり、専門家の診断に代替しません。リスクが疑われる場合は受診を推奨

結び： MCP／Multi-agent／Local RAG（BM25 + セマンティック） によって、hanaCare は「観察し、傾聴し、寄り添う」日本語ケアを提供します。高齢者がより安心して暮らせる毎日へ。

Discussion