curl \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"Explain how AI works"}]}]}' \
  -X POST 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent?key=YOUR_API_KEY'

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": "## Understanding AI: A Simplified Breakdown\n\nArtificial intelligence (AI) is a broad field that encompasses a wide range of technologies and techniques. Here's a simplified explanation:\n\n**1. AI's Goal: Mimicking Human Intelligence**\n\nAI aims to create systems that can perform tasks that typically require human intelligence, such as:\n\n* **Learning and adapting:** AI systems can learn from data and improve their performance over time.\n* **Problem-solving:** AI can analyze information and find solutions to complex problems.\n* **Decision-making:** AI systems can make decisions based on data and reasoning.\n* **Understanding and responding to language:** AI can understand and generate natural language, like English or Spanish.\n* **Recognizing patterns and objects:** AI can identify patterns and objects in images, videos, and sounds.\n\n**2. How AI Learns: Machine Learning**\n\nMachine learning is a subset of AI that focuses on enabling systems to learn from data without explicit programming. This is done through algorithms that:\n\n* **Train on data:** These algorithms are fed large datasets of labeled examples to learn patterns and relationships.\n* **Make predictions:** Based on the learned patterns, the algorithms can make predictions or decisions on new, unseen data.\n\n**3. Different Types of Machine Learning:**\n\n* **Supervised learning:** Algorithms learn from labeled data, where each example has a known outcome.\n* **Unsupervised learning:** Algorithms learn from unlabeled data, identifying patterns and structures on their own.\n* **Reinforcement learning:** Algorithms learn through trial and error, receiving feedback and rewards for their actions.\n\n**4. Beyond Machine Learning: Other AI Techniques**\n\n* **Deep learning:** A powerful subset of machine learning that uses artificial neural networks with multiple layers to learn complex patterns from data.\n* **Natural language processing (NLP):** This focuses on enabling computers to understand and generate human language.\n* **Computer vision:** This allows computers to \"see\" and interpret images and videos.\n\n**5. AI Applications:**\n\nAI is used in numerous fields, including:\n\n* **Healthcare:** Diagnosis, treatment planning, drug discovery.\n* **Finance:** Fraud detection, risk assessment, personalized investing.\n* **Transportation:** Self-driving cars, traffic optimization.\n* **E-commerce:** Personalized recommendations, customer service.\n* **Education:** Personalized learning, automated grading.\n\n**Important Note:** AI is a rapidly evolving field. This explanation provides a basic understanding but is not exhaustive. There are many other aspects and nuances to explore.\n\n**In short, AI seeks to emulate human intelligence through data-driven learning and problem-solving techniques. Its applications are constantly expanding, transforming various industries and aspects of our lives.** \n"
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "index": 0,
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE"
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 4,
    "candidatesTokenCount": 562,
    "totalTokenCount": 566
  }
}

rye add google-cloud-storage google-generativeai

export GOOGLE_CLOUD_CREDENTIALS=JSON形式のServiceAccountキーをBase64エンコードした値
export GOOGLE_CLOUD_PROJECT_ID=GoogleCloudのProject名
export GOOGLE_CLOUD_REGION="asia-northeast1"

base64 -i your-service-account-secret.json

import os
import json
import base64
from domain.repository.video_repository_interface import (
    VideoRepositoryInterface,
    AnalysisVideoDto,
    AnalysisVideoResult,
)
from google.cloud import storage
from google.oauth2 import service_account
import vertexai
from vertexai.generative_models import GenerativeModel, Part


class GeminiVideoRepository(VideoRepositoryInterface):
    def __init__(self) -> None:
        encoded_service_account_key = os.getenv("GOOGLE_CLOUD_CREDENTIALS")
        if encoded_service_account_key is None:
            raise Exception("GOOGLE_CLOUD_CREDENTIALS is not set.")

        decoded_service_account_key = base64.b64decode(
            encoded_service_account_key
        ).decode("utf-8")
        service_account_info = json.loads(decoded_service_account_key)

        self.google_cloud_storage_client = storage.Client.from_service_account_info(
            service_account_info
        )

        self.credentials = service_account.Credentials.from_service_account_info(
            # service_account_info の型チェックは難しいので type: ignore で回避している
            service_account_info  # type: ignore
        )

        vertexai.init(
            project=os.getenv("GOOGLE_CLOUD_PROJECT_ID"),
            location=os.getenv("GOOGLE_CLOUD_REGION"),
            credentials=self.credentials,
        )

    async def video_analysis(self, dto: AnalysisVideoDto) -> AnalysisVideoResult:
        model = GenerativeModel(
            "gemini-1.5-flash-001",
        )

        video = Part.from_uri(
            // video/quicktime とハードコードされていますが `video_url` の拡張子から判定すべきです
            mime_type="video/quicktime",
            uri=dto["video_url"],
        )

        contents = [
            video,
            """
            # Instruction
            動画の内容を確認して要約の作成をお願いします。
            
            # 制約条件
            - 以下のJSON形式で返すようにお願いします。
              - {"summary": "動画の要約文章をここに設定"}
                - "summary" には動画の要約文章を設定します。
            - ハルシネーションを起こさないでください。
            """,
        ]

        generation_config = {
            "response_mime_type": "application/json",
        }

        response = await model.generate_content_async(
            contents,
            generation_config=generation_config,
        )

        response_content = response.text

        result: AnalysisVideoResult = json.loads(response_content)

        return result

{'summary': '動画は、飼い主らしき人物が「ねこちゃん、こっちおいで」と呼びかける様子から始まります。ねこちゃんは最初は振り返り、その後飼い主の方へ歩み寄ります。しかし、飼い主はねこちゃんを呼ぶのをやめてしまい、ねこちゃんは少し寂しそうな表情を見せます。最終的にねこちゃんは飼い主の元へ寄り添い、動画は終了します。'}

from typing import Protocol, TypedDict


class AnalysisVideoDto(TypedDict):
    video_url: str


class AnalysisVideoResult(TypedDict):
    summary: str


class VideoRepositoryInterface(Protocol):
    async def video_analysis(self, dto: AnalysisVideoDto) -> AnalysisVideoResult: ...