cover-agent を使ってみる

$ rye tools install cover-agent --git https://github.com/Codium-ai/cover-agent.git

$ git clone https://github.com/Codium-ai/cover-agent
$ cd cover-agent
$ rye init -r requirements.txt
# テスト実行
$ rye test

$ cover-agent \
  --source-file-path "app.py" \
  --test-file-path "test_app.py" \
  --code-coverage-report-path "coverage.xml" \
  --test-command "rye test -- --cov=. --cov-report=xml --cov-report=term" \
  --test-command-dir "." \
  --coverage-type "cobertura" \
  --desired-coverage 70 \
  --max-iterations 10

2024-06-10 19:30:46,995 - cover_agent.UnitTestGenerator - INFO - Running build/test command to generate coverage report: "rye test -- --cov=. --cov-report=xml --cov-report=term"
Streaming results from LLM model...
```yaml
language: python
testing_framework: pytest
number_of_tests: 2
test_headers_indentation: 0
```

Streaming results from LLM model...
```yaml
language: python
testing_framework: pytest
number_of_tests: 2
relevant_line_number_to_insert_after: 22
```

2024-06-10 19:30:51,850 - cover_agent.CoverAgent - INFO - Current Coverage: 62.79%
2024-06-10 19:30:51,850 - cover_agent.CoverAgent - INFO - Desired Coverage: 70%
Streaming results from LLM model...
```yaml
language: python
existing_test_function_signature: |
  def test_root():
new_tests:
- test_behavior: |
    Test the add endpoint by sending a GET request to "/add/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_add
  test_code: |
    def test_add():
        response = client.get("/add/3/4")
        assert response.status_code == 200
        assert response.json() == {"result": 7}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the subtract endpoint by sending a GET request to "/subtract/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_subtract
  test_code: |
    def test_subtract():
        response = client.get("/subtract/10/4")
        assert response.status_code == 200
        assert response.json() == {"result": 6}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the multiply endpoint by sending a GET request to "/multiply/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_multiply
  test_code: |
    def test_multiply():
        response = client.get("/multiply/3/4")
        assert response.status_code == 200
        assert response.json() == {"result": 12}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the divide endpoint by sending a GET request to "/divide/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_divide
  test_code: |
    def test_divide():
        response = client.get("/divide/8/2")
        assert response.status_code == 200
        assert response.json() == {"result": 4.0}
  new_imports_code: |
    ""
  test_tags: happy path
```

2024-06-10 19:31:03,382 - cover_agent.UnitTestGenerator - INFO - Total token used count for LLM model gpt-4o: 2422
2024-06-10 19:31:03,385 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:04,992 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 65.12%
2024-06-10 19:31:04,992 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:06,607 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 67.44%
2024-06-10 19:31:06,607 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:08,219 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 69.77%
2024-06-10 19:31:08,220 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:09,828 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 74.42%
2024-06-10 19:31:09,828 - cover_agent.CoverAgent - INFO - Reached above target coverage of 70% (Current Coverage: 74.42%) in 1 iterations.
python-fastapi

from fastapi import FastAPI, HTTPException
from datetime import date

app = FastAPI()


@app.get("/")
async def root():
    """
    A simple function that serves as the root endpoint for the FastAPI application.
    No parameters are passed into the function.
    Returns a dictionary with a welcome message.
    """
    return {"message": "Welcome to the FastAPI application!"}


@app.get("/current-date")
async def current_date():
    """
    Get the current date as an ISO-formatted string.
    """
    return {"date": date.today().isoformat()}


@app.get("/add/{num1}/{num2}")
async def add(num1: int, num2: int):
    """
    An asynchronous function that takes two integer parameters 'num1' and 'num2', and returns a dictionary containing the result of adding 'num1' and 'num2' under the key 'result'.
    """
    return {"result": num1 + num2}


@app.get("/subtract/{num1}/{num2}")
async def subtract(num1: int, num2: int):
    """
    A function that subtracts two numbers and returns the result as a dictionary.

    Parameters:
        num1 (int): The first number to be subtracted.
        num2 (int): The second number to subtract from the first.

    Returns:
        dict: A dictionary containing the result of the subtraction.
    """
    return {"result": num1 - num2}


@app.get("/multiply/{num1}/{num2}")
async def multiply(num1: int, num2: int):
    """
    Multiply two numbers and return the result as a dictionary.

    Parameters:
    - num1 (int): The first number to be multiplied.
    - num2 (int): The second number to be multiplied.

    Returns:
    - dict: A dictionary containing the result of the multiplication.
    """
    return {"result": num1 * num2}


@app.get("/divide/{num1}/{num2}")
async def divide(num1: int, num2: int):
    """
    An asynchronous function that handles a GET request to divide two numbers.
    Parameters:
    - num1: an integer representing the numerator
    - num2: an integer representing the denominator
    Returns:
    - A dictionary containing the result of the division
    Raises:
    - HTTPException with status code 400 if num2 is 0
    """
    if num2 == 0:
        raise HTTPException(status_code=400, detail="Cannot divide by zero")
    return {"result": num1 / num2}


@app.get("/square/{number}")
async def square(number: int):
    """
    Return the square of a number.
    """
    return {"result": number**2}


@app.get("/sqrt/{number}")
async def sqrt(number: float):
    """
    Return the square root of a number. Returns an error for negative numbers.
    """
    if number < 0:
        raise HTTPException(
            status_code=400, detail="Cannot take square root of a negative number"
        )
    return {"result": math.sqrt(number)}


@app.get("/is-palindrome/{text}")
async def is_palindrome(text: str):
    """
    Check if a string is a palindrome.
    """
    return {"is_palindrome": text == text[::-1]}


@app.get("/days-until-new-year")
async def days_until_new_year():
    """
    Calculates the number of days until the next New Year.
    """
    today = date.today()
    next_new_year = date(today.year + 1, 1, 1)
    delta = next_new_year - today
    return {"days_until_new_year": delta.days}


@app.get("/echo/{message}")
async def echo(message: str):
    """
    Returns the same message that is sent to it.
    """
    return {"message": message}

from datetime import date


from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_root():
    """
    Test the root endpoint by sending a GET request to "/" and checking the response status code and JSON body.
    """
    response = client.get("/")
    assert response.status_code == 200
    assert response.json() == {"message": "Welcome to the FastAPI application!"}


def test_current_date():
    response = client.get("/current-date")
    assert response.status_code == 200
    assert "date" in response.json()
    assert response.json()["date"] == date.today().isoformat()

from datetime import date

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_root():
    """
    Test the root endpoint by sending a GET request to "/" and checking the response status code and JSON body.
    """
    response = client.get("/")
    assert response.status_code == 200
    assert response.json() == {"message": "Welcome to the FastAPI application!"}


def test_current_date():
    response = client.get("/current-date")
    assert response.status_code == 200
    assert "date" in response.json()
    assert response.json()["date"] == date.today().isoformat()


def test_divide():
    response = client.get("/divide/8/2")
    assert response.status_code == 200
    assert response.json() == {"result": 4.0}


def test_multiply():
    response = client.get("/multiply/3/4")
    assert response.status_code == 200
    assert response.json() == {"result": 12}


def test_subtract():
    response = client.get("/subtract/10/4")
    assert response.status_code == 200
    assert response.json() == {"result": 6}


def test_add():
    response = client.get("/add/3/4")
    assert response.status_code == 200
    assert response.json() == {"result": 7}

2024-06-10 19:30:51,850 - cover_agent.CoverAgent - INFO - Current Coverage: 62.79%
2024-06-10 19:30:51,850 - cover_agent.CoverAgent - INFO - Desired Coverage: 70%

language: python
existing_test_function_signature: |
  def test_root():
new_tests:
- test_behavior: |
    Test the add endpoint by sending a GET request to "/add/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_add
  test_code: |
    def test_add():
        response = client.get("/add/3/4")
        assert response.status_code == 200
        assert response.json() == {"result": 7}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the subtract endpoint by sending a GET request to "/subtract/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_subtract
  test_code: |
    def test_subtract():
        response = client.get("/subtract/10/4")
        assert response.status_code == 200
        assert response.json() == {"result": 6}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the multiply endpoint by sending a GET request to "/multiply/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_multiply
  test_code: |
    def test_multiply():
        response = client.get("/multiply/3/4")
        assert response.status_code == 200
        assert response.json() == {"result": 12}
  new_imports_code: |
    ""
  test_tags: happy path

- test_behavior: |
    Test the divide endpoint by sending a GET request to "/divide/{num1}/{num2}" and checking the response status code and JSON body.
  test_name: |
    test_divide
  test_code: |
    def test_divide():
        response = client.get("/divide/8/2")
        assert response.status_code == 200
        assert response.json() == {"result": 4.0}
  new_imports_code: |
    ""
  test_tags: happy path

2024-06-10 19:31:03,382 - cover_agent.UnitTestGenerator - INFO - Total token used count for LLM model gpt-4o: 2422
2024-06-10 19:31:03,385 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:04,992 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 65.12%
2024-06-10 19:31:04,992 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:06,607 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 67.44%
2024-06-10 19:31:06,607 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:08,219 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 69.77%
2024-06-10 19:31:08,220 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "rye test -- --cov=. --cov-report=xml --cov-report=term"
2024-06-10 19:31:09,828 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 74.42%
2024-06-10 19:31:09,828 - cover_agent.CoverAgent - INFO - Reached above target coverage of 70% (Current Coverage: 74.42%) in 1 iterations.
python-fastapi

user="""\
## Overview
You are a code assistant that accepts a {{ language }} source file, and a {{ language }} test file.
Your goal is to generate additional unit tests to complement the existing test suite, in order to increase the code coverage against the source file.

Additional guidelines:
- Carefully analyze the provided code. Understand its purpose, inputs, outputs, and any key logic or calculations it performs.
- Brainstorm a list of test cases you think will be necessary to fully validate the correctness of the code and achieve 100% code coverage.
- After each individual test has been added, review all tests to ensure they cover the full range of scenarios, including how to handle exceptions or errors.
- If the original test file contains a test suite, assume that each generated test will be a part of the same suite. Ensure that the new tests are consistent with the existing test suite in terms of style, naming conventions, and structure.

## Source File
Here is the source file that you will be writing tests against, called `{{ source_file_name }}`.
Note that we have manually added line numbers for each line of code, to help you understand the code coverage report.
Those numbers are not a part of the original code.
=========
{{ source_file_numbered|trim }}
=========


## Test File
Here is the file that contains the existing tests, called `{{ test_file_name }}`.
=========
{{ test_file| trim }}
=========


{%- if additional_includes_section|trim %}


{{ additional_includes_section|trim }}
{% endif %}


{%- if failed_tests_section|trim  %}

{{ failed_tests_section|trim }}

{% endif %}

{%- if additional_instructions_text|trim  %}

{{ additional_instructions_text|trim }}
{% endif %}


## Code Coverage
The following is the existing code coverage report. Use this to determine what tests to write, as you should only write tests that increase the overall coverage:
=========
{{ code_coverage_report|trim }}
=========


## Response
The output must be a YAML object equivalent to type $NewTests, according to the following Pydantic definitions:
=====
class SingleTest(BaseModel):
    test_behavior: str = Field(description="Short description of the behavior the test covers")
{%- if language in ["python","java"] %}
    lines_to_cover: str = Field(description="A list of line numbers, currently uncovered, that this specific new test aims to cover")
    test_name: str = Field(description=" A short test name, in snake case, that reflects the behaviour to test")
{%- else %}
    test_name: str = Field(description=" A short unique test name, that should reflect the test objective")
{%- endif %}
    test_code: str = Field(description="A single test function, that tests the behavior described in 'test_behavior'. The test should be a written like its a part of the existing test suite, if there is one, and it can use existing helper functions, setup, or teardown code.")
    new_imports_code: str = Field(description="New imports that are required to run the new test function, and are not already imported in the test file. Give an empty string if no new imports are required. If relevant, add new imports as  'import ...' lines.")
    test_tags: str = Field(description="A single label that best describes the test, out of: ['happy path', 'edge case','other']")

class NewTests(BaseModel):
    language: str = Field(description="The programming language of the source code")
    existing_test_function_signature: str = Field(description="A single line repeating a signature header of one of the existing test functions")
    new_tests: List[SingleTest] = Field(min_items=1, max_items={{ max_tests }}, description="A list of new test functions to append to the existing test suite, aiming to increase the code coverage. Each test should run as-is, without requiring any additional inputs or setup code. Don't introduce new dependencies")
=====

Example output:
```yaml
language: {{ language }}
existing_test_function_signature: |
  ...
new_tests:
- test_behavior: |
    Test that the function returns the correct output for a single element list
{%- if language in ["python","java"] %}
  lines_to_cover: |
    [1,2,5, ...]
  test_name: |
    test_single_element_list
{%- else %}
  test_name: |
    ...
{%- endif %}
  test_code: |
{%- if language in ["python"] %}
    def ...
{%- else %}
    ...
{%- endif %}
  new_imports_code: |
    ""
  test_tags: happy path
    ...
```

Use block scalar('|') to format each YAML output.


{%- if additional_instructions_text|trim  %}

{{ additional_instructions_text|trim }}
{% endif %}


Response (should be a valid YAML, and nothing else):
```yaml
"""

MAX_TESTS_PER_RUN = 4

# Markdown text used as conditional appends
ADDITIONAL_INCLUDES_TEXT = """
## Additional Includes
The following is a set of included files used as context for the source code above. This is usually included libraries needed as context to write better tests:
======
{included_files}
======
"""

ADDITIONAL_INSTRUCTIONS_TEXT = """
## Additional Instructions
======
{additional_instructions}
======
"""

FAILED_TESTS_TEXT = """
## Previous Iterations Failed Tests
Below is a list of failed tests that you generated in previous iterations. Do not generate the same tests again, and take the failed tests into account when generating new tests.
======
{failed_test_runs}
======
"""

あなたは、{{ language }} のソースファイルと {{ language }} のテストファイルを受け付けるコードアシスタントです。
あなたの目標は、ソースファイルに対するコードカバレッジを増やすために、既存のテストスイートを補完する追加のユニットテストを生成することです。

追加のガイドライン:

提供されたコードを注意深く分析してください。その目的、入力、出力、および実行する重要なロジックや計算を理解してください。
コードの正確性を完全に検証し、100%のコードカバレッジを達成するために必要と思われるテストケースのリストを考案してください。
個々のテストを追加した後、すべてのテストを見直し、例外やエラーの処理方法を含む全範囲のシナリオをカバーしていることを確認してください。
元のテストファイルにテストスイートが含まれている場合、生成された各テストが同じスイートの一部になると想定してください。新しいテストが既存のテストスイートのスタイル、命名規則、構造に一致することを確認してください。

以下は、あなたがテストを書く対象となるソースファイル {{ source_file_name }} です。
コードカバレッジレポートを理解するために、手動で各コード行に行番号を追加しています。これらの番号は元のコードの一部ではありません。

以下は、既存のテストを含むファイル {{ test_file_name }} です。

出力は次のPydantic定義に従って、$NewTests 型と同等のYAMLオブジェクトでなければなりません:

以下は既存のコードカバレッジレポートです。これを使用して書くべきテストを決定してください。全体のカバレッジを増やすテストのみを書くようにしてください。

language: {{ language }}
existing_test_function_signature: |
  ...
new_tests:
- test_behavior: |
    一つの要素のリストに対して関数が正しい出力を返すことをテストする
{%- if language in ["python","java"] %}
  lines_to_cover: |
    [1,2,5, ...]
  test_name: |
    test_single_element_list
{%- else %}
  test_name: |
    ...
{%- endif %}
  test_code: |
{%- if language in ["python"] %}
    def ...
{%- else %}
    ...
{%- endif %}
  new_imports_code: |
    ""
  test_tags: happy path
    ...