Zenn
🧪

最適化したプロンプトを使って Azure OpenAI Service でドキュメントからテストコードを生成する #2

2023/04/12に公開
Azure
Python
unittest
TDD
OpenAI
tech

PromptGenerator メソッドによって最適化されたプロンプトを使って Azure OpenAI Service で OpenAPI と PlantUML からテストコードを生成する #2

前回、Azure OpenAI Service を使って、まず ChatGPT にプロンプトを生成させる PromptGenerator メソッドを用いてプロンプトのチューニングを行い、そのプロンプトを用いて前々回同様のアプローチでテストコードを生成し、その内容を評価しました。

前回の大きな問題としては、テストダブルをテスト対象にまで使用していることでした。本記事ではこの解決を試みます。

ゴール

  • PromptGenerator メソッドを用いて、前回のプロンプトに代わる最適化されたプロンプトを生成する。
  • 前項のプロンプトを使ってドキュメントからテストコードを生成し、その内容を評価する。
  • その際に、テスト対象にテストダブルを用いないようにする。

免責

  • 本記事の内容は、執筆時点のものです。LLM の変化やゆらぎもあるため、再現性は保証されません。
  • 本記事の内容は検証レベルのものです。完全な手法に関する情報を提供するものではありません。
  • 本記事で使用する PlantUML は、細部まで作り込んでいるわけではありません。細かい部分で間違いがある可能性があります。

TL;DR

GPT-3 を使うのをやめて、GPT-4 を使うことで、この問題を高い確率で解決することができます。

ストラテジー

  1. PromptGenerator メソッドを使って、要件を追加する
  2. 前項のプロンプトを箇条書きにする
  3. Few-shot として追加のインストラクションを行う
  4. GPT-4 を使う

1. PromptGenerator メソッドを使って、要件を追加する

ゼロから PromptGenerator を行ったため、前回とは異なる質問が生成されており、前回の内容をそのまま適用できるわけではありませんでした。そのため、都度質問に回答していきます。
なお、このような再現性の問題を軽減または解消するため、今回は Temperature を 0 に変更しました。
初回プロンプトは前回と同様です。
最終的に 6 回のやり取りを行うことになりました。

生成されたプロンプト #6

a) Revised Prompt:
Generate unit test code in Python using the unittest framework and unittest.mock's patch module for given text materials such as OpenAI Specifications (JSON or YAML), PlantUML sequence diagrams (PlantUML format), and PlantUML class diagrams (PlantUML format), following the PEP 8 coding standards. Aim for more than 90% code coverage, with 100% being preferred. Consider edge cases and error handling from the given materials step-by-step, and follow the AAA (Arrange-Act-Assert) style without including comments in the test code. Generate test cases and prepare test data step-by-step, allowing the user to specify test targets/objectives based on the actors of the sequence diagrams. Base test cases on class diagrams first, then sequence diagrams, and finally OpenAPI specifications for API servers. Define immutable test data in the setUp method of the test class that inherits unittest.TestCase, and specify additional test data in test cases if needed. Use the @patch('...') decorator for mocking instead of with patch('...') as mock_foo: statement. Apply the spy test double pattern using the assert_called_once_with method of patch.object when applicable, and use the with patch.object('...') as mock_foo: statement in spy test double pattern cases. Be mindful of token limitations and avoid exceeding them when generating test code. Never apply test doubles like stub, mock, and the rest for the test targets/objectives. Ensure that the test code is MECE (mutually exclusive and collectively exhaustive), structured, organized, and clean.

b) Suggestions:

  • Provide test case scenarios when using the revised prompt in another session
  • Ensure proper handling of exceptions and edge cases based on the given materials
  • Validate the test code against the desired code coverage
  • Follow the specified guidelines for test data, mocking, test double patterns, and coding style

c) Questions:

  1. Are you satisfied with the revised prompt, or do you have any further changes or additions you'd like to make?

上記の Revised Prompt を使って、テストコード生成のためのプロンプトを作って実行します。

テストコード生成結果

5 回生成しましたが、結果としては、いずれにおいても patch がテスト対象に用いられていました。

結果1
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    """Test class for PetStore API"""

    def setUp(self):
        """Set up immutable test data"""
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01 00:00:00'
        self.created_at = '2020-01-01 00:00:00'
        self.updated_at = '2020-01-01 00:00:00'
        self.pet_response = {
            'id': 1,
            'name': 'Fido',
            'type': 'dog',
            'tag': 'pet'
        }
        self.error_response = {
            'code': 500,
            'message': 'unexpected error'
        }

    def test_find_pets(self):
        """Test findPets API"""
        with patch('petstore.findPets') as mock_find_pets:
            mock_find_pets.return_value = self.pet_response
            response = petstore.findPets()
            self.assertEqual(response, self.pet_response)
            mock_find_pets.assert_called_once_with()

    def test_find_pet_by_id(self):
        """Test findPetById API"""
        with patch('petstore.findPetById') as mock_find_pet_by_id:
            mock_find_pet_by_id.return_value = self.pet_response
            response = petstore.findPetById(self.pet_id)
            self.assertEqual(response, self.pet_response)
            mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    def test_add_pet_success(self):
        """Test addPet API success"""
        with patch('petstore.addPet') as mock_add_pet:
            with patch.object(petstore, 'db') as mock_db:
                mock_add_pet.return_value = self.pet_response
                response = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                self.assertEqual(response, self.pet_response)
                mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.transaction.assert_called_once_with()
                mock_db.INSERT.assert_called_once_with('orders', self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.DELETE.assert_called_once_with('pets', self.pet_id)
                mock_db.commit.assert_called_once_with()

    def test_add_pet_failure(self):
        """Test addPet API failure"""
        with patch('petstore.addPet') as mock_add_pet:
            with patch.object(petstore, 'db') as mock_db:
                mock_add_pet.return_value = self.error_response
                response = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                self.assertEqual(response, self.error_response)
                mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.transaction.assert_called_once_with()
                mock_db.INSERT.assert_called_once_with('orders', self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.DELETE.assert_not_called()
                mock_db.rollback.assert_called_once_with()

    def test_delete_pet(self):
        """Test deletePet API"""
        with patch('petstore.deletePet') as mock_delete_pet:
            with patch.object(petstore, 'db') as mock_db:
                mock_delete_pet.return_value = None
                response = petstore.deletePet(self.user_id, self.pet_id)
                self.assertEqual(response, None)
                mock_delete_pet.assert_called_once_with(self.user_id, self.pet_id)
                mock_db.DELETE.assert_called_once_with('pets', self.pet_id)
                mock_db.DELETE.assert_called_once_with('orders', self.user_id, self.pet_id)
結果2
mport unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-08-08T12:00:00Z'
        self.created_at = '2020-08-08T12:00:00Z'
        self.updated_at = '2020-08-08T12:00:00Z'
        self.tags = ['cat', 'dog']
        self.limit = 10
        self.pet = {
            'name': 'test_pet',
            'tag': 'test_tag'
        }
        self.pet_response = [
            {
                'id': 1,
                'name': 'test_pet',
                'type': 'test_type',
                'tag': 'test_tag'
            }
        ]
        self.error_model = {
            'code': 500,
            'message': 'unexpected error'
        }

    def test_find_pets(self):
        with patch('petstore.findPets') as mock_find_pets:
            mock_find_pets.return_value = self.pet_response
            response = petstore.findPets(tags=self.tags, limit=self.limit)
            self.assertEqual(response, self.pet_response)
            mock_find_pets.assert_called_once_with(tags=self.tags, limit=self.limit)

    def test_find_pet_by_id(self):
        with patch('petstore.findPetById') as mock_find_pet_by_id:
            mock_find_pet_by_id.return_value = self.pet_response
            response = petstore.findPetById(pet_id=self.pet_id)
            self.assertEqual(response, self.pet_response)
            mock_find_pet_by_id.assert_called_once_with(pet_id=self.pet_id)

    def test_add_pet(self):
        with patch('petstore.addPet') as mock_add_pet:
            mock_add_pet.return_value = self.pet_response
            response = petstore.addPet(pet=self.pet)
            self.assertEqual(response, self.pet_response)
            mock_add_pet.assert_called_once_with(pet=self.pet)

    def test_delete_pet(self):
        with patch('petstore.deletePet') as mock_delete_pet:
            mock_delete_pet.return_value = None
            response = petstore.deletePet(pet_id=self.pet_id)
            self.assertIsNone(response)
            mock_delete_pet.assert_called_once_with(pet_id=self.pet_id)

    def test_add_pet_failure(self):
        with patch('petstore.addPet') as mock_add_pet:
            mock_add_pet.return_value = self.error_model
            response = petstore.addPet(pet=self.pet)
            self.assertEqual(response, self.error_model)
            mock_add_pet.assert_called_once_with(pet=self.pet)
結果3
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01T00:00:00'
        self.created_at = '2020-01-01T00:00:00'
        self.updated_at = '2020-01-01T00:00:00'
        self.pet_list = [
            {
                'id': 1,
                'name': 'pet1',
                'type': 'dog',
                'tag': 'cute'
            },
            {
                'id': 2,
                'name': 'pet2',
                'type': 'cat',
                'tag': 'playful'
            }
        ]
        self.pet_detail = {
            'id': 1,
            'name': 'pet1',
            'type': 'dog',
            'tag': 'cute'
        }
        self.pet_data = {
            'name': 'pet3',
            'tag': 'lazy'
        }
        self.pet_response = {
            'id': 3,
            'name': 'pet3',
            'type': 'dog',
            'tag': 'lazy'
        }
        self.error_response = {
            'code': 500,
            'message': 'unexpected error'
        }

    @patch('petstore.findPets')
    def test_find_pets(self, mock_find_pets):
        mock_find_pets.return_value = self.pet_list
        response = petstore.findPets()
        self.assertEqual(response, self.pet_list)
        mock_find_pets.assert_called_once_with()

    @patch('petstore.findPetById')
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        mock_find_pet_by_id.return_value = self.pet_detail
        response = petstore.findPetById(self.pet_id)
        self.assertEqual(response, self.pet_detail)
        mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    @patch('petstore.addPet')
    def test_add_pet(self, mock_add_pet):
        mock_add_pet.return_value = self.pet_response
        response = petstore.addPet(self.pet_data)
        self.assertEqual(response, self.pet_response)
        mock_add_pet.assert_called_once_with(self.pet_data)

    @patch('petstore.deletePet')
    def test_delete_pet(self, mock_delete_pet):
        mock_delete_pet.return_value = None
        response = petstore.deletePet(self.pet_id)
        self.assertIsNone(response)
        mock_delete_pet.assert_called_once_with(self.pet_id)

    @patch('petstore.addPet')
    def test_add_pet_failure(self, mock_add_pet):
        mock_add_pet.return_value = self.error_response
        response = petstore.addPet(self.pet_data)
        self.assertEqual(response, self.error_response)
        mock_add_pet.assert_called_once_with(self.pet_data)
結果4
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01 00:00:00'
        self.created_at = '2020-01-01 00:00:00'
        self.updated_at = '2020-01-01 00:00:00'
        self.pet_name = 'dog'
        self.pet_type = 'dog'
        self.pet_tag = 'dog'
        self.pet_list = [{'id': 1, 'name': 'dog', 'type': 'dog', 'tag': 'dog'}]
        self.pet_response = {'id': 1, 'name': 'dog', 'type': 'dog', 'tag': 'dog'}
        self.error_response = {'code': 500, 'message': 'unexpected error'}

    @patch('petstore.findPets')
    def test_find_pets(self, mock_find_pets):
        mock_find_pets.return_value = self.pet_list
        result = petstore.findPets()
        self.assertEqual(result, self.pet_list)
        mock_find_pets.assert_called_once_with()

    @patch('petstore.findPetById')
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        mock_find_pet_by_id.return_value = self.pet_response
        result = petstore.findPetById(self.pet_id)
        self.assertEqual(result, self.pet_response)
        mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    @patch('petstore.addPet')
    def test_add_pet(self, mock_add_pet):
        mock_add_pet.return_value = self.pet_response
        result = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertEqual(result, self.pet_response)
        mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)

    @patch('petstore.deletePet')
    def test_delete_pet(self, mock_delete_pet):
        mock_delete_pet.return_value = None
        result = petstore.deletePet(self.pet_id)
        self.assertIsNone(result)
        mock_delete_pet.assert_called_once_with(self.pet_id)

    @patch('petstore.addPet')
    def test_add_pet_failure(self, mock_add_pet):
        mock_add_pet.return_value = self.error_response
        result = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertEqual(result, self.error_response)
        mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
結果5
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01T00:00:00Z'
        self.created_at = '2020-01-01T00:00:00Z'
        self.updated_at = '2020-01-01T00:00:00Z'
        self.pet_list = [
            {
                'id': 1,
                'name': 'dog',
                'type': 'dog',
                'tag': 'dog'
            },
            {
                'id': 2,
                'name': 'cat',
                'type': 'cat',
                'tag': 'cat'
            }
        ]
        self.pet_detail = {
            'id': 1,
            'name': 'dog',
            'type': 'dog',
            'tag': 'dog'
        }
        self.new_pet = {
            'name': 'rabbit',
            'type': 'rabbit',
            'tag': 'rabbit'
        }
        self.error_model = {
            'code': 500,
            'message': 'unexpected error'
        }

    def test_find_pets(self):
        with patch('petstore.api.findPets') as mock_find_pets:
            mock_find_pets.return_value = self.pet_list
            response = mock_find_pets()
            self.assertEqual(response, self.pet_list)
            mock_find_pets.assert_called_once_with()

    def test_find_pet_by_id(self):
        with patch('petstore.api.findPetById') as mock_find_pet_by_id:
            mock_find_pet_by_id.return_value = self.pet_detail
            response = mock_find_pet_by_id(self.pet_id)
            self.assertEqual(response, self.pet_detail)
            mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    def test_add_pet_success(self):
        with patch('petstore.api.addPet') as mock_add_pet:
            with patch.object('petstore.api', 'db') as mock_db:
                mock_add_pet.return_value = self.pet_detail
                response = mock_add_pet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                self.assertEqual(response, self.pet_detail)
                mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.transaction.assert_called_once_with()
                mock_db.INSERT.assert_called_once_with('orders', self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.DELETE.assert_called_once_with('pets', self.pet_id)
                mock_db.commit.assert_called_once_with()

    def test_add_pet_failure(self):
        with patch('petstore.api.addPet') as mock_add_pet:
            with patch.object('petstore.api', 'db') as mock_db:
                mock_add_pet.return_value = self.error_model
                response = mock_add_pet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                self.assertEqual(response, self.error_model)
                mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.transaction.assert_called_once_with()
                mock_db.INSERT.assert_called_once_with('orders', self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
                mock_db.DELETE.assert_not_called()
                mock_db.rollback.assert_called_once_with()

    def test_delete_pet(self):
        with patch('petstore.api.deletePet') as mock_delete_pet:
            with patch.object('petstore.api', 'db') as mock_db:
                mock_delete_pet.return_value = None
                response = mock_delete_pet(self.user_id, self.pet_id)
                self.assertEqual(response, None)
                mock_delete_pet.assert_called_once_with(self.user_id, self.pet_id)
                mock_db.DELETE.assert_called_once_with('pets', self.pet_id)
                mock_db.DELETE.assert_called_once_with('orders', self.user_id, self.pet_id)

Completions call failed:
This model's maximum context length is 4097 tokens, however you requested 4099 tokens (3999 in your prompt; 100 for the completion). Please reduce your prompt; or completion length.

2. 前項のプロンプトを箇条書きにする

Revised Prompt における要件が箇条書きになることで、より伝わりやすくなるかを検証しました。

PromptGenerator のプロンプトに対して、以下のようにフィードバックを行いました。

As my answer for your question 1, I would like you to organize and bulletize the revised prompt.

生成されたプロンプトは以下の通りです。

a) Revised Prompt:
Generate unit test code in Python using the unittest framework and unittest.mock's patch module for given text materials such as OpenAI Specifications (JSON or YAML), PlantUML sequence diagrams (PlantUML format), and PlantUML class diagrams (PlantUML format), following the PEP 8 coding standards.

Key aspects to consider:

  • Aim for more than 90% code coverage, with 100% being preferred.
  • Consider edge cases and error handling from the given materials step-by-step.
  • Follow the AAA (Arrange-Act-Assert) style without including comments in the test code.
  • Generate test cases and prepare test data step-by-step, allowing the user to specify test targets/objectives based on the actors of the sequence diagrams.
  • Base test cases on class diagrams first, then sequence diagrams, and finally OpenAPI specifications for API servers.
  • Define immutable test data in the setUp method of the test class that inherits unittest.TestCase, and specify additional test data in test cases if needed.
  • Use the @patch('...') decorator for mocking instead of with patch('...') as mock_foo: statement.
  • Apply the spy test double pattern using the assert_called_once_with method of patch.object when applicable, and use the with patch.object('...') as mock_foo: statement in spy test double pattern cases.
  • Be mindful of token limitations and avoid exceeding them when generating test code.
  • Never apply test doubles like stub, mock, and the rest for the test targets/objectives.
  • Ensure that the test code is MECE (mutually exclusive and collectively exhaustive), structured, organized, and clean.

b) Suggestions:

  • Provide test case scenarios when using the revised prompt in another session
  • Ensure proper handling of exceptions and edge cases based on the given materials
  • Validate the test code against the desired code coverage
  • Follow the specified guidelines for test data, mocking, test double patterns, and coding style

c) Questions:

  1. Are you satisfied with the revised prompt, or do you have any further changes or additions you'd like to make?

テストコード生成結果

5 回生成しましたが、結果としては、いずれにおいても patch がテスト対象に用いられていました。

ただ、5 回中 4回、テスト対象をテストコードの冒頭で import してくれるようになったのは、小さな前進でした。

結果1
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01 00:00:00'
        self.created_at = '2020-01-01 00:00:00'
        self.updated_at = '2020-01-01 00:00:00'
        self.pet_response = {
            'id': 1,
            'name': 'dog',
            'type': 'dog',
            'tag': 'pet'
        }
        self.error_response = {
            'code': 500,
            'message': 'unexpected error'
        }

    @patch('petstore.findPets')
    def test_find_pets(self, mock_find_pets):
        mock_find_pets.return_value = self.pet_response
        response = petstore.findPets()
        self.assertEqual(response, self.pet_response)
        mock_find_pets.assert_called_once_with()

    @patch('petstore.findPetById')
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        mock_find_pet_by_id.return_value = self.pet_response
        response = petstore.findPetById(self.pet_id)
        self.assertEqual(response, self.pet_response)
        mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    @patch('petstore.addPet')
    def test_add_pet(self, mock_add_pet):
        mock_add_pet.return_value = self.pet_response
        response = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertEqual(response, self.pet_response)
        mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)

    @patch('petstore.deletePet')
    def test_delete_pet(self, mock_delete_pet):
        mock_delete_pet.return_value = None
        response = petstore.deletePet(self.pet_id, self.user_id)
        self.assertEqual(response, None)
        mock_delete_pet.assert_called_once_with(self.pet_id, self.user_id)

    @patch('petstore.addPet')
    def test_add_pet_failure(self, mock_add_pet):
        mock_add_pet.return_value = self.error_response
        response = petstore.addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertEqual(response, self.error_response)
        mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)

if __name__ == '__main__':
    unittest.main()
結果2
import unittest
from unittest.mock import patch

from petstore_api import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    """Test cases for the PetStoreAPI class"""

    def setUp(self):
        """Set up test data"""
        self.user_id = 1
        self.pet_id = 2
        self.datetime = '2020-01-01T00:00:00Z'
        self.created_at = '2020-01-01T00:00:00Z'
        self.updated_at = '2020-01-01T00:00:00Z'
        self.pet_list = [
            {'id': 1, 'name': 'cat', 'type': 'cat', 'tag': 'cute'},
            {'id': 2, 'name': 'dog', 'type': 'dog', 'tag': 'loyal'},
            {'id': 3, 'name': 'fish', 'type': 'fish', 'tag': 'colorful'}
        ]
        self.pet_detail = {'id': 2, 'name': 'dog', 'type': 'dog', 'tag': 'loyal'}
        self.pet_data = {'name': 'rabbit', 'type': 'rabbit', 'tag': 'furry'}
        self.error_model = {'code': 500, 'message': 'Internal Server Error'}

    def test_find_pets(self):
        """Test the findPets method"""
        with patch.object(PetStoreAPI, 'findPets', return_value=self.pet_list) as mock_find_pets:
            pet_store_api = PetStoreAPI()
            response = pet_store_api.findPets()
            mock_find_pets.assert_called_once_with()
            self.assertEqual(response, self.pet_list)

    def test_find_pet_by_id(self):
        """Test the findPetById method"""
        with patch.object(PetStoreAPI, 'findPetById', return_value=self.pet_detail) as mock_find_pet_by_id:
            pet_store_api = PetStoreAPI()
            response = pet_store_api.findPetById(self.pet_id)
            mock_find_pet_by_id.assert_called_once_with(self.pet_id)
            self.assertEqual(response, self.pet_detail)

    def test_add_pet(self):
        """Test the addPet method"""
        with patch.object(PetStoreAPI, 'addPet', return_value=self.pet_data) as mock_add_pet:
            pet_store_api = PetStoreAPI()
            response = pet_store_api.addPet(self.pet_data)
            mock_add_pet.assert_called_once_with(self.pet_data)
            self.assertEqual(response, self.pet_data)

    def test_delete_pet(self):
        """Test the deletePet method"""
        with patch.object(PetStoreAPI, 'deletePet', return_value=None) as mock_delete_pet:
            pet_store_api = PetStoreAPI()
            response = pet_store_api.deletePet(self.pet_id)
            mock_delete_pet.assert_called_once_with(self.pet_id)
            self.assertIsNone(response)

    def test_add_pet_transaction_failure(self):
        """Test the addPet method when transaction fails"""
        with patch.object(PetStoreAPI, 'addPet', return_value=self.error_model) as mock_add_pet:
            pet_store_api = PetStoreAPI()
            response = pet_store_api.addPet(self.pet_data)
            mock_add_pet.assert_called_once_with(self.pet_data)
            self.assertEqual(response, self.error_model)
結果3
import unittest
from unittest.mock import patch

from petstore_api import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-01-01 00:00:00'
        self.created_at = '2020-01-01 00:00:00'
        self.updated_at = '2020-01-01 00:00:00'
        self.pet = {
            'name': 'Fido',
            'type': 'dog',
            'tag': 'friendly'
        }
        self.pet_list = [
            {
                'id': 1,
                'name': 'Fido',
                'type': 'dog',
                'tag': 'friendly'
            },
            {
                'id': 2,
                'name': 'Mittens',
                'type': 'cat',
                'tag': 'playful'
            }
        ]
        self.pet_response = {
            'code': 200,
            'message': 'pet response'
        }
        self.error_response = {
            'code': 500,
            'message': 'unexpected error'
        }

    def test_find_pets(self):
        """Test findPets method of PetStoreAPI"""
        with patch.object(PetStoreAPI, 'findPets') as mock_find_pets:
            mock_find_pets.return_value = self.pet_list
            pet_list = PetStoreAPI().findPets()
            mock_find_pets.assert_called_once_with()
            self.assertEqual(pet_list, self.pet_list)

    def test_find_pet_by_id(self):
        """Test findPetById method of PetStoreAPI"""
        with patch.object(PetStoreAPI, 'findPetById') as mock_find_pet_by_id:
            mock_find_pet_by_id.return_value = self.pet
            pet = PetStoreAPI().findPetById(self.pet_id)
            mock_find_pet_by_id.assert_called_once_with(self.pet_id)
            self.assertEqual(pet, self.pet)

    def test_add_pet(self):
        """Test addPet method of PetStoreAPI"""
        with patch.object(PetStoreAPI, 'addPet') as mock_add_pet:
            mock_add_pet.return_value = self.pet_response
            response = PetStoreAPI().addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
            mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
            self.assertEqual(response, self.pet_response)

    def test_delete_pet(self):
        """Test deletePet method of PetStoreAPI"""
        with patch.object(PetStoreAPI, 'deletePet') as mock_delete_pet:
            mock_delete_pet.return_value = self.pet_response
            response = PetStoreAPI().deletePet(self.user_id, self.pet_id)
            mock_delete_pet.assert_called_once_with(self.user_id, self.pet_id)
            self.assertEqual(response, self.pet_response)

    def test_add_pet_failure(self):
        """Test addPet method of PetStoreAPI with failure"""
        with patch.object(PetStoreAPI, 'addPet') as mock_add_pet:
            mock_add_pet.return_value = self.error_response
            response = PetStoreAPI().addPet(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
            mock_add_pet.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
            self.assertEqual(response, self.error_response)
結果4
import unittest
from unittest.mock import patch

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.user_id = 1
        self.pet_id = 1
        self.datetime = '2020-10-01'
        self.created_at = '2020-10-01'
        self.updated_at = '2020-10-01'
        self.tags = ['cat', 'dog']
        self.limit = 10
        self.pet = {
            'name': 'Tom',
            'tag': 'cat'
        }

    @patch('petstore.api.findPets')
    def test_find_pets(self, mock_find_pets):
        mock_find_pets.return_value = 200, 'pet response'
        status_code, response = petstore.api.findPets(self.tags, self.limit)
        self.assertEqual(status_code, 200)
        self.assertEqual(response, 'pet response')
        mock_find_pets.assert_called_once_with(self.tags, self.limit)

    @patch('petstore.api.findPetById')
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        mock_find_pet_by_id.return_value = 200, 'pet response'
        status_code, response = petstore.api.findPetById(self.pet_id)
        self.assertEqual(status_code, 200)
        self.assertEqual(response, 'pet response')
        mock_find_pet_by_id.assert_called_once_with(self.pet_id)

    @patch('petstore.api.addPet')
    def test_add_pet(self, mock_add_pet):
        mock_add_pet.return_value = 200, 'pet response'
        status_code, response = petstore.api.addPet(self.pet)
        self.assertEqual(status_code, 200)
        self.assertEqual(response, 'pet response')
        mock_add_pet.assert_called_once_with(self.pet)

    @patch('petstore.api.deletePet')
    def test_delete_pet(self, mock_delete_pet):
        mock_delete_pet.return_value = 204, 'pet deleted'
        status_code, response = petstore.api.deletePet(self.pet_id)
        self.assertEqual(status_code, 204)
        self.assertEqual(response, 'pet deleted')
        mock_delete_pet.assert_called_once_with(self.pet_id)

    @patch('petstore.api.addPet')
    @patch('petstore.db.transaction')
    @patch('petstore.db.INSERT')
    @patch('petstore.db.DELETE')
    @patch('petstore.db.commit')
    @patch('petstore.db.rollback')
    def test_add_pet_transaction_failure(self, mock_rollback, mock_commit, mock_delete, mock_insert, mock_transaction, mock_add_pet):
        mock_add_pet.return_value = 500, 'unexpected error'
        mock_transaction.return_value = None
        mock_insert.return_value = None
        mock_delete.return_value = None
        mock_commit.return_value = None
        mock_rollback.return_value = None
        status_code, response = petstore.api.addPet(self.pet)
        self.assertEqual(status_code, 500)
        self.assertEqual(response, 'unexpected error')
        mock_add_pet.assert_called_once_with(self.pet)
        mock_transaction.assert_called_once()
        mock_insert.assert_called_once_with('orders', self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        mock_delete.assert_called_once_with('pets', self.pet_id)
        mock_commit.assert_not_called()
        mock_rollback.assert_called_once()
結果5
import unittest
from unittest.mock import patch

from petstore_api import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    """Test cases for the PetStoreAPI class"""

    def setUp(self):
        self.pet_store_api = PetStoreAPI()
        self.user_id = 1
        self.pet_id = 2
        self.datetime = '2020-01-01T00:00:00Z'
        self.created_at = '2020-01-01T00:00:00Z'
        self.updated_at = '2020-01-01T00:00:00Z'

    @patch('petstore_api.PetStoreAPI.findPets')
    def test_find_pets(self, mock_find_pets):
        """Test the findPets method"""
        # Arrange
        tags = ['dog', 'cat']
        limit = 10

        # Act
        self.pet_store_api.findPets(tags, limit)

        # Assert
        mock_find_pets.assert_called_once_with(tags, limit)

    @patch('petstore_api.PetStoreAPI.findPetById')
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        """Test the findPetById method"""
        # Arrange
        pet_id = 1

        # Act
        self.pet_store_api.findPetById(pet_id)

        # Assert
        mock_find_pet_by_id.assert_called_once_with(pet_id)

    @patch('petstore_api.PetStoreAPI.addPet')
    def test_add_pet(self, mock_add_pet):
        """Test the addPet method"""
        # Arrange
        pet = {
            'name': 'Fido',
            'tag': 'dog'
        }

        # Act
        self.pet_store_api.addPet(pet)

        # Assert
        mock_add_pet.assert_called_once_with(pet)

    @patch('petstore_api.PetStoreAPI.deletePet')
    def test_delete_pet(self, mock_delete_pet):
        """Test the deletePet method"""
        # Arrange
        pet_id = 1

        # Act
        self.pet_store_api.deletePet(pet_id)

        # Assert
        mock_delete_pet.assert_called_once_with(pet_id)

    @patch('petstore_api.PetStoreAPI.transaction')
    def test_transaction_success(self, mock_transaction):
        """Test the transaction method with successful transaction"""
        # Arrange
        mock_transaction.return_value = True

        # Act
        result = self.pet_store_api.transaction(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)

        # Assert
        mock_transaction.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertTrue(result)

    @patch('petstore_api.PetStoreAPI.transaction')
    def test_transaction_failure(self, mock_transaction):
        """Test the transaction method with failed transaction"""
        # Arrange
        mock_transaction.return_value = False

        # Act
        result = self.pet_store_api.transaction(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)

        # Assert
        mock_transaction.assert_called_once_with(self.user_id, self.pet_id, self.datetime, self.created_at, self.updated_at)
        self.assertFalse(result)

3. Few-shot として追加のインストラクションを行う

Revised Prompt の後に、以下のインストラクションを追加しました。

## Hint: When to use `patch` of `unittest.mock`:

Given the test target/objective is "Foo" class that has two methods named "bar" and "baz". The class is imported like `from foo import Foo` and instantiated as `self.foo = Foo()` in the `setUp` function.
In `test_bar` test function, you do not mock `self.foo.bar` because it is the target/objective of the test case. You only mock `self.foo.baz` if needed in the test case.
In `test_baz` test function, you do not mock `self.foo.baz` because it is the target/objective of the test case. You only mock `self.foo.bar` if needed in the test case.

テストコード生成結果

以下のようなテストコードを生成するようになってしまいました。

import unittest
from unittest.mock import patch

from foo import Foo

class TestFoo(unittest.TestCase):
    def setUp(self):
        self.foo = Foo()

    @patch('foo.Foo.baz')
    def test_bar(self, mock_baz):
        # Arrange
        expected = 'bar'

        # Act
        actual = self.foo.bar()

        # Assert
        self.assertEqual(expected, actual)
        mock_baz.assert_not_called()

    @patch('foo.Foo.bar')
    def test_baz(self, mock_bar):
        # Arrange
        expected = 'baz'

        # Act
        actual = self.foo.baz()

        # Assert
        self.assertEqual(expected, actual)
        mock_bar.assert_not_called()

if __name__ == '__main__':
    unittest.main()

import unittest
from unittest.mock import patch

from foo import Foo

class TestFoo(unittest.TestCase):
    def setUp(self):
        self.foo = Foo()

    def test_bar(self):
        # Arrange
        expected_result = 'bar'
        expected_params = {'param1': 'value1', 'param2': 'value2'}

        # Act
        result = self.foo.bar(**expected_params)

        # Assert
        self.assertEqual(result, expected_result)

    @patch('foo.Foo.baz')
    def test_baz(self, mock_baz):
        # Arrange
        expected_result = 'baz'
        expected_params = {'param1': 'value1', 'param2': 'value2'}

        # Act
        result = self.foo.baz(**expected_params)

        # Assert
        mock_baz.assert_called_once_with(**expected_params)
        self.assertEqual(result, expected_result)

そこで、試しに以下のインストラクションも追加してみましたが、結果は変わりませんでした。

If you understand when to use `patch` of `unittest.mock`, forget the hint including `Foo`, `bar`, `baz`. Remember the instruction above again.

4. GPT-4 を使う

箇条書きにしたプロンプトを、GPT-4 (gpt-4-32k) で動かしました。

テストコード生成結果

結果は非常に良好でした。5 回中 4 回、 patch を正しい対象に適用してくれました。

気になる点としては、テストケースがこれまでの GPT-3 から明らかに減少していることでした。

結果1
import unittest
from unittest.mock import patch
from pet_store import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.api = PetStoreAPI()
        self.sample_pet = {"id": 1, "name": "Fluffy", "tag": "dog"}
        self.sample_pets = [
            {"id": 1, "name": "Fluffy", "tag": "dog"},
            {"id": 2, "name": "Whiskers", "tag": "cat"},
        ]

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pets(self, mock_query_database):
        mock_query_database.return_value = self.sample_pets
        response = self.api.find_pets()
        self.assertEqual(response, self.sample_pets)
        mock_query_database.assert_called_once_with("SELECT * FROM pets")

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pet_by_id(self, mock_query_database):
        mock_query_database.return_value = self.sample_pet
        response = self.api.find_pet_by_id(1)
        self.assertEqual(response, self.sample_pet)
        mock_query_database.assert_called_once_with("SELECT * FROM pets WHERE id = 1")

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_add_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = self.sample_pet
        response = self.api.add_pet(self.sample_pet)
        self.assertEqual(response, self.sample_pet)
        mock_execute_transaction.assert_called_once_with(
            "INSERT INTO pets (name, tag) VALUES ('Fluffy', 'dog')"
        )

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_delete_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = None
        self.api.delete_pet(1)
        mock_execute_transaction.assert_called_once_with(
            "DELETE FROM pets WHERE id = 1"
        )

if __name__ == "__main__":
    unittest.main()
結果2
import unittest
from unittest.mock import patch
from pet_store import PetStoreAPI, Pet, Order

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.api = PetStoreAPI()
        self.sample_pet = Pet(id=1, name="Fluffy", tag="dog")
        self.sample_pets = [
            Pet(id=1, name="Fluffy", tag="dog"),
            Pet(id=2, name="Whiskers", tag="cat"),
        ]
        self.sample_order = Order(id=1, user_id=1, pet_id=1)

    @patch("pet_store.PetStoreAPI.find_pets")
    def test_find_pets(self, mock_find_pets):
        mock_find_pets.return_value = self.sample_pets
        response = self.api.find_pets()
        self.assertEqual(response, self.sample_pets)
        mock_find_pets.assert_called_once()

    @patch("pet_store.PetStoreAPI.find_pet_by_id")
    def test_find_pet_by_id(self, mock_find_pet_by_id):
        mock_find_pet_by_id.return_value = self.sample_pet
        response = self.api.find_pet_by_id(1)
        self.assertEqual(response, self.sample_pet)
        mock_find_pet_by_id.assert_called_once_with(1)

    @patch("pet_store.PetStoreAPI.add_pet")
    def test_add_pet(self, mock_add_pet):
        mock_add_pet.return_value = self.sample_pet
        response = self.api.add_pet(self.sample_pet)
        self.assertEqual(response, self.sample_pet)
        mock_add_pet.assert_called_once_with(self.sample_pet)

    @patch("pet_store.PetStoreAPI.delete_pet")
    def test_delete_pet(self, mock_delete_pet):
        mock_delete_pet.return_value = None
        self.api.delete_pet(1)
        mock_delete_pet.assert_called_once_with(1)

if __name__ == "__main__":
    unittest.main()
結果3
import unittest
from unittest.mock import patch
from pet_store import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.api = PetStoreAPI()
        self.sample_pet = {"id": 1, "name": "Fluffy", "tag": "dog"}
        self.sample_pets = [
            {"id": 1, "name": "Fluffy", "tag": "dog"},
            {"id": 2, "name": "Whiskers", "tag": "cat"},
        ]

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pets(self, mock_query_database):
        mock_query_database.return_value = self.sample_pets
        response = self.api.find_pets()
        self.assertEqual(response, self.sample_pets)
        mock_query_database.assert_called_once_with("SELECT * FROM pets")

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pet_by_id(self, mock_query_database):
        mock_query_database.return_value = self.sample_pet
        response = self.api.find_pet_by_id(1)
        self.assertEqual(response, self.sample_pet)
        mock_query_database.assert_called_once_with("SELECT * FROM pets WHERE id = 1")

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_add_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = self.sample_pet
        response = self.api.add_pet(self.sample_pet)
        self.assertEqual(response, self.sample_pet)
        mock_execute_transaction.assert_called_once_with(
            "INSERT INTO pets (name, tag) VALUES ('Fluffy', 'dog')"
        )

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_delete_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = None
        self.api.delete_pet(1)
        mock_execute_transaction.assert_called_once_with(
            "DELETE FROM pets WHERE id = 1"
        )

if __name__ == "__main__":
    unittest.main()
結果4
import unittest
from unittest.mock import patch
from pet_store import PetStoreAPI

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.api = PetStoreAPI()
        self.sample_pet = {"id": 1, "name": "Fluffy", "tag": "dog"}
        self.sample_pets = [
            {"id": 1, "name": "Fluffy", "tag": "dog"},
            {"id": 2, "name": "Whiskers", "tag": "cat"},
        ]

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pets(self, mock_query_database):
        mock_query_database.return_value = self.sample_pets
        response = self.api.find_pets()
        self.assertEqual(response, self.sample_pets)
        mock_query_database.assert_called_once_with("SELECT * FROM pets")

    @patch("pet_store.PetStoreAPI._query_database")
    def test_find_pet_by_id(self, mock_query_database):
        mock_query_database.return_value = self.sample_pet
        response = self.api.find_pet_by_id(1)
        self.assertEqual(response, self.sample_pet)
        mock_query_database.assert_called_once_with("SELECT * FROM pets WHERE id = 1")

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_add_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = self.sample_pet
        pet_data = {"name": "Fluffy", "tag": "dog"}
        response = self.api.add_pet(pet_data)
        self.assertEqual(response, self.sample_pet)
        mock_execute_transaction.assert_called_once_with(
            "INSERT INTO pets (name, tag) VALUES ('Fluffy', 'dog')"
        )

    @patch("pet_store.PetStoreAPI._execute_transaction")
    def test_delete_pet(self, mock_execute_transaction):
        mock_execute_transaction.return_value = None
        self.api.delete_pet(1)
        mock_execute_transaction.assert_called_once_with(
            "DELETE FROM pets WHERE id = 1"
        )

if __name__ == "__main__":
    unittest.main()
結果5
import unittest
from unittest.mock import patch, MagicMock
from pet_store import PetStoreAPI, Pet, Order

class TestPetStoreAPI(unittest.TestCase):
    def setUp(self):
        self.api = PetStoreAPI()
        self.sample_pet = Pet(id=1, name="Fluffy", tag="dog")
        self.sample_pets = [
            Pet(id=1, name="Fluffy", tag="dog"),
            Pet(id=2, name="Whiskers", tag="cat"),
        ]
        self.sample_order = Order(id=1, user_id=1, pet_id=1)

    @patch("pet_store.PetStoreAPI.get_pets_from_db")
    def test_find_pets(self, mock_get_pets_from_db):
        mock_get_pets_from_db.return_value = self.sample_pets
        response = self.api.find_pets()
        self.assertEqual(response, self.sample_pets)
        mock_get_pets_from_db.assert_called_once()

    @patch("pet_store.PetStoreAPI.get_pet_by_id_from_db")
    def test_find_pet_by_id(self, mock_get_pet_by_id_from_db):
        mock_get_pet_by_id_from_db.return_value = self.sample_pet
        response = self.api.find_pet_by_id(1)
        self.assertEqual(response, self.sample_pet)
        mock_get_pet_by_id_from_db.assert_called_once_with(1)

    @patch("pet_store.PetStoreAPI.add_pet_to_db")
    def test_add_pet(self, mock_add_pet_to_db):
        mock_add_pet_to_db.return_value = self.sample_pet
        pet_data = {"name": "Fluffy", "tag": "dog"}
        response = self.api.add_pet(pet_data)
        self.assertEqual(response, self.sample_pet)
        mock_add_pet_to_db.assert_called_once_with(pet_data)

    @patch("pet_store.PetStoreAPI.delete_pet_from_db")
    def test_delete_pet(self, mock_delete_pet_from_db):
        mock_delete_pet_from_db.return_value = None
        self.api.delete_pet(1)
        mock_delete_pet_from_db.assert_called_once_with(1)

if __name__ == "__main__":
    unittest.main()

まとめ

テストダブルを正しく使うには、その意義を理解する必要があります。テスト対象そのものをモックすると、それはテストとして機能しません。
この概念を GPT-3 は持っていないようで、形式でしか判断できない様子でした。対応策として Few-shot を用いると、かえってそちらにバイアスがかかってしまい、PromptGenerator メソッドで生成したプロンプトの指示内容がほぼ無視されてしまう程でした。
それに対して、GPT-4 は高い確率で正しくモックの適用ができていて、この問題を乗り越えることができることが分かりました。

よって、テストコード生成には GPT-4 を使うことが事実上必須であると言えます。

Discussion