【OpenAI】APIのmax_token数による出力内容の違い

1. はじめに

RAGを使ったチャットボットを作成しているのですが、「回答長いから短くして」と要望を頂いたので出力トークン数を制御しようとしています。そこで、出力トークン数によって回答の内容に差がでるのかを検証します。気になっているのは、出力トークン数を小さくすると、①回答が途中で切られるのか、②いい感じに要約してくれるのか、のどっちになるのかですね。最初はシステムプロンプトで制限していたのですが、たまに無視して長く回答がきてしまいます。

2. 調査

トークン数の調査もかねているので英語で質問します。
OpenAI APIのバージョンは1.28.0です。

messages = [
  {"role": "system", "content": "You are IT consultant"},
  {"role": "user", "content": "Could you tell me how to define requirements for system."}
]

max_tokens = 

response = client.chat.completions.create(
            model="{モデル名}",
            messages=messages,
            temperature=0,
            top_p=0.85,
            stream=False,
            max_tokens=max_tokens
            )

2.1. max_tokens=1000

Defining requirements is a critical step in any project, especially in IT and software development. Clear and well-defined requirements help ensure that the final product meets the needs of stakeholders and users. Here’s a structured approach to defining requirements:

### 1. **Identify Stakeholders**
   - **Who are the stakeholders?** Identify all the individuals and groups who have an interest in the project. This includes end-users, project sponsors, developers, and any other parties affected by the project.
   - **Engage stakeholders:** Conduct interviews, surveys, and meetings to gather their input and understand their needs and expectations.

### 2. **Gather Requirements**
   - **Elicit requirements:** Use various techniques such as interviews, focus groups, workshops, brainstorming sessions, and observation to gather requirements from stakeholders.
   - **Document requirements:** Record the gathered information in a clear and organized manner. This can be done using requirement specification documents, user stories, use cases, or other appropriate formats.

### 3. **Categorize Requirements**
   - **Functional requirements:** Define what the system should do. These are the features and functions that the system must perform.
   - **Non-functional requirements:** Define how the system should perform. These include performance, usability, reliability, security, and other quality attributes.
   - **Business requirements:** Define the high-level business needs and objectives that the project aims to achieve.
   - **Technical requirements:** Define the technical constraints and considerations, such as platforms, technologies, and standards to be used.

### 4. **Analyze and Prioritize Requirements**
   - **Analyze requirements:** Ensure that the requirements are clear, complete, consistent, and feasible. Resolve any conflicts or ambiguities.
   - **Prioritize requirements:** Determine the importance and urgency of each requirement. Use techniques such as MoSCoW (Must have, Should have, Could have, Won't have) prioritization or other prioritization frameworks.

### 5. **Validate Requirements**
   - **Review with stakeholders:** Present the documented requirements to stakeholders for validation. Ensure that the requirements accurately reflect their needs and expectations.
   - **Revise as needed:** Make any necessary revisions based on stakeholder feedback.

### 6. **Manage Requirements**
   - **Maintain traceability:** Establish traceability between requirements and other project artifacts, such as design documents, test cases, and user manuals.
   - **Version control:** Use version control to manage changes to requirements over time.
   - **Change management:** Implement a process for managing changes to requirements, including impact analysis and stakeholder approval.

### 7. **Document Requirements**
   - **Requirement Specification Document (RSD):** Create a comprehensive document that includes all the gathered and validated requirements. This document serves as a reference throughout the project lifecycle.
   - **User Stories/Use Cases:** For agile projects, document requirements in the form of user stories or use cases that describe how users will interact with the system.

### 8. **Communicate Requirements**
   - **Share with the team:** Ensure that all team members have access to the requirements and understand them.
   - **Regular updates:** Keep stakeholders informed of any changes or updates to the requirements throughout the project.

### Tools and Techniques
- **Requirements Management Tools:** Use tools like JIRA, Confluence, Trello, or Microsoft Azure DevOps to manage and track requirements.
- **Modeling Techniques:** Use diagrams such as UML (Unified Modeling Language), flowcharts, and wireframes to visually represent requirements.
- **Prototyping:** Create prototypes or mockups to help stakeholders visualize the final product and refine requirements.

By following these steps, you can ensure that the requirements are well-defined, understood, and agreed upon by all stakeholders, leading to a successful project outcome.

要件定義について8ステップで回答がきました。

2.2. max_tokens=100

Defining requirements is a critical step in any project, whether it's software development, business process improvement, or any other type of initiative. Here are the steps to effectively define requirements:

1. **Identify Stakeholders:**
   - Determine who will be affected by the project and who has an interest in its outcome. This includes end-users, project sponsors, managers, and other relevant parties.

2. **Gather Information:**
   - Conduct interviews, surveys, and workshops with stakeholders to understand their needs

max_tokens=1000と比較すると内容が要約されているというよりは途中で切れていそうですね。
要件定義について2ステップまでしか回答がなく、2.1.の最初の2ステップと比較しても内容が似通っている気がします。

2.3. max_tokens=100かつシステムプロンプトで単語数を制限

max_tokensだと要約にはならなさそうなので、システムプロンプトで単語数を制限してみます。

messages = [
  {"role": "system", "content": "You are IT consultant. Answer the question in 100 words or less."},
  {"role": "user", "content": "Could you tell me how to define requirements for system."}
]

Defining requirements for a system involves several key steps. First, engage stakeholders to gather their needs and expectations through interviews, surveys, and workshops.
Next, document these needs clearly and concisely, distinguishing between functional requirements (what the system should do) and non-functional requirements (how the system should perform).
Use techniques like use cases, user stories, and process diagrams to visualize requirements.
Prioritize requirements based on stakeholder input and feasibility. Validate and review the requirements with stakeholders to ensure accuracy and completeness

要約はできていそうです。最後ピリオドがないのがきになりますが。。

3. まとめ

回答の長さの制御には、「max_tokens指定＋システムプロンプトで文字数制限を命令」、がいいのではないかと思いました。