概要
Phi-3がGPT-3.5に匹敵する記事を見ました。
ただ、実際のところどれぐらい出力に差があるのか、気になったので検証してみます。
比較方法
下記の３つのプロンプトで確認する。プロンプトは、英語で行う。
- Few-shot Learning
- CoT（Chain of Thought）
- Zero-shot CoT（≒ step by step）
参考記事
検証方法
Azure AI studio上で検証を行う。
Few-shot Learning
Regarding the following job:
This is a job that involves conducting tests that need to be performed manually for the portion of the project that was managed by a senior employee at an SIer (System Integration and Engineering) company. Specialized knowledge is not particularly required, and the necessary information for the tests will be provided in advance.
Is this job:
(A) Suitable for new graduates (B) Requiring more specialized experience
Answer: (A) Suitable for new graduates
Regarding the following job:
This is a role in which the consultant engages in consultative sales, explaining and persuading the management team of the client about the importance of the company's own system for the client's business and obtaining approval for system implementation in the board of directors.
Is this job:
(A) Suitable for new graduates (B) Requiring more specialized experience
Answer:
Phi-3
(B) Requiring more specialized experience
The job involving consultative sales and persuading management teams
about system implementation typically requires a strong
understanding of the industry, the ability to analyze business needs,
and experience in sales and business development.
While a new graduate could potentially start in an entry-level position
and gain experience over time,
this particular role would likely benefit from someone
with more specialized experience in the field.
GPT-3.5
(B) Requiring more specialized experience
CoT（Chain of Thought）
Below are the translations:
Regarding the following job: This is a job that involves conducting tests that need to be performed manually for the portion of the project that was managed by a senior employee at an SIer (System Integration and Engineering) company. Specialized knowledge is not particularly required, and the necessary information for the tests will be provided in advance.
Is this job: (A) Suitable for new graduates (B) Requiring more specialized experience
Answer: (A) Suitable for new graduates
Regarding the following job: This is a role in which the consultant engages in consultative sales, explaining and persuading the management team of the client about the importance of the company's own system for the client's business and obtaining approval for system implementation in the board of directors.
Is this job: (A) Suitable for new graduates (B) Requiring more specialized experience
Answer:
phi-3
(B) Requiring more specialized experience
The job described involves consultative sales and persuading management
teams, which typically requires a deep understanding of the industry,
the client's business, and the systems being proposed. This level of expertise
is usually gained through experience, making it more suitable for individuals
with specialized experience rather than new graduates. However,
it's worth noting that some entry-level positions in sales or consulting
might be available to new graduates with the potential for growth and
learning on the job. But based on the description provided,
the job seems to lean towards requiring more specialized
GPT-3.5
(B) Requiring more specialized experience
Zero-shot CoT（≒ step by step）
Regarding the following job:
This is a role in which the consultant engages in consultative sales, explaining and persuading the management team of the client about the importance of the company's own system for the client's business and obtaining approval for system implementation in the board of directors.
Is this job: (A) Suitable for new graduates (B) Requiring more specialized experience
Take a deep breath and work on this problem step-by-step. Please answer according to the following output format:
Output Format
Required Experience: Describe the experience that is believed to be necessary for the job. Answer: Write the answer.
Answer: Required Experience: Business consulting experience, client negotiation experience, presentation skills.
Answer:
Phi-3
Output Format Required Experience: Describe the experience that is believed to be necessary for the job.
Answer: Required Experience: Business consulting experience, client negotiation experience, presentation skills, understanding of IT systems, knowledge of the client's industry, and strong communication skills.
Output Format: (A) Suitable for new graduates (B) Requiring more specialized experience
Answer: (B) Requiring more specialized experience
Explanation: While a new graduate may have the foundational skills needed for this job, such as communication and presentation skills
GPT3.5
(B) Requiring more specialized experience
まとめ
３つのプロンプトで検証を行いました。
答えは一緒でしたが、GPTは端的,Phi-3は補足情報を付与してくれました。
Discussion