やること
Text generation web UIをAPIとして利用する方法について紹介します。
前提
- Text generation web UIをinstall済みであること
下記を参考にinstallを行う
- モデルは、Phi-3を使う
参考資料
利用手順
- text generation web UIのディレクトリに移動
- text generation web UIを起動
- Modelをセット
- 下記のコードを実行
import requests
url = "http://127.0.0.1:5000/v1/chat/completions"
headers = {
"Content-Type": "application/json"
}
history = []
while True:
user_message = input("> ")
history.append({"role": "user", "content": user_message})
data = {
"mode": "chat",
"character": "Example",
"messages": history
}
response = requests.post(url, headers=headers, json=data, verify=False)
assistant_message = response.json()['choices'][0]['message']['content']
history.append({"role": "assistant", "content": assistant_message})
print(assistant_message)
- cmd上に > が表示されるので会話してみる
会話内容
hello
Hello! How can I help you today?
very sleepy
Oh, you should definitely get some rest. Taking care of yourself is important.
and very hungey
I understand, a good meal can help you feel better. What would you like to eat?
everything but vegetables
No problem, we can find some delicious options without the vegetables. What's your favorite type of cuisine?
junkfood
Well, we do have some healthier alternatives with a slightly indulgent taste. How about we find a balance?
No Thank you
Alright, just let me know when you're ready to order.
- 下記のコードを実行
import json
import requests
import sseclient # pip install sseclient-py
url = "http://127.0.0.1:5000/v1/completions"
headers = {
"Content-Type": "application/json"
}
data = {
"prompt": "This is a cake recipe:\n\n1.",
"max_tokens": 200,
"temperature": 1,
"top_p": 0.9,
"seed": 10,
"stream": True,
}
stream_response = requests.post(url, headers=headers, json=data, verify=False, stream=True)
client = sseclient.SSEClient(stream_response)
print(data['prompt'], end='')
for event in client.events():
payload = json.loads(event.data)
print(payload['choices'][0]['text'], end='')
print()
- 以下が出力される
This is a cake recipe:
1. Preheat oven to 350°F (175°C).
2. Grease and flour a 9x13 inch baking pan.
3. In a large bowl, cream together 1 cup of softened butter and 2 cups of white sugar.
4. Beat in 4 eggs, one at a time.
5. Stir in 1 teaspoon of vanilla extract.
6. Combine 3 cups of all-purpose flour and 1 teaspoon of baking powder; gradually add to the creamed mixture.
7. Pour batter into the prepared pan.
8. Bake for 30-35 minutes, or until a toothpick inserted into the center of the cake comes out clean.
9. Cool the cake in the pan for 10 minutes, then remove to wire ra
まとめ
Text generation web UIをAPIとして利用する方法を試しました。
Jetsonなどのデバイスに組み込めば、面白いプロダクトできそうだなーと思いました。
Discussion