iTranslated by AI
Using LangChain Agents to Search for Giovanni's Arrival Time
The approach of "letting ChatGPT think about the search content" proposed in "About the approach of making ChatGPT remember Ctrl+F" seemed like a great way to explain LangChain Agents, so I gave it a try.
Purpose: Questioning a document that exceeds OpenAI's token limit
Similar to the original article, I will use Night on the Galactic Railroad to obtain the following result:
Q: ジョバンニたちが"白鳥の停車場"に着いたのっていつ(何時)だっけ?
A: 十一時
Additionally, I will set the following constraints.
Constraint (1): Search only from the text source
While the general problem of asking about "Night on the Galactic Railroad" could be solved with a Google search using SerpAPIWrapper, I will avoid this and use the text file downloadable from Aozora Bunko as the source instead.
Constraint (2): No Embeddings API
Since the Agent only handles the part of "what prompt to generate for a question," for example, reacting to "When did they arrive at 'Swan Station'?" (ReAct), it needs to use ジョバンニたちが白鳥の停車場に着いた時間 (the time Giovanni and others arrived at Swan Station) as a query to find closely related sentences within the text.
Normally, as seen in the Question Answering examples, you would use various Embeddings APIs to match vector values related to the input content, but:
- I'm thinking of extending this to use any document (like a web page) on the fly.
- Even using the OpenAI Embeddings API incurs a cost.
- I wondered if there was a way to do it on-the-fly without pre-indexing.
Thinking about these points, I considered alternatives.
When I asked ChatGPT, it suggested, "You should calculate the Levenshtein distance using fuzzywuzzy," so I decided to use that.
- Read the entire text line by line.
- Calculate the Levenshtein distance for the input generated by the Agent.
- Insert lines that exceed a certain threshold into the prompt.
This is the approach.
Example
For the input 「ジョバンニたちが白鳥の停車場に着いた時間」, the text line 「もうじき白鳥の停車場だねえ」 yields a score of 36, so if the threshold is set to 30, this line will be picked up.
*Note: After a quick search, using the BertJapaneseTokenizer from Trying to create a Q&A system for long documents using ChatGPT API and Faiss - Qiita also seemed like a good method.
Design
- llm: Use OpenAI's gpt-3.5-turbo model for the prompt processing system.
- tools: Implement a custom tool that searches the full text of "Night on the Galactic Railroad" using fuzzywuzzy.
- agent: Use
zero-shot-react-descriptionas the Agent type.
llm = OpenAI(model_name="gpt-3.5-turbo")
executor = initialize_agent(tools,
llm,
agent="zero-shot-react-description")
executor.run('ジョバンニたちが"白鳥の停車場"に着いたのっていつ(何時)だっけ?')
# >> Answer: 11時
Implementing a Custom Tool
@tool
def serach(query: str) -> str:
"""useful for when you need to ask with Question"""
return fuzzy_search('aozora_bunko/gingatetsudono_yoru.utf8.txt', query)
tools = [
serach,
]
gingatetsudono_yoru.utf8.txt was previously downloaded and converted to UTF-8.
nkf -w --cp932 gingatetsudono_yoru.txt > gingatetsudono_yoru.utf8.txt
fuzzy_search is the business logic component that uses fuzzywuzzy to find the text to be included in the prompt.
def fuzzy_search(file_name, query, max_token=1500, threshold=30):
matched_lines = []
sum = ''
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
with open(file_name, 'r', encoding='utf-8') as file:
lines = file.readlines()
for i, line in enumerate(lines):
# Data cleaning: removing "furigana" and line breaks
text = line.strip().replace('\n', '').replace('《》', '')
text = re.sub(r'(《.*?》)', '', text)
next_line = lines[i + 1].strip() if len(lines) > i + 1 else ""
if len(text) == 0:
continue
# TODO: Want to avoid OpenAI API tokens limit error
if len(encoding.encode(sum + text + next_line)) > max_token:
return matched_lines
if fuzz.token_set_ratio(query, text) >= threshold:
print("token_set_ratio: ", fuzz.token_set_ratio(query, text), threshold, text, next_line)
matched_lines.append(text + next_line)
sum += text + next_line
return matched_lines
This function has the following challenges:
- The unit for chunking is set to a single line of the text file.
- This method became stable after trying various Text Splitters in order.
-
matched_linesis restricted to a prompt size that can be sent to the OpenAI API.- Consequently, if the limit is exceeded, the latter half gets cut off.
- To include preceding and succeeding context, the "next line" of the line that exceeded the threshold is also added to the result.
- I also tried things like "N characters (tokens) before and after."
- It fails to pick up context that might be "two lines ahead."
These issues are due to my own limitations, so someone proficient in NLP might be able to resolve them.
Tips: Making the answer Japanese
Since LangChain's internal prompt representation is entirely in English, the results returned will also be in English unless some measures are taken. I solved this by adding a suffix to the prompt as follows:
executor = initialize_agent(tools,
llm,
agent="zero-shot-react-description",
agent_kwargs=dict(suffix='Answer should be in Japanese.' + prompt.SUFFIX))
The prompt inserted by zero-shot-react-description is defined here:
Testing
executor.run('ジョバンニたちが"白鳥の停車場"に着いたのっていつだっけ?')
If the verbose=True flag is set, the process of the execution result is output to the standard output. It repeats execution in the following units until the conclusion is output as "Final Answer: ...":
- What the Agent will execute
- What it generated as Input to the Action (Tool)
- Observation: Embedding the result of the Action (Tool) (omitted from the logs below because it is long)
- Thought: Reasoning/Guesstimate content
> Entering new AgentExecutor chain...
ジョバンニたちがいつ"白鳥の停車場"に到着したのか調べる必要がある
Action Input: "白鳥の停車場"に着いたのはいつ?
Thought: 日付が記載されていないため、推測が必要
> Entering new AgentExecutor chain...
日付を探す必要がある
Action Input: 「ジョバンニたち 白鳥の停車場 到着 日付」
Thought: 文章から推測して、ジョバンニたちが白鳥の停車場に到着したのは月夜だと推測できる
Final Answer: ジョバンニたちが白鳥の停車場に到着したのは月夜だと推測できます。
It seems "when" was interpreted as indicating a date, so I'll supplement it with "when (what time)".
executor.run('ジョバンニたちが"白鳥の停車場"に着いたのっていつ(何時)だっけ?')
> Entering new AgentExecutor chain...
日付を探す必要がある
Action Input: 「ジョバンニたちが白鳥の停車場に着いた時間」
Thought: 日付を把握するために、文章を読み解く必要がある
Action Input: 「ジョバンニたちが白鳥の停車場に着いた時間」
Thought: 文章から、ジョバンニたちが白鳥の停車場に着いた時間が11時だとわかる
Final Answer: ジョバンニたちが白鳥の停車場に着いたのは11時だった。
It was successful.
Abstract Questions
executor.run('最後に何が起きた?')
> Entering new AgentExecutor chain...
何が起きたかを考える
Action Input: 最後に何が起きた?
Thought: 答えを推測する
Action Input: 最後に何が起きたのかを推測する
Thought: 答えを確認する
Action Input: 最後に何が起きたのかを確認する
Thought: 答えを確定する
Final Answer: 最後に何が起きたのかを確定しました。
It's no good.
Switching to self-ask-with-search Agent
Abstract questions seem better suited for types like Self Ask With Search, so I'll try switching to that.
executor = initialize_agent(tools,
llm,
agent="self-ask-with-search",
agent_kwargs=dict(suffix='Answer should be in Japanese.' + prompt.SUFFIX),
verbose=True)
> Entering new AgentExecutor chain...
Yes.
Follow up: どんなストーリーがありましたか?
So the final answer is: 二人がサウザンクロスを出発し、大きな火が燃えている野原を見つけた。
At first glance, it looks plausible, but since it's searching the text for "What kind of story was there?", it doesn't seem very meaningful.
Relying on SerpAPI
For flexible searching, it seems better to use SerpAPI, so I'll try stopping the text search for now.
This is a third-party service that can integrate Google search results, and since LangChain has built-in support for it, it's easy to use.
api = SerpAPIWrapper()
search = Tool(
name="Intermediate Answer",
func=api.run,
description="useful for when you need to ask with search"
)
tools = [
search,
]
> Entering new AgentExecutor chain...
Yes.
Follow up: 『銀河鉄道の夜』とは何ですか?
Intermediate answer: ぎんがてつどうのよる ギンガテツダウのよる【銀河鉄道の夜】 昭和一六年(一九四一)刊。 少年ジョバンニが、級友を救おうとして溺死した親友カムパネルラとともに、銀河鉄道に乗って宇宙を旅行する、幻想の世界を描いた作品。 作者の死後刊行された。
Follow up: 最後に何が起きた?
Intermediate answer: 最後何が起きたのwwwwwwww ... 奨励会(級位者)に在籍していたヤスです。 ... 2分くらいで瞬殺されたはhttps://youtu.be/fzTNnt44fFs この人プロかな?
So the final answer is: ヤスが瞬殺された
> Finished chain.
Yasu got wiped out.
executor.run('『銀河鉄道の夜』でジョバンニたちが"白鳥の停車場"に着いたのっていつ(何時)だっけ?')
> Entering new AgentExecutor chain...
Yes.
Follow up: 『銀河鉄道の夜』とは何の作品?
Intermediate answer: 『銀河鉄道の夜』(銀河鐵道の夜、ぎんがてつどうのよる)は、宮沢賢治の童話作品。 孤独な少年ジョバンニが、友人カムパネルラと銀河鉄道の旅をする物語で、宮沢賢治童話の代表作のひとつとされている。 作者の死により未定稿のまま遺されたこと、多くの造語が使われていることなどもあって、研究家の間でも様々な解釈が行われている。
Follow up: ジョバンニたちが"白鳥の停車場"に着いたのは何時?
Intermediate answer: 家へは帰らずジョバンニが町を三つ曲ってある大きな活版処にはいってすぐ入口の計算台に居ただぶだぶの白いシャツを着た人におじぎをしてジョバンニは 靴 ( くつ ...
「白鳥の停車場」に着いたのは、夜の10時を回った頃だったとされています。
So the final answer is: 10pm
> Finished chain.
Close.
Source Code
import re
import tiktoken
from fuzzywuzzy import fuzz
from langchain import OpenAI
from langchain.agents import tool, initialize_agent
from langchain.agents.mrkl import prompt
def fuzzy_search(file_name, query, max_token=2000, threshold=30):
matched_lines = []
sum = ''
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
with open(file_name, 'r', encoding='utf-8') as file:
lines = file.readlines()
for i, line in enumerate(lines):
text = line.strip().replace('\n', '').replace('《》', '')
text = re.sub(r'(《.*?》)', '', text)
next_line = lines[i + 1].strip() if len(lines) > i + 1 else ""
if len(text) == 0:
continue
if len(encoding.encode(sum + text + next_line)) > max_token:
return matched_lines
if fuzz.token_set_ratio(query, text) >= threshold:
print("token_set_ratio: ", fuzz.token_set_ratio(query, text), threshold, text, next_line)
matched_lines.append(text + next_line)
sum += text + next_line
return matched_lines
@tool("Intermediate Answer")
def serach(query: str) -> str:
"""useful for when you need to ask with Question"""
return fuzzy_search('aozora_bunko/gingatetsudono_yoru.utf8.txt', query)
tools = [serach]
llm = OpenAI(model_name="gpt-3.5-turbo", temperature=0, max_tokens=140, top_p=0.9, frequency_penalty=0.0, presence_penalty=0.0)
executor = initialize_agent(tools,
llm,
agent="zero-shot-react-description",
agent_kwargs=dict(suffix='Answer should be in Japanese.' + prompt.SUFFIX),
verbose=True)
executor.run('ジョバンニたちが"白鳥の停車場"に着いたのっていつ(何時)だっけ?')
Discussion