iTranslated by AI
Building a Lateral Thinking Puzzle Site Powered by Generative AI
What am I doing?
I have released a web app that uses LLMs to solve the challenges of playing lateral thinking puzzles.
What is a lateral thinking puzzle?
A lateral thinking puzzle is a deduction game where a seemingly mysterious situation is presented, and the solver repeatedly asks questions to uncover the truth.
Questions are limited to those that can be answered with "Yes" or "No," and the solver must find the answer within the restricted information.
For example, suppose there is a problem like the following:
Hanako headed to her workplace and took a city bus. However, even when she reached the last stop, she did not get off the bus and remained seated. Why?
In response to this, the solver narrows down the possibilities of the answer through the following exchange.
Solver: "Was there a bus hijacking?"
Questioner: "No"
Solver: "Is Hanako in a condition where she can move?"
Questioner: "Yes"
The appeal of lateral thinking puzzles is the fun of exploratively deriving the answer yourself from infinite possibilities.
What are the challenges?
Since lateral thinking puzzles are games played by interacting between two or more people, they basically cannot be played alone.
Therefore, I wanted to automate the side that responds to questions, but with previous technology (within the scope of my understanding), it was difficult to not only parse and classify the natural language of the questions but also to obtain logically correct responses.
So, I decided to try using the currently popular OpenAI.
What I Made
I released Yesonor, a submission site for lateral thinking puzzles where you can enjoy "Sea Turtle Soup" with AI.
Service details
This site provides the following services:
- You can post your own lateral thinking puzzles.
- You can challenge posted lateral thinking puzzles. (Since the AI responds to questions, you can challenge them alone whenever you have free time.)
Features and Implementation
1. Feature to respond to questions
The AI answers "Yes" or "No" to questions.
By providing a prompt that combines the truth of the puzzle and the question to the Chat Completions API, I made it so that the response is obtained as output.
Example prompt (actually in English) ↓
## Instruction
Based on the following information and examples, answer the user's question with True (Yes), False (No), or Unknown (Unknown/Not mentioned).
## Problem
Taro is wearing glasses even though his eyesight is good. Why?
## Truth
Taro loves fashion and is wearing non-prescription glasses.
## Question (Sentence entered by the solver)
Is Taro wearing glasses of his own volition?
Responses are returned as JSON containing one of the values True/False/Unknown using the Function Calling feature. The reason for limiting the return values is as follows:
- By reducing the number of tokens used for the answer, the response is made faster.
- To prevent the AI from giving away spoilers by talking too much.
Few-shot Learning to Improve Response Accuracy
Lateral thinking puzzles often involve a twist, and with standard LLM text generation, you may not always get the response intended by the author.
Therefore, authors are asked to input at least three examples of questions and their answers (hereafter referred to as QuestionExample) for each problem, and these are included in the prompt (so-called Few-shot Learning).
However, since a phenomenon occurred where response accuracy dropped if too many QuestionExamples were included in the prompt, I made it so that only QuestionExamples with a close vector distance to the question asked by the solver are sent.
To calculate the distance between questions, I use OpenAI's Embeddings.
Custom Message Feature to Improve Solver Satisfaction
Due to the nature of the puzzles and to prevent spoilers, responses to questions only return "Yes" or "No," which inevitably leads to a lack of information from the solver's perspective.
As a countermeasure, I implemented a feature that allows returning a unique response set by the author (hereafter referred to as CustomMessage) in addition to the answer.
CustomMessage can be set in association with QuestionExample. When the vector distance between the solver's question and a QuestionExample (see above) is closer than a certain threshold, the CustomMessage set for that QuestionExample is displayed as a message.
Through this, I aim to provide an experience as if the "AI is being thoughtful and giving an appropriate response."
2. Feature to Judge Correct/Incorrect Answers
The AI evaluates the solver's answer (the solution to the puzzle) and determines whether it is correct or incorrect.
Similar to the question feature mentioned above, a prompt combining the truth of the puzzle and the answer is given to the Chat Completions API to make the determination.
However, unlike the question feature, judging the correctness of puzzles that receive sentences as answers is quite difficult because the LLM needs to understand what constitutes a correct answer.
This time, I included the following as a judgment condition in the prompt: "Does the input cover all the elements explained in the truth?"
Additionally, by using Embeddings to quantify to some extent how close the answer is and showing it to the user, I give the user a sense that they are "gradually approaching the truth."
Operational Experience
It has been about half a year since I started operating the service, so here are some casual thoughts on what it's like to run a system using LLMs.
LLMs are reasonably smart
As a user myself, I have asked various questions, and I was happy to see that the LLM surprisingly gives proper answers.
However, it is nowhere near a human level of thoughtfulness, and it probably doesn't provide a great experience for users who expect strictly accurate responses.
Expensive
The amount I'm paying OpenAI is starting to exceed the scope of a hobby.
Thanks to everyone, a certain number of people are playing, but even with a user base that hasn't exactly gone viral, it's costing several thousand yen per month. (I don't know the exact figure as it's mixed with my personal LLM usage.)
My impression is that with the current OpenAI cost structure, it would be difficult to generate a serious profit with this service.
Conclusion
There is still plenty of room for improvement in this service.
In particular, I expect the experience of the question feature to improve as model accuracy increases, so I'm looking forward to future OpenAI updates.
Actually, since there are various AI solutions other than GPT, I hope to try out different options.
Please come and check out Yesonor!
Discussion