iTranslated by AI
A Sample Project: Training AutoML with ChatGPT-Generated Single-Label Text Data to Evaluate My Own Comments
I wanted to investigate the tendencies in my own writing. Google Cloud's AutoML significantly lowers the barrier to using machine learning. As an experiment, I will try training a single-label text classification model and have it assign labels to my sentences.
Overview of the Workflow
Here are the steps to have AutoML label my sentences:
- Prepare training data
- Register it as a Vertex AI text classification (single-label) dataset
- Train with AutoML
- Deploy the model and create an endpoint
- Submit my own sentences for prediction
This is the flow. It is particularly helpful that AutoML handles the training part.
Preparing the Training Data
The hardest part is preparing the training data. Initially, I considered manually labeling my own blog posts and training on those, but I rejected that idea because it would be too time-consuming.
Getting Help from ChatGPT
One of ChatGPT's strengths is generating dummy and sample data. For this project, I used the following prompt to request the creation of training sentences:
There are labels as follows:
positive: Sentences containing positive emotions, such as "I like" or "I prefer."
Please generate 50 sentences that you think are appropriate for this label, each at least 100 characters long. Please make the sentences related to IT technology.
For a single-label text dataset, at least 10 pieces of data corresponding to each label are required (a requirement of Vertex AI). Although I generated 50 here just in case, I think about 15 would be sufficient. I repeated this process for the following labels:
- positive: Sentences containing positive emotions, such as "I like" or "I prefer."
- negative: Sentences containing negative emotions, such as "I hate" or "I'm not good at."
- fail: Sentences explaining that something did not go well.
- quote: Sentences starting with ">" representing a quotation.
- opinion: Sentences containing an opinion.
- try: Sentences describing what happened after actually trying something.
- issue: Sentences representing a problem or task.
- code: Source code
I am still exploring which labels are best, but I came up with these based on the elements that seem to fill my own writing.
Entering Sample Data into a Labeled Spreadsheet
Initially, I thought about combining the OpenAI API with a Spreadsheet SDK. However, since the response content was unstable (for example, it would include irrelevant answers like "Yes, I can help you output sentences for such labels"), I entered the data manually based on ChatGPT's output.
As shown in the image, I put the sample text in the first column and the label in the second column.
Export this sheet as a CSV. We will register it as a dataset.
Creating a Dataset
Create a Google Cloud Vertex AI dataset. When selecting the dataset type, choose Text classification (single-label).
Next, add documents to the dataset. Use the CSV created earlier. Select Upload import files from your computer and choose that CSV.
For data splitting, the "Default" random setting should be fine. After that, check the Analyze tab among the Import, Browse, and Analyze tabs. If there are no errors regarding insufficient labels, you are ready to go.
Training with AutoML
This mode automatically tests appropriate algorithms and applies the one judged to be the most accurate. It's very helpful because it handles the training just by clicking through the screens. However, in exchange for that convenience, it took 9 hours for the training to complete in my case. I recommend starting the execution before going to bed.
Select Training from the left pane menu of Vertex AI and press + Create. You should be able to select the dataset and annotation set (the set of labels and sentences) you created earlier. Set the training method to "AutoML."
In 2. Model details, since this is the first time creating the model, Train new model is fine. Start the training. Then, go to sleep. Good night.
Deploying the Model and Creating an Endpoint
Once the training is complete, you will see that the job is finished in the Training Pipelines section.
After this, you can create a prediction endpoint. Select Endpoints from the left pane menu and click Create Endpoint. During the Model settings step of creation, select the model that was just created. After waiting a short while, the endpoint will be ready.
Testing It Out
Since it provide a convenient request example, let's try it immediately in Cloud Shell.
> ENDPOINT_ID="00000000000000"
> PROJECT_ID="1111111111111"
> curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" https://us-central1-aiplatform.googleapis.com/ui/projects/${PROJECT_ID}/locations/us-central1/endpoints/${ENDPOINT_ID}:predict -d '{
"instances": {
"mimeType": "text/plain",
"content": "Next, deploy the created model to create an endpoint."
}
}'
{
"predictions": [
{
"ids": [
"8641602292656635904",
"5772809330021629952",
"9218063044960059392",
"8201806987395072",
"1147612512712130560",
"5759298531139518464",
"1161123311594242048",
"2300534017318977536",
"2314044816201089024",
"4606377026532671488",
"6925730834628476928",
"4619887825414782976",
"6912220035746365440"
],
"confidences": [
0.0073390332981944084,
0.70815819501876831,
0.00072439218638464808,
0.00010403751366538927,
0.16298328340053558,
0.0024225716479122639,
0.038506362587213516,
0.013907374814152718,
0.017786769196391106,
0.0088923797011375427,
0.0070339511148631573,
0.01147755142301321,
0.02066413126885891
],
"displayNames": [
"code",
"operation",
"fail",
"negative",
"fact",
"issue",
"try",
"link",
"background",
"opinion",
"succeed",
"quote",
"positive"
]
}
],
"modelVersionId": "1"
}
The results are a bit hard to read, but looking at the console screen for help, the figure indicates a prediction that "there is a high probability that the sentence represents an operation."
Let's test a few more.
I believe that going to the office once in a while can lead to good communication.
I was hoping for opinion to be more prominent.
I feel good because the test I tried yesterday passed successfully.
negative!? Did I make a mistake with the registered data?
curl -X POST
This is correctly detected as code as intended. It seems like the meaning of operation might be too broad, which could be negatively affecting the predictions.
Conclusion
AutoML is a convenient service, but I had been putting off trying it because the effort of finding sentences for labels or labeling sentences felt tedious. Since ChatGPT has made generating dummy data much easier, I decided to try them together. The ability to have the system handle training automatically while you sleep is a very welcome feature in an environment where you also want to focus on business.
On the other hand, it must be said that the accuracy of the model itself is not yet great. If you want to evaluate each sentence with high precision, you will likely need a much larger amount of labeled data. As a way to utilize predictions even with a small dataset, I thought it could be immediately useful for pinpointing specific expressions, such as detecting whether something is a spam-like expression or an aggressive sentence. Let's have ChatGPT create some sample data right away.
I see, so that's not allowed. I will think of another way for the data generation method.
References
Discussion