iTranslated by AI
Amazon Review Classification using Natural Language Processing (BERT Edition)
In this article, we will handle the task of determining whether an Amazon review is a high rating or a low rating based on the content of the review.
The models used are Naive Bayes and BERT, and this time we will focus on BERT.
Please see this article for the Naive Bayes implementation.
What is BERT?
BERT is a model proposed by Google in 2018.
In addition to the advantages of being able to conduct deep analysis and having high computational efficiency by using Attention, it has become popular due to the ease of training using Pre-training and Fine-tuning.
About Transformer
BERT uses an Attention-based NN called the Transformer Encoder, which was proposed as part of the Transformer model.
The Transformer is a model proposed in the extremely famous paper "Attention is all you need".
The Transformer is an Encoder-Decoder model, similar to Seq2seq.
The ENCODER extracts meaning, and the DECODER translates that meaning into Japanese.
By constructing it with Attention layers, it became possible to "perform more accurate translations while accelerating computation" compared to models such as RNNs.
For those who want to know more about the Transformer, click here.
About BERT
In Transformer, there is a class for document classification called BertForSequenceClassification, which has the following structure:
During inference, you obtain classification scores for each category through this model. Document classification is performed by taking the label with the highest classification score as the predicted value. BERT consists of 12 Transformer layers, and in BertForSequenceClassification, the output corresponding to the first token [CLS] of the final layer is input to the classifier. The classifier calculates classification scores using a linear transformation and the tanh function on the input. Additionally, during fine-tuning, parameters are updated to minimize the loss between the output and the labels.
Preprocessing
My execution environment is as follows:
- Execution environment
- Google Colaboratory (Pro +)
- Python
- 3.8.10
- janome
- 0.4.2
- Transformers
- 4.18.0
- fugashi
- 1.1.0
- ipadic
- 1.0.0
- pytorch_lightning
- 1.6.1
Among the files available from here, we will use cls-acl10-unprocessed.
There are pre-processed datasets as well, but this time I will start from the pre-processing stage for study purposes.
I referred to this book for the pre-processing code:
I will post the code for now, but since it is almost the same as the Naive Bayes edition, I will omit detailed explanations.
code
# Load tokenizer
tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)
max_length = 128 # BERT has a maximum of 512 tokens
def get_tokenized_sentences_and_labels(file_name: str):
# Load data
with open(file_name) as f:
data = f.read()
data = data.replace('\n', '').replace('\r', '')
reviews = re.findall(pattern=r'(.+?)',string=data) # Remove reviews that have a rating but no text
dataset_for_loader = []
for item in reviews:
raiting = re.findall(pattern=r'(.+?)',string=item)
text = re.findall(pattern=r'',string=item)
text = '\n'.join(text)
encoding = tokenizer(
text,
max_length=max_length,
padding='max_length',
truncation=True
)
raiting = int(float(raiting[0])) - 1 # Get the 5-star rating
label = 0 if raiting >= 3 else 1 # positive -> 0, negative -> 1
encoding['labels'] = label
encoding = {k: torch.tensor(v) for k, v in encoding.items()}
dataset_for_loader.append(encoding)
return dataset_for_loader
dataset_learn = get_tokenized_sentences_and_labels(config.input_path + 'train.review')
dataset_test = get_tokenized_sentences_and_labels(config.input_path + 'test.review')
div_index = int(len(dataset_learn) * 0.8)
dataset_train = dataset_learn[:div_index]
dataset_val = dataset_learn[div_index:]
dataloader_train = DataLoader(dataset_train, batch_size=32, shuffle=True)
dataloader_val = DataLoader(dataset_val, batch_size=256)
dataloader_test = DataLoader(dataset_test, batch_size=256)
Training
BERT can perform inference without fine-tuning, but generally, fine-tuning improves accuracy, so this time I performed inference after fine-tuning. If you are interested, it might be interesting to try it without fine-tuning and compare the results.
I referred to the following book for the training code. This is the book I read when I first studied natural language processing, and while the difficulty level is slightly high, I highly recommend it as it is very easy to understand.
First, define the BertForSequenceClassification_pl class. In this class, we load the BERT model to be fine-tuned and define functions related to training and testing.
class BertForSequenceClassification_pl(pl.LightningModule):
def __init__(self, model_name, num_labels, lr):
# model_name: Name of the Transformers model
# num_labels: Number of labels
# lr: Learning rate
super().__init__()
# Save the num_labels and lr arguments.
# For example, lr can be accessed via self.hparams.lr.
# It is also automatically saved during checkpoint creation.
self.save_hyperparameters()
# Load BERT
self.bert_sc = BertForSequenceClassification.from_pretrained(
model_name,
num_labels=num_labels
)
# Write a function that outputs the loss when a mini-batch (`batch`) of training data is given.
# batch_idx is the mini-batch number, but it is not used this time.
def training_step(self, batch, batch_idx):
output = self.bert_sc(**batch)
loss = output.loss
self.log('train_loss', loss) # Log the loss with the name 'train_loss'.
return loss
# Write a function that calculates the metrics for evaluating validation data
# when a mini-batch of validation data is given.
def validation_step(self, batch, batch_idx):
output = self.bert_sc(**batch)
val_loss = output.loss
self.log('val_loss', val_loss) # Log the loss with the name 'val_loss'.
# Write a function that calculates the metrics for evaluating test data
# when a mini-batch of test data is given.
def test_step(self, batch, batch_idx):
labels = batch.pop('labels') # Get labels from the batch
output = self.bert_sc(**batch)
labels_predicted = output.logits.argmax(-1)
num_correct = ( labels_predicted == labels ).sum().item()
accuracy = num_correct/labels.size(0) # Accuracy
self.log('accuracy', accuracy) # Log the accuracy with the name 'accuracy'.
# Write a function that returns the optimizer used for training.
def configure_optimizers(self):
return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)
Next, set up the fine-tuning configuration. In this case, we use the Trainer class from PyTorch Lightning for training. We specify settings such as the number of epochs and checkpoints required for training.
# Specify conditions for saving model weights during training
checkpoint = pl.callbacks.ModelCheckpoint(
monitor='val_loss',
mode='min',
save_top_k=1,
save_weights_only=True,
dirpath='model/',
)
# Specify the training method
trainer = pl.Trainer(
gpus=1,
max_epochs=10,
callbacks = [checkpoint]
)
Fine-tuning is performed based on these conditions. For this project, I used the Japanese pre-trained model cl-tohoku/bert-base-japanese-whole-word-masking. Also, the learning rate was set to 1e-5. I haven't performed hyperparameter tuning this time, so if you're interested, please try various settings.
# Load PyTorch Lightning model
model = BertForSequenceClassification_pl(
'cl-tohoku/bert-base-japanese-whole-word-masking', num_labels=2, lr=1e-5
)
# Perform fine-tuning
trainer.fit(model, dataloader_train, dataloader_val)
I performed predictions using the fine-tuned model.
test = trainer.test(dataloaders=dataloader_test)
print(f'Accuracy: {test[0]["accuracy"]:.2f}')
Checking the accuracy, it was found to be 0.83. The confusion matrix is as follows:
It can be seen that both positive and negative classes are being predicted well.
Finally, I extracted some reviews where classification failed.
Here is an example where a positive review was incorrectly classified as negative.
This book mainly covers the management philosophy the author reached through their experience at the Industrial Revitalization Corporation and their long consulting career. I feel like I would have liked more stories from actual turnaround sites, but I am satisfied with this in its own way. One passage that personally left the biggest impression was the following criticism of Japanese companies that survived the post-bubble recession by suppressing new graduate recruitment and increasing non-regular employment to protect regular employees: "In Japan, many companies have said 'People are important.' But that meant the people inside the company were important. That became clear then. People outside the company didn't matter at all. It was never a case of management based on a noble philosophy like 'respect for humanity'."
It seems possible that the model was swayed by the ambiguity of expressions like "satisfied with this in its own way" or by the fact that the quoted section contains negative content.
Here is an example where a negative review was incorrectly classified as positive.
It might not have felt so out of place in a bubble-like atmosphere, but at this point, it's a stock book that makes you wonder "Why now?". Its selling point is its flippancy. It explains stock investing not as buying and selling ownership in a company, but as gambling, using manga with a thick art style. Although it is flippant, I felt a strange sense of fondness for the way it emphasizes flippancy rather than depicting flippant content as something serious.
It is quite difficult to classify this one correctly according to the label.
The code implemented this time can be found here. It's messy because it wasn't written for sharing.
You should be able to run it by rewriting the PATH.
Please try playing around with it by changing parameters, etc.
Summary
How was it? BERT is indeed powerful.
This time we used Amazon reviews, but it can also be used for other tasks like sentiment analysis, so please give it a try.
Also, this article is a relay article for the "AcademiX: Online Community for Learning AI and Machine Learning" to which I belong.
If you want to learn more about AI, please join the community!
Discussion