【ML】Hugging face Introduction

This time, I will explain about Hugging Face Hub.

What is Hugging Face Hub?

Hugging Face Hub is a site for sharing models and datasets that specialize in natural language processing (NLP). (It's like a github for machine learning models.)

What can you do?

You can download trained models, and download and save datasets. You can also perform transfer learning, which is becoming standard in large-scale learning, and save models.

How do you do it?

I will show you the specific code.

*This assumes that the transformers and sentencepiece libraries have already been installed.

・Examples of various natural language processing tasks

from transformers import pipeline
import numpy as np
import pandas as pd


text = """Dear Amazon, last week I ordered an Optimus Prime action figure \
from your online store in Germany. Unfortunately, when I opened the package, \
I discovered to my horror that I had been sent an action figure of Megatron \
instead! As a lifelong enemy of the Decepticons, I hope you can understand my \
dilemma. To resolve the issue, I demand an exchange of Megatron for the \
Optimus Prime figure I ordered. Enclosed are copies of my records concerning \
this purchase. I expect to hear from you soon. Sincerely, Bumblebee."""

# Emotion classification
classifier = pipeline("text-classification")
outputs = classifier(text)
classified = pd.DataFrame(outputs)

# Named entity recognition # Extracting proper nouns and other
ner_tagger = pipeline("ner", aggregation_strategy="simple")
outputs = ner_tagger(text)
ner = pd.DataFrame(outputs)

# Question and Answer
reader = pipeline("question-answering")
question = "What does customer want?"
outputs = reader(question=question, context=text)
answer = pd.DataFrame([outputs])

# Summarize
summarizer = pipeline("summarization")
outputs = summarizer(text, max_length=45, clean_up_tokenization_spaces=True)
summary = (outputs[0]["summary_text"])

# Trasnlation
translator = pipeline("translation_en_to_de", model="Helsinki-NLP/opus-mt-en-de")
outputs = translator(text, clean_up_tokenization_spaces=True, min_length=100)
translated = outputs[0]["translation_text"]

# Text Generation
generator = pipeline("text-generation")
response = "Dear Bumblebee, I am sorry to hear that your order was mixed up."
prompt = text + "\n\nCustomer service response:\n" + response
outputs = generator(prompt, max_length=200)
generated = outputs[0]["generated_text"]

# Output the results of your prefer tasks
print(generated)

Hugging Face allows you to download many models using a simple interface called pipeline. You can check the license for each model by checking the model name, which can be obtained with the following code, on the Hugging Face web page.

・Check the model name

# Example
print(ner_pipeline.model)

Please try and enjoy.

Summary

The Hugging Face hub makes it very easy to use pre-trained models. Please try to make good use of them.