Differences between tokenize, encode, and encode_plus in Hugging Face Tokenizers

When using the Hugging Face library, you often encounter tokenize, encode, and encode_plus, which can be confusing. Here is a summary of their differences.

tokenize

This method splits an input sentence into tokens (word segments) according to the language model's vocabulary.

$ pip install transformers[ja]

tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
tokenizer.tokenize("私は元気です。")

['私', 'は', '元気', 'です', '。']

Additionally, you can use convert_tokens_to_ids to convert the tokens into an array of token IDs corresponding to the vocabulary.

tokenizer.convert_tokens_to_ids(['私', 'は', '元気', 'です', '。'])

[1325, 9, 12453, 2992, 8]

encode

This method performs the previously mentioned tokenize and convert_tokens_to_ids steps simultaneously, converting the input sentence directly into an array of token IDs.

tokenizer.encode("私は元気です。")

[2, 1325, 9, 12453, 2992, 8, 3]

Note that in the current version of the transformers library (v4.11.3), the encode method automatically inserts special tokens at the beginning and end of the array by default, controlled by the add_special_tokens option. (This is the recommended behavior as it matches how the language models were pre-trained.)
In the case of BERT, the special tokens are [CLS] at the beginning and [SEP] at the end, which correspond to IDs 2 and 3, respectively.
In older versions, this behavior is not enabled by default, so you need to include the argument add_special_tokens=True.

# For older versions of transformers
tokenizer.encode("私は元気です。", add_special_tokens=True)

encode_plus

In addition to the encode method described earlier, this outputs other IDs required for the language model input. For BERT, it outputs token_type_ids and attention_mask along with the IDs.

tokenizer.encode_plus("私は元気です。")

{'input_ids': [2, 1325, 9, 12453, 2992, 8, 3], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

You can generally think of token_type_ids as 0 unless you are performing specific fine-tuning.
The attention_mask is important for defining the actual length of the tokens. When using a language model, it is common to set a fixed maximum number of tokens, but since the input sentence doesn't always occupy the entire length, the mask ensures that the model does not apply attention to the unused portions.

tokenizer.encode_plus("私は元気です。", padding='max_length', max_length=16)

{'input_ids': [2, 1325, 9, 12453, 2992, 8, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]}

In the example above, you can see that the attention_mask is set to 1 only for the parts where actual tokens exist.

Also, setting the return_tensors='pt' option in encode_plus is convenient because it returns the data in a format that can be directly fed into the language model.

from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")

inputs = tokenizer.encode_plus("私はとても[MASK]です。", return_tensors='pt')
outputs = model(**inputs)
tokenizer.convert_ids_to_tokens(outputs.logits[0][1:-1].argmax(axis=-1))

['私', 'は', 'とても', '幸せ', 'です', '。']

batch_encode_plus

This is convenient when you want to process multiple sentences at once.

from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")

sentences = [
  "私はとても[MASK]です。", 
  "昨日とても[MASK]なことがあった。", 
  "それはとても[MASK]な食べ物だね！"
]

inputs = tokenizer.batch_encode_plus(sentences, padding='max_length', max_length=16, return_tensors='pt')
outputs = model(**inputs)

for i in range(3):
    print(tokenizer.convert_ids_to_tokens(outputs.logits[i][1:-1].argmax(axis=-1)))

['私', 'は', 'とても', '幸せ', 'です', '。', '。', '」', '」', '」', 'そして', 'そう', 'そう', '。']
['昨', '##日', 'とても', '大変', 'な', 'こと', 'が', 'あっ', 'た', '。', '。', '残念', '残念', '」']
['それ', 'は', 'とても', '美味', 'な', '食べ物', 'だ', 'ね', '!', '。', '!', '大好き', '大好き', '!']