👩‍💻

# 言語処理100本ノック 2020 (Rev 2) 第6章: 機械学習 56. 適合率，再現率，F1スコアの計測

2023/01/14に公開約4,700字

### 問題

#### 56. 適合率，再現率，F1スコアの計測

52で学習したロジスティック回帰モデルの適合率，再現率，F1スコアを，評価データ上で計測せよ．カテゴリごとに適合率，再現率，F1スコアを求め，カテゴリごとの性能をマイクロ平均（micro-average）とマクロ平均（macro-average）で統合せよ．

solution56.py
``````import pandas as pd
import numpy as np
import re
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score, f1_score

def preprocessing(data):
x = []
y = []
label = {"b":0, "e":1, "t":2, "m":3}

for title, category in data:
title = re.sub("[0-9]+", "0", title)
x.append(title.lower())
y.append(label[category])

return x,y

def score(model, X):
pred = model.predict([X])
pred_proba = model.predict_proba([X])[0, pred]
return pred[0], pred_proba[0]

def calc(Y, pred):
ppv = precision_score(Y, pred, average=None)
ppv_micro = precision_score(Y, pred, average="micro").reshape(1)
ppv_macro = precision_score(Y, pred, average="macro").reshape(1)
ppv = np.concatenate([ppv, ppv_micro, ppv_macro])

recall = recall_score(Y, pred, average=None)
recall_micro = recall_score(Y, pred, average="micro").reshape(1)
recall_macro = recall_score(Y, pred, average="macro").reshape(1)
recall = np.concatenate([recall, recall_micro, recall_macro])

f1 = f1_score(Y, pred, average=None)
f1_micro = f1_score(Y, pred, average="micro").reshape(1)
f1_macro = f1_score(Y, pred, average="macro").reshape(1)
f1 = np.concatenate([f1, f1_micro, f1_macro])

index = ["0", "1", "2", "3", "micro", "macro"]
scores = pd.DataFrame({"ppv": ppv, "recall": recall, "f1":f1}, index=index)

return scores

np.random.seed(123)
df = pd.read_csv("chapter06/NewsAggregatorDataset/newsCorpora.csv", header=None, sep="\t", names=["ID", "TITLE", "URL", "PUBLISHER", "CATEGORY", "STORY", "HOSTNAME", "TIMESTAMP"])
df = df.loc[df["PUBLISHER"].isin(["Reuters", "Huffington Post", "Businessweek", "Contactmusic.com", "Daily Mail"]), ["TITLE", "CATEGORY"]]

train, valid_test = train_test_split(df, test_size=0.2, shuffle=True)
valid, test = train_test_split(valid_test, test_size=0.5, shuffle=True)

train = np.array(train)
valid = np.array(valid)
test = np.array(test)

X_train, Y_train = preprocessing(train)
X_valid, Y_valid = preprocessing(valid)
X_test, Y_test = preprocessing(test)

tfidfvectorizer = TfidfVectorizer(min_df=0.001)

X_train = tfidfvectorizer.fit_transform(X_train).toarray()
X_valid = tfidfvectorizer.transform(X_valid).toarray()
X_test = tfidfvectorizer.transform(X_test).toarray()

model = LogisticRegression()
model.fit(X_train, Y_train)

train_pred = []
test_pred = []

for X in X_train:
train_pred.append(score(model, X)[0])

for X in X_test:
test_pred.append(score(model, X)[0])

print(calc(Y_test, test_pred))
``````
output
``````            ppv    recall        f1
0      0.885572  0.930314  0.907392
1      0.861060  0.949597  0.903164
2      0.808333  0.584337  0.678322
3      0.859375  0.561224  0.679012
micro  0.867316  0.867316  0.867316
macro  0.853585  0.756368  0.791972
``````

calc関数では、与えられた正解ラベル`Y`と予測ラベル`pred`に対して、`Precision``Recall``F1`スコアを計算し、それらをまとめたデータフレームを返すことです。

`ppv(Precision)`を計算するために、`precision_score(Y, pred, average=None)`を使用し、各クラスごとの`Precision`を計算します。`ppv(Precision)`のマイクロ平均を計算するために、`precision_score(Y, pred, average="micro").reshape(1)`を使用し、マイクロ平均を計算します。`ppv(Precision)`のマクロ平均を計算するために、`precision_score(Y, pred, average="macro").reshape(1)`を使用し、マクロ平均を計算します。

ログインするとコメントできます