マルチリンガルなEmbeddingモデル Voyage AI "voyage-multilingual-2"・upstage "solar-embedding-1-large"を評価してみる。

#    Model                                   Hit Rate@3    MRR@3
---  --------------------------------------  ------------  ----------
a    amazon-titan-embeddings-g1              0.745         0.645
b    baai-bge-m3                             0.853ᵃʰ       0.782ᵃᵍʰ
c    cohere-embed-multilingual-v3.0          0.856ᵃᵍʰ      0.779ᵃʰ
d    infloat-multilingual-e5-large-instruct  0.866ᵃᵍʰ      0.783ᵃʰ
e    infloat-multilingual-e5-large           0.866ᵃᵍʰ      0.801ᵃᶜᵍʰ
f    openai-text-embedding-3-large           0.870ᵃᵍʰ      0.798ᵃᵍʰ
g    openai-text-embedding-3-small           0.825ᵃʰ       0.750ᵃʰ
h    openai-text-embedding-ada-002           0.792ᵃ        0.712ᵃ
i    solar-embedding-1-large                 0.869ᵃᵍʰ      0.809ᵃᵇᶜᵍʰ
j    voyage-multilingual-2                   0.888ᵃᵇᶜᵍʰ    0.813ᵃᵇᶜᵍʰ

===== @5 =====

#    Model                                   Hit Rate@5    MRR@5
---  --------------------------------------  ------------  ----------
a    amazon-titan-embeddings-g1              0.799         0.658
b    baai-bge-m3                             0.882ᵃʰ       0.789ᵃᵍʰ
c    cohere-embed-multilingual-v3.0          0.897ᵃᵍʰ      0.788ᵃʰ
d    infloat-multilingual-e5-large-instruct  0.903ᵃᵍʰ      0.792ᵃᵍʰ
e    infloat-multilingual-e5-large           0.902ᵃᵍʰ      0.810ᵃᶜᵍʰ
f    openai-text-embedding-3-large           0.906ᵃᵍʰ      0.806ᵃᵍʰ
g    openai-text-embedding-3-small           0.866ᵃ        0.759ᵃʰ
h    openai-text-embedding-ada-002           0.849ᵃ        0.726ᵃ
i    solar-embedding-1-large                 0.903ᵃᵍʰ      0.817ᵃᵇᶜᵍʰ
j    voyage-multilingual-2                   0.918ᵃᵇᵍʰ     0.820ᵃᵇᶜᵍʰ

===== @10 =====

#    Model                                   Hit Rate@10    MRR@10
---  --------------------------------------  -------------  -----------
a    amazon-titan-embeddings-g1              0.860          0.666
b    baai-bge-m3                             0.914ᵃʰ        0.793ᵃᵍʰ
c    cohere-embed-multilingual-v3.0          0.924ᵃᵍʰ       0.792ᵃʰ
d    infloat-multilingual-e5-large-instruct  0.935ᵃᵍʰ       0.796ᵃᵍʰ
e    infloat-multilingual-e5-large           0.938ᵃᵇᵍʰ      0.815ᵃᵇᶜᵍʰ
f    openai-text-embedding-3-large           0.935ᵃᵍʰ       0.810ᵃᵍʰ
g    openai-text-embedding-3-small           0.900ᵃ         0.764ᵃʰ
h    openai-text-embedding-ada-002           0.881          0.730ᵃ
i    solar-embedding-1-large                 0.923ᵃᵍʰ       0.820ᵃᵇᶜᵍʰ
j    voyage-multilingual-2                   0.953ᵃᵇᶜᵍʰᶦ    0.824ᵃᵇᶜᵈᵍʰ