<p>OpenAIの高性能な音声認識モデルであるWhisperを、オフラインでかつGPUが無くても簡単に試せるようにしてくれたリポジトリを知ったのでご紹介。<br style="display:none">
<span class="embed-block zenn-embedded zenn-embedded-card"><iframe id="zenn-embedded__5d29b06e214ad" src="https://embed.zenn.studio/card#zenn-embedded__5d29b06e214ad" data-content="https%3A%2F%2Fgithub.com%2Fggerganov%2Fwhisper.cpp" frameborder="0" scrolling="no" loading="lazy"></iframe></span><a href="https://github.com/ggerganov/whisper.cpp" style="display:none" target="_blank" rel="nofollow noopener noreferrer">https://github.com/ggerganov/whisper.cpp</a></p>
<h2 id="%E7%A7%81%E3%81%AE%E7%92%B0%E5%A2%83">
<a class="header-anchor-link" href="#%E7%A7%81%E3%81%AE%E7%92%B0%E5%A2%83" aria-hidden="true"></a> 私の環境</h2>
<ul>
<li>Mac mini (M1, 2020)</li>
<li>Chip Apple M1</li>
<li>Memory 16 GB</li>
<li>macOS Monterey Version 12.6</li>
</ul>
<h2 id="%E8%A9%A6%E3%81%97%E3%81%A6%E3%81%BF%E3%82%8B">
<a class="header-anchor-link" href="#%E8%A9%A6%E3%81%97%E3%81%A6%E3%81%BF%E3%82%8B" aria-hidden="true"></a> 試してみる</h2>
<h3 id="%E9%9F%B3%E5%A3%B0%E3%83%87%E3%83%BC%E3%82%BF%E3%81%8B%E3%82%89%E6%96%87%E5%AD%97%E8%B5%B7%E3%81%93%E3%81%97">
<a class="header-anchor-link" href="#%E9%9F%B3%E5%A3%B0%E3%83%87%E3%83%BC%E3%82%BF%E3%81%8B%E3%82%89%E6%96%87%E5%AD%97%E8%B5%B7%E3%81%93%E3%81%97" aria-hidden="true"></a> 音声データから文字起こし</h3>
<h4 id="%E3%83%AA%E3%83%9D%E3%82%B8%E3%83%88%E3%83%AA%E3%82%92%E3%82%AF%E3%83%AD%E3%83%BC%E3%83%B3">
<a class="header-anchor-link" href="#%E3%83%AA%E3%83%9D%E3%82%B8%E3%83%88%E3%83%AA%E3%82%92%E3%82%AF%E3%83%AD%E3%83%BC%E3%83%B3" aria-hidden="true"></a> リポジトリをクローン</h4>
<div class="code-block-container"><pre><code>git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
</code></pre></div><h4 id="%E3%83%A2%E3%83%87%E3%83%AB%E3%82%92%E3%83%80%E3%82%A6%E3%83%B3%E3%83%AD%E3%83%BC%E3%83%89">
<a class="header-anchor-link" href="#%E3%83%A2%E3%83%87%E3%83%AB%E3%82%92%E3%83%80%E3%82%A6%E3%83%B3%E3%83%AD%E3%83%BC%E3%83%89" aria-hidden="true"></a> モデルをダウンロード</h4>
<p>tiny, base, small, medium, large とあり、左から順に精度が上がるがサイズと使用するメモリの量が上がる。詳細はこちらへ <a href="https://github.com/ggerganov/whisper.cpp#memory-usage" target="_blank" rel="nofollow noopener noreferrer">https://github.com/ggerganov/whisper.cpp#memory-usage</a></p>
<div class="code-block-container"><pre><code>bash ./models/download-ggml-model.sh base
</code></pre></div><h4 id="%E3%83%93%E3%83%AB%E3%83%89">
<a class="header-anchor-link" href="#%E3%83%93%E3%83%AB%E3%83%89" aria-hidden="true"></a> ビルド</h4>
<div class="code-block-container"><pre><code>make
</code></pre></div><h4 id="%E5%AE%9F%E8%A1%8C%E3%81%99%E3%82%8B">
<a class="header-anchor-link" href="#%E5%AE%9F%E8%A1%8C%E3%81%99%E3%82%8B" aria-hidden="true"></a> 実行する</h4>
<aside class="msg message"><span class="msg-symbol">!</span><div class="msg-content">
<p>読み込める音声ファイルが現在16ビットのWAVファイルのみであることに注意。<br>
必要に応じて次のようなコマンドなどで変換して下さい。</p>
<div class="code-block-container"><pre class="language-shell"><code class="language-shell">ffmpeg <span class="token parameter variable">-i</span> input.mp3 <span class="token parameter variable">-ar</span> <span class="token number">16000</span> <span class="token parameter variable">-ac</span> <span class="token number">1</span> <span class="token parameter variable">-c:a</span> pcm_s16le output.wav
</code></pre></div>
</div></aside>
<div class="code-block-container"><pre><code>./main -l ja -f ~/Music/output.wav -m models/ggml-base.bin
</code></pre></div><table>
<thead>
<tr>
<th>オプション</th>
<th>説明</th>
</tr>
</thead>
<tbody>
<tr>
<td>l</td>
<td>認識する言語</td>
</tr>
<tr>
<td>f</td>
<td>読み込むファイル</td>
</tr>
<tr>
<td>m</td>
<td>使用するモデル</td>
</tr>
</tbody>
</table>
<p>これだけで認識できちゃいます。<br>
ちなみに、<code>-tr</code>というオプションもあり、これを入れると英語に翻訳してくれます（すごい）</p>
<details><summary>出力</summary><div class="details-content"><div class="code-block-container"><pre class="language-shell"><code class="language-shell">whisper_model_load: loading model from <span class="token string">'models/ggml-base.bin'</span>
whisper_model_load: n_vocab       <span class="token operator">=</span> <span class="token number">51865</span>
whisper_model_load: n_audio_ctx   <span class="token operator">=</span> <span class="token number">1500</span>
whisper_model_load: n_audio_state <span class="token operator">=</span> <span class="token number">512</span>
whisper_model_load: n_audio_head  <span class="token operator">=</span> <span class="token number">8</span>
whisper_model_load: n_audio_layer <span class="token operator">=</span> <span class="token number">6</span>
whisper_model_load: n_text_ctx    <span class="token operator">=</span> <span class="token number">448</span>
whisper_model_load: n_text_state  <span class="token operator">=</span> <span class="token number">512</span>
whisper_model_load: n_text_head   <span class="token operator">=</span> <span class="token number">8</span>
whisper_model_load: n_text_layer  <span class="token operator">=</span> <span class="token number">6</span>
whisper_model_load: n_mels        <span class="token operator">=</span> <span class="token number">80</span>
whisper_model_load: f16           <span class="token operator">=</span> <span class="token number">1</span>
whisper_model_load: <span class="token builtin class-name">type</span>          <span class="token operator">=</span> <span class="token number">2</span>
whisper_model_load: adding <span class="token number">1608</span> extra tokens
whisper_model_load: mem_required  <span class="token operator">=</span>  <span class="token number">506.00</span> MB
whisper_model_load: ggml ctx size <span class="token operator">=</span>  <span class="token number">140.60</span> MB
whisper_model_load: memory size   <span class="token operator">=</span>   <span class="token number">22.83</span> MB
whisper_model_load: model size    <span class="token operator">=</span>  <span class="token number">140.54</span> MB

system_info: n_threads <span class="token operator">=</span> <span class="token number">4</span> / <span class="token number">8</span> <span class="token operator">|</span> AVX <span class="token operator">=</span> <span class="token number">0</span> <span class="token operator">|</span> AVX2 <span class="token operator">=</span> <span class="token number">0</span> <span class="token operator">|</span> AVX512 <span class="token operator">=</span> <span class="token number">0</span> <span class="token operator">|</span> NEON <span class="token operator">=</span> <span class="token number">1</span> <span class="token operator">|</span> FP16_VA <span class="token operator">=</span> <span class="token number">1</span> <span class="token operator">|</span> WASM_SIMD <span class="token operator">=</span> <span class="token number">0</span> <span class="token operator">|</span> BLAS <span class="token operator">=</span> <span class="token number">1</span> <span class="token operator">|</span>

main: processing <span class="token string">'/Users/kaisei/Music/output.wav'</span> <span class="token punctuation">(</span><span class="token number">96597</span> samples, <span class="token number">6.0</span> sec<span class="token punctuation">)</span>, <span class="token number">4</span> threads, <span class="token number">1</span> processors, lang <span class="token operator">=</span> ja, task <span class="token operator">=</span> transcribe, timestamps <span class="token operator">=</span> <span class="token number">1</span> <span class="token punctuation">..</span>.


<span class="token punctuation">[</span>00:00:00.000 --<span class="token operator">&gt;</span> 00:00:05.740<span class="token punctuation">]</span>  音声認識テスト中


whisper_print_timings:     load <span class="token function">time</span> <span class="token operator">=</span>   <span class="token number">121.41</span> ms
whisper_print_timings:      mel <span class="token function">time</span> <span class="token operator">=</span>    <span class="token number">13.86</span> ms
whisper_print_timings:   sample <span class="token function">time</span> <span class="token operator">=</span>     <span class="token number">1.96</span> ms
whisper_print_timings:   encode <span class="token function">time</span> <span class="token operator">=</span>   <span class="token number">371.95</span> ms / <span class="token number">61.99</span> ms per layer
whisper_print_timings:   decode <span class="token function">time</span> <span class="token operator">=</span>    <span class="token number">35.89</span> ms / <span class="token number">5.98</span> ms per layer
whisper_print_timings:    total <span class="token function">time</span> <span class="token operator">=</span>   <span class="token number">545.44</span> ms
</code></pre></div></div></details>
<h3 id="%E3%83%AA%E3%82%A2%E3%83%AB%E3%82%BF%E3%82%A4%E3%83%A0%E9%9F%B3%E5%A3%B0%E8%AA%8D%E8%AD%98">
<a class="header-anchor-link" href="#%E3%83%AA%E3%82%A2%E3%83%AB%E3%82%BF%E3%82%A4%E3%83%A0%E9%9F%B3%E5%A3%B0%E8%AA%8D%E8%AD%98" aria-hidden="true"></a> リアルタイム音声認識</h3>
<p>リアルタイムでの音声認識も簡単にできました。</p>
<h4 id="%E5%BF%85%E8%A6%81%E3%81%AA%E3%82%82%E3%81%AE">
<a class="header-anchor-link" href="#%E5%BF%85%E8%A6%81%E3%81%AA%E3%82%82%E3%81%AE" aria-hidden="true"></a> 必要なもの</h4>
<p>sdlというものが必要らしいので、パッケージマネージャで入れる。</p>
<div class="code-block-container"><pre><code>brew install sdl2
</code></pre></div><h4 id="%E3%83%93%E3%83%AB%E3%83%89-1">
<a class="header-anchor-link" href="#%E3%83%93%E3%83%AB%E3%83%89-1" aria-hidden="true"></a> ビルド</h4>
<div class="code-block-container"><pre><code>make stream
</code></pre></div><h4 id="%E5%AE%9F%E8%A1%8C%E3%81%99%E3%82%8B-1">
<a class="header-anchor-link" href="#%E5%AE%9F%E8%A1%8C%E3%81%99%E3%82%8B-1" aria-hidden="true"></a> 実行する</h4>
<div class="code-block-container"><pre><code>./stream -l ja  -m models/ggml-base.bin
</code></pre></div><details><summary>出力</summary><div class="details-content"><div class="code-block-container"><pre><code>audio_sdl_init: found 1 capture devices:
audio_sdl_init:    - Capture device #0: 'AT2020USB+'
audio_sdl_init: attempt to open default capture device ...
audio_sdl_init: obtained spec for input device (SDL Id = 2):
audio_sdl_init:     - sample rate:       16000
audio_sdl_init:     - format:            33056 (required: 33056)
audio_sdl_init:     - channels:          1 (required: 1)
audio_sdl_init:     - samples per frame: 1024
whisper_model_load: loading model from 'models/ggml-base.bin'
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 2
whisper_model_load: adding 1608 extra tokens
whisper_model_load: mem_required  =  506.00 MB
whisper_model_load: ggml ctx size =  140.60 MB
whisper_model_load: memory size   =   22.83 MB
whisper_model_load: model size    =  140.54 MB

main: processing 48000 samples (step = 3.0 sec / len = 10.0 sec), 4 threads, lang = ja, task = transcribe, timestamps = 0 ...
main: n_new_line = 2

日本語の音声にしき
サクランボ
こんにちは
コンバンは
GPUがなくてもオフラインで
簡単音声認識
</code></pre></div></div></details>
<p>とっても簡単に出来ちゃうのでぜひ試してみて下さい！</p>


【Whisper】GPUが無くてもオフラインで簡単音声認識

Discussion