whisper.cppでlarge-v3を動かす with Metal

./models/generate-coreml-model.sh large
Traceback (most recent call last):
  File "/Users/〇〇/Documents/GitHub/whisper.cpp/models/convert-whisper-to-coreml.py", line 306, in <module>
    raise ValueError("Invalid model name")
ValueError: Invalid model name
coremlc: error: Model does not exist at models/coreml-encoder-large.mlpackage -- file:///Users/〇〇/Documents/GitHub/whisper.cpp/
mv: rename models/coreml-encoder-large.mlmodelc to models/ggml-large-encoder.mlmodelc: No such file or directory

あっ、ふーん

ミツヒコ・イクルミ

対応してそうなIssue見つけた

ミツヒコ・イクルミ

./models/generate-coreml-model.sh large-v3

動きました、エイリアスなわけじゃないんだね

ミツヒコ・イクルミ

make large-v3

でいいです、binがほしいだけなのだ

ミツヒコ・イクルミ

whisper_init_state: loading Core ML model from 'models/ggml-large-v3-encoder.mlmodelc' whisper_init_state: first run on a device may take a while ... whisper_init_state: failed to load Core ML model from 'models/ggml-large-v3-encoder.mlmodelc' ggml_metal_free: deallocating error: failed to initialize whisper context
DL失敗している

ミツヒコ・イクルミ

ナウいデータセットはここらしい

ミツヒコ・イクルミ

だいたい理解した
CoreMLサポートを受けるには
./models/download-ggml-model.sh large-v3 ./models/generate-coreml-model.sh large-v3
なんだと思われる
ただ使うだけなら
./models/download-ggml-model.sh large-v3 ./main -m models/ggml-large-v3.bin -f samples/jfk.wav
で動作確認できる

ミツヒコ・イクルミ

実際動かしてみるとこんななので、普通に動かすだけだとCoreMLが動いてないことはわかる

> ./main -m models/ggml-large-v3.bin -f samples/jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: loading '/Users/〇〇/Documents/GitHub/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =  3117.88 MB, ( 3118.53 / 51539.61)
whisper_model_load:    Metal buffer size =  3117.87 MB
whisper_model_load: model size    = 3117.39 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: loading '/Users/〇〇/Documents/GitHub/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =   220.20 MB, ( 3338.73 / 51539.61)
whisper_init_state: kv self size  =  220.20 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =   245.76 MB, ( 3584.49 / 51539.61)
whisper_init_state: kv cross size =  245.76 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.51 / 51539.61)
whisper_init_state: compute buffer (conv)   =   32.36 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.52 / 51539.61)
whisper_init_state: compute buffer (encode) =  212.36 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.54 / 51539.61)
whisper_init_state: compute buffer (cross)  =    9.32 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.56 / 51539.61)
whisper_init_state: compute buffer (decode) =   99.17 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =    30.72 MB, ( 3615.28 / 51539.61)
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =   210.73 MB, ( 3826.01 / 51539.61)
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     7.68 MB, ( 3833.69 / 51539.61)
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =    97.53 MB, ( 3931.23 / 51539.61)

system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 | 

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.300 --> 00:00:09.180]   And so, my fellow Americans, ask not what your country can do for you, ask what you
[00:00:09.180 --> 00:00:11.000]   can do for your country.


whisper_print_timings:     load time =  2202.78 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     9.56 ms
whisper_print_timings:   sample time =    40.29 ms /   147 runs (    0.27 ms per run)
whisper_print_timings:   encode time =   992.49 ms /     1 runs (  992.49 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time =   985.82 ms /   145 runs (    6.80 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  4238.08 ms
ggml_metal_free: deallocating
ggml_metal_free: deallocating```

ミツヒコ・イクルミ

./models/generate-coreml-model.sh large-v3
動かしてるので、結果はそのうち。でかすぎてしばらく終わらんと思う

ミツヒコ・イクルミ

もう終わった！！！
実行結果はこんな感じ、CoreMLが有効になってるらしいことがわかる

> ./main -m models/ggml-large-v3.bin -f samples/jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: loading '/Users/〇〇/Documents/GitHub/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =  3117.88 MB, ( 3118.53 / 51539.61)
whisper_model_load:    Metal buffer size =  3117.87 MB
whisper_model_load: model size    = 3117.39 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: loading '/Users/〇〇/Documents/GitHub/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =   220.20 MB, ( 3338.73 / 51539.61)
whisper_init_state: kv self size  =  220.20 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =   245.76 MB, ( 3584.49 / 51539.61)
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: loading Core ML model from 'models/ggml-large-v3-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.51 / 51539.61)
whisper_init_state: compute buffer (conv)   =   10.85 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.52 / 51539.61)
whisper_init_state: compute buffer (cross)  =    9.32 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     0.02 MB, ( 3584.54 / 51539.61)
whisper_init_state: compute buffer (decode) =   99.17 MB
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     9.22 MB, ( 3593.76 / 51539.61)
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =     7.68 MB, ( 3601.45 / 51539.61)
ggml_metal_add_buffer: allocated 'backend         ' buffer, size =    97.53 MB, ( 3698.98 / 51539.61)

system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 1 | OPENVINO = 0 | 

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.300 --> 00:00:09.180]   And so, my fellow Americans, ask not what your country can do for you, ask what you
[00:00:09.180 --> 00:00:11.000]   can do for your country.


whisper_print_timings:     load time =  1125.50 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     7.79 ms
whisper_print_timings:   sample time =    41.47 ms /   147 runs (    0.28 ms per run)
whisper_print_timings:   encode time =  2558.57 ms /     1 runs ( 2558.57 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time =  1001.30 ms /   145 runs (    6.91 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  7439.88 ms
ggml_metal_free: deallocating
ggml_metal_free: deallocating

ミツヒコ・イクルミ

今更だけどモデルのビルドに15分とかしかかからなかったので、なんかとてもはやなっていることがわかる
マシンスペックはM1 Maxのメモリ64GBなのでM2以降ならもっといいのかもしれない

このスクラップは2023/11/23にクローズされました