Closed9
CloudWatch Evidently の EvaluateFeature と BatchEvaluateFeature の比較
michimaniEvaluateFeature
- project と
entityId, featureのセットを一つを指定して value を取得 - quota は 200 TPS (transactions per second)
- EvaluateFeature - CloudWatch Evidently
BatchEvaluateFeature
- project と
entityId, featureのセットを複数指定してそれぞれの value をリストで取得 - quota は 50 TPS
- BatchEvaluateFeature - CloudWatch Evidently
michimaniひとつの EntityID に対して複数の Feature の value を取得するような場合の選択肢
- 各 Feature に対して EvaluateFeature API を叩いて value を取得
- BatchEvaluateFeature にすべての Feature の情報を渡して value のリストを取得
michimaniそれぞれの API の quota を考えると、 Feature の数が 5 個以上であれば BatchEvaluateFeature を使ったほうが上限に達しにくいはず。
- 横が Feature の数
- 縦が使用する API
- その組み合わせで、 quota 内ですべての Feature に対する value を取得できる EntityID の数
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
|---|---|---|---|---|---|---|---|---|---|---|
| EvaluateFeature | 200 | 100 | 66 | 50 | 40 | 33 | 28 | 25 | 22 | 20 |
| BatchEvaluateFeature | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 |
| EvaluateFeature BatchEvaluateFeature 使い分け |
200 | 100 | 66 | 50 | 50 | 50 | 50 | 50 | 50 | 50 |
michimani使い分けるのが良さそう、というのは正しそう。それぞれでどれくらい処理時間に差が出るか。
michimaniListed features. count: 10
---- get 10 features by EvaluateFeature API ----
feature: feature-1, value: {'boolValue': True}
feature: feature-2, value: {'boolValue': True}
feature: feature-3, value: {'boolValue': True}
feature: feature-4, value: {'boolValue': True}
feature: feature-5, value: {'boolValue': True}
feature: feature-6, value: {'boolValue': True}
feature: feature-7, value: {'boolValue': True}
feature: feature-8, value: {'boolValue': True}
feature: feature-9, value: {'boolValue': True}
feature: feature-10, value: {'boolValue': True}
elapsed: 0.16936558300221805
---- get 10 features by BatchEvaluateFeature API ----
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-1, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-2, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-3, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-4, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-5, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-6, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-7, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-8, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-9, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-10, value: {'boolValue': True}
elapsed: 0.014566375000867993
BatchEvaluateFeature めっちゃはやい。ほんまか？
michimani処理順逆にしてもやっぱり BatchEvaluateFeature のほうがはやい。
Listed features. count: 10
---- get 10 features by BatchEvaluateFeature API ----
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-1, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-2, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-3, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-4, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-5, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-6, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-7, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-8, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-9, value: {'boolValue': True}
feature: arn:aws:evidently:ap-northeast-1:699028004073:project/ef-vs-bef/feature/feature-10, value: {'boolValue': True}
elapsed: 0.05244791699806228
---- get 10 features by EvaluateFeature API ----
feature: feature-1, value: {'boolValue': True}
feature: feature-2, value: {'boolValue': True}
feature: feature-3, value: {'boolValue': True}
feature: feature-4, value: {'boolValue': True}
feature: feature-5, value: {'boolValue': True}
feature: feature-6, value: {'boolValue': True}
feature: feature-7, value: {'boolValue': True}
feature: feature-8, value: {'boolValue': True}
feature: feature-9, value: {'boolValue': True}
feature: feature-10, value: {'boolValue': True}
elapsed: 0.10206466600357089
10 回ずつ実行してみた。
Listed features. count: 10
---- get 10 features by EvaluateFeature API ----
elapsed: 0.1720401669954299
elapsed: 0.12039266699866857
elapsed: 0.12361987500480609
elapsed: 0.12408404199959477
elapsed: 0.1212201250018552
elapsed: 0.11948174999997718
elapsed: 0.10677008399943588
elapsed: 0.11592154199752258
elapsed: 0.12358333299926016
elapsed: 0.12295774999802234
---- get 10 features by BatchEvaluateFeature API ----
elapsed: 0.0610867089999374
elapsed: 0.016109041003801394
elapsed: 0.014233082998543978
elapsed: 0.016069457997218706
elapsed: 0.016354667000996415
elapsed: 0.015939832999720238
elapsed: 0.014508166997984517
elapsed: 0.014131541000097059
elapsed: 0.013616000003821682
elapsed: 0.014615457999752834
概ね BatchEvaluateFeature のほうが EvaluateFeature をループで回すより 10 倍はやい。
michimaniコードはこちらで。
このスクラップは2023/10/09にクローズされました