iTranslated by AI
Benchmarking HashiCorp Vault with vault-benchmark
Target Audience
- Those who want to perform benchmarks on HashiCorp Vault.
Vault Benchmarking Tools
It seems there are various benchmarking tools available, but here we will use vault-benchmark, which was recently added to the tutorials, to run a benchmark.
Please note that while there is a tutorial on the HashiCorp Learn site, this tool is community-supported.
Running Benchmarks with vault-benchmark
0. Prerequisites
Gather the necessary components to run the benchmark.
vault server
You can use either Self-Hosted or HCP Vault, but avoid running benchmarks against a production Vault environment.
Here, as usual, we will run a dev server.
$ mkdir vault-benchmark-test
$ cd vault-benchmark-test
$ vault server -dev -dev-root-token-id root > vault-server.log 2>&1 &
# Vault settings
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ export VAULT_TOKEN=root
# Verify it is running correctly
$ vault status
If you do not have the Vault binary yet, please follow this guide to install it.
In the tests that follow, we are using the Enterprise binary, so some outputs may differ from the Community version.
vault-benchmark
Download it from here and place it in a convenient location.
Here, I will put it in the directory I set up for logs.
$ ls
vault-benchmark* vault-server.log
$ ./vault-benchmark version
vault-benchmark v0.2.0
1. Preparing the Configuration File
vault-benchmark configuration is done using HCL, similar to other HashiCorp products.
Referring to the Usage, we prepare a file like the following:
# Vault server configuration
vault_addr = "http://127.0.0.1:8200"
vault_token = "root"
vault_namespace="root"
# Benchmark measurement duration
duration = "30s"
# If this is not set to true, the artifacts used for the benchmark will remain in Vault.
cleanup = true
# AppRole Auth Method test
test "approle_auth" "approle_logins" {
weight = 50 # Execute approle_auth in 50% of the overall tests
config {
role {
role_name = "benchmark-role"
token_ttl="2m"
}
}
}
# KV v2 Secret Engine test
test "kvv2_write" "static_secret_writes" {
weight = 50 # Execute kvv2_write in 50% of the overall tests
config {
numkvs = 100 # Number of k/v pairs
kvsize = 100 # 100 bytes
}
}
In the example above, we measure benchmarks for AppRole and KVv2.
You do not need to prepare AppRole or KVv2 before running the benchmark; they are automatically created when the benchmark starts and deleted upon completion (if cleanup=true is set).
2. Running the Benchmark
Once the configuration file is ready, let's run it.
$ ./vault-benchmark run -config=config.hcl
2023-12-07T17:04:06.823+0900 [INFO] vault-benchmark: setting up targets
2023-12-07T17:04:08.881+0900 [INFO] vault-benchmark: starting benchmarks: duration=30s
2023-12-07T17:04:38.882+0900 [INFO] vault-benchmark: cleaning up targets
2023-12-07T17:04:55.063+0900 [INFO] vault-benchmark: benchmark complete
Target: http://127.0.0.1:8200
op count rate throughput mean 95th% 99th% successRatio
approle_logins 155550 5185.101815 5184.816465 1.432215ms 2.431935ms 5.580876ms 100.00%
static_secret_writes 155250 5174.983900 5174.899341 490.932µs 1.117813ms 1.979946ms 100.00%
3. Checking the Results
The output metrics are interpreted as follows:
- op: Name of the test
- count: Number of successful operations
- rate: Successes per second (count/duration)
- throughput: Number of successful tests within the rate
- mean: Average test time per operation (ms)
- 95th%/99th%: 95th and 99th percentiles, respectively
- successRatio: Percentage of successful operations
Summary
What did you think?
Although it is a simple method, it is useful to get an idea of how many requests can be handled easily.
Additionally, if you can also monitor the load on the infrastructure side (network, or CPU/storage IO for self-hosted setups), it helps in anticipating potential issues before deploying new use cases.
References
Vault benchmark testing tool
Benchmark Vault performance
(At this reference site, you can also try out vault-benchmark in your browser.)
Vault Benchmark Document
Discussion