Open7
kTLSの検証
goforbroke環境情報
Docker上でkTLSを動かす．
- ホスト
AMD Ryzen 7 PRO 5750GE
64GB Memory
% uname -a
Linux 5.15.0-79-generic #86~20.04.2-Ubuntu SMP Mon Jul 17 23:27:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
kTLSはホストのtlsモジュールが必要なため，入っていることを確認する．
% lsmod | grep tls
tls 114688 0
- Docker
DockerでkTLS検証用のimageがあるために利用する
ホストにtlsモジュールを入れる．
kTLSのモジュールが無いと警告が出る．
docker run --rm -it ktls_sendfile test_sslapi.sh
make depend && make _tests
make[1]: Entering directory '/work/openssl'
make[1]: Leaving directory '/work/openssl'
make[1]: Entering directory '/work/openssl'
( SRCTOP=. \
BLDTOP=. \
PERL="/usr/bin/perl" \
FIPSKEY="f4556650ac31d35461610bac4ed81b1a181b2d8a43ea2854cbae22ca74560813" \
EXE_EXT= \
/usr/bin/perl ./test/run_tests.pl test_sslapi )
00-prep_fipsmodule_cnf.t ..
# The results of this test will end up in test-runs/prep_fipsmodule
1..0 # SKIP FIPS module config file only supported in a fips build
skipped: FIPS module config file only supported in a fips build
Files=1, Tests=0, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.10 cusr 0.00 csys = 0.11 CPU)
Result: NOTESTS
90-test_sslapi.t ..
# The results of this test will end up in test-runs/test_sslapi
1..1
# Subtest: ../../test/sslapitest
1..1
# Subtest: test_ktls_sendfile
1..8
# SKIP: @ test/sslapitest.c:1296
# Kernel does not support KTLS
ok 1 - iteration 1 # skipped
# SKIP: @ test/sslapitest.c:1296
# Kernel does not support KTLS
ok 2 - iteration 2 # skipped
# SKIP: @ test/sslapitest.c:1296
# Kernel does not support KTLS
ok 3 - iteration 3 # skipped
# SKIP: @ test/sslapitest.c:1296
# Kernel does not support KTLS
ok 4 - iteration 4 # skipped
tlsモジュールを入れたら警告はでない．
% sudo modprobe tls
% lsmod | grep tls
tls 114688 0
事前にfil descriptorの上限も引き上げておく
% diff /etc/security/limits.conf*
55,56d54
< * soft nofile 65536
< * hard nofile 65536
kTLS有効化されているかのテスト
ktls_sendfileが呼ばれており，有効化されている．
% docker run --rm -it ktls_sendfile test_sslapi.sh
make depend && make _tests
make[1]: Entering directory '/work/openssl'
make[1]: Leaving directory '/work/openssl'
make[1]: Entering directory '/work/openssl'
( SRCTOP=. \
BLDTOP=. \
PERL="/usr/bin/perl" \
FIPSKEY="f4556650ac31d35461610bac4ed81b1a181b2d8a43ea2854cbae22ca74560813" \
EXE_EXT= \
/usr/bin/perl ./test/run_tests.pl test_sslapi )
00-prep_fipsmodule_cnf.t ..
# The results of this test will end up in test-runs/prep_fipsmodule
1..0 # SKIP FIPS module config file only supported in a fips build
skipped: FIPS module config file only supported in a fips build
Files=1, Tests=0, 0 wallclock secs ( 0.00 usr 0.00 sys + 0.09 cusr 0.01 csys = 0.10 CPU)
Result: NOTESTS
90-test_sslapi.t ..
# The results of this test will end up in test-runs/test_sslapi
1..1
# Subtest: ../../test/sslapitest
1..1
# Subtest: test_ktls_sendfile
1..6
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c16740, clientssl=0x5627e0c18500, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c16740.
DEBUG_KTLS: Skip ktls for s=0x5627e0c18500 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c18500 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c16740, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c16740 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c16740 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c16740, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c16740 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c16740 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: Skip ktls for s=0x5627e0c18500 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c18500 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c16740, wfd=5, fd=3, offset=0, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c16740, wfd=5, fd=3, offset=16384, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c16740, wfd=5, fd=3, offset=32768, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c16740, wfd=5, fd=3, offset=49152, size=16384, flags=0
ok 1 - iteration 1
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c263f0, clientssl=0x5627e0c29360, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c263f0.
DEBUG_KTLS: Skip ktls for s=0x5627e0c29360 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c29360 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c263f0, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c263f0 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c263f0 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c263f0, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c263f0 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c263f0 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: Skip ktls for s=0x5627e0c29360 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c29360 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c263f0, wfd=5, fd=3, offset=0, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c263f0, wfd=5, fd=3, offset=16384, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c263f0, wfd=5, fd=3, offset=32768, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c263f0, wfd=5, fd=3, offset=49152, size=16384, flags=0
ok 2 - iteration 2
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c29360, clientssl=0x5627e0c2b120, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c29360.
DEBUG_KTLS: Skip ktls for s=0x5627e0c2b120 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c2b120 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c29360, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c29360 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x21, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x0
DEBUG_KTLS: s=0x5627e0c29360 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: calling BIO_set_ktls s=0x5627e0c29360, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c29360 BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE) returned true, which=0x22, SSL3_CC_WRITE=0x2, which & SSL3_CC_WRITE=0x2
DEBUG_KTLS: s=0x5627e0c29360 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: Skip ktls for s=0x5627e0c2b120 because compressed or ktls disabled, s->compress=(nil), s->options=0x120000, disabled=1.
DEBUG_KTLS: s=0x5627e0c2b120 after s->statem.enc_write_state = ENC_WRITE_STATE_VALID.
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c29360, wfd=5, fd=3, offset=0, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c29360, wfd=5, fd=3, offset=16384, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c29360, wfd=5, fd=3, offset=32768, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c29360, wfd=5, fd=3, offset=49152, size=16384, flags=0
ok 3 - iteration 3
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c40440, clientssl=0x5627e0c42200, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c40440.
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40440, wfd=5, fd=3, offset=0, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40440, wfd=5, fd=3, offset=16384, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40440, wfd=5, fd=3, offset=32768, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40440, wfd=5, fd=3, offset=49152, size=16384, flags=0
ok 4 - iteration 4
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c40440, clientssl=0x5627e0c42200, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c40440.
DEBUG_KTLS: Skip ktls for s=0x5627e0c40440 NID_aes_128_ccm, TLS 1.3 or tag_length mismatch, (s->version==TLS_1_3_VERSION)=1.
# SKIP: @ test/sslapitest.c:1338
# Failed to enable KTLS for TLS 1.3 cipher TLS_AES_128_CCM_SHA256
ok 5 - iteration 5 # skipped
DEBUG_KTLS: after create_ssl_objects2, serverssl=0x5627e0c40f40, clientssl=0x5627e0c42d00, clientssl->options=0x120000
DEBUG_KTLS: SSL_set_options(serverssl, SSL_OP_ENABLE_KTLS) OK, serverssl->options=0x120108, SSL_OP_ENABLE_KTLS=0x8, serverssl=0x5627e0c40f40.
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40f40, wfd=5, fd=3, offset=0, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40f40, wfd=5, fd=3, offset=16384, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40f40, wfd=5, fd=3, offset=32768, size=16384, flags=0
DEBUG_KTLS: ktls_sendfile ret=16384, s=0x5627e0c40f40, wfd=5, fd=3, offset=49152, size=16384, flags=0
ok 6 - iteration 6
ok 1 - test_ktls_sendfile
../../util/wrap.pl ../../test/sslapitest ../../test/certs ../../test/recipes/90-test_sslapi_data/passwd.txt /tmp/NCupO7d_wZ default ../../test/default.cnf => 0
ok 1 - running sslapitest
ok
All tests successful.
Files=1, Tests=1, 1 wallclock secs ( 0.00 usr 0.00 sys + 0.20 cusr 0.15 csys = 0.35 CPU)
Result: PASS
make[1]: Leaving directory '/work/openssl'
READMEの通り実行してみる
nginxを起動します
% docker run --rm -it ktls_sendfile
別の端末でcurlを実行するとTLS 1.3で暗号スイートは TLS_AES_256_GCM_SHA384 が実行されています．
% docker exec -it $(docker ps -q) curl -sSkv -o /dev/null https://localhost
* Trying 127.0.0.1:443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [879 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=JP; ST=Osaka; L=Osaka City; CN=example.com
* start date: Aug 17 06:30:52 2023 GMT
* expire date: Aug 16 06:30:52 2024 GMT
* issuer: C=JP; ST=Osaka; L=Osaka City; CN=example.com
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} [5 bytes data]
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.4
< Date: Thu, 17 Aug 2023 07:39:03 GMT
< Content-Type: text/html
< Content-Length: 615
< Last-Modified: Thu, 17 Aug 2023 06:34:52 GMT
< Connection: keep-alive
< ETag: "64ddbf8c-267"
< Accept-Ranges: bytes
<
{ [5 bytes data]
* Connection #0 to host localhost left intact
*
サーバのログに kTLSのDebugログが呼び出されて実行できています．
2023/08/17 07:38:58 [notice] 1#0: using the "epoll" event method
2023/08/17 07:38:58 [notice] 1#0: nginx/1.21.4
2023/08/17 07:38:58 [notice] 1#0: built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
2023/08/17 07:38:58 [notice] 1#0: OS: Linux 5.15.0-79-generic
2023/08/17 07:38:58 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/08/17 07:38:58 [notice] 1#0: start worker processes
2023/08/17 07:38:58 [notice] 1#0: start worker process 7
2023/08/17 07:39:03 [notice] 7#0: *1 ngx_ssl_handshake: calling BIO_get_ktls_send() while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443
2023/08/17 07:39:03 [notice] 7#0: *1 ngx_ssl_handshake: BIO_get_ktls_send(): 1 while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443
DEBUG_KTLS: ktls_sendfile ret=615, s=0x5561060926c0, wfd=3, fd=10, offset=0, size=615, flags=0
Linuxのkernel TLSでnginxのSSL_sendfileを試してみたの通りOpenSSL側のsend_fileも出ているので動作しています．
goforbroke検証準備
外部公開する
負荷をかけるためにDockerfile に外部公開する変更を加えます．
合わせてスループットを計測するためのファイルを置きます．
diff Dockerfile Dockerfile.org
32,33d31
< RUN truncate -s 10m /usr/local/nginx/html/10M
< EXPOSE 443
8443ポートで外部公開します
docker build -t ktls_sendfile .
docker run -d -p 8443:8443 ktls_sendfile
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2d44e6decf19 ktls_sendfile "/usr/local/nginx/sb…" 10 seconds ago Up 10 seconds 443/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp youthful_pascal
8443ポートでアクセスできるようになりました
% curl -sSkv -o /dev/null https://localhost:8443/index.html
* Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [879 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=JP; ST=Osaka; L=Osaka City; CN=example.com
* start date: Aug 17 06:30:52 2023 GMT
* expire date: Aug 16 06:30:52 2024 GMT
* issuer: C=JP; ST=Osaka; L=Osaka City; CN=example.com
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} [5 bytes data]
> GET /index.html HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.21.4
< Date: Thu, 17 Aug 2023 08:01:09 GMT
< Content-Type: text/html
< Content-Length: 615
< Last-Modified: Thu, 17 Aug 2023 06:34:52 GMT
< Connection: keep-alive
< ETag: "64ddbf8c-267"
< Accept-Ranges: bytes
<
{ [5 bytes data]
* Connection #0 to host localhost left intact
vegeta で試しに負荷をかけると200が返って成功しています．
echo GET https://{your ip}:8443 | vegeta attack -insecure -duration=10s | vegeta report
Requests [total, rate, throughput] 500, 50.10, 50.09
Duration [total, attack, wait] 9.982s, 9.98s, 1.672ms
Latencies [min, mean, 50, 90, 95, 99, max] 927.625µs, 31.706ms, 2.123ms, 104.117ms, 260.461ms, 457.648ms, 481.053ms
Bytes In [total, mean] 307500, 615.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:500
Error Set:
kTLS無効化したDocker imageの作成
比較のためにkTLSの設定を無効化するDocker imageを作成します．
% diff Dockerfile Dockerfile.disable-ktls
29c29
< COPY nginx.conf /usr/local/nginx/conf/
---
> COPY nginx.conf.disable_ktls /usr/local/nginx/conf/nginx.conf
% diff nginx.conf nginx.conf.disable_ktls
108c108
< ssl_conf_command Options KTLS;
---
> #ssl_conf_command Options KTLS;
docker build -t disable_ktls_sendfile -f Dockerfile.disable-ktls .
並列で8444ポートで起動させておきます
docker run -d -p 8444:443 disable_ktls_sendfile
sendfileが使われていないことの確認
ホストに auditctl を導入して sendfile() が呼ばれていないことを確認します．
sudo auditctl -a always,exit -S sendfile -F arch=b64
sendfile() を使うと監査ログに残るため，無効化されているか確認できます．
% sudo ausearch -sc sendfile
----
time->Sat Aug 19 13:39:38 2023
type=PROCTITLE msg=audit(1692419978.122:144): proctitle=6E67696E783A20776F726B65722070726F63657373
type=SYSCALL msg=audit(1692419978.122:144): arch=c000003e syscall=40 success=yes exit=615 a0=3 a1=a a2=7fff29a7d120 a3=267 items=0 ppid=215446 pid=215447 auid=4294967295 uid=65534 gid=65534 euid=65534 suid=65534 fsuid=65534 egid=65534 sgid=65534 fsgid=65534 tty=pts0 ses=4294967295 comm="nginx" exe="/usr/local/nginx/sbin/nginx" subj=docker-default key=(null)
----
time->Sat Aug 19 13:39:57 2023
type=PROCTITLE msg=audit(1692419997.319:199): proctitle=6E67696E783A20776F726B65722070726F63657373
type=SYSCALL msg=audit(1692419997.319:199): arch=c000003e syscall=40 success=yes exit=615 a0=3 a1=a a2=7fff29a7d120 a3=267 items=0 ppid=215446 pid=215447 auid=4294967295 uid=65534 gid=65534 euid=65534 suid=65534 fsuid=65534 egid=65534 sgid=65534 fsgid=65534 tty=pts0 ses=4294967295 comm="nginx" exe="/usr/local/nginx/sbin/nginx" subj=docker-default key=(null)
----
time->Sat Aug 19 13:49:52 2023
type=PROCTITLE msg=audit(1692420592.156:1759): proctitle=6E67696E783A20776F726B65722070726F63657373
type=SYSCALL msg=audit(1692420592.156:1759): arch=c000003e syscall=40 success=yes exit=615 a0=3 a1=a a2=7ffd36b8cea0 a3=267 items=0 ppid=227302 pid=227303 auid=4294967295 uid=65534 gid=65534 euid=65534 suid=65534 fsuid=65534 egid=65534 sgid=65534 fsgid=65534 tty=pts0 ses=4294967295 comm="nginx" exe="/usr/local/nginx/sbin/nginx" subj=docker-default key=(null)
監査ログもパフォーマンスのペナルティとなるため sendfile() が呼び出されていることを調べたら，無効化する．
sudo auditctl -d always,exit -F arch=b64 -S sendfile
計測
:8443がkTLSで:8444が無効化したバージョン．
スループットに違いはない．
スループット
- kTLS有効
% echo GET https://{your ip}:8443 | vegeta attack -insecure -duration=5m -timeout=1m -rate=5000| vegeta report
Requests [total, rate, throughput] 1499996, 4999.99, 4999.95
Duration [total, attack, wait] 5m0s, 5m0s, 2.417ms
Latencies [min, mean, 50, 90, 95, 99, max] 167.683µs, 155.204ms, 11.114ms, 503.695ms, 578.517ms, 675.208ms, 816.476ms
Bytes In [total, mean] 922497540, 615.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:1499996
- kLTSなし
% echo GET https://{your ip}:8444 | vegeta attack -insecure -duration=5m -timeout=1m -rate=5000| vegeta report
Requests [total, rate, throughput] 1499996, 4999.99, 4999.95
Duration [total, attack, wait] 5m0s, 5m0s, 2.25ms
Latencies [min, mean, 50, 90, 95, 99, max] 186.443µs, 99.395ms, 2.368ms, 387.799ms, 463.372ms, 545.537ms, 750.165ms
Bytes In [total, mean] 922497540, 615.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:1499996
Error Set:
CPU使用率
- kTLS有効
22:06:53 CPU %user %nice %system %iowait %steal %idle
22:07:03 all 2.88 0.00 0.58 0.08 0.00 96.47
22:07:13 all 0.03 0.00 0.08 0.03 0.00 99.87
22:07:23 all 0.05 0.00 0.00 0.00 0.00 99.95
22:07:33 all 4.53 0.00 5.31 0.00 0.00 90.16
22:07:43 all 19.63 0.00 14.60 0.08 0.00 65.70
22:07:53 all 18.58 0.00 14.52 0.08 0.00 66.83
22:08:03 all 20.45 0.00 13.74 0.15 0.00 65.66
22:08:13 all 21.03 0.00 14.23 0.18 0.00 64.56
22:08:23 all 19.26 0.00 14.24 0.13 0.00 66.37
22:08:33 all 19.69 0.00 13.79 0.05 0.00 66.47
22:08:43 all 19.03 0.00 14.26 0.28 0.00 66.44
22:08:53 all 17.33 0.00 14.21 0.10 0.00 68.36
22:09:03 all 15.54 0.00 15.37 0.05 0.00 69.04
22:09:13 all 16.72 0.00 13.13 0.26 0.00 69.89
22:09:23 all 17.87 0.00 14.33 0.05 0.00 67.74
22:09:33 all 17.18 0.00 14.50 0.05 0.00 68.27
22:09:43 all 20.06 0.00 14.89 0.23 0.00 64.82
22:09:53 all 16.94 0.00 14.50 0.08 0.00 68.49
22:10:03 all 16.95 0.00 16.20 0.03 0.00 66.83
22:10:13 all 18.85 0.00 13.22 0.18 0.00 67.75
22:10:23 all 18.93 0.00 14.00 0.08 0.00 67.00
22:10:33 all 21.25 0.00 13.21 0.10 0.00 65.44
22:10:43 all 19.52 0.00 14.63 0.18 0.00 65.67
22:10:43 CPU %user %nice %system %iowait %steal %idle
22:10:53 all 16.45 0.00 12.46 0.08 0.00 71.01
22:11:03 all 20.71 0.00 14.51 0.05 0.00 64.73
22:11:13 all 21.89 0.00 14.78 0.15 0.00 63.18
22:11:23 all 21.07 0.00 14.85 0.05 0.00 64.03
22:11:33 all 18.72 0.00 14.81 0.10 0.00 66.36
22:11:43 all 15.24 0.00 14.62 0.16 0.00 69.99
22:11:53 all 16.19 0.00 14.48 0.08 0.00 69.26
22:12:03 all 17.02 0.00 14.10 0.03 0.00 68.85
22:12:13 all 20.25 0.00 16.71 0.23 0.00 62.81
22:12:23 all 16.68 0.00 15.52 0.08 0.00 67.72
22:12:33 all 11.22 0.00 10.35 0.08 0.00 78.35
22:12:43 all 0.43 0.00 0.00 0.00 0.00 99.57
- kTLSなし
2:15:23 all 0.05 0.00 0.00 0.03 0.00 99.92
22:15:33 all 3.06 0.00 2.74 0.02 0.00 94.17
22:15:43 all 11.67 0.00 6.07 0.05 0.00 82.21
22:15:53 all 15.98 0.00 6.32 0.05 0.00 77.65
22:16:03 all 16.79 0.00 6.40 0.07 0.00 76.74
22:16:13 all 18.20 0.00 6.76 0.10 0.00 74.94
22:16:23 all 17.01 0.00 6.33 0.05 0.00 76.61
22:16:33 all 17.33 0.00 6.52 0.08 0.00 76.07
22:16:43 all 17.22 0.00 7.26 0.03 0.00 75.49
22:16:53 all 17.27 0.00 6.10 0.03 0.00 76.61
22:17:03 all 15.70 0.00 6.15 0.12 0.00 78.03
22:17:13 all 17.43 0.00 6.49 0.03 0.00 76.06
22:17:23 all 18.18 0.00 6.58 0.10 0.00 75.15
22:17:33 all 18.97 0.00 6.47 0.08 0.00 74.48
22:17:43 all 20.30 0.00 7.78 0.12 0.00 71.80
22:17:53 all 17.24 0.00 6.60 0.07 0.00 76.08
22:18:03 all 17.68 0.00 7.29 0.05 0.00 74.98
22:18:13 all 17.38 0.00 7.47 0.10 0.00 75.04
22:18:23 all 16.35 0.00 6.77 0.02 0.00 76.85
22:18:23 CPU %user %nice %system %iowait %steal %idle
22:18:33 all 15.87 0.00 6.01 0.05 0.00 78.07
22:18:43 all 15.89 0.00 7.47 0.12 0.00 76.52
22:18:53 all 11.70 0.00 6.44 0.05 0.00 81.81
22:19:03 all 12.59 0.00 6.23 0.05 0.00 81.13
22:19:13 all 18.98 0.00 6.96 0.05 0.00 74.01
22:19:23 all 19.39 0.00 6.80 0.05 0.00 73.76
22:19:33 all 17.08 0.00 6.10 0.08 0.00 76.74
22:19:43 all 17.40 0.00 6.62 0.03 0.00 75.95
22:19:53 all 17.46 0.00 5.68 0.10 0.00 76.76
22:20:03 all 18.29 0.00 5.76 0.08 0.00 75.87
22:20:13 all 16.86 0.00 6.87 0.05 0.00 76.22
22:20:23 all 15.23 0.00 5.80 0.08 0.00 78.90
22:20:33 all 5.45 0.00 3.14 0.03 0.00 91.39
22:20:43 all 0.05 0.00 0.00 0.00 0.00 99.95
計測 part2
下記で使っているOpenSSLとnginxはdebugプリントを有効化しているため，kTLSが有効化されていない場合に比べてパフォーマンスにペナルティが発生する．
(リポジトリの目的はsendile()が呼び出されていることを確認する目的のため，当然
そこでdebugを無効化してパフォーマンスを比較する
スループット
- kTLS有効
% echo GET https://{your ip}:8443 | vegeta attack -insecure -duration=5m -timeout=10s -rate=5000| vegeta report
Requests [total, rate, throughput] 1500000, 5000.00, 4992.81
Duration [total, attack, wait] 5m0s, 5m0s, 432.346ms
Latencies [min, mean, 50, 90, 95, 99, max] 176.422µs, 113.81ms, 2.545ms, 416.88ms, 492.825ms, 605.936ms, 714.755ms
Bytes In [total, mean] 922500000, 615.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:1500000
Error Set:```
- kLTSなし
```bash
% echo GET https://{your ip}:8444 | vegeta attack -insecure -duration=5m -timeout=10s -rate=5000| vegeta report
Requests [total, rate, throughput] 1499996, 4999.99, 4999.95
Duration [total, attack, wait] 5m0s, 5m0s, 2.25ms
Latencies [min, mean, 50, 90, 95, 99, max] 186.443µs, 99.395ms, 2.368ms, 387.799ms, 463.372ms, 545.537ms, 750.165ms
Bytes In [total, mean] 922497540, 615.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:1499996
Error Set:
CPU使用率
- kTLS有効
23:09:33 CPU %user %nice %system %iowait %steal %idle
23:09:43 all 4.40 0.00 7.29 0.05 0.00 88.26
23:09:53 all 10.88 0.00 8.11 0.03 0.00 80.98
23:10:03 all 14.13 0.00 9.29 0.07 0.00 76.51
23:10:13 all 15.46 0.00 7.92 0.15 0.00 76.46
23:10:23 all 17.10 0.00 9.21 0.08 0.00 73.61
23:10:33 all 17.01 0.00 9.08 0.05 0.00 73.87
23:10:43 all 15.46 0.00 8.85 0.13 0.00 75.57
23:10:53 all 12.37 0.00 7.94 0.05 0.00 79.64
23:11:03 all 17.41 0.00 8.67 0.08 0.00 73.84
23:11:13 all 16.84 0.00 8.13 0.05 0.00 74.97
23:11:23 all 14.19 0.00 8.12 0.10 0.00 77.59
23:11:33 all 15.60 0.00 8.60 0.08 0.00 75.72
23:11:43 all 15.06 0.00 8.54 0.10 0.00 76.30
23:11:53 all 16.03 0.00 8.83 0.05 0.00 75.09
23:12:03 all 15.78 0.00 8.17 0.05 0.00 76.01
23:12:13 all 17.45 0.00 8.16 0.13 0.00 74.27
23:12:23 all 16.08 0.00 8.94 0.08 0.00 74.91
23:12:33 all 16.95 0.00 9.06 0.05 0.00 73.94
23:12:43 all 16.39 0.00 8.50 0.10 0.00 75.01
23:12:53 all 16.32 0.00 8.69 0.10 0.00 74.88
23:13:03 all 17.05 0.00 7.96 0.05 0.00 74.94
23:13:13 all 18.15 0.00 8.61 0.08 0.00 73.16
23:13:23 all 14.09 0.00 8.39 0.08 0.00 77.44
23:13:33 all 14.52 0.00 10.17 0.07 0.00 75.23
23:13:43 all 17.94 0.00 8.53 0.05 0.00 73.48
23:13:53 all 15.14 0.00 10.35 0.07 0.00 74.43
23:14:03 all 15.17 0.00 10.79 0.05 0.00 73.99
23:14:13 all 17.80 0.00 8.71 0.05 0.00 73.45
23:14:23 all 15.82 0.00 7.80 0.13 0.00 76.26
23:14:33 all 11.19 0.00 4.15 0.05 0.00 84.61
23:14:43 all 0.23 0.00 0.05 0.05 0.00 99.67
23:14:53 all 0.08 0.00 0.05 0.08 0.00 99.80
- kTLSなし
23:15:23 all 3.02 0.00 2.64 0.03 0.00 94.32
23:15:33 all 11.28 0.00 6.20 0.07 0.00 82.44
23:15:43 all 18.15 0.00 6.80 0.05 0.00 75.00
23:15:53 all 15.24 0.00 6.05 0.10 0.00 78.60
23:16:03 all 18.15 0.00 6.24 0.05 0.00 75.56
23:16:13 all 16.63 0.00 5.64 0.05 0.00 77.69
23:16:23 all 19.35 0.00 6.45 0.05 0.00 74.15
23:16:33 all 16.11 0.00 6.16 0.10 0.00 77.63
23:16:43 all 17.33 0.00 6.05 0.05 0.00 76.57
23:16:53 all 18.42 0.00 6.06 0.08 0.00 75.45
23:17:03 all 20.32 0.00 6.68 0.08 0.00 72.92
23:17:13 all 17.57 0.00 6.04 0.05 0.00 76.34
23:17:23 all 18.57 0.00 6.22 0.05 0.00 75.16
23:17:33 all 18.98 0.00 7.40 0.03 0.00 73.59
23:17:43 all 16.77 0.00 7.62 0.05 0.00 75.57
23:17:53 all 19.19 0.00 6.46 0.08 0.00 74.27
23:18:03 all 18.68 0.00 6.14 0.05 0.00 75.13
23:18:13 all 18.56 0.00 7.62 0.10 0.00 73.72
23:18:23 all 18.05 0.00 6.42 0.05 0.00 75.48
23:18:33 all 15.27 0.00 6.62 0.08 0.00 78.03
23:18:43 all 15.39 0.00 5.49 0.10 0.00 79.02
23:18:43 CPU %user %nice %system %iowait %steal %idle
23:18:53 all 17.51 0.00 6.66 0.05 0.00 75.78
23:19:03 all 13.14 0.00 5.55 0.05 0.00 81.27
23:19:13 all 18.65 0.00 6.07 0.08 0.00 75.20
23:19:23 all 20.47 0.00 6.07 0.05 0.00 73.41
23:19:33 all 17.29 0.00 7.39 0.03 0.00 75.29
23:19:43 all 19.29 0.00 6.47 0.08 0.00 74.16
23:19:53 all 15.18 0.00 6.80 0.08 0.00 77.94
23:20:03 all 14.94 0.00 6.14 0.03 0.00 78.89
23:20:13 all 16.35 0.00 6.52 0.05 0.00 77.08
23:20:23 all 10.89 0.00 3.34 0.18 0.00 85.60
23:20:33 all 0.03 0.00 0.05 0.05 0.00 99.87
23:20:43 all 0.03 0.00 0.05 0.05 0.00 99.87
ダウンロードのスループット
あまり変わらない
- kTLS有効
% for i in $(seq 1 100); do curl -k -s -o /dev/null -w '%{speed_download}\n' https://{your ip}:8443/1G; done | ministat
x <stdin>
+-----------------------------------------------------------------------------------------------------------------------------------------+
| x |
| x |
| x x |
| x x x |
| x x x |
| x x x x |
| x x x x x |
| x x x x x x |
| x x x x x x x |
| x x x x x x x |
| x x x x x x x x |
| x x x x x x x x |
| x x x x x x x x |
| x x x x x x x x |
| x x x x x x x x x x x x |
|x x x x x x x x x x x x x x x x x|
| |______________________AM_____________________| |
+-----------------------------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 100 1.1630652e+08 1.1655903e+08 1.1648316e+08 1.1648154e+08 42810.698
- kTLSなし
% for i in $(seq 1 100); do curl -k -s -o /dev/null -w '%{speed_download}\n' https://{your ip}:8444/1G; done | ministat
x <stdin>
+-----------------------------------------------------------------------------------------------------------------------------------------+
| x x |
| x x x |
| x x x |
| x x x |
| x x x |
| x x x |
| x x x x |
| x x x x |
| x x x x |
| x x x x |
| x x x x |
| x x x x x |
| x x x x x x |
| x x x x x x x |
| x x x x x x x x x x |
| x x x x x x x x x x x x x x x|
|x x x x x x x x x x x x x x x x x x x x|
| |_____________________________A___________M_________________| |
+-----------------------------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 100 1.1629393e+08 1.1655903e+08 1.1650844e+08 1.1648522e+08 57933.295
カーネルのtlsモジュール削除
x <stdin>
+-----------------------------------------------------------------------------------------------------------------------------------------+
| x |
| x |
| x |
| x x |
| x x |
| x x |
| x x |
| x x |
| x x |
| x x |
| x x x |
| x x x |
| x x x |
| x x x |
| x x x x x x |
| x x x x x x |
| x x x x x x |
| x x x x x x x |
| x x x x x x x x x |
| x x x x x x x x x x x|
| x x x x x x x x x x x|
|x x x x x x x x x x x x x x x|
| |___________________A__M________________| |
+-----------------------------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 100 1.1626874e+08 1.1655903e+08 1.1650844e+08 1.1650138e+08 42458.168