iTranslated by AI
Handling ECH Key Rotation Mismatches (Go, nginx, and Caddy)
ECH (Encrypted ClientHello) is cached on the client side because ECHConfigList is distributed via DNS. However, for security reasons, servers must rotate keys periodically. This means that at the exact moment a key is rotated, there will inevitably be "clients arriving with the old ECHConfigList."
Actually, ECH is designed with the assumption that requests with old keys will arrive, and it has a mechanism called retry_configs.
In this article, I will organize how nginx, Go (crypto/tls), and Caddy handle "keys for acceptance (decryption)" and "configurations returned via retry_configs."
By the way, if you want to know about ECH itself, I recommend the article I wrote previously.
What is retry_configs?
When a server cannot decrypt (or does not accept) an ECH, there is a mechanism to return a message to the client saying, "try again with this ECHConfigList next time," instead of just ending the connection. This is retry_configs.
- The client's DNS cache is outdated
- Immediately after the server rotated keys
- Public name or configuration is in the process of migration
These situations are bound to happen if you are operating a service. This specification exists for that purpose.
In the first place, DNS and TLS are completely different mechanisms, so there is no way to change them simultaneously. It can be said that this is a necessary specification to cope with reality.
And when you try to operate this realistically, you will immediately realize that you need to consider the following:
- Do not invalidate decryption keys immediately; create a period where both old and new keys coexist
- Keep the ECHConfigList returned in
retry_configsto a minimum
I will explain how this is handled in each implementation.
Go (crypto/tls): Returning SendAsRetry
Go's crypto/tls uses a model where ECH keys are passed to the server as []tls.EncryptedClientHelloKey.
This structure contains SendAsRetry. As the name suggests, the behavior is:
- If the server rejects the ECH, only keys where the
SendAsRetryfield istrue(corresponding to that ECHConfig) are included inretry_configs.
Furthermore, tls.Config.GetEncryptedClientHelloKeys was added in Go 1.25. This is not for "directly replacing static EncryptedClientHelloKeys," but a callback that can return "the set of keys accepted right now" for each connection.
The background for this addition was precisely the awareness of the problem that "key rotation is difficult if ECH keys are static fields."
In short, the design of Go (crypto/tls) is as follows:
- For decryption: Hold current + old keys in an array
- For
retry_configs: Return only keys withSendAsRetry=true - Key rotation: Since Go 1.25, they can be returned dynamically via
GetEncryptedClientHelloKeys, making it easier to rotate without downtime
nginx: Only the first file is targeted for retry
nginx can utilize ECH by combining it with the OpenSSL ECH feature branch. By the way, I wrote about how to use it in the following article.
ECHConfig (and private keys) are passed via ssl_ech_file. In the current documentation, it is not explicitly stated how key rotation should be implemented.
On the other hand, looking at the implementation, when multiple ssl_ech_file directives are specified, the behavior differs depending on whether it is the first one specified (undocumented).
- Only the first file is used for
retry_configs(FOR_RETRY) - Second and subsequent files are for decryption (
NO_RETRY) and are not included inretry_configs
In other words, the design is thought to be as follows:
- You want to accept old keys to rescue outdated DNS caches (so load multiple files)
- However, you want to keep the ECHConfigList returned in
retry_configssmall, so the first specified file is considered the current file
Thus, nginx achieves the role of SendAsRetry in Go by "attaching FOR_RETRY only to the first file." As a result, it is possible to minimize the response size of retry_configs.
However, I find it quite questionable that this specification is not documented, so I have posted about it in the Nginx Community.
Caddy: Using the Go SendAsRetry model as-is while automating various tasks
Since Caddy is built on Go's crypto/tls, the basic strategy is the Go model itself. In other words, the "decryption key set" and the "configuration returned in retry_configs" can be separated by SendAsRetry.
Caddy explicitly sets SendAsRetry when constructing []tls.EncryptedClientHelloKey. Specifically, it has a condition where "already replaced keys (where Replaced is set) are not sent in a retry."
In short, Caddy works like this (at least that is how it reads from the implementation):
- For decryption: Holds current + old keys and can decrypt with old keys (retains them for a certain period)
- For
retry_configs: SetsSendAsRetry=falsefor replaced keys, bringing the ECHConfigList returned inretry_configscloser to the current one
Caddy implements key rotation on its own and handles updates to DNS providers automatically. I personally feel that you cannot operate this realistically without reading the source code.
Summary
-
retry_configsis a mechanism to absorb discrepancies caused by DNS caches or immediate key rotation - However, it is necessary to prevent the size of
retry_configsfrom increasing - Go allows limiting
retry_configswithEncryptedClientHelloKey.SendAsRetry(Key rotation became easier withGetEncryptedClientHelloKeysin Go 1.25) - nginx achieves the same goal by "making only the first key file the target for retry (
FOR_RETRY)" to minimize the response - Caddy follows the Go model by setting
SendAsRetryon the implementation side so that "replaced keys are not sent in a retry," while also attempting to automate everything itself
Discussion