<h1 id="%22lisbon%22%3A-etcd-ssl-cert-troubles">
<a class="header-anchor-link" href="#%22lisbon%22%3A-etcd-ssl-cert-troubles" aria-hidden="true"></a> "Lisbon": etcd SSL cert troubles</h1>
<p>SadServersの <a href="https://sadservers.com/newserver/lisbon" target="_blank" rel="nofollow noopener noreferrer">"Lisbon": etcd SSL cert troubles</a> の解説です。</p>
<p>SadServers って何？ って人は、 <a href="https://zenn.dev/kenma/articles/373d382bf4f1db" target="_blank">SadServers解説</a> を見てください。<br>
一言でいうと、LeetCode (コーディング問題集) の Linuxサーバ設定版のようなものです。</p>
<h2 id="%E5%95%8F%E9%A1%8C">
<a class="header-anchor-link" href="#%E5%95%8F%E9%A1%8C" aria-hidden="true"></a> 問題</h2>
<p>問題を見ていきます。 <code>&gt;</code> の箇所は <a href="https://www.deepl.com/ja/translator" target="_blank" rel="nofollow noopener noreferrer">DeepL</a>での翻訳です。</p>
<div class="code-block-container"><pre><code>Scenario: "Lisbon": etcd SSL cert troubles

Level: Medium

Description: There's an etcd server running on https://localhost:2379 , get the value for the key "foo", ie etcdctl get foo or curl https://localhost:2379/v2/keys/foo
&gt; https://localhost:2379 で etcd サーバーが動いているので、キー "foo" の値を取得します。etcdctl get foo または curl https://localhost:2379/v2/keys/foo です。

Test: etcdctl get foo returns bar.

Time to Solve: 20 minutes.

OS: Debian 11
</code></pre></div><p><code>etcd</code> は <a href="https://www.redhat.com/ja/topics/containers/what-is-etcd" target="_blank" rel="nofollow noopener noreferrer">'etcd とは'</a>を読むと、Key Value Store の サーバのようです。<br>
<code>etcdctl</code>が<code>etcd</code>専用のクライアントのようです。が、<code>curl</code>でも叩けるので、HTTPサーバということがわかります。</p>
<h2 id="%E8%A7%A3%E8%AA%AC">
<a class="header-anchor-link" href="#%E8%A7%A3%E8%AA%AC" aria-hidden="true"></a> 解説</h2>
<p>早速、<code>etcdctl</code>を実行します。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ etcdctl get foo
Error:  client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate has expired or is not yet valid: current time 2024-02-05T14:12:40Z is after 2023-01-30T00:02:48Z

error #0: x509: certificate has expired or is not yet valid: current time 2024-02-05T14:12:40Z is after 2023-01-30T00:02:48Z
</code></pre></div><blockquote>
<p>error #0: x509: certificate has expired or is not yet valid: current time 2024-02-05T14:12:40Z is after 2023-01-30T00:02:48Z</p>
</blockquote>
<p>エラー内容と見ると、SSL証明書の有効期限が<code>2023-01-30T00:02:48Z</code>ですが、現在時刻が<code>2024-02-05T14:12:40Z</code>のため、SSL証明書の有効期限切れ(expired)です。</p>
<p><code>date</code>コマンドで、システムの現在日時を確認します。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ date
Mon Feb  5 14:12:52 UTC 2024
</code></pre></div><p>たしかに、<code>Feb 5 2024</code>となっているので、SSL証明書の有効期限<code>2023-01-30</code>を満たすように、<code>date -s</code>コマンドで日時を<code>2023-01-01</code>に変更します。<br>
（実際に実施した日時は、<code>2023 Feb 5</code>なので、どうやらこのVMの日時は一年後になっているようです。）</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ sudo date -s "01/01 13:00 2023"
Sun Jan  1 13:00:00 UTC 2023
</code></pre></div><p>その後、再度 <code>ectctl</code>を実施します。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ etcdctl get foo
Error:  client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
</code></pre></div><p>エラー内容が変わりました。不正なJSONを返しているとエラーは言っています。<br>
ちょっと、わからないので、<code>curl</code>も実行してみます。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ curl -v https://localhost:2379/v2/keys/foo
*   Trying 127.0.0.1:2379...
* Connected to localhost (127.0.0.1) port 2379 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  start date: Dec 31 00:02:48 2022 GMT
*  expire date: Jan 30 00:02:48 2023 GMT
*  common name: localhost (matched)
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  SSL certificate verify ok.
&gt; GET /v2/keys/foo HTTP/1.1
&gt; Host: localhost:2379
&gt; User-Agent: curl/7.74.0
&gt; Accept: */*
&gt; 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
&lt; HTTP/1.1 404 Not Found
&lt; Server: nginx/1.18.0
&lt; Date: Sun, 01 Jan 2023 13:00:25 GMT
&lt; Content-Type: text/html
&lt; Content-Length: 153
&lt; Connection: keep-alive
&lt; 
&lt;html&gt;
&lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.18.0&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
* Connection #0 to host localhost left intact
</code></pre></div><p><code>404</code>が返ってきています。また、レスポンスヘッダーから、</p>
<blockquote>
<p>&lt; Server: nginx/1.18.0</p>
</blockquote>
<p>と、<code>etcd</code>ではなく <code>nginx</code> がレスポンスを変えているようです。  これはどういうことでしょうか？？</p>
<p><code>lsof</code>で、port 2379を使っているプロセスを調べます。 が、残念ながら<code>lsof</code>は入っていません。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ lsof -i:2379
bash: lsof: command not found
</code></pre></div><p><code>systemctl status etcd</code>で、<code>ectd</code>の状態を確認します。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ systemctl status etcd
● etcd.service - etcd - highly-available key value store
     Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-02-05 14:22:39 UTC; 1 months 4 days left
       Docs: https://etcd.io/docs
             man:etcd
   Main PID: 580 (etcd)
      Tasks: 8 (limit: 521)
     Memory: 25.5M
        CPU: 1.022s
     CGroup: /system.slice/etcd.service
             └─580 /usr/bin/etcd --cert-file /etc/ssl/certs/localhost.crt --key-file /etc/ssl/certs/loca&gt;

Feb 05 14:22:38 ip-10-0-0-138 etcd[580]: ClientTLS: cert = /etc/ssl/certs/localhost.crt, key = /etc/ssl/&gt;
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: 8e9e05c52164694d is starting a new election at term 18
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: 8e9e05c52164694d became candidate at term 19
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at &gt;
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: 8e9e05c52164694d became leader at term 19
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at &gt;
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: ready to serve client requests
Feb 05 14:22:39 ip-10-0-0-138 systemd[1]: Started etcd - highly-available key value store.
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: published {Name:ip-10-0-0-138 ClientURLs:[https://localhost:237&gt;
Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: serving client requests on 127.0.0.1:2379
</code></pre></div><blockquote>
<p>Feb 05 14:22:39 ip-10-0-0-138 etcd[580]: serving client requests on 127.0.0.1:2379</p>
</blockquote>
<p><code>etcd</code>が、<code>port 2379</code>で待っていることは確認できます。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ sudo /usr/sbin/iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:2379 redir ports 443
DOCKER     all  --  anywhere            !ip-127-0-0-0.us-east-2.compute.internal/8  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  ip-172-17-0-0.us-east-2.compute.internal/16  anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
</code></pre></div><blockquote>
<p>REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:2379 redir ports 443</p>
</blockquote>
<p>port 2379が、 port 443にリダイレクトされています。 そのため、<code>nginx</code>が返答していたのです。</p>
<p><code>iptables -t nat -F</code> で、設定をリセットします。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ sudo /usr/sbin/iptables -t nat -F

# リセット後確認
admin@ip-10-0-0-138:/$ sudo /usr/sbin/iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (0 references)
target     prot opt source               destination        
</code></pre></div><p>で、再度、 <code>etcdctl</code> を実施します。</p>
<div class="code-block-container"><pre><code>admin@ip-10-0-0-138:/$ etcdctl get foo
bar
</code></pre></div><p>正しく動きました。</p>


[SadServers] 解説 "Lisbon": etcd SSL cert troubles

Discussion