🫰

Debian11 で Pacemeker と DRBD

2023/08/10に公開

前提

https://zenn.dev/mnod/articles/2c13763b1387ff で作成したDRBDをPacemaker に組み込みます。

作業前の Pacemaker の状況です。

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:01:14 2023
  * Last change:  Sat Aug  5 20:46:46 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 2 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# pcs resource
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0

# pcs resource config ClusterIP
 Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: cidr_netmask=24 ip=10.2.0.244 nic=ens3
  Operations: monitor interval=10s timeout=20s (ClusterIP-monitor-interval-10s)
              start interval=0s timeout=20s (ClusterIP-start-interval-0s)
              stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)

# pcs resource config WebServer
 Resource: WebServer (class=systemd type=nginx)
  Operations: monitor interval=60 timeout=100 (WebServer-monitor-interval-60)
              start interval=0s timeout=100 (WebServer-start-interval-0s)
              stop interval=0s timeout=100 (WebServer-stop-interval-0s)

シングルマスターDRBD

リソースの追加

マスター側でリソース設定を追加

# pcs property set no-quorum-policy=ignore
# pcs resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0
# pcs resource promotable drbd_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs resource create fs_drbd_r0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=ext4
# pcs constraint colocation add drbd_r0-clone with Master ClusterIP
# pcs constraint colocation add fs_drbd_r0 with drbd_r0-clone INFINITY with-rsc-role=Master
# pcs constraint order promote drbd_r0-clone then start fs_drbd_r0
Adding drbd_r0-clone fs_drbd_r0 (kind: Mandatory) (Options: first-action=promote then-action=start)
# pcs constraint colocation add WebServer with fs_drbd_r0 INFINITY

Pacemakerの状態確認

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:08:07 2023
  * Last change:  Sun Aug  6 05:07:44 2023 by hacluster via crmd on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

DRBDの確認

マスター側で状態を確認

# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:4 nr:0 dw:4 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:1653006 nr:710 dw:3037061 dr:1391858 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd0      1.3G   32K  1.2G   1% /mnt

スタンバイ側

# drbdadm status r0
r0 role:Secondary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:4 dw:4 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Pacemaker の動作確認

マスター側をスタンバイにしてみる。

# pcs node standby debian11-0

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:22:06 2023
  * Last change:  Sun Aug  6 05:21:59 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Node debian11-0: standby
  * Online: [ debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-1
  * WebServer   (systemd:nginx):         Started debian11-1
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-1 ]
    * Stopped: [ debian11-0 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

マスター側のリソースが無くなった。
DRBDも停止している。

# df | grep mnt

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:1653006 nr:710 dw:3037061 dr:1391858 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

スタンバイ側で確認すると、リソースを持っているのが分かる。

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:4 dw:4 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@debian11-1:~#
root@debian11-1:~# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:23:59 2023
  * Last change:  Sun Aug  6 05:21:59 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Node debian11-0: standby
  * Online: [ debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-1
  * WebServer   (systemd:nginx):         Started debian11-1
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-1 ]
    * Stopped: [ debian11-0 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  peer connection:Connecting

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:0 nr:8 dw:12 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd0      1.3G   32K  1.2G   1% /mnt

マスター側をアンスタンバイする。

# pcs node unstandby debian11-0

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:25:51 2023
  * Last change:  Sun Aug  6 05:25:43 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

リソースが戻ってきた。

# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate


# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:4 nr:8 dw:12 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:1653006 nr:710 dw:3037061 dr:1391858 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd0      1.3G   32K  1.2G   1% /mnt

スタンバイ側で確認すると、リソースは手放している。

# drbdadm status r0
r0 role:Secondary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:8 nr:12 dw:20 dr:21 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

リソースを移動してみる。

# pcs resource move WebServer debian11-1

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:28:40 2023
  * Last change:  Sun Aug  6 05:28:34 2023 by root via crm_resource on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-1
  * WebServer   (systemd:nginx):         Started debian11-1
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-1 ]
    * Slaves: [ debian11-0 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

マスター側はリソースを手放している。
DRBD自体は動いている。

# drbdadm status r0
r0 role:Secondary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:8 nr:12 dw:20 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:1653006 nr:710 dw:3037061 dr:1391858 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

スタンバイ側で確認すると、リソースを持っている。

# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate


# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:12 nr:16 dw:28 dr:38 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd0      1.3G   32K  1.2G   1% /mnt

リソースを元のノードに戻す

# pcs resource move WebServer debian11-0

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 05:31:10 2023
  * Last change:  Sun Aug  6 05:31:04 2023 by root via crm_resource on debian11-0
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:12 nr:16 dw:28 dr:34 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:1653006 nr:710 dw:3037061 dr:1391858 al:14 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd0      1.3G   32K  1.2G   1% /mnt

スタンバイ側で確認すると、リソースは手放している。

# drbdadm status r0
r0 role:Secondary
  disk:UpToDate
  peer role:Primary

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:16 nr:20 dw:36 dr:38 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:710 nr:1653006 dw:4420405 dr:3857 al:12 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

デュアルマスター DRBD

必要なパッケージをインストールします。

# apt install --no-install-recommends dlm-controld

マスター側でリソース設定を追加

# pcs resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1
# pcs resource promotable drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs resource create controld ocf:pacemaker:controld --group locking
# pcs resource create o2cb ocf:pacemaker:o2cb --group locking
# pcs resource clone locking interleave=true
# pcs resource create fs_drbd_r1 ocf:heartbeat:Filesystem device=/dev/drbd1 directory=/opt fstype=ocfs2 --group shared_fs
# pcs resource clone shared_fs interleave=true
# pcs constraint order promote drbd_r1-clone then start shared_fs-clone
Adding drbd_r1-clone shared_fs-clone (kind: Mandatory) (Options: first-action=promote then-action=start)
# pcs constraint colocation add drbd_r1-clone with Master ClusterIP

設定を反映

# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 8 replies from the controller........ OK

Pacemakerの状態確認。
いくつかのリソースの動作が失敗している。

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Sun Aug  6 07:49:29 2023
  * Last change:  Sun Aug  6 07:46:20 2023 by hacluster via crmd on debian11-1
  * 2 nodes configured
  * 13 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0
  * Clone Set: drbd_r1-clone [drbd_r1] (promotable):
    * Masters: [ debian11-0 debian11-1 ]
  * Clone Set: locking-clone [locking]:
    * Stopped: [ debian11-0 debian11-1 ]
  * Clone Set: shared_fs-clone [shared_fs]:
    * Started: [ debian11-0 debian11-1 ]

Failed Resource Actions:
  * o2cb_monitor_0 on debian11-0 'not installed' (5): call=163, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-06 07:46:20 -04:00', queued=0ms, exec=29ms
  * o2cb_monitor_0 on debian11-1 'not installed' (5): call=153, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-06 07:46:20 -04:00', queued=1ms, exec=77ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Wed Aug  9 16:49:05 2023
  * Last change:  Tue Aug  8 19:21:46 2023 by hacluster via crmd on debian11-0
  * 2 nodes configured
  * 13 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0
  * Clone Set: drbd_r1-clone [drbd_r1] (promotable):
    * Masters: [ debian11-0 debian11-1 ]
  * Clone Set: shared_fs-clone [shared_fs]:
    * Started: [ debian11-0 debian11-1 ]
  * Clone Set: locking-clone [locking]:
    * Stopped: [ debian11-0 debian11-1 ]

Failed Resource Actions:
  * o2cb_monitor_0 on debian11-0 'not installed' (5): call=180, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=86ms
  * o2cb_monitor_0 on debian11-1 'not installed' (5): call=216, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=48ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

DRBDの確認

マスター側で確認

# drbdadm status r1
r1 role:Primary
  volume:1 disk:UpToDate
  peer role:Primary
    volume:1 replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:4 nr:4 dw:8 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:60025 nr:59917 dw:119942 dr:120603 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# df -h /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      1.4G  147M  1.2G  11% /opt

# ls -l /opt/*txt
-rw-r--r-- 1 root root 20 Aug  6 04:29 /opt/helloworld1.txt
-rw-r--r-- 1 root root 12 Aug  6 04:28 /opt/helloworld.txt

# rm /opt/helloworld.txt

# ls -l /opt/*txt
-rw-r--r-- 1 root root 20 Aug  6 04:29 /opt/helloworld1.txt

同様にスタンバイ側で確認

# drbdadm status r1
r1 role:Primary
  volume:1 disk:UpToDate
  peer role:Primary
    volume:1 replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:4 nr:4 dw:4 dr:4 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:59938 nr:60046 dw:119952 dr:120418 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# df -h /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      1.4G  147M  1.2G  11% /opt

# ls -l /opt/*txt
-rw-r--r-- 1 root root 20 Aug  6 04:29 /opt/helloworld1.txt

# rm /opt/helloworld1.txt

# ls -l /opt/*txt
ls: cannot access '/opt/*txt': No such file or directory

Pacemaker の動作確認

マスター側をスタンバイにしてみる。

# pcs node standby debian11-0

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Wed Aug  9 16:52:16 2023
  * Last change:  Wed Aug  9 16:52:09 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 13 resource instances configured

Node List:
  * Node debian11-0: standby (with active resources)
  * Online: [ debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-1
  * WebServer   (systemd:nginx):         Started debian11-1
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * drbd_r0   (ocf::linbit:drbd):      Promoting debian11-1
    * Stopped: [ debian11-0 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Stopped
  * Clone Set: drbd_r1-clone [drbd_r1] (promotable):
    * drbd_r1   (ocf::linbit:drbd):      Demoting debian11-0
    * Masters: [ debian11-1 ]
  * Clone Set: shared_fs-clone [shared_fs]:
    * Started: [ debian11-1 ]
    * Stopped: [ debian11-0 ]
  * Clone Set: locking-clone [locking]:
    * Stopped: [ debian11-0 debian11-1 ]

Failed Resource Actions:
  * o2cb_monitor_0 on debian11-0 'not installed' (5): call=180, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=86ms
  * o2cb_monitor_0 on debian11-1 'not installed' (5): call=216, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=48ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

マスター側のDRBD が停止した。

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

スタンバイ側で確認すると、DRBDは動いている。

# drbdadm status r1
r1 role:Primary
  volume:1 disk:UpToDate
  peer connection:Connecting

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:4 nr:8 dw:12 dr:21 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:59952 nr:60096 dw:120071 dr:120555 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4

# df -h /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      1.4G  147M  1.2G  11% /opt

マスター側をアンスタンバイする。

# pcs node unstandby debian11-0

# pcs status
Cluster name: web_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: debian11-0 (version 2.0.5-ba59be7122) - partition with quorum
  * Last updated: Wed Aug  9 16:57:39 2023
  * Last change:  Wed Aug  9 16:57:08 2023 by root via cibadmin on debian11-0
  * 2 nodes configured
  * 13 resource instances configured

Node List:
  * Online: [ debian11-0 debian11-1 ]

Full List of Resources:
  * ClusterIP   (ocf::heartbeat:IPaddr2):        Started debian11-0
  * WebServer   (systemd:nginx):         Started debian11-0
  * Clone Set: drbd_r0-clone [drbd_r0] (promotable):
    * Masters: [ debian11-0 ]
    * Slaves: [ debian11-1 ]
  * fs_drbd_r0  (ocf::heartbeat:Filesystem):     Started debian11-0
  * Clone Set: drbd_r1-clone [drbd_r1] (promotable):
    * Masters: [ debian11-0 debian11-1 ]
  * Clone Set: shared_fs-clone [shared_fs]:
    * Started: [ debian11-0 debian11-1 ]
  * Clone Set: locking-clone [locking]:
    * Stopped: [ debian11-0 debian11-1 ]

Failed Resource Actions:
  * o2cb_monitor_0 on debian11-0 'not installed' (5): call=180, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=86ms
  * o2cb_monitor_0 on debian11-1 'not installed' (5): call=216, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/ocfs2_controld.pcmk', last-rc-change='2023-08-08 19:21:46 -04:00', queued=0ms, exec=48ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# drbdadm status r1
r1 role:Primary
  volume:1 disk:UpToDate
  peer role:Primary
    volume:1 replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:4 nr:8 dw:12 dr:17 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:29 nr:22 dw:51 dr:318 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

# df -h /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      1.4G  147M  1.2G  11% /opt

スタンバイ側でも異常なし

# drbdadm status r1
r1 role:Primary
  volume:1 disk:UpToDate
  peer role:Primary
    volume:1 replication:Established peer-disk:UpToDate

# cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:12 nr:12 dw:20 dr:25 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:59990 nr:60141 dw:120168 dr:120663 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
GitHubで編集を提案

Discussion