Quickly Setting Up Distributed Storage with DRBD9

Introduction

I have been building and operating Nextcloud for some time, and out of a strong desire to "not lose even a single millimeter of data!", this is a summary of the redundancy process for the storage HDDs as a personal memo.

Recently, with the constant talk of the "Nankai Trough earthquake coming soon," I have established a backup system between Nagoya and Hokkaido to ensure that even if the main HDD fails, things will be fine.

Since this is for my own notes, I haven't written it in great detail, but please use it as a reference.

Configuration

• DRBD Version: 9.0
• OS for each node: Ubuntu 22.04 LTS
• Node 1 (192.168.100.1, VM on Proxmox)
  • Hostname: node-1
• Node 2 (192.168.100.2, Raspberry Pi 4B)
  • Hostname: node-2

From here on, I will refer to them by their hostnames.
The backbone network between regions uses Wireguard already built on OCI, allowing communication between each region. (By the way, everything, including my parents' and siblings' houses, is connected and able to communicate via the Wireguard backbone network.)

The HDDs being used are as follows:
https://www.amazon.co.jp/dp/B07BK6696F?psc=1&ref=ppx_yo2ov_dt_b_product_details

Installing DRBD

Execute on node-1 and node-2

# Add repository
$ sudo add-apt-repository ppa:linbit/linbit-drbd9-stack

# Update repository
$ sudo apt update

# Install DRBD required for setup
$ sudo apt install drbd-utils drbd-dkms -y

# Load kernel module
$ sudo modprobe drbd

# Verify installation
$ cat /proc/drbd

Preparing and Initializing Disks for Redundancy

Execute on node-1 and node-2

# Find the device name of the target HDD using fdisk or similar.
# We will use "/dev/sda" here, but please replace it with the appropriate device name for your environment.
$ sudo pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.
$ sudo vgcreate  vg0 /dev/sda
  Volume group "vg0" successfully created
$ sudo lvcreate -l100%FREE -n lv0 vg0
  Logical volume "lv0" created.

Verify creation

$ lsblk

Verify hostnames (node-1, node-2)

$ uname -n
node-1

Execute on node-1 and node-2

$ sudo tee /etc/drbd.d/global_common.conf <<EOF > /dev/null 
global {
    usage-count no;
}
common {
    net {
        protocol C;
    }
}
EOF

Execute on node-1 and node-2

$ sudo tee /etc/drbd.d/r0.res <<EOF > /dev/null 
resource r0 {
    device /dev/drbd0;
    disk /dev/vg0/lv0;
    meta-disk internal;
    on <node-1 hostname> {
        address 192.168.100.101:7789;
    }
    on <node-2 hostname> {
        address 192.168.100.102:7799;
    }
}
EOF

Execute on node-1 and node-2

# Create metadata
$ sudo drbdadm create-md r0
# Start DRBD
$ sudo drbdadm up r0
# Check status
$ sudo drbdadm status r0

At this point, it has only been started and synchronization has not yet begun, so execute the following command on node-1.

$ sudo drbdadm primary --force r0

Wait for synchronization, as it takes some time.

If there is no data on either HDD, the initial full synchronization can be skipped.
In that case, use the following command (execute on node-1) instead of $ sudo drbdadm primary --force r0. (Official Documentation)

$ sudo drbdadm new-current-uuid --clear-bitmap r0

Notes

Addendum (2023/12/31): Procedure for changing node IP addresses

Execute on node-1 and node-2

# Down
$ sudo drbdadm down r0
# Change configuration (example)
$ sudo nano /etc/drbd.d/r0.res
# Apply settings
$ sudo drbdadm adjust r0
# Recreate metadata
$ sudo drbdadm create-md r0
# Check status
$ sudo drbdadm status r0