In my Pi Homelab post, I talked about my Raspberry Pi Kubernetes cluster.

Since that post, I’ve switched from Ubuntu to Arch Linux ARM (aarch64) for my Pi nodes, and made some refinements on the provisioning process.

Why switch?

I like Arch’s rolling-release model. The release model of other distros works well for prioritizing stability, but for my personal stuff I prefer using up-to-date versions of software, even at risk of incompatibilities. (It’s all a learning experience!)

My laptop already runs Arch (technically, Endeavour OS) so I’m much more familiar with it than Ubuntu. I’m trying to standardize as much of my homelab stuff on Arch as possible.

Also, I like the way Arch Linux ARM is distributed. Rather than distributing a pre-baked .img file you can flash directly, they provide an archive of the OS filesystem and expect you to partition, mount and install the files on your SD card “manually.” This means it’s easier to make tweaks.

Tweaks

I make some tweaks to the default files (beyond what’s specified by Arch Linux ARM install guide). They’re meant to support secure, automated headless provisioning.

  • The alarm user is deleted.
  • The root user is password-locked.
  • SSH password auth is disabled.
  • An SSH key is added to /root/.ssh/authorized_keys.
  • A static IP addess is used. (e.g. 192.168.123.1)
  • A predictable hostname is used. (e.g. pi1)

The provisioned node is available via SSH only if you have the right private key. (No passwords!)

The only other tweaks are enabling kernel features required for containerd/kubernetes. Specifically:

  • Options set: cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
  • Modules enabled: overlay nf_conntrack br_netfilter

mksdcard.sh

Since I have five Pis to provision, I made a script, mksdcard.sh, to automate the SD card creation process. Usage:

  1. Download the OS archive into the same directory as the script (using unencrypted HTTP :barf:):

    wget http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz

    (This step only has to be done once.)

  2. Create an SSH key if you haven’t already (e.g. with ssh-keygen).

  3. Insert your SD card and note the block device it uses (e.g. /dev/mmcblk0 or /dev/sdb).

  4. Run the mksdcard.sh script:

    sudo ./mksdcard.sh -d BLOCK-DEVICE -p PUBLIC-KEY-FILE -n 1

    It’s VERY important that you get the block device right: this script will ERASE all existing data on the specified device!

    The -n/--node flag is an integer used to determine the node’s IP and hostname. If the node number is 1, for example, the hostname is pi1 and the IP is 192.168.123.1. (IP address blocks can be configured in the script.)

  5. Once the script finishes, remove the SD card and put it in a Raspberry Pi 4. Connect it to Ethernet and power.

    Assuming you’re on the same network, you should be able to connect to it with:

    ssh root@192.168.123.1

  6. Repeat steps 3-5 for the other Pis, incrementing the value of -n/--node each time.

That’s it — So, here’s the contents of the mksdcard.sh script:

The latest version of this script is available on Github.

#!/usr/bin/env bash
# Expected to run as root
set -euo pipefail

cleanup() {
    EXIT_CODE="$?"
    if [[ "$TMPDIR" != "" ]]; then
        echo '*** Unmounting...'
        umount "$BOOTDIR" "$ROOTDIR"
        rm -r "$TMPDIR"
    fi

    if [[ "$EXIT_CODE" == 0 ]]; then
        echo '*** SUCCESS (SAFE TO EJECT)'
    else
        echo '*** Exit code:' $EXIT_CODE
        echo '*** FAILURE (SAFE TO EJECT)'
    fi

}

trap cleanup EXIT

# Argument defaults

# download at: http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz
OS_ARCHIVE="ArchLinuxARM-rpi-aarch64-latest.tar.gz"
USER_SSH_KEY="id_rsa.pub"

# Parse user-specified arguments

while (( "$#" )); do
  case "$1" in
    -h|--help)
      echo "Usage: mksdcard.sh -n NUM -a ARCHIVE -d DEVICE -u USER -p KEY-FILE"
      exit 0
      ;;
    -a|--archive)
      OS_ARCHIVE="$2"
      shift 2
      ;;
    -d|--device)
      SDDEV="$2"
      shift 2
      ;;
    -p|--public-key)
      USER_SSH_KEY="$2"
      shift 2
      ;;
    -n|--node)
      NODE_NUM="$2"
      shift 2
      ;;
  esac
done

# Ensure required arguments were specified

SDDEV="${SDDEV?Must specify -d/--device}"
NODE_NUM="${NODE_NUM?Must specify -n/--node}"

# Calculate static IP information
# These are hardcoded for my LAN subnet -- adjust to your needs
IP_ADDR="192.168.123.$NODE_NUM/24"
IP_ROUTER="192.168.123.0"

# Partition SD card with sfdisk
# 1. Wipe any existing partitions
# 2. Allocate first 200M to a partition
# 3. Allocate rest to a partition
# type=c is partition type: W95 FAT32 (LBA)
# type=83 is partition type: Linux
sfdisk "$SDDEV" << EOF
label: dos
label-id: 0x2aff57d5
device: $SDDEV
unit: sectors
sector-size: 512

size=+200MiB, type=c
type=83
EOF

# Make filesystems
mkfs.vfat "$SDDEV"1
mkfs.ext4 "$SDDEV"2

# Mount filesystems in temporary directory
TMPDIR="$(mktemp -d)"
BOOTDIR="$TMPDIR/boot"
ROOTDIR="$TMPDIR/root"
mkdir -p "$BOOTDIR"
mkdir -p "$ROOTDIR"
mount "$SDDEV"1 "$BOOTDIR"
mount "$SDDEV"2 "$ROOTDIR"

# Extract OS archive into root partition
bsdtar -xpf "$OS_ARCHIVE" -C "$ROOTDIR"

# Delete alarm user that exists by default
userdel -P "$ROOTDIR" alarm

# Lock root so password login is not possible
usermod -P "$ROOTDIR" -L root

cat >> "$ROOTDIR/etc/ssh/sshd_config" << EOF
PasswordAuthentication no
ChallengeResponseAuthentication no
PermitRootLogin without-password
EOF

# Add public key for SSH login
mkdir -p "$ROOTDIR/root/.ssh"
cp "$USER_SSH_KEY" "$ROOTDIR/root/.ssh/authorized_keys"

# update /etc/fstab for the different SD block device compared to the Raspberry Pi 3
sed -i 's/mmcblk0/mmcblk1/g' "$ROOTDIR/etc/fstab"

# set hostname
echo "pi$NODE_NUM" > "$ROOTDIR/etc/hostname"

# Static network configuration
cat >> "$ROOTDIR/etc/dhcpcd.conf" << EOF
interface eth0
static ip_address=$IP_ADDR
static routers=$IP_ROUTER
EOF

# Enable cgroups (needed for containerd)
echo "cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1" > "$ROOTDIR/boot/cmdline.txt"

# Enable kernel modules needed for k0s
cat > "$ROOTDIR/etc/modules-load.d/k0s-modules.conf" << EOF
overlay
nf_conntrack
br_netfilter
EOF

# flush writes
sync

# Move boot stuff to boot partition
mv "$ROOTDIR/boot/"* "$BOOTDIR"

echo '*** Preparation complete!'

# # Run on the Pi after login:
# # pacman-key --init
# # pacman-key --populate archlinuxarm