Real Cluster on DigitalOcean with kubeadm

This chapter leaves the local cluster behind and builds a real one, twice. You'll provision DigitalOcean droplets, install Kubernetes with kubeadm (no DOKS), make the cluster cloud-aware so that Service type=LoadBalancer provisions a real DO Load Balancer in the next chapter, install the pieces every later chapter depends on (CSI, metrics-server), and create the per-environment Namespaces. Then you'll repeat the process to get a second cluster so staging and production are physically separate.

A few terms to know before you start:

kubeadm is the official command-line tool for bootstrapping a Kubernetes cluster on machines you own. It installs the control-plane components, generates certificates, and prints a kubeadm join command you run on each worker. It does not provision the machines or install a network plugin; those are your job.
DOKS (DigitalOcean Kubernetes Service) is DO's managed offering where they run the control plane for you. The course skips it on purpose: you only learn how Kubernetes really works by installing it yourself once.
A droplet is DigitalOcean's name for a Linux VM. In this chapter each cluster gets three of them — one control plane, two workers.

The commands here live in the cluster-bootstrap repository. Treat that repo as the source of truth for cluster setup. The app repo never installs cluster-wide components.

Create the `cluster-bootstrap` repo

You don't have this repo yet — make it now. It stays local for this chapter; later chapters add more files to it (CI service accounts in Part 7, the backup script in Part 11, ArgoCD Applications in Part 13), but nothing is pushed to GitHub:

mkdir -p ~/code/cluster-bootstrap && cd ~/code/cluster-bootstrap
git init -b master
mkdir scripts

Every command in this chapter runs from ~/code/cluster-bootstrap. When you see a path like cluster-bootstrap/scripts/node-prep.sh (Step 4) or cluster-bootstrap/ci-deployer/staging.yaml (Part 7), that's the repo root — i.e. ~/code/cluster-bootstrap/scripts/node-prep.sh on disk.

Prerequisites

The Express app runs on kind and metrics-server works locally (Part 2).
doctl and gh are authenticated.

You have an SSH keypair registered with DigitalOcean. If not:

doctl compute ssh-key import k8slearn-key --public-key-file ~/.ssh/id_ed25519.pub
doctl compute ssh-key list

You have your DigitalOcean API token saved as DO_API_TOKEN in your shell. The same token you fed to doctl auth init is fine. DigitalOcean doesn't let you read it back from doctl, so if you didn't save it, generate a new one in the DO control panel (API → Personal access tokens, full read+write) and store it in your password manager. Export it for this chapter:
```
export DO_API_TOKEN='dop_v1_xxxxxxxx...'   # paste the value
```
The DO cloud-controller-manager you install below needs this in a Secret inside the cluster (Step 8). Don't commit it.
A DigitalOcean project to scope the cluster's resources. Projects are how DO groups droplets, load balancers, and volumes in the control panel — pinning the course resources to a dedicated project keeps them from getting tangled with anything else in your account. Droplets don't pick a project on their own; you either create them inside one or assign them afterwards.

If you don't have one yet, create it from the CLI:
```
doctl projects create \
  --name "k8slearn" \
  --purpose "Class project / Educational purposes" \
  --environment "Development" \
  --description "Resources for the k8slearn course"
```
Or do the equivalent in the dashboard: Manage → Projects → New Project, then fill in the same fields. Valid purposes are listed in the DO API reference; pick whichever fits.

Either way — newly created or already created by hand — capture the project ID, since doctl projects resources assign (used in Step 2) takes the ID, not the name:
```
doctl projects list
export DO_PROJECT_ID="$(doctl projects list --format ID,Name --no-header | awk '/k8slearn/ {print $1}')"
echo "$DO_PROJECT_ID"   # should print a UUID
```
Substitute k8slearn in the awk filter with whatever you named the project. Keep DO_PROJECT_ID exported across both the staging and prod passes — it doesn't change between them.

What you will do, twice

You'll repeat every step in this chapter for two environments: staging first, then prod. To keep the prose short, the commands use a single environment variable ENV (staging or prod). Run each block once with ENV=staging, then again with ENV=prod.

Step 1 — Pick a region, size, and naming convention

Pick a region close to you. The examples use fra1 (Frankfurt); substitute any region you prefer.

Droplet size: s-2vcpu-4gb is the minimum for a kubeadm node that runs Cilium and a few workloads without thrashing. Three of them per cluster (1 CP + 2 workers) is ~$30–40/month per cluster.

Names:

Cluster: k8s-staging, k8s-prod
Control plane: ${cluster}-cp-1
Workers: ${cluster}-w-1, ${cluster}-w-2

export ENV=staging                # or prod, on the second pass
export CLUSTER="k8s-${ENV}"
export REGION="fra1"
export SIZE="s-2vcpu-4gb"
export IMAGE="ubuntu-24-04-x64"
export SSH_KEY_NAME="k8slearn-key" # change to match the name in `doctl compute ssh-key list`
export SSH_FP="$(doctl compute ssh-key list --format Name,FingerPrint --no-header | awk -v k="$SSH_KEY_NAME" '$1==k {print $2}')"
test -n "$SSH_FP" || { echo "no SSH key found named '$SSH_KEY_NAME' — check 'doctl compute ssh-key list'"; }

Step 2 — Create the droplets and capture their IPs

Create the droplets, then read both the public and private IPs of each into shell variables. Every command later in the chapter assumes these variables exist in your current shell.

for n in cp-1 w-1 w-2; do
  doctl compute droplet create "${CLUSTER}-${n}" \
    --region "$REGION" --size "$SIZE" --image "$IMAGE" \
    --ssh-keys "$SSH_FP" \
    --tag-names "k8slearn,k8slearn-${ENV}" \
    --enable-private-networking --enable-monitoring \
    --wait
done

# Capture droplet ID + public + private IP for each role
read CP_ID CP_IP CP_PRIV < <(doctl compute droplet get "${CLUSTER}-cp-1" --format ID,PublicIPv4,PrivateIPv4 --no-header)
read W1_ID W1_IP W1_PRIV < <(doctl compute droplet get "${CLUSTER}-w-1"  --format ID,PublicIPv4,PrivateIPv4 --no-header)
read W2_ID W2_IP W2_PRIV < <(doctl compute droplet get "${CLUSTER}-w-2"  --format ID,PublicIPv4,PrivateIPv4 --no-header)
export CP_ID W1_ID W2_ID CP_IP W1_IP W2_IP CP_PRIV W1_PRIV W2_PRIV

printf 'CP   : %s (private %s)\nW1   : %s (private %s)\nW2   : %s (private %s)\n' \
  "$CP_IP" "$CP_PRIV" "$W1_IP" "$W1_PRIV" "$W2_IP" "$W2_PRIV"

Move the new droplets into the project from the Prerequisites. doctl compute droplet create has no --project-id flag, so this is a separate step. The resource URN format is do:droplet:<id>:

doctl projects resources assign "$DO_PROJECT_ID" \
  --resource="do:droplet:${CP_ID}" \
  --resource="do:droplet:${W1_ID}" \
  --resource="do:droplet:${W2_ID}"

# Verify they show up under the project
doctl projects resources list "$DO_PROJECT_ID" | grep droplet

Resources that later chapters create — the firewall in Step 3, the DO Load Balancer in Part 4, and the CSI-provisioned volumes in Part 9 — also won't land in this project automatically. Assign them the same way as they appear, using these URNs:

Firewall: do:firewall:<id> (from doctl compute firewall list)
Load Balancer: do:loadbalancer:<id> (from doctl compute load-balancer list)
Volume: do:volume:<id> (from doctl compute volume list)

Recover shell state in a new terminal

All the droplet IDs and IPs you captured above live only in this shell. If you close the terminal, switch machines, or come back tomorrow, run this self-contained block to rebuild the variables every later step depends on. Set ENV to whichever pass you're on (staging or prod):

export ENV=staging                # or prod
export CLUSTER="k8s-${ENV}"

read CP_ID CP_IP CP_PRIV < <(doctl compute droplet get "${CLUSTER}-cp-1" --format ID,PublicIPv4,PrivateIPv4 --no-header)
read W1_ID W1_IP W1_PRIV < <(doctl compute droplet get "${CLUSTER}-w-1"  --format ID,PublicIPv4,PrivateIPv4 --no-header)
read W2_ID W2_IP W2_PRIV < <(doctl compute droplet get "${CLUSTER}-w-2"  --format ID,PublicIPv4,PrivateIPv4 --no-header)
export CP_ID W1_ID W2_ID CP_IP W1_IP W2_IP CP_PRIV W1_PRIV W2_PRIV

printf 'CP : %s (private %s)\nW1 : %s (private %s)\nW2 : %s (private %s)\n' \
  "$CP_IP" "$CP_PRIV" "$W1_IP" "$W1_PRIV" "$W2_IP" "$W2_PRIV"

If a read line prints nothing, the droplet doesn't exist under that name yet — re-check doctl compute droplet list --tag-name "k8slearn-${ENV}" and confirm Step 2 actually finished.

Some other things later steps assume are also shell-scoped and need re-exporting in a new terminal:

DO_PROJECT_ID (from Prerequisites) — export DO_PROJECT_ID="$(doctl projects list --format ID,Name --no-header | awk '/k8slearn/ {print $1}')".
DO_API_TOKEN (from Prerequisites, used in Step 8) — paste from your password manager. DO won't let you read it back from doctl.
KUBECONFIG (from Step 7 onward) — export KUBECONFIG=~/.kube/config-${ENV}.

Step 3 — Lock down with a DO firewall

The DO firewall is the only thing between the open internet and your control plane. Only allow what you need.

doctl splits --inbound-rules / --outbound-rules on spaces, not newlines, so the rules have to end up on a single space-separated line with no leading or trailing whitespace (an extra space produces an empty rule and the API rejects it as "must be a key:value pair"). Put each rule in an array, then join with "${arr[*]}" so the shell stitches them together cleanly:

MY_IP="$(curl -s https://ifconfig.me)/32"

INBOUND_RULES=(
  "protocol:tcp,ports:22,address:${MY_IP}"
  "protocol:tcp,ports:6443,address:${MY_IP}"
  "protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0"
  "protocol:tcp,ports:443,address:0.0.0.0/0,address:::/0"
  "protocol:tcp,ports:all,tag:k8slearn-${ENV}"
  "protocol:udp,ports:all,tag:k8slearn-${ENV}"
)

OUTBOUND_RULES=(
  "protocol:tcp,ports:all,address:0.0.0.0/0,address:::/0"
  "protocol:udp,ports:all,address:0.0.0.0/0,address:::/0"
  "protocol:icmp,address:0.0.0.0/0,address:::/0"
)

doctl compute firewall create \
  --name "k8s-${ENV}-fw" \
  --tag-names "k8slearn-${ENV}" \
  --inbound-rules "${INBOUND_RULES[*]}" \
  --outbound-rules "${OUTBOUND_RULES[*]}"

"${arr[*]}" joins array elements with the first character of IFS (a space, by default), so the resulting string is rule1 rule2 rule3 — exactly what doctl wants. For TCP/UDP rules, the port spec must be a number or the literal all — ports:0 is rejected. ICMP has no ports, so its rule omits the ports: key entirely.

The firewall binds to the droplets through the --tag-names "k8slearn-${ENV}" flag on firewall create: DO automatically attaches the firewall to every droplet that carries that tag, and Step 2 already tagged the three droplets with k8slearn,k8slearn-${ENV} at creation time. There is no explicit "attach droplet X to firewall Y" step — adding the tag is the attachment, and any future droplet you launch with that tag inherits the same firewall.

The tag-to-tag rule (tag:k8slearn-${ENV}) inside the inbound rules is a separate mechanism on top: it lets traffic from any droplet with that tag reach any droplet with that tag, on every port. Cilium needs that plus a few system ports. 80/443 are open to the world for the DO Load Balancer health-check path in the next chapter; 6443 (the Kubernetes API) is open only to your IP.

Confirm the firewall is set up with the right tag, and that the droplets carry that tag — that's how the binding works. DropletIDs will stay empty; it's only populated when droplets are attached explicitly by ID, which isn't what's happening here.

FW_ID="$(doctl compute firewall list --format ID,Name --no-header | awk -v n="k8s-${ENV}-fw" '$2==n {print $1}')"
doctl compute firewall get "$FW_ID" --format Name,Tags,DropletIDs
doctl compute droplet list --tag-name "k8slearn-${ENV}" --format Name,ID,Tags

Happy-path output (IDs will differ):

Name              Tags                Droplet IDs
k8s-staging-fw    k8slearn-staging

Name                ID           Tags
k8s-staging-cp-1    572038432    k8slearn,k8slearn-staging
k8s-staging-w-1     572038556    k8slearn,k8slearn-staging
k8s-staging-w-2     572038733    k8slearn,k8slearn-staging

Three things to check: the firewall row has k8slearn-${ENV} under Tags, the Droplet IDs column is empty (tag-attachment, as discussed), and all three droplets appear in the second list with k8slearn-${ENV} in their Tags column.

If the droplet list is empty or missing nodes, those droplets weren't tagged at create time — re-check doctl compute droplet get "${CLUSTER}-cp-1" --format Tags and add the tag with doctl compute droplet tag "${CLUSTER}-cp-1" --tag-name "k8slearn-${ENV}" if needed.

Step 4 — Prepare the OS on every node

Copy this script to every droplet and run it as root. It's the OS prep kubeadm needs. The script lives in the cluster-bootstrap repo on your laptop — every command in this chapter assumes you're running it from the root of that repo.

If $CP_IP, $W1_IP, $W2_IP are empty (new terminal, closed shell, etc.), rebuild them with the Recover shell state in a new terminal block from Step 2 before running the loop below.

cluster-bootstrap/scripts/node-prep.sh:

#!/usr/bin/env bash
set -euo pipefail

# 1. swap off
swapoff -a
sed -i.bak '/ swap / s/^/#/' /etc/fstab

# 2. kernel modules
cat <<'EOF' >/etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

# 3. sysctls
cat <<'EOF' >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

# 4. containerd — the container runtime the kubelet talks to via CRI. Docker
#    isn't used on Kubernetes nodes anymore; containerd is the lightweight
#    daemon that actually pulls images and runs containers.
apt-get update
apt-get install -y containerd
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml >/dev/null
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd

# 5. kubeadm, kubelet, kubectl — pin the minor here, bump deliberately
KVER=v1.34
apt-get install -y apt-transport-https ca-certificates curl gpg
install -m 0755 -d /etc/apt/keyrings
curl -fsSL "https://pkgs.k8s.io/core:/stable:/${KVER}/deb/Release.key" \
  | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
  https://pkgs.k8s.io/core:/stable:/${KVER}/deb/ /" \
  > /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

# 6. kubelet must run with --cloud-provider=external so DO CCM can take over
mkdir -p /etc/systemd/system/kubelet.service.d
cat <<'EOF' >/etc/systemd/system/kubelet.service.d/20-cloud-provider.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"
EOF
systemctl daemon-reload

Kubernetes follows a version-skew policy that maintains the most recent three minor releases. The course pins v1.34 as a currently-supported minor; bump it deliberately when 1.34 ages out, and re-check the Cilium, DO CCM, DO CSI, Traefik, cert-manager, kube-prometheus-stack, and ArgoCD chart versions against the new minor at the same time.

Push and run on every droplet:

SSH_OPTS="-o StrictHostKeyChecking=accept-new"

for IP in "$CP_IP" "$W1_IP" "$W2_IP"; do
  scp $SSH_OPTS cluster-bootstrap/scripts/node-prep.sh "root@${IP}:/root/node-prep.sh"
  ssh $SSH_OPTS "root@${IP}" "bash /root/node-prep.sh"
done

StrictHostKeyChecking=accept-new skips the interactive Are you sure you want to continue connecting (yes/no)? prompt on first connection: ssh silently writes the droplet's host key to ~/.ssh/known_hosts and proceeds. It's a deliberate weaker setting than the default — fine here because you're connecting to droplets you just created seconds ago and the IPs came straight from doctl, so there's nothing for a MITM to impersonate. On subsequent connections the recorded key is enforced normally, so a swapped-out host does fail loud. Don't use this on machines you don't own or didn't just provision. Later steps reuse $CP_IP, $W1_IP, $W2_IP and won't prompt again because the keys are now cached.

Step 5 — Bootstrap the control plane

SSH to the control plane droplet and run kubeadm init. The Pod CIDR is the IP range Kubernetes hands out to pods; every CNI plugin needs to know it, and the value here (10.244.0.0/16) matches what Cilium expects below.

ssh "root@${CP_IP}" bash -s <<EOF
set -euo pipefail
PRIVATE_IP=\$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
kubeadm init \\
  --pod-network-cidr=10.244.0.0/16 \\
  --apiserver-advertise-address="\${PRIVATE_IP}" \\
  --apiserver-cert-extra-sans="${CP_IP}" \\
  --upload-certs | tee /root/kubeadm-init.log
EOF

When it finishes, the kubeadm join command is in /root/kubeadm-init.log on the control plane. Pull it back to your laptop and capture it for the worker join step:

JOIN_CMD=$(ssh "root@${CP_IP}" \
  "tail -n 20 /root/kubeadm-init.log | grep -A1 'kubeadm join' | tr -d '\\\\\n' | tr -s ' '")
echo "$JOIN_CMD"

Set up kubectl access on the control plane droplet so you can verify locally if needed:

ssh "root@${CP_IP}" '
  mkdir -p ~/.kube
  cp -i /etc/kubernetes/admin.conf ~/.kube/config
  kubectl get nodes
'
# k8s-staging-cp-1   NotReady   control-plane   ...

The control plane is NotReady until the CNI (Container Network Interface — the plugin that gives pods IPs and lets them talk to each other) is installed. That's expected.

Step 6 — Join the workers

Run the captured join command on each worker. node-prep.sh already set --cloud-provider=external on the kubelet:

for IP in "$W1_IP" "$W2_IP"; do
  ssh "root@${IP}" "$JOIN_CMD"
done

ssh "root@${CP_IP}" 'kubectl get nodes'
# all three appear, all NotReady (CNI not yet installed)

Step 7 — Install Cilium

Cilium is the CNI plugin this course uses. It builds pod networking on top of eBPF (a kernel facility for safely running small programs in the network stack), which gets you good performance, the option to skip kube-proxy entirely, and a built-in observability tool called Hubble. Any other CNI (Calico, Flannel, Weave) would work; Cilium is the modern default.

First, copy the kubeconfig back to your laptop. A kubeconfig is the YAML file kubectl reads to learn which clusters exist, how to authenticate to each one, and which one is currently selected (the "context"). One per environment, merged in Step 12 so contexts never collide.

mkdir -p ~/.kube
scp "root@${CP_IP}:/etc/kubernetes/admin.conf" "$HOME/.kube/config-${ENV}"
chmod 600 "$HOME/.kube/config-${ENV}"

# Point the kubeconfig at the public IP (kubeadm wrote the private one)
sed -i.bak "s#server: https://[^:]*:6443#server: https://${CP_IP}:6443#" "$HOME/.kube/config-${ENV}"
rm -f "$HOME/.kube/config-${ENV}.bak"

export KUBECONFIG=~/.kube/config-${ENV}
kubectl get nodes

Install Cilium with Helm. Helm is the package manager for Kubernetes: a chart is a templated bundle of manifests plus a values.yaml for customization, and helm upgrade --install either creates the release or updates it in place. The rest of the course leans on Helm for cert-manager, Traefik, sealed-secrets, kube-prometheus-stack, and ArgoCD.

helm repo add cilium https://helm.cilium.io/
helm repo update
helm upgrade --install cilium cilium/cilium \
  --version 1.16.5 \
  --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="$CP_PRIV" \
  --set k8sServicePort=6443 \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Wait for it to settle:

kubectl -n kube-system rollout status ds/cilium
kubectl get nodes
# all three Ready

Optional but useful, the cilium CLI:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz"
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
cilium status

Step 8 — Install the DO cloud-controller-manager

The cloud-controller-manager (CCM) is the component that bridges Kubernetes with your cloud provider's APIs. When a workload asks for a Service type=LoadBalancer, the DO CCM calls the DigitalOcean API and provisions a real Load Balancer; when a node is deleted from DO, the CCM removes the corresponding Node object. Managed offerings like DOKS bake this in; on a self-installed cluster you install it yourself.

Create the API-token Secret from DO_API_TOKEN (set in Prerequisites):

test -n "$DO_API_TOKEN" || { echo "set DO_API_TOKEN first"; exit 1; }
kubectl -n kube-system create secret generic digitalocean \
  --from-literal=access-token="$DO_API_TOKEN"

Install the CCM. Pin a version that matches the Kubernetes minor in node-prep.sh:

DO_CCM_VERSION=v0.1.65   # check https://github.com/digitalocean/digitalocean-cloud-controller-manager/releases
kubectl apply -f "https://raw.githubusercontent.com/digitalocean/digitalocean-cloud-controller-manager/${DO_CCM_VERSION}/releases/digitalocean-cloud-controller-manager.yaml"

kubectl -n kube-system rollout status deploy/digitalocean-cloud-controller-manager
kubectl -n kube-system logs deploy/digitalocean-cloud-controller-manager | tail

Check that nodes pick up DO-aware labels and providerID:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\n"}{end}'
# k8s-staging-cp-1   digitalocean://<id>
# k8s-staging-w-1    digitalocean://<id>
# k8s-staging-w-2    digitalocean://<id>

If the providerID field is empty, the CCM hasn't reconciled yet. The most common cause is the kubelet running without --cloud-provider=external. Re-check /etc/systemd/system/kubelet.service.d/20-cloud-provider.conf on each node.

Step 9 — Install the DigitalOcean CSI driver

CSI is the Container Storage Interface, a standard plugin format for hooking external storage systems into Kubernetes. Platform components (Prometheus, VictoriaMetrics, Grafana, Alertmanager) request a PersistentVolumeClaim (a "give me N GiB of storage" request) in Part 9, and the DO CSI driver fulfils each one by creating a DO block-storage volume and attaching it to the right node. The available kinds of storage are exposed as StorageClass objects — picking do-block-storage (the default below) is what tells Kubernetes "use the DO CSI driver."

DO_CSI_VERSION=v4.13.0   # check https://github.com/digitalocean/csi-digitalocean/releases
kubectl apply -f "https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/deploy/kubernetes/releases/csi-digitalocean-${DO_CSI_VERSION}.yaml"

The CSI install also creates a do-block-storage StorageClass. Make it the default:

kubectl patch storageclass do-block-storage \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl get storageclass
# do-block-storage (default)   dobs.csi.digitalocean.com   ...

Smoke-test the driver with a throwaway PVC:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: csi-smoke, namespace: default }
spec:
  accessModes: [ ReadWriteOnce ]
  resources: { requests: { storage: 1Gi } }
EOF
kubectl get pvc csi-smoke -w   # should reach Bound within ~30s
kubectl delete pvc csi-smoke

If the PVC stays Pending, look at kubectl describe pvc csi-smoke and kubectl -n kube-system logs -l app=csi-do-controller -c csi-provisioner.

Step 10 — Install metrics-server

The built-in HPA needs the metrics.k8s.io API. On a real cluster the default install works without --kubelet-insecure-tls:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system rollout status deploy/metrics-server
sleep 30
kubectl top nodes
# numbers, not <unknown>

Step 11 — Create the per-environment app Namespace

The CI deploy account in Part 7 will be namespace-scoped and cannot create namespaces. They must exist before CI runs.

kubectl create namespace "expressapp-${ENV}"

The app repo's base/ does not contain a Namespace. That stays here, in cluster-bootstrap.

Step 12 — Wire up your laptop

After running steps 1–11 for both environments you have ~/.kube/config-staging and ~/.kube/config-prod. Each one still uses the default kubeadm names (kubernetes-admin@kubernetes, kubernetes). Rename them per environment before merging, so contexts never collide.

Run the renaming on each kubeconfig separately, while pointing KUBECONFIG at just that file:

for ENV in staging prod; do
  cfg="$HOME/.kube/config-${ENV}"
  [ -f "$cfg" ] || continue
  KUBECONFIG="$cfg" kubectl config rename-context kubernetes-admin@kubernetes "do-${ENV}"
  KUBECONFIG="$cfg" kubectl config set-cluster kubernetes        --server="https://$(grep -oE 'https://[^:]+' "$cfg" | head -1 | sed 's|https://||')"  # no-op if already set
  KUBECONFIG="$cfg" kubectl config delete-cluster kubernetes 2>/dev/null || true   # rename below replaces it
done

A simpler equivalent is to rewrite the cluster and user names with sed so the merged file doesn't need any post-merge surgery:

for ENV in staging prod; do
  cfg="$HOME/.kube/config-${ENV}"
  [ -f "$cfg" ] || continue
  sed -i.bak \
    -e "s/^- name: kubernetes-admin$/- name: do-${ENV}-admin/" \
    -e "s/^- name: kubernetes$/- name: do-${ENV}/" \
    -e "s/cluster: kubernetes$/cluster: do-${ENV}/" \
    -e "s/user: kubernetes-admin$/user: do-${ENV}-admin/" \
    "$cfg"
  rm -f "${cfg}.bak"
  KUBECONFIG="$cfg" kubectl config rename-context "kubernetes-admin@kubernetes" "do-${ENV}" 2>/dev/null || true
done

Now merge into a single file with kubectl config view --flatten:

KUBECONFIG=~/.kube/config-staging:~/.kube/config-prod kubectl config view --flatten > ~/.kube/config.merged
mv ~/.kube/config.merged ~/.kube/config
chmod 600 ~/.kube/config
unset KUBECONFIG

kubectl config get-contexts
# CURRENT   NAME          CLUSTER       AUTHINFO
#           do-staging    do-staging    do-staging-admin
#           do-prod       do-prod       do-prod-admin

A safety-net trick: color your prompt by context so you never type the wrong command into prod:

# ~/.bashrc
kubeprompt() {
  local ctx; ctx=$(kubectl config current-context 2>/dev/null)
  case "$ctx" in
    do-prod)    printf "\e[31m[%s]\e[0m " "$ctx" ;;   # red
    do-staging) printf "\e[33m[%s]\e[0m " "$ctx" ;;   # yellow
    *)          printf "\e[32m[%s]\e[0m " "$ctx" ;;
  esac
}
export PS1='$(kubeprompt)\u@\h:\w\$ '

Step 13 — Repeat for the second cluster

Re-run steps 1–11 with ENV=prod. Same droplet sizes, same scripts. After both passes, run Step 12 once with both kubeconfigs in place.

Verification checklist

Run on each context:

kubectl --context "do-${ENV}" get nodes
# 3 Ready

kubectl --context "do-${ENV}" get nodes -o jsonpath='{range .items[*]}{.spec.providerID}{"\n"}{end}'
# digitalocean://... three times (proves DO CCM has labelled them)

kubectl --context "do-${ENV}" -n kube-system get pods | grep -E 'cilium|cloud-controller|csi-do|metrics-server'
# all Running

kubectl --context "do-${ENV}" get storageclass
# do-block-storage (default)

kubectl --context "do-${ENV}" top nodes
# numbers

kubectl --context "do-${ENV}" get ns "expressapp-${ENV}"
# Active

Completion signal

kubectl --context do-staging get nodes and kubectl --context do-prod get nodes both show three Ready nodes. On each cluster: Cilium is healthy, the DO cloud-controller-manager has annotated nodes with their DO providerID, the DO CSI driver is installed with do-block-storage as the default StorageClass, metrics-server is serving the metrics API, and the per-environment app namespace exists. Switching contexts is documented and obvious; your prompt color is different in do-staging and do-prod.

You're ready for 04. Edge: Ingress and TLS.

Create the cluster-bootstrap repo​

Prerequisites​

What you will do, twice​

Step 1 — Pick a region, size, and naming convention​

Step 2 — Create the droplets and capture their IPs​

Recover shell state in a new terminal​

Step 3 — Lock down with a DO firewall​

Step 4 — Prepare the OS on every node​

Step 5 — Bootstrap the control plane​

Step 6 — Join the workers​

Step 7 — Install Cilium​

Step 8 — Install the DO cloud-controller-manager​

Step 9 — Install the DigitalOcean CSI driver​

Step 10 — Install metrics-server​

Step 11 — Create the per-environment app Namespace​

Step 12 — Wire up your laptop​

Step 13 — Repeat for the second cluster​

Verification checklist​

Completion signal​