Manifests with Kustomize and Push-Based Deploys

The Express app is running by hand. This chapter makes the deployment repeatable, environment-aware, and continuous. You'll lay out the app repository so manifests live next to source, use Kustomize to express the differences between staging and prod, and wire GitHub Actions to apply them to the right cluster on the right trigger.

Kustomize is a manifest layering tool that ships inside kubectl (the -k flag) and as a standalone binary. You write a base — a complete set of manifests good enough to deploy on its own — and then per-environment overlays that point at the base and describe what's different (different namespace, different replica count, different image tag, an extra annotation, etc.). Overlays apply patches on top of the base; nothing is duplicated. The killer feature is that kustomize build overlays/prod/ deterministically prints the final YAML, so you can diff, validate, and kubectl apply -k it like any other manifest.

The base/ you write here is deliberately minimal. CRD-typed resources (SealedSecret, ServiceMonitor) are added later, in the chapter where their controllers and CRDs become available. Applying a manifest that references a CRD before its controller is installed will fail. See the manifest evolution table at the end.

Prerequisites

Images publish to DOCR on every push (Part 5).
PR status checks gate merges (Part 6).
Each cluster already has the per-environment app Namespace (created in Part 3 by cluster-bootstrap). The base does not include a Namespace.

Final layout

By the end of this chapter the app repo looks like this:

expressapp/
  app/
    server.js
    Dockerfile
    package.json
  k8s/
    base/
      deployment.yaml
      service.yaml
      ingress.yaml
      configmap.yaml
      hpa.yaml
      serviceaccount.yaml
      kustomization.yaml
    overlays/
      staging/
        kustomization.yaml
        deployment-patch.yaml
        ingress-patch.yaml
      prod/
        kustomization.yaml
        deployment-patch.yaml
        ingress-patch.yaml
  .github/workflows/
    pr.yml
    build.yml
    deploy-staging.yml
    deploy-prod.yml

Namespaces stay in cluster-bootstrap. See Part 3.

Step 1 — Write the base

A ServiceAccount is the identity a pod runs as when it talks to the Kubernetes API. Every pod gets one, even if you don't declare it (default). Giving the app its own ServiceAccount means you can grant it permissions later (or, as here, deny them by turning off the auto-mounted token) without affecting anything else.

k8s/base/serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expressapp
automountServiceAccountToken: false

k8s/base/configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: expressapp
data:
  LOG_LEVEL: "info"

k8s/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expressapp
  labels: { app.kubernetes.io/name: expressapp }
spec:
  replicas: 2
  selector:
    matchLabels: { app.kubernetes.io/name: expressapp }
  template:
    metadata:
      labels: { app.kubernetes.io/name: expressapp }
    spec:
      serviceAccountName: expressapp
      imagePullSecrets:
        - name: docr
      containers:
        - name: app
          # The full image name (registry + path) is set per environment by
          # the CI deploy workflows via `kustomize edit set image`. The
          # placeholder below is rewritten before `kubectl apply -k`.
          image: REGISTRY_PLACEHOLDER/expressapp:placeholder
          imagePullPolicy: IfNotPresent
          ports:
            - { name: http, containerPort: 3000 }
          envFrom:
            - configMapRef: { name: expressapp }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 500m, memory: 256Mi }
          startupProbe:   { httpGet: { path: /readyz, port: http }, failureThreshold: 30, periodSeconds: 1 }
          readinessProbe: { httpGet: { path: /readyz, port: http }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz, port: http }, periodSeconds: 10, failureThreshold: 3 }

k8s/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: expressapp
spec:
  selector: { app.kubernetes.io/name: expressapp }
  ports:
    - { name: http, port: 80, targetPort: http }

k8s/base/hpa.yaml (uses metrics-server, not kube-prometheus-stack):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: expressapp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: expressapp
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }

k8s/base/ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: expressapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod    # overlay can change this
    # The HTTP->HTTPS redirect is handled by Traefik's EntryPoint config from Part 4.
    # The security-headers Middleware lives in the `traefik` namespace.
    traefik.ingress.kubernetes.io/router.middlewares: traefik-security-headers@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: app.example.com                              # overlay must change this
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: expressapp, port: { number: 80 } }
  tls:
    - hosts: [ app.example.com ]
      secretName: expressapp-tls

k8s/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
  app.kubernetes.io/name: expressapp
  app.kubernetes.io/part-of: expressapp

resources:
  - serviceaccount.yaml
  - configmap.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - hpa.yaml

images:
  - name: REGISTRY_PLACEHOLDER/expressapp
    newName: REGISTRY_PLACEHOLDER/expressapp
    newTag: placeholder

REGISTRY_PLACEHOLDER is the literal string the deploy workflow rewrites. Each workflow does:

kustomize edit set image REGISTRY_PLACEHOLDER/expressapp=${DOCR_REGISTRY}/expressapp:${SHA}

where DOCR_REGISTRY is the repository variable you set in Part 5 (e.g. registry.digitalocean.com/k8slearn-yourname). Keeping the registry out of the committed manifests means the same base/ works for any reader who runs the course with their own DOCR.

Notice what's not there: no Namespace, no SealedSecret, no ServiceMonitor. Those come later.

Step 2 — Write the staging overlay

k8s/overlays/staging/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: expressapp-staging
namePrefix: ""

resources:
  - ../../base

patches:
  - path: deployment-patch.yaml
  - path: ingress-patch.yaml

configMapGenerator:
  - name: expressapp
    behavior: merge
    literals:
      - ENV=staging
      - LOG_LEVEL=debug

replicas:
  - name: expressapp
    count: 2

k8s/overlays/staging/deployment-patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata: { name: expressapp }
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 250m, memory: 128Mi }

k8s/overlays/staging/ingress-patch.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: expressapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  rules:
    - host: app.staging.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: expressapp, port: { number: 80 } }
  tls:
    - hosts: [ app.staging.example.com ]
      secretName: expressapp-tls

Step 3 — Write the prod overlay

k8s/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: expressapp-prod

resources:
  - ../../base

patches:
  - path: deployment-patch.yaml
  - path: ingress-patch.yaml

configMapGenerator:
  - name: expressapp
    behavior: merge
    literals:
      - ENV=prod
      - LOG_LEVEL=info

replicas:
  - name: expressapp
    count: 3

k8s/overlays/prod/deployment-patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata: { name: expressapp }
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { cpu: 500m, memory: 256Mi }

k8s/overlays/prod/ingress-patch.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: expressapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: expressapp, port: { number: 80 } }
  tls:
    - hosts: [ app.example.com ]
      secretName: expressapp-tls

Step 4 — Verify the manifests render

kustomize build k8s/overlays/staging | less
kustomize build k8s/overlays/prod    | less

Read each rendered output. Does the Deployment go into expressapp-staging/expressapp-prod? Does the Ingress use the right hostname and the right cluster-issuer? Does the HPA target the right Deployment? If anything is off, fix the overlay until both renders look right.

Run a diff against the live cluster to see what would change:

kubectl --context do-staging diff -k k8s/overlays/staging
kubectl --context do-prod    diff -k k8s/overlays/prod

A non-zero exit code means there are differences; that's fine and expected on the first apply. The point is that you have a tool that previews every change before it lands.

Step 5 — First apply, by hand

Just once, apply by hand so you understand what CI will be doing. Read DOCR_REGISTRY back from doctl so you don't hardcode the value:

DOCR_REGISTRY=$(doctl registry get --format Endpoint --no-header)
SHA=$(gh run list --workflow build.yml --limit 1 --json headSha --jq '.[0].headSha')

cd k8s/overlays/staging
kustomize edit set image \
  REGISTRY_PLACEHOLDER/expressapp="${DOCR_REGISTRY}/expressapp:${SHA}"
cd -

kubectl --context do-staging apply -k k8s/overlays/staging
kubectl --context do-staging -n expressapp-staging rollout status deploy/expressapp

(Reset kustomization.yaml before committing. CI will set the image tag on every run; you don't want a real SHA committed in the overlay.)

cd k8s/overlays/staging
git checkout kustomization.yaml
cd -

Step 6 — CI service accounts (namespace-scoped)

Time for RBAC (Role-Based Access Control), Kubernetes' authorization model. A Role is a list of allowed verbs (get, list, create, patch, ...) on specific resource kinds inside one namespace. A ClusterRole is the same idea but cluster-wide. A RoleBinding (or ClusterRoleBinding) attaches a Role to a subject — usually a ServiceAccount or a user. RBAC denies by default: if no binding grants you a verb, you can't do it.

Create a dedicated ServiceAccount in each cluster for CI, with a Role (not ClusterRole) that grants only what kubectl apply -k needs. Do this in cluster-bootstrap so it survives cluster rebuilds:

cluster-bootstrap/ci-deployer/staging.yaml:

apiVersion: v1
kind: ServiceAccount
metadata: { name: ci-deployer, namespace: expressapp-staging }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: ci-deployer, namespace: expressapp-staging }
# Note: `kubectl apply` is authorized as create + update + patch. There's no
# "apply" RBAC verb. Server-side apply uses patch. `delete` is included so CI
# can drop resources that get removed from the kustomization.
rules:
  - apiGroups: [""]
    resources: ["configmaps","secrets","services","serviceaccounts"]
    verbs: ["get","list","watch","create","update","patch","delete"]
  - apiGroups: ["apps"]
    resources: ["deployments","replicasets"]
    verbs: ["get","list","watch","create","update","patch","delete"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get","list","watch","create","update","patch","delete"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get","list","watch","create","update","patch","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: ci-deployer, namespace: expressapp-staging }
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: expressapp-staging
---
apiVersion: v1
kind: Secret
metadata:
  name: ci-deployer-token
  namespace: expressapp-staging
  annotations:
    kubernetes.io/service-account.name: ci-deployer
type: kubernetes.io/service-account-token

The same shape, with expressapp-prod substituted, lives at cluster-bootstrap/ci-deployer/prod.yaml.

Apply on each cluster:

kubectl --context do-staging apply -f cluster-bootstrap/ci-deployer/staging.yaml
kubectl --context do-prod    apply -f cluster-bootstrap/ci-deployer/prod.yaml

Build a kubeconfig from the service-account token and store it as a GitHub secret. A helper script:

build_kubeconfig() {
  local ctx="$1" ns="$2" outfile="$3"
  local server token ca
  server=$(kubectl --context "$ctx" config view --minify -o jsonpath='{.clusters[].cluster.server}')
  ca=$(kubectl --context "$ctx" -n "$ns" get secret ci-deployer-token -o jsonpath='{.data.ca\.crt}')
  token=$(kubectl --context "$ctx" -n "$ns" get secret ci-deployer-token -o jsonpath='{.data.token}' | base64 -d)
  cat > "$outfile" <<EOF
apiVersion: v1
kind: Config
clusters:
  - name: cluster
    cluster:
      server: ${server}
      certificate-authority-data: ${ca}
users:
  - name: ci
    user: { token: ${token} }
contexts:
  - name: ci
    context: { cluster: cluster, user: ci, namespace: ${ns} }
current-context: ci
EOF
}

build_kubeconfig do-staging expressapp-staging /tmp/kubeconfig-staging
build_kubeconfig do-prod    expressapp-prod    /tmp/kubeconfig-prod

gh secret set KUBECONFIG_STAGING < /tmp/kubeconfig-staging
gh secret set KUBECONFIG_PROD    < /tmp/kubeconfig-prod
rm -f /tmp/kubeconfig-staging /tmp/kubeconfig-prod

This is the credential the deploy workflows below read. The service account can't create namespaces, can't read other namespaces, and can't bind itself to anything new. If CI is compromised, the blast radius is exactly that one namespace.

Step 7 — The deploy workflows

Both workflows live in the app repo's .github/workflows/. They weren't written in Part 6 because they need the Kustomize overlays (Step 1–4 above) and the namespace-scoped CI service accounts (Step 6) to exist first.

`.github/workflows/deploy-staging.yml`

Triggered on push to master. No environment approval; the staging cluster is the safety net for the prod path.

name: deploy-staging

on:
  push:
    branches: [master]

concurrency:
  group: deploy-staging
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging
    env:
      DOCR_REGISTRY: ${{ vars.DOCR_REGISTRY }}
    steps:
      - uses: actions/checkout@v4
      - name: kubectl
        run: |
          curl -sL -o /usr/local/bin/kubectl \
            "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable-1.34.txt)/bin/linux/amd64/kubectl"
          chmod +x /usr/local/bin/kubectl
      - name: kustomize
        run: |
          curl -sL "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
          sudo mv kustomize /usr/local/bin/
      - name: Write kubeconfig
        run: |
          mkdir -p ~/.kube
          printf '%s' '${{ secrets.KUBECONFIG_STAGING }}' > ~/.kube/config
          chmod 600 ~/.kube/config
      - name: Pin staging image to this commit
        working-directory: k8s/overlays/staging
        run: |
          kustomize edit set image \
            REGISTRY_PLACEHOLDER/expressapp="${DOCR_REGISTRY}/expressapp:${GITHUB_SHA}"
      - name: Apply
        run: kubectl apply -k k8s/overlays/staging
      - name: Rollout status
        run: kubectl -n expressapp-staging rollout status deploy/expressapp --timeout=5m

`.github/workflows/deploy-prod.yml`

Triggered on a v* tag. First job is the provenance check defined in Part 6, inlined here. The deploy job is needs: provenance-check so no cluster credentials are mounted unless the check passes.

name: deploy-prod

on:
  push:
    tags: [ 'v*' ]

concurrency:
  group: deploy-prod
  cancel-in-progress: false

permissions:
  contents: read

jobs:
  provenance-check:
    runs-on: ubuntu-latest
    outputs:
      tag-sha: ${{ steps.resolve.outputs.tag-sha }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          ref: ${{ github.ref }}
      - id: resolve
        run: |
          TAG="${GITHUB_REF##refs/tags/}"
          TAG_SHA="$(git rev-list -n 1 "$TAG")"
          echo "tag-sha=$TAG_SHA" >> "$GITHUB_OUTPUT"
      - name: Tag must be reachable from origin/master
        env:
          TAG_SHA: ${{ steps.resolve.outputs.tag-sha }}
        run: |
          git fetch origin master:refs/remotes/origin/master
          if ! git merge-base --is-ancestor "$TAG_SHA" origin/master; then
            echo "::error::Tag commit $TAG_SHA is not an ancestor of origin/master."
            exit 1
          fi

  deploy:
    needs: provenance-check
    runs-on: ubuntu-latest
    environment: prod
    env:
      DOCR_REGISTRY: ${{ vars.DOCR_REGISTRY }}
      TAG_SHA: ${{ needs.provenance-check.outputs.tag-sha }}
    steps:
      - uses: actions/checkout@v4
      - name: kubectl
        run: |
          curl -sL -o /usr/local/bin/kubectl \
            "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable-1.34.txt)/bin/linux/amd64/kubectl"
          chmod +x /usr/local/bin/kubectl
      - name: kustomize
        run: |
          curl -sL "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
          sudo mv kustomize /usr/local/bin/
      - name: Write kubeconfig
        run: |
          mkdir -p ~/.kube
          printf '%s' '${{ secrets.KUBECONFIG_PROD }}' > ~/.kube/config
          chmod 600 ~/.kube/config
      - name: Pin prod image to the tagged SHA
        working-directory: k8s/overlays/prod
        run: |
          kustomize edit set image \
            REGISTRY_PLACEHOLDER/expressapp="${DOCR_REGISTRY}/expressapp:${TAG_SHA}"
      - name: Apply
        run: kubectl apply -k k8s/overlays/prod
      - name: Rollout status
        run: kubectl -n expressapp-prod rollout status deploy/expressapp --timeout=5m

Push and confirm

git add k8s/ .github/workflows/deploy-staging.yml .github/workflows/deploy-prod.yml
git commit -m "feat(k8s): kustomize base + staging/prod overlays + deploy workflows"
git push origin master
gh run watch

The staging deploy runs automatically. Tag a release to exercise the prod path:

git tag v0.2.0
git push origin v0.2.0
gh run watch
# provenance-check passes, deploy waits for prod environment approval, then rolls

Confirm in both clusters:

kubectl --context do-staging -n expressapp-staging get deploy,svc,ing,hpa
kubectl --context do-prod    -n expressapp-prod    get deploy,svc,ing,hpa

Browse to https://app.staging.example.com and https://app.example.com. Both should return the Express app's JSON.

Manifest evolution

The base and overlays grow as later chapters install the matching controllers. This chapter ends with the row after 07:

After part	`base/` contains	overlay additions
04 (edge)	`Deployment`, `Service`, `Ingress`, `ConfigMap`, `HPA`, `ServiceAccount`	—
07 (this chapter)	unchanged	per-env: `replicas`, `host`, `cluster-issuer`
08 (sealed-secrets)	unchanged	+ `SealedSecret` per env
09 (observability)	+ `ServiceMonitor` (CRD now exists in cluster)	unchanged
12 (polish)	+ `PodDisruptionBudget`, topology spread, refined requests/limits	unchanged

Completion signal

Merging a PR to master deploys to staging without manual intervention. kubectl --context do-staging -n expressapp-staging rollout history deploy/expressapp shows the new revision.
Pushing a v* tag against a master commit triggers the provenance check, asks an approver, and then deploys the same image to prod.
kustomize build k8s/overlays/staging and kustomize build k8s/overlays/prod render cleanly with no references to CRDs.

The CI service account in each namespace can apply the overlay but can't list namespaces or create new ones. Server-side apply is authorized as patch, so that's what we check:

SA=system:serviceaccount:expressapp-staging:ci-deployer
kubectl --context do-staging --as="$SA" auth can-i get    namespaces                                 # no
kubectl --context do-staging --as="$SA" auth can-i create namespaces                                 # no
kubectl --context do-staging --as="$SA" auth can-i get    secrets -n kube-system                     # no
kubectl --context do-staging --as="$SA" auth can-i create deployments -n expressapp-staging          # yes
kubectl --context do-staging --as="$SA" auth can-i update deployments -n expressapp-staging          # yes
kubectl --context do-staging --as="$SA" auth can-i patch  deployments -n expressapp-staging          # yes (server-side apply)

You're ready for 08. Sealed Secrets.

Prerequisites​

Final layout​

Step 1 — Write the base​

Step 2 — Write the staging overlay​

Step 3 — Write the prod overlay​

Step 4 — Verify the manifests render​

Step 5 — First apply, by hand​

Step 6 — CI service accounts (namespace-scoped)​

Step 7 — The deploy workflows​

.github/workflows/deploy-staging.yml​

.github/workflows/deploy-prod.yml​

Push and confirm​

Manifest evolution​

Completion signal​