Databases on Kubernetes (operator)

Symptom

Goal: Run the database on Kubernetes with an operator; instead of a bare StatefulSet, use the automatic failover, persistent storage and backup features the operator provides.

1. Why an operator: a StatefulSet alone is not enough

A plain StatefulSet gives you pods with stable identities and PVCs, but not the logic for failover, replication setup, backup and upgrades. An operator packages this database-specific operational knowledge as a controller + custom resource (CRD). You write a CR called 'Cluster', and the operator manages the StatefulSet, Service, Secret and backup jobs behind it. In production, running databases with an operator is generally preferred.

PostgreSQL

# operators for PostgreSQL:
#  - CloudNativePG (CNPG)
#  - Zalando postgres-operator
#  - Crunchy PGO

2. Choose and install the operator

Choose an operator for your engine: CloudNativePG or Zalando postgres-operator for PostgreSQL; the Percona operators (Percona Operator for MySQL / for MongoDB) are common for MySQL and MongoDB. Install the operator with Helm or a bundle manifest; this step adds the CRDs and the controller pod to the cluster.

PostgreSQL

# CloudNativePG installation
kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.24.0.yaml

MySQL

# Percona Operator for MySQL (Helm)
helm install my-op percona/pxc-operator

MongoDB

# Percona Operator for MongoDB (Helm)
helm install psmdb-op percona/psmdb-operator

3. Plan storage (StorageClass / PVC)

Database state lives in PVCs; specify the StorageClass to use in the CR. Choose a StorageClass that provides low-latency block storage, ideally local-like SSD; network file systems (NFS) are not recommended for most databases. Set the PVC size and expandability (allowVolumeExpansion) correctly from the start, because shrinking is hard.

PostgreSQL

spec:
  storage:
    storageClass: fast-ssd
    size: 50Gi

4. Apply the Cluster CR (replicas and automatic failover)

Specify the number of instances/replicas in the CR; the operator sets up synchronous/asynchronous replication and, if the primary fails, performs automatic failover by promoting a replica. The application should connect to the '-rw' (read-write) Service created by the operator; this service always points to the current primary. The connection target therefore does not change during failover.

PostgreSQL

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-db
spec:
  instances: 3
  storage:
    storageClass: fast-ssd
    size: 50Gi
# the application connects to: app-db-rw

5. Write backups to S3 via the CR

Operators also manage backups declaratively: you define the primary backup destination (object storage/S3) and WAL archiving in the CR. In CloudNativePG, S3 details are given with spec.backup.barmanObjectStore (deprecated since 1.26; for new deployments use the Barman Cloud Plugin and its ObjectStore resource); continuous WAL archiving + base backups enable point-in-time recovery. Reference credentials from a Kubernetes Secret.

PostgreSQL

spec:
  backup:
    barmanObjectStore:
      destinationPath: s3://company-backup/app-db
      s3Credentials:
        accessKeyId:
          name: s3-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: s3-creds
          key: SECRET_ACCESS_KEY

6. Scheduled backup and restore CRs

Create a Backup CR for a one-off backup and a ScheduledBackup CR for scheduled ones; take regular base backups with a cron expression. Restores are also declarative: you bootstrap a new Cluster from an existing object store backup with 'bootstrap.recovery'. Run restore drills regularly; taking backups is not enough, test the recovery.

PostgreSQL

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: nightly-backup
spec:
  schedule: "0 0 2 * * *"
  cluster:
    name: app-db