Check Patroni DCS (etcd/Consul) and REST API health

Command

# etcd v3 (default API in etcd 3.4+): member health, leader, raft term and DB size
etcdctl --endpoints=https://<etcd1>:2379,https://<etcd2>:2379,https://<etcd3>:2379 --cacert=/etc/etcd/ca.crt endpoint health --cluster -w table
etcdctl --endpoints=https://<etcd1>:2379 --cacert=/etc/etcd/ca.crt endpoint status --cluster -w table
etcdctl --endpoints=https://<etcd1>:2379 --cacert=/etc/etcd/ca.crt alarm list
# Consul:
consul members
consul operator raft list-peers
# Patroni REST API (on each member):
curl -s http://<member>:8008/cluster | python3 -m json.tool
curl -s -o /dev/null -w '%{http_code}\n' http://<member>:8008/primary   # 200 = leader, 503 = not leader

What it does

Checks quorum, leader and capacity of the DCS where Patroni holds the leader lock, and each member's own view of its role through the Patroni REST API. The /primary endpoint is the basis of load balancer (HAProxy) health checks.

When to use it

When Patroni logs 'DCS is not accessible', after an unexpected demote/failover, and as a routine health check.

Caution

If the DCS loses quorum, Patroni cannot renew the leader lock and demotes the primary to read-only (unless failsafe_mode, Patroni 3.0+, is enabled); an etcd outage is therefore a database write outage. When the etcd DB size reaches its quota (default 2 GB) a NOSPACE alarm stops writes. Run the DCS on at least 3 nodes on separate disks.