Redis / Valkey day-to-day operations and failure scenarios (Cluster and Sentinel)

Not an installation: a scenario-based runbook for managing an installed Redis Cluster or a Sentinel-managed master/replica setup day to day and recovering it when it breaks (Redis 7.2, Redis 8 and Valkey). Cluster: health check, adding/removing nodes, rebalance/reshard, manual failover (FORCE vs TAKEOVER), replacing a failed node and --cluster fix. Sentinel: status and quorum, SENTINEL FAILOVER, adding/removing nodes and SENTINEL RESET, changing settings. For both: memory pressure and big/hot key diagnosis. If you are not sure what to do, start with the health check; restore from backup and security hardening are covered in separate Redis guides.

Topology: Redis Cluster (3+ masters + replicas) or Sentinel (1 master + replicas + 3 Sentinels)

Scenarios

• Cluster health check (--cluster check, CLUSTER INFO / NODES) — Read-only: are all slots covered, do the nodes agree on the configuration, is there a failed (fail) node or a master without a replica. Start here if you are not sure what to do.
• Add a node to the cluster (master or replica) — A new instance (cluster-enabled yes, empty) is ready: it joins the cluster as a master and receives slots, or becomes a replica of a master. No downtime.
• Remove a node from the cluster (empty its slots + del-node) — A master or replica is being retired: the master's slots are moved to the other masters, then the node is removed from the cluster. No downtime; ASK redirections occur during the migration.
• Redistribute slots (rebalance / reshard) — Slots or memory are unbalanced across the masters, a new master has received no slots, or a given number of slots has to be moved from one master to another.
• Manual failover: promote a replica to master (CLUSTER FAILOVER, FORCE, TAKEOVER) — Maintenance is planned on the master's server (planned, no data loss), or the master is unreachable and no automatic failover happened (FORCE / TAKEOVER; risk of data loss).
• Replace a failed master / node (FORGET, new replica, --cluster fix) — A node's server is permanently lost: if its replica became master, the old node is forgotten and a new replica is added in its place; if it had no replica, its slots are left without an owner and are closed with --cluster fix (the data in those slots is lost).
• Sentinel status: master, replicas, Sentinels, quorum — Read-only: the master address the Sentinels see, the replicas, the other Sentinels, CKQUORUM and the recent failover events. Start here if you are not sure what to do in a Sentinel setup.
• Manual failover with Sentinel (SENTINEL FAILOVER) — Maintenance is planned on the master's server: Sentinel promotes the most suitable replica to master and points the other replicas and the clients to the new master. Downtime 1–2 s; no data loss while the master is running.
• Add / remove / replace a node in a Sentinel setup (SENTINEL RESET) — A replica or Sentinel has been permanently removed or newly added: the new replica connects with REPLICAOF, and the Sentinels' stale entries are cleared with SENTINEL RESET.
• Change the quorum and Sentinel settings (SENTINEL SET) — The number of Sentinels has changed, or failover triggers too late/too early: quorum, down-after-milliseconds, failover-timeout and parallel-syncs are changed on all Sentinels.
• Memory pressure: usage, fragmentation, eviction diagnosis — used_memory is approaching maxmemory, 'OOM command not allowed' errors, eviction is increasing, RSS is larger than expected, or the replica/AOF buffers have grown. The diagnosis is read-only; the remedies are described separately.
• Find big and hot keys (--bigkeys, --memkeys, --hotkeys) — One node's memory or CPU is higher than the others', there are latency spikes, DEL takes long: the largest and most frequently accessed keys are found and cleaned up safely (UNLINK).

Before you start

• … is installed on the administration server (it comes with …, /usr/bin/…) and it can reach …the … instance…all cluster nodes on both the client port and the cluster bus port (port + 10000)…the Sentinels (…) and Redis (…)….
• You have the … password (requirepass / masterauth); it is prompted for in the steps or taken from the REDIS_PASS environment variable. If you use ACLs, the user needs +@admin (cluster/sentinel commands) and +@keyspace.
• A change record has been opened and the application team informed. Failover and slot migration can cause brief MOVED/ASK redirections or dropped connections on the clients; cluster-aware clients follow these by themselves.
• The new instance … is ready as in the Redis Cluster installation guide: same … version, cluster-enabled yes, same requirepass/masterauth, empty data directory (no old nodes-.conf, RDB/AOF), client and bus port (port + 10000) open in the firewall.

Phases of the guide

• Status: slot coverage, nodes and configuration consistency (read-only) — Check the cluster (--cluster check) · Show the nodes, roles and replica distribution · Interpret the result
• 1. Diagnosis: is the new node empty and reachable? — Check the new node and the cluster
• 2. Apply: add the node — Add the node to the cluster (--cluster add-node) · Distribute slots to the new master (rebalance) · Plan a replica for the new master
• 3. Verification — Verify the cluster and the new node
• 1. Diagnosis: is the node a master, does it have slots? — Show the role and slots of the node to remove
• 2. Apply: drain the slots and remove the node — Move the master's slots to the other masters · Remove the node from the cluster (--cluster del-node) · Disable the service on the removed server
• 3. Verification — Verify the cluster
• 1. Diagnosis: current distribution — Slots, keys and memory per master
• 2. Apply — …Balance the slots (rebalance)…Move … slots (reshard)…
• 1. Diagnosis — Show the replica's role and master link
• 2. Apply — Start the failover (CLUSTER FAILOVER… FORCE… TAKEOVER…)
• 3. Verification — Verify the new layout
• 1. Diagnosis: lost node and slot coverage — Show the nodes in fail state and the slots without an owner
• 2. Apply: forget the lost node — Make all nodes forget the lost node (CLUSTER FORGET) · Close the slots without an owner (--cluster fix) · Add the new node as a replica
• 3. Verification and follow-up — Verify the cluster
• Status: master, replicas, Sentinels (read-only) — The master and quorum as seen by Sentinel · List the replicas and the other Sentinels · Search the Sentinel log for recent failover events
• 1. Diagnosis — Verify that the quorum and the replicas are ready
• 2. Apply — Start the failover (SENTINEL FAILOVER)
• 3. Verification — Verify the new master and the replicas
• 1. Diagnosis — Replica and Sentinel counts known to the Sentinels
• 2. Apply — Attach the new replica to the master · Shut down the service of the removed replica · Start the new Sentinel · Shut down the removed Sentinel · SENTINEL RESET on all Sentinels
• 3. Verification — Check the counts and the quorum again
• 1. Diagnosis — Show the current settings on each Sentinel
• 2. Apply — Set … to … on all Sentinels
• 1. Diagnosis: memory, limit and eviction — Show memory usage and the limit (INFO memory) · MEMORY DOCTOR and MEMORY STATS · Show the memory settings
• 2. Remedies — Remedy by situation
• 1. Diagnosis: big and hot keys — Biggest keys per type (--bigkeys) · Biggest keys by memory (--memkeys) · Hot keys (--hotkeys) · Inspect the size and type of a key
• 2. Remedy — Delete the big key safely (UNLINK)