MySQL InnoDB Cluster day-to-day operations and failure scenarios (MySQL Shell 8.4)

Not an installation: a scenario-based runbook for the day-to-day operation and failure recovery of an InnoDB Cluster that is already running (Group Replication + MySQL Router, deployed with the MySQL Shell AdminAPI). Pick a scenario card: health check, adding members (clone / incremental) and removing them (including force), rejoining a failed member, restarting after the whole cluster went down (rebootClusterFromCompleteOutage), forceQuorumUsingPartitionOf after losing the majority, planned primary switchover, adding / re-bootstrapping Router, cluster options (consistency, expelTimeout, memberWeight, autoRejoinTries, exitStateAction) and rescan() for the metadata. Every scenario follows the pattern diagnosis → safety net → execution → verification → follow-up. If you are building the cluster for the first time, open the InnoDB Cluster installation guide; for classic source–replica replication, the replication operations guide; for the 8.0 → 8.4 migration, the upgrade guide.

Topology: 3–9 MySQL nodes (single-primary) + MySQL Router

Scenarios

• Cluster status and health check (is everything OK?) — Read-only: cluster.status({extended: 1}), member roles/states/versions, applier queues, the Router list and the error log; nothing changes, no downtime. Start here if you are unsure or an alert has fired; the last step points you to the right scenario.
• Add a new member (addInstance: clone / incremental) — You want to add a new MySQL node to the cluster, increase fault tolerance (3 → 5) or bring back a node that was removed. The cluster stays open for writes; during clone the donor sees disk/network load and the new node restarts.
• Remove a member (removeInstance, force if needed) — You want to remove a node from the cluster permanently (hardware return, downsizing), or the node is lost and has to be deleted from the metadata (force). Removing a secondary causes no downtime; removing the primary causes a write outage of a few seconds.
• Rejoin a failed member (MISSING / ERROR / OFFLINE) — A member restarted, lost the network or was expelled from the group; the majority is running and the application can write. With rejoinInstance the member catches up on what it missed and returns; no downtime for the cluster.
• Whole cluster went down: restart it (rebootClusterFromCompleteOutage) — All nodes went down and came back up because of a power / data center / hypervisor outage or similar; Group Replication is not running on any member and the application cannot write (the outage is ongoing). The cluster is restarted from the most up-to-date member.
• Majority lost (NO_QUORUM): force quorum with the remaining members — Most members disappeared at the same time (e.g. 2 out of 3) and the survivors stopped accepting writes with NO_QUORUM (writes hang). If you are certain the lost members are really down, the cluster is brought back with the surviving partition. Misuse creates a split-brain; losing a single member (the majority is still up) is not this scenario.
• Planned primary switchover (setPrimaryInstance) — You want to hand over the primary role to a specific secondary in a planned way for maintenance (OS patch, hardware), load balancing or a DR drill. Writes wait for a few seconds and open transactions are allowed to finish; Router redirects to the new primary on its own.
• Add Router or re-bootstrap it — You want to install Router on a new application server, re-bootstrap Router (version upgrade, rebuilt cluster, routerErrors warning) or clean up stale Router entries. During a re-bootstrap the application connections going through that Router drop briefly.
• Change cluster options (consistency, expelTimeout, memberWeight…) — You want to tune failover consistency, how long before a member is expelled during network flapping, failover priority, automatic rejoin, or how a member behaves on error. These are changed while the cluster is running, no restart needed; consistency takes effect for new sessions.
• Sync the metadata with the actual group (rescan) — cluster.status() mentions a member that is not in the metadata, or a stale entry that is no longer in the group (START GROUP_REPLICATION run by hand, old member never removed, server rebuilt). No downtime; only the metadata is updated.

Before you start

• MySQL Shell … and MySQL … are installed on the nodes involved; the cluster was built with the InnoDB Cluster installation guide or in an equivalent way with the AdminAPI (metadata exists). The operating system root can connect to the local mysql without a password. Take the cluster name in the form from dba.getCluster().getName() and the member addresses from the ip:port lines of the cluster.status() topology.
• The password of the … account is known (the same on every node). MySQL Shell asks for the password on every connection; with --save-passwords=never it does not store it.
• The nodes can reach each other on the MySQL ports (……:……, ……) and on the Group Replication communication ports.
• MySQL … is installed on the new node at the same minor version as the cluster members (required for clone), GTID is enabled, server_id differs from the other nodes and the Group Replication settings from the installation guide are in my.cnf; the firewall/SELinux ports are open.
• The Router server can reach the nodes on the MySQL port… and can reach repo.mysql.com….
• Approval for the change, a maintenance window (if needed) and communication with the application team are in place.

Phases of the guide

• Status: cluster, members and Router (read-only) — Get and save the detailed cluster status · Check members and queues with SQL · Scan the Group Replication error records · Check RW/RO routing through Router · Interpret the result and pick the right scenario
• 1. Diagnosis: are the new node and the cluster ready? — Check the new node's version and identity · Check the cluster status and member versions
• 2. Safety net — Save the current topology
• 3. Execution: add the member — Configure the new node and create the administrator account · Have the cluster check the new node's eligibility · Add the node to the cluster (…)
• 4. Verification — Verify that the new member is ONLINE
• 5. Follow-up — Update Router, monitoring and inventory
• 1. Diagnosis: is it safe to remove the member? — Show the target member's state and the remaining tolerance
• 2. Safety net — Save the current topology
• 3. Execution: remove the member — Remove the member from the cluster… (force)…
• If the node removed with force comes back — Stop and clean up Group Replication on the returning node
• 4. Verification — Verify the remaining members
• 5. Follow-up — Odd count, Router and monitoring
• 1. Diagnosis: why did the member drop out, can it rejoin? — The target member's state as seen by the cluster · Show the state of MySQL and Group Replication on the target node · Errant transaction check (on the target but not in the cluster)
• 2. Safety net — Save the current state
• 3. Execution: rejoin the member — Rejoin the member with rejoinInstance
• 4. Verification — Verify that the member is ONLINE
• 5. Follow-up — Root cause and automatic rejoin
• 1. Diagnosis: is the whole cluster really down, which member is the most up to date? — Start MySQL on every node and show the Group Replication state · Ask Shell for the most up-to-date member (dryRun)
• 2. Safety net — Save the GTID set of every node
• 3. Execution: start the cluster from the most up-to-date member — Restart the cluster
• 4. Verification — Verify that all members are ONLINE
• 5. Follow-up — Router, application and root cause
• 1. Diagnosis: is the quorum really lost? — Show the cluster status from the surviving node
• 2. Safety net: isolate the lost members — Make sure the lost nodes are down (fencing) · Save the state of the surviving partition
• 3. Execution: force the quorum — Establish quorum with the surviving partition
• 4. Verification and bringing back the lost members — Verify that writes are enabled · Rejoin the lost members (…)
• 5. Follow-up — Review the topology
• 1. Diagnosis: is the target eligible to become primary? — Check the members' state and versions
• 2. Safety net — Save the current state
• 3. Execution: change the primary — Hand the primary role over to …