MongoDB day-to-day operations and failure scenarios (replica set and sharded cluster)

Not an installation: a scenario-based runbook for running an installed MongoDB replica set and sharded cluster day to day and recovering them when they break. Health check, adding/removing members, priority/hidden/delayed members, replacing an arbiter with a data-bearing member, resyncing a lagging member, force reconfig after losing the majority, planned stepDown, oplog size, FCV; adding/removing shards, balancer, jumbo chunks and reshardCollection. If you are not sure what to do, start with the health check; restore from backup, security settings and major version upgrades are covered in separate MongoDB guides.

Topology: Replica set (3+ members) or sharded cluster (config server + shard replica sets + mongos)

Scenarios

• Health check: members, lag, oplog window (shards and balancer when sharded) — Read-only: member states, replication lag, oplog window, configuration; on a sharded cluster the shard list, balancer and chunk distribution. Start here if you are not sure what to do.
• Add a new member to the replica set (initial sync) — A new server has been built (same version, keyfile, replSetName); it is added with priority 0 / votes 0 and given a vote and priority once initial sync finishes and it becomes SECONDARY. No downtime.
• Remove a member from the replica set — The server is being retired or is permanently lost: the member is removed from the configuration and the remaining votes are checked to still form a majority. If it is the primary, step it down first.
• Change a member's role: priority, hidden, delayed member — Change a member's priority for becoming primary, make it a hidden member for reporting/backup or a delayed member as protection against accidental deletes, or return it to normal.
• Replace the arbiter with a data-bearing member (PSA → PSS) — Replica set with primary + secondary + arbiter: when a data member goes down, majority writes and cache pressure become a problem. A new data member is added and the arbiter is removed.
• Resync a lagging or corrupted member — The member is stuck in RECOVERING and reports 'too stale to catch up', or its data files are corrupted: the member's data directory is emptied and it is resynced from scratch with an initial sync. The other members keep running.
• Majority lost: force a reconfiguration with the surviving members (force reconfig) — Most members are permanently lost (site/server loss) and the survivors cannot elect a primary: rs.reconfig(cfg, {force: true}) with the surviving members. Last resort; used incorrectly it risks data loss and split brain.
• Planned primary change (rs.stepDown, making a specific member primary) — Maintenance/patching is due on the primary server: the primary steps down and a caught-up secondary becomes primary. The application may see write errors for a few seconds.
• Change the oplog size (replSetResizeOplog) — The oplog window is short: secondaries become 'too stale' after maintenance, the backup cannot capture the oplog, or it is not long enough for point-in-time recovery. It is grown on each member without downtime.
• Check and finalize the Feature Compatibility Version (FCV) — After an upgrade the FCV is still at the old version, or members run different versions: the binary version and FCV of every member are checked and the FCV is raised to the new version.
• Add a shard to the cluster (sh.addShard) — Capacity is running out: a new shard replica set (shardsvr) has been built and is added to the cluster; the balancer migrates chunks to it gradually.
• Remove a shard (removeShard: drain and movePrimary) — A shard is being retired: its chunks are drained to the other shards, the databases for which it is the primary shard are moved (movePrimary), and then it is removed from the cluster.
• Balancer: time window, stop / start, disable per collection — Chunk migrations cause load during peak hours, migrations must stop for maintenance/backup, or a single collection must not be balanced.
• Uneven distribution or jumbo chunks (diagnosis and fix) — One shard is much fuller than the others, the balancer does not migrate, there are chunks flagged 'jumbo', or moveChunk fails: the distribution, jumbo chunks and balancer errors are examined.
• Change the shard key (reshardCollection) — The current shard key produces a hot shard, jumbo chunks or scatter-gather queries: the collection is redistributed online with a new shard key (5.0+). At the end writes are blocked for ~2 s.

Before you start

• mongosh is installed on the management server and it has name resolution and port access …to mongos (…)…to all members (…)…. The names in the connection must match the member names in rs.conf().
• You have the password of the … user; its role is root or at least clusterAdmin + readAnyDatabase.… Connecting directly to a shard or config server replica set requires the shard-local admin user (created in the mongo-sharded guide).…
• A change record / maintenance window has been opened and the application team informed. Reconfig, stepDown and shard operations can cause an election or routing delay of a few seconds.
• The new member … has been prepared with the operating system, installation, keyfile and configuration phases of the replica set installation guide: same MongoDB version (…), same keyfile, replication.replSetName: …, mongod running but rs.initiate NOT run. Name resolution and port access work in both directions with the other members; the data disk has at least as much free space as the current data.
• During the initial sync all data is rewritten on the member's disk…; because the old directory is moved aside, free space of twice the data size is required…. The remaining members keep the majority, and the oplog window of the source member is longer than the initial sync will take.

Phases of the guide

• Status: members, lag and oplog (read-only) — Show member state and lag · Show the configuration (priority, votes, hidden, delay) · Interpret the result and pick the right scenario
• Status: shards, balancer and chunk distribution (read-only) — Show the cluster summary · Chunk distribution and jumbo chunks · Check the status of the shard replica sets
• 1. Diagnosis: is the new member ready? — Check the new member's service and version · See the current set's state and data size · Verify access to the new member
• 2. Safety net — Save the current replica set configuration
• 3. Apply: add the member and wait for it to sync — Add the member with priority 0 / votes 0 · Wait for the initial sync to finish · Give the member a vote and priority
• 4. Verification and follow-up — Verify the state · Update the application connection strings and monitoring
• 1. Diagnosis: is the member to remove the primary, is the majority kept? — Determine the member's role and the remaining vote count
• 2. Safety net — Save the current replica set configuration
• 3. Apply: remove the member — Remove the member from the configuration (rs.remove) · Stop mongod on the removed server
• 4. Verification and follow-up — Verify the state · Update connection strings, monitoring and firewall
• 1. Diagnosis — See the member's current settings and the oplog window
• 2. Safety net — Save the current replica set configuration
• 3. Apply — Change the member setting (rs.reconfig)
• 4. Verification — Verify the result
• 1. Diagnosis: arbiter and new member — See the arbiter and the write concern · Verify that the new member is ready
• 2. Safety net — Save the current replica set configuration
• 3. Apply: add the data member, remove the arbiter — Add the new member with priority 0 / votes 0 · Wait for the initial sync to finish · Give the new member a vote and priority · Remove the arbiter · Stop mongod on the arbiter server
• 4. Verification and follow-up — Verify the state and the write concern
• 1. Diagnosis: does the member really need a resync? — See the member's state and last message · See the error in the member's log
• 2. Safety net — Save the current replica set configuration
• 3. Apply: empty the data directory and run an initial sync — Stop mongod · …Move the data directory aside…Empty the data directory… · Start mongod (the initial sync starts) · Wait for the initial sync to finish
• 4. Verification and follow-up — Verify the state · Delete the old data directory
• 1. Diagnosis: is the majority really permanently lost? — See the state and optimes from a surviving member · Verify that the lost members are down
• 2. Safety net — Save the surviving member's configuration
• 3. Apply: force reconfig — Force a reconfiguration with the surviving members only · Verify that a primary has been elected
• 4. Verification and follow-up — Try a majority write · Re-add the lost members and fix the root cause
• 1. Diagnosis: are the secondaries caught up? — See the primary and the lag
• 2. Apply — Freeze the other secondaries (so that the target is elected) · Step down the primary (rs.stepDown)
• 3. Verification — Verify the new state
• 1. Diagnosis: current oplog size and window — Oplog size and window on each member