MongoDB sharded cluster installation (config servers + shards + mongos)

A sharded cluster for when the data or write load outgrows a single replica set: a three-member config server replica set, two or three shard replica sets and two mongos routers; internal authentication with a keyfile, authorization, sh.addShard, shard key selection, sharding a collection, the balancer and verification of the distribution. The result is a horizontally scalable cluster that applications reach only through the mongos routers. If the data fits on a single server, a replica set installation is simpler and sufficient.

Topology: 3 config servers + 2×3 shard members + 2 mongos (at least 11 servers; smaller for testing)

Before you start

• All servers run the same, up-to-date operating system; time is synchronized with NTP (chrony). Clock skew between members causes session and oplog problems.
• RAM and CPU match the values in the form (the WiredTiger cache is calculated from this value). Swap is configured but small.
• The data directory (…) is on a separate disk/LV, formatted with XFS, mounted with noatime and persistent in fstab. ext4 with WiredTiger can cause performance problems.
• Every server name resolves from all servers and from the application servers (DNS or /etc/hosts). The replica set configuration uses names, not IPs.
• Ports …, …, … are open between the servers; the application network can reach these ports. These ports are not used by another service.
• The servers can reach repo.mongodb.org and pgp.mongodb.com over HTTPS, or an internal mirror is configured.
• Neither the operating system's own mongodb package nor an old MongoDB installation is present.
• scp works from the first server to the others as root (or a sudo-capable user) (to copy the keyfile). Otherwise move the file manually by a secure method.
• You are logged in with root or sudo privileges; the maintenance window and a server snapshot (fallback) plan are ready.
• The shard key was decided together with the application team: the filter field of the most frequent queries, high cardinality, an even distribution of values. The shard key can be changed later with reshardCollection, but it is costly.

Phases of the guide

• Operating system preparation (all servers) — Hostname: … · Hostname: … · Hostname: … · Hostname: … · Hostname: … · System update and required tools …
• Installing the MongoDB software — Add the MongoDB … repository · Add the MongoDB … repository · Install the MongoDB server software (config servers and shards) · Install mongos and mongosh (routers) · Prepare the data directory · SELinux policy, labels and ports …
• Keyfile (internal cluster authentication) — Create the keyfile · Copy the keyfile to all servers · Keyfile permissions
• Config server replica set — mongod.conf (config server): … · Start mongod · rs.initiate (config server)
• Shard A (replica set shardA) — mongod.conf (shard A): … · Start mongod · rs.initiate (shard A) · Shard-local administrator user (shard A)
• Shard B (replica set shardB) — mongod.conf (shard B): … · Start mongod · rs.initiate (shard B) · Shard-local administrator user (shard B)
• Shard C (replica set shardC) — mongod.conf (shard C): … · Start mongod · rs.initiate (shard C) · Shard-local administrator user (shard C)
• mongos (router) — mongos.conf: … · mongos systemd service · Start mongos · Create the cluster administrator · Add the shards (sh.addShard) · Create the application user
• Shard key and sharding the collection — Choosing the shard key (decision step) · Shard the collection · Balancer state · Balancer time window
• Verification — Cluster status · Distribution test · Config server replica set status · Application connection string
• Post-installation — Log rotation · Backup strategy · Monitoring