Elasticsearch snapshot and restore (fs / S3 repository, SLM)

Sets up snapshots (backups) for an installed Elasticsearch 8.x / 9.x cluster and covers recovery: a shared file system (NFS + path.repo) or S3 (keystore) repository, activation through a rolling restart and verification, manual snapshots (indices, include_global_state, feature_states), scheduled snapshots and retention with SLM (expire_after/min_count/max_count), restore under a new name or over a closed index, feature state restore, monitoring, and migration to another cluster by registering the same repository read-only. The result is daily automatic snapshots that have been proven restorable. Snapshots are the only valid backup method in Elasticsearch; use reindex for mapping/shard changes within the same cluster, and the scenarios in the operations guide to bring back a single index quickly.

Topology: Elasticsearch 8.x / 9.x cluster (1–N nodes) + shared directory (NFS) or S3-compatible object storage

Before you start

• The cluster is healthy (green) and you have the password of elastic (or of a user with manage + snapshot privileges). Because the repository settings require restarting the nodes, make sure the cluster can tolerate the loss of one node (replicas are assigned).
• The HTTP CA file (…) exists on every node; API calls are verified against this CA.
• … is mounted from the same share (NFS) on all master and data nodes and is persistent in fstab. The UID/GID of the elasticsearch user must be the same on all nodes, or the share must map everything to a single user with all_squash,anonuid=…,anongid=…; otherwise the nodes cannot read each other's files. A local (non-shared) directory fails repository verification on a multi-node cluster.
• The … bucket has been created; the access key (access key / secret key) has s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, s3:ListMultipartUploadParts and s3:ListBucketMultipartUploads permissions on this bucket. The nodes can reach … over HTTPS.
• The share has enough free space: the first snapshot takes as much space as the on-disk size of the indices; later ones only add the changed segments (incremental).
• You know the version rules: a snapshot can be restored to a cluster on the same or a newer version; it cannot be restored to an older version. An index can be restored at most to the major version following the one it was created in (an index created in 8.x can go to 9.x; an index created in 7.x cannot be restored to 9.x as a regular index).
• You have root access on all nodes and a maintenance window for the rolling restart.

Phases of the guide

• API access (all nodes) — Temporary credentials file for the API · Check the cluster and existing repositories
• Shared file system preparation (all nodes) — Mount the NFS share · Set the directory permissions and test a write · Add the path.repo setting
• S3 credentials and client settings (all nodes) — Add the S3 access keys to the keystore · Add the S3 client settings
• Register and verify the repository — Register the repository · Verify the repository
• Manual snapshot — Create the test index · Take a snapshot · Monitor the snapshot status
• Automatic snapshots with SLM — Create the SLM policy · Run the policy now · Check the SLM status and statistics · Run retention manually
• Restore — Restore the index under a new name · Monitor the restore and compare document counts · Delete the restored copy · Close the index and restore it in place · Restore the feature state
• Migration to another cluster — Preparing the target cluster · Register the repository read-only · Restore the indices on the target cluster · Delete the credentials file on the target
• Monitoring and maintenance — Monitor snapshot health · Repository cleanup · Delete the credentials files · Regular recovery drills