Elasticsearch cluster installation (3+ nodes, TLS, Kibana)

Builds an Elasticsearch 8.x / 9.x cluster of 3 or more nodes from scratch: Elastic repository, heap sized to RAM, memory lock, vm.max_map_count, automatic security on the first node (TLS + elastic password), the other nodes joining with an enrollment token, combined or separate master/data roles, Kibana installation and enrollment, a shared snapshot repository and a nightly SLM policy. The result is a green cluster protected by TLS and passwords, with Kibana and a snapshot taken every night. Day-to-day operation of the installed cluster (shards, disk, adding/removing nodes), security hardening, snapshot/restore and version upgrades are covered in separate Elasticsearch guides.

Topology: 3+ Elasticsearch nodes (+ optional Kibana server)

Before you start

• All servers run the same, supported operating system release and are up to date. Clocks are synchronized (chrony): with clock skew, TLS certificates and tokens can be treated as invalid.
• Each node has … GB RAM (heap will be … MB) and at least 2 CPUs. Elasticsearch ships with its own JDK; do not install Java separately.
• … has enough free space (when the disk is 85% full no new shards are allocated to the data node; at 90% shards start moving to other nodes). SSD is recommended.
• Hostnames and IPs are static: ……=……, ………, …=……. Open: 9300 (transport) between the nodes, 9200 (HTTPS) from clients to the nodes…, 5601 from users to Kibana….
• The servers can reach artifacts.elastic.co over HTTPS (or an internal mirror is configured). Once the repository is added, the command below lists the installable versions.
• There is no old Elasticsearch installation or data directory on the servers (data from an old cluster prevents the node from joining the new cluster).
• … is mounted from the same NFS share on all master and data nodes (persistent in fstab). The elasticsearch user must have write permission on the share: if the user's UID is not the same on all nodes, use all_squash,anonuid=… on the NFS side or align the UIDs.
• Root/sudo access, a maintenance window and server snapshots are ready. Enrollment tokens are valid for 30 minutes; run the steps that join the other nodes without a break.

Phases of the guide

• Operating system preparation — Set the hostname · Set the hostname (Kibana) · Configure name resolution (/etc/hosts) · Install basic tools and synchronize the clock · Write kernel settings (vm.max_map_count) · Apply the sysctl settings …
• Elastic repository and software installation — Import the Elastic GPG key · Define the Elastic repository · Define the Elastic repository · List the installable versions · Install Elasticsearch · Lock the version
• Heap, memory lock and directories — Set the heap size · Set file ownership (heap.options) · Allow memory locking in systemd · Reload systemd · Prepare the data directory · Set permissions on the snapshot directory
• First node: bootstrap the cluster — Write elasticsearch.yml (node 1) · Set file ownership (elasticsearch.yml) · Start the service (node 1) · Set the elastic user's password · Create a temporary credentials file for API access · Check the cluster status (single node) …
• Complete cluster formation — Verify that all nodes have joined · Remove the cluster.initial_master_nodes line · Convert node 1 to a dedicated master: drain its shards · Set node 1's role to master and clean its data directory · Remove the temporary allocation rule
• Verification — Check cluster health · Check memory lock and heap · Check master election and the voting configuration · Check shard distribution with a test index · Check the log file
• Kibana installation and enrollment — Install Kibana · Set the listen address · Create a Kibana enrollment token · Connect Kibana to the cluster (kibana-setup) · Connect Kibana to multiple nodes · Generate the encryption keys …
• Snapshot repository — Register and verify the repository · Define the nightly snapshot policy (SLM)
• Post-installation — Create the application user and role · Prepare the CA certificate for clients · Apply the monitoring and backup checklist · Delete the temporary credentials file