Elastic Stack Deployment

Environment

Server Name	IP Address
elasticsearch-01	10.30.185.200
elasticsearch-02	10.30.185.201
elasticsearch-03	10.30.185.202
elasticsearch-04	10.30.185.203
elasticsearch-05	10.30.185.204
kibana-01	10.30.185.205

Configure Elasticsearch

1. Add Parameters Required by Elasticsearch (all ES nodes)

Elasticsearch uses a mmapfs directory by default to store its indices. The Linux default limits on mmaps is usually too low, which can result in out-of-memory exceptions. This limit should be raised to 262144.

Run the following command to add the file /etc/sysctl.d/70-elasticsearch.conf with the attribute vm.max_map_count=262144:

echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/70-elasticsearch.conf > /dev/null

2. Tune Network Parameters (all ES nodes)

The default Linux network parameters are not optimal for high throughput applications, in particular a high volume of ingress UDP packets. This can result in dropped packets and lost data. Linux network performance for ElastiFlow can optimized by changing the parameters below.

Run the following command to add the file /etc/sysctl.d/60-net.conf with the recommended changes.

echo -e "net.core.netdev_max_backlog=4096\nnet.core.rmem_default=262144\nnet.core.rmem_max=67108864\nnet.ipv4.udp_rmem_min=131072\nnet.ipv4.udp_mem=2097152 4194304 8388608" | sudo tee /etc/sysctl.d/60-net.conf > /dev/null

3. Apply Changes (all ES nodes)

For changes to the above parameters to take effect the system can be restarted. Alternatively the following commands can be run to apply the changes without a reboot:

sudo sysctl -w vm.max_map_count=262144 && \
  sudo sysctl -w net.core.netdev_max_backlog=4096 && \
  sudo sysctl -w net.core.rmem_default=262144 && \
  sudo sysctl -w net.core.rmem_max=67108864 && \
  sudo sysctl -w net.ipv4.udp_rmem_min=131072 && \
  sudo sysctl -w net.ipv4.udp_mem='2097152 4194304 8388608'

4. Configure JVM Heap Size (all ES nodes)

If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason it’s best to start the JVM with the initial and maximum heap sizes set to equal values.

Add the file heap.options to /etc/elasticsearch/jvm.options.d and set -Xms and -Xmx to about one third of the system memory, but do not exceed 31g. For this example we will use 12GB of the available 32GB of memory for JVM heap.

echo -e "-Xms12g\n-Xmx12g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null

5. Increase System Limits (all ES nodes)

Increased system limits should be specified in a systemd attributes file for the elasticsearch service.

sudo mkdir /etc/systemd/system/elasticsearch.service.d && \
  echo -e "[Service]\nLimitNOFILE=131072\nLimitNPROC=8192\nLimitMEMLOCK=infinity\nLimitFSIZE=infinity\nLimitAS=infinity" | \
  sudo tee /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf > /dev/null

6. Copy Certificates to Elasticsearch Configuration Path

Copy TLS certificates to /etc/elasticsearch/certs.

7. Edit elasticsearch.yml (all ES nodes)

Edit the Elasticsearch configuration file, /etc/elasticsearch/elasticsearch.yml, replacing the contents of the file with the provided configurations.

NOTE: If you want Elasticsearch data to be stored on a different mount point, you must first create the directory and assign permissions to elasticsearch. For example, to store data on /mnt/data0, run sudo mkdir /mnt/data0/elasticsearch && sudo chown -R elasticsearch:elasticsearch /mnt/data0/elasticsearch. Then edit the path.data option in elasticsearch.yml specifying this path.

8. Enable and Start Elasticsearch Master Nodes

Execute the following commands on elasticsearch-01, elasticsearch-02 and elasticsearch-03 to start Elsticsearch and enable it run automatically when the server boots:

sudo systemctl daemon-reload && \
  sudo systemctl enable elasticsearch && \
  sudo systemctl start elasticsearch

Confirm Elasticsearch started successfully by executing:

sudo systemctl status elasticsearch

9. Set Passwords for Elasticsearch Built-in Accounts

Execute the following command on one of the running Elasticsearch nodes to setup passwords for the various built-in accounts:

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

The following will be displayed:

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]

Answer y, then enter and confirm passwords for the built-in Elasticsearch accounts.

10. Verify Elasticsearch

Ensure that the Elasticsearch REST API is available by running the following:

curl -XGET -k "https://elastic:PASSWORD@10.30.185.200:9200"

The output should be similar to the following:

{
  "name" : "elasticsearch-01",
  "cluster_name" : "elastiflow",
  "cluster_uuid" : "S5Y3Z2USSq2sR2TyOkLe3A",
  "version" : {
    "number" : "8.7.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

11. Enable and Start Elasticsearch Data Nodes

Execute the following commands on elasticsearch-04 and elasticsearch-05 to start Elasticsearch and enable it run automatically when the server boots:

sudo systemctl daemon-reload && \
  sudo systemctl enable elasticsearch && \
  sudo systemctl start elasticsearch

Configure Kibana

1. Copy CA and Certificates

Copy TLS certificates to /etc/kibana/certs.

2. Edit kibana.yml

Edit the Kibana configuration file /etc/kibana/kibana.yml, replacing the contents of the file with the provided configuration. Edit as necessary for your environment (especially elasticsearch.password).

4. Enable and Start Kibana

Execute the following commands:

sudo systemctl daemon-reload && \
  sudo systemctl enable kibana && \
  sudo systemctl start kibana

Confirm Kibana started successfully by executing:

sudo systemctl status kibana

You should now be able to access Kibana at https://10.30.185.205:5601.