# Elastic Stack Deployment

### Environment

| Server Name      | IP Address    |
| ---------------- | ------------- |
| elasticsearch-01 | 10.30.185.200 |
| elasticsearch-02 | 10.30.185.201 |
| elasticsearch-03 | 10.30.185.202 |
| elasticsearch-04 | 10.30.185.203 |
| elasticsearch-05 | 10.30.185.204 |
| kibana-01        | 10.30.185.205 |

### Configure Elasticsearch

#### 1. Add Parameters Required by Elasticsearch (all ES nodes)

Elasticsearch uses a `mmapfs` directory by default to store its indices. The Linux default limits on mmaps is usually too low, which can result in out-of-memory exceptions. This limit should be raised to `262144`.

Run the following command to add the file `/etc/sysctl.d/70-elasticsearch.conf` with the attribute `vm.max_map_count=262144`:

```shell
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/70-elasticsearch.conf > /dev/null
```

#### 2. Tune Network Parameters (all ES nodes)

The default Linux network parameters are not optimal for high throughput applications, in particular a high volume of ingress UDP packets. This can result in dropped packets and lost data. Linux network performance for ElastiFlow can optimized by changing the parameters below.

Run the following command to add the file `/etc/sysctl.d/60-net.conf` with the recommended changes.

```shell
echo -e "net.core.netdev_max_backlog=4096\nnet.core.rmem_default=262144\nnet.core.rmem_max=67108864\nnet.ipv4.udp_rmem_min=131072\nnet.ipv4.udp_mem=2097152 4194304 8388608" | sudo tee /etc/sysctl.d/60-net.conf > /dev/null
```

#### 3. Apply Changes (all ES nodes)

For changes to the above parameters to take effect the system can be restarted. Alternatively the following commands can be run to apply the changes without a reboot:

```shell
sudo sysctl -w vm.max_map_count=262144 && \
  sudo sysctl -w net.core.netdev_max_backlog=4096 && \
  sudo sysctl -w net.core.rmem_default=262144 && \
  sudo sysctl -w net.core.rmem_max=67108864 && \
  sudo sysctl -w net.ipv4.udp_rmem_min=131072 && \
  sudo sysctl -w net.ipv4.udp_mem='2097152 4194304 8388608'
```

#### 4. Configure JVM Heap Size (all ES nodes)

If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason it’s best to start the JVM with the initial and maximum heap sizes set to equal values.

Add the file `heap.options` to `/etc/elasticsearch/jvm.options.d` and set `-Xms` and `-Xmx` to about one third of the system memory, but do not exceed `31g`. For this example we will use 12GB of the available 32GB of memory for JVM heap.

```shell
echo -e "-Xms12g\n-Xmx12g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null
```

#### 5. Increase System Limits (all ES nodes)

Increased system limits should be specified in a `systemd` attributes file for the `elasticsearch` service.

```shell
sudo mkdir /etc/systemd/system/elasticsearch.service.d && \
  echo -e "[Service]\nLimitNOFILE=131072\nLimitNPROC=8192\nLimitMEMLOCK=infinity\nLimitFSIZE=infinity\nLimitAS=infinity" | \
  sudo tee /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf > /dev/null
```

#### 6. Copy Certificates to Elasticsearch Configuration Path

Copy TLS certificates to `/etc/elasticsearch/certs`.

#### 7. Edit elasticsearch.yml (all ES nodes)

Edit the Elasticsearch configuration file, `/etc/elasticsearch/elasticsearch.yml`, replacing the contents of the file with the provided configurations.

> **NOTE**: If you want Elasticsearch data to be stored on a different mount point, you must first create the directory and assign permissions to `elasticsearch`. For example, to store data on `/mnt/data0`, run `sudo mkdir /mnt/data0/elasticsearch && sudo chown -R elasticsearch:elasticsearch /mnt/data0/elasticsearch`. Then edit the `path.data` option in `elasticsearch.yml` specifying this path.

#### 8. Enable and Start Elasticsearch Master Nodes

Execute the following commands on `elasticsearch-01`, `elasticsearch-02` and `elasticsearch-03` to start Elsticsearch and enable it run automatically when the server boots:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable elasticsearch && \
  sudo systemctl start elasticsearch
```

Confirm Elasticsearch started successfully by executing:

```shell
sudo systemctl status elasticsearch
```

#### 9. Set Passwords for Elasticsearch Built-in Accounts

Execute the following command on one of the running Elasticsearch nodes to setup passwords for the various built-in accounts:

```shell
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
```

The following will be displayed:

```shell
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]
```

Answer `y`, then enter and confirm passwords for the built-in Elasticsearch accounts.

#### 10. Verify Elasticsearch

Ensure that the Elasticsearch REST API is available by running the following:

```shell
curl -XGET -k "https://elastic:PASSWORD@10.30.185.200:9200"
```

The output should be similar to the following:

```json
{
  "name" : "elasticsearch-01",
  "cluster_name" : "elastiflow",
  "cluster_uuid" : "S5Y3Z2USSq2sR2TyOkLe3A",
  "version" : {
    "number" : "8.7.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```

#### 11. Enable and Start Elasticsearch Data Nodes

Execute the following commands on `elasticsearch-04` and `elasticsearch-05` to start Elasticsearch and enable it run automatically when the server boots:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable elasticsearch && \
  sudo systemctl start elasticsearch
```

### Configure Kibana

#### 1. Copy CA and Certificates

Copy TLS certificates to `/etc/kibana/certs`.

#### 2. Edit kibana.yml

Edit the Kibana configuration file `/etc/kibana/kibana.yml`, replacing the contents of the file with the provided configuration. Edit as necessary for your environment (especially `elasticsearch.password`).

#### 4. Enable and Start Kibana

Execute the following commands:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable kibana && \
  sudo systemctl start kibana
```

Confirm Kibana started successfully by executing:

```shell
sudo systemctl status kibana
```

You should now be able to access Kibana at `https://10.30.185.205:5601`.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://elastiflow.gitbook.io/docs/elastic-stack-deployment.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.