What Are Snapshots?

Elasticsearch Snapshots allow you to create backups of your Elasticsearch indices and cluster metadata. These backups are stored in a remote or local repository (such as a local or remote file system, Microsoft Azure, Amazon S3, or Google Cloud Storage). The snapshots can then be used to restore data in case of data loss, or cluster migrations.

The main components and concepts of Elasticsearch Snapshot and Restore include:

1. Snapshot: A snapshot is a backup of one or more Elasticsearch indices, including all the data and metadata associated with them. Snapshots are incremental, meaning they only store data that has changed since the last snapshot. This makes them more space-efficient and faster to create compared to full backups.

2. Repository: A repository is a storage location where snapshots are saved. It can be a local file system, a remote file system, or a cloud-based storage service (such as Amazon S3, Microsoft Azure or Google Cloud Storage). Repositories need to be registered with Elasticsearch before they can be used for snapshot and restore operations.

3. Restore: The restore process involves retrieving a snapshot from a repository and applying it to the Elasticsearch cluster. This process can be used to recover lost data, migrate data to a new cluster, or to create a new cluster with the same data as the source cluster.

Snapshot and Restore is an essential tool for Elasticsearch cluster administration and disaster recovery, as it provides an efficient and reliable way to back up and recover your Elasticsearch data.

Setup Snapshot & Restore via Kibana

⚠️ If you will be using a shared file system you must add "path.repo: /your/path/here" to elasticsearch.yml make sure you have read/write permissions to this path before proceeding

1. Access Kibana: Open Kibana in your web browser by navigating to the Kibana URL. (typically http://your_kibana_host:5601))

2. Open Stack Management: Click on the "Stack Management" tab on the left-hand side of the Kibana UI.

3. Navigate to Snapshot and Restore: In the Stack Management menu, click on "Snapshot and Restore" under the "Data" section.

image

Register a repository

1. Before creating snapshots, you need to register a snapshot repository. Click on the "Repositories" tab and then click the "Register a repository" button.

   

   image
2. Choose a repository type (e.g., Shared file system, Microsoft Azure, Amazon S3, or Google Cloud Storage) and be ready to provide the necessary settings, such as the repository location, access keys, and bucket name (depending on the repository type). We will be registering a remote NFS share that we've already mounted on the server and added to the path.repo field in our elasticsearch.yml configuration file. We will give our Repository a name select "Shared file system" and click next.

   

   image

   (Note: If you're using Azure, S3 or GCS, make sure you've already installed the required plugins on your Elasticsearch cluster.)

3. We will now use the path.repo settings we previously added to elasticsearch.yml in the "Location" field, specify our Chunk size, Max snapshots bytes per second, Max restore bytes per second and click register.

   

   image
4. Now click on verify repository to verify everything is working as expected and Elasticsearch can connect:

   

   image

   

   image

Create a snapshot - Kibana

1. Click on the "Policies" tab, and then click the "Create policy" button.
2. Choose the repository you registered earlier, provide a unique snapshot name, and select the time interval you would like to take snapshots.
3. Next configure the data streams and/or indices you would like to snapshot. I have unchecked All and used an index pattern of elastiflow*. This will cover all ElastiFlow indices. Next configure whether you would like to ignore unavailable indices, allow partial indices, include global and feature states then click next.
4. Next configure your snapshot retention policy. I will be deleting snapshots after 4 days with a maximum of 100 snapshots.
5. Review your policy and when ready click "Save policy"
6. Once this policy is saved you can manually run it by selecting the "Run now" button followed by "Run policy".

Monitor snapshot progress

You can monitor the progress of a snapshot by clicking on its name in the "Snapshots" tab. This will show the snapshot's details, including the current status and progress.

Restore a snapshot - Kibana

1. To restore a snapshot, click on the snapshot name in the "Snapshots" tab and then click the "Restore" button. You can choose which indices to restore and customize the restore operation with additional settings, like renaming indices or changing index settings.
2. Uncheck "All data streams and indices" unless you would like to restore everything included in the snapshot then manually select the indices you wish to restore. If you would like to rename indices, partially restore indices, or restore aliases this is where you would configure these options:
3. This is where you would override index settings saved when the snapshot was taken:
4. Click "Restore snapshot" to start the restore process.

Monitor restore progress - Kibana

You can monitor the progress of a restore operation by navigating to the "Restore Status" tab:

Setup Snapshot & Restore via API

Install required plugins (optional)

Some repository types, like S3 or GCS, require installing plugins. Install the required plugin for your chosen repository type using the following command:

(Replace repository-s3 with the appropriate plugin name for your chosen repository type.)

bin/elasticsearch-plugin install repository-s3

Register a snapshot repository

To create and manage snapshots, you need to register a snapshot repository. Choose a repository type that suits your needs (e.g., fs for local file system, s3 Amazon S3, or gcs for Google Cloud Storage).

Example: Registering an fs (file system) repository:

PUT /_snapshot/my_backup_repository
{
"type": "fs",
"settings": {
"location": "/path/to/your/backup/directory",
"compress": true
}
}

Replace /path/to/your/backup/directory with the path to your desired backup directory.

Create a snapshot - API

To create a snapshot, use the following API call:

PUT /_snapshot/my_backup_repository/snapshot_name?wait_for_completion=true

Replace my_backup_repository with your repository name and snapshot_name with a unique name for your snapshot. The wait_for_completion=true parameter makes the API call wait until the snapshot is created.

Monitor snapshot progress (optional)

You can monitor the progress of a snapshot by calling the following API:

GET /_snapshot/my_backup_repository/snapshot_name/_status

Replace my_backup_repository and snapshot_name with your repository and snapshot names, respectively.

Retrieve snapshot information

You can retrieve information about a specific snapshot or all snapshots in a repository using the following API call:

GET /_snapshot/my_backup_repository/snapshot_name

Replace my_backup_repository and snapshot_name with your repository and snapshot names, respectively.

Restore a snapshot - API

To restore a snapshot, use the following API call:

POST /_snapshot/my_backup_repository/snapshot_name/_restore

Replace my_backup_repository and snapshot_name with your repository and snapshot names, respectively. You can also specify additional parameters, like indices or rename_pattern, to customize the restore operation.

Monitor restore progress - API

You can monitor the progress of a restore operation by calling the following API:

GET /_cat/recovery?v&active_only=true

This command will return information about ongoing restore operations.

By following these steps, you can enable and use Elasticsearch Snapshot and Restore to create snapshots of your indices and restore them as needed.