Wednesday, December 11, 2019

Sven Mollinga

Sven Mollinga

This is my personal tech blog. This is the place where I put up updates about my current projects.

Elasticsearch is very good in collecting data. A lot of this data only usable for a short amount of time and Elasticsearch is well known to consume lots of ram when there are many indices(shards) added. To archive or delete these indices we can use the tool Curator. It’s very easy to install and configure and can be up and running within 10 minutes. I added a small example to delete Filebeat indices after 10 days.

Git repo:
Configfile Language: YML
Operating system: Debian/Ubuntu

How to install

First we install the Curator through the PIP installer.

pip install elasticsearch-curator

After the installation we add the necessary configuration files. The YML configuration files can be added in the /etc/elasticsearch folder. These files are found within my SysadminScripts git repo.

We start by doing a dry run of the curator command to see if it has the desired outcome.

/usr/local/bin/curator /etc/elasticsearch/delete_filebeat.yml --config /etc/elasticsearch/curator.yml --dry-run

If the output of the command is correct we wil add the following command to your crontab (/etc/crontab). This will run the Curator command every night at half past 12.

30 0 * * * root /usr/local/bin/curator /etc/elasticsearch/delete_filebeat.yml –config /etc/elasticsearch/curator.yml