Lambda to delete indices older than ‘x’ days on Elastic Search

This is simple example of how we can delete the indices older than ‘x’ days.

import boto3
from requests_aws4auth import AWS4Auth
from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

host = 'XXXXXXXXXXXXXXXX.us-east-1.es.amazonaws.com' # Provide the elasticsearch endpoint
region = 'us-east-1' # Provide the region
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Lambda execution starts here.
def lambda_handler(event, context):

# Build the Elasticsearch client.
es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)

index_list = curator.IndexList(es)
# Delete the indices for the pattern yyyy-mm-dd* with creation_date greater than x days.
# Source https://curator.readthedocs.io/en/latest/examples.html
index_list.filter_by_age(source='creation_date', direction='older', timestring='%Y-%m-%d', unit='days', unit_count=7)

print("Found %s indices to delete" % len(index_list.indices))

if index_list.indices:
curator.DeleteIndices(index_list).do_action()

print('Indices deleted successfully')

This example needs aws4auth, elasticsearch, curator modules installed. You can build these modules on using linux machine. I’ve used one of the ec2 instance that has amazon linux installed.
# Install Dependancies
yum -y install python-pip zip
pip install virtualenv

# Create the virtual environment
mkdir -p /var/es-cleanup && cd /var/es-cleanup
virtualenv /var/es-cleanup
cd /var/es-cleanup && source bin/activate
pip install requests_aws4auth -t .
pip install elasticsearch -t .
pip install elasticsearch-curator -t .

# Copy the code to current directory and set the file permission to execute mode
chmod 754 es-cleanup.py

# Package the lambda
zip -r /var/es-cleanup.zip *

# Send the package to S3 Bucket
# aws s3 cp /var/es-cleanup s3://BUCKET_NAME/

Hope you enjoyed the post.

Cheers

Ramasankar Molleti

LinkedIn