Lambda to delete indices older than ‘x’ days on Elastic Search

This is simple example of how we can delete the indices older than ‘x’ days.


import boto3
from requests_aws4auth import AWS4Auth
from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

host = 'XXXXXXXXXXXXXXXX.us-east-1.es.amazonaws.com' # Provide the elasticsearch endpoint
region = 'us-east-1' # Provide the region
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Lambda execution starts here.
def lambda_handler(event, context):

# Build the Elasticsearch client.
es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)

index_list = curator.IndexList(es)
# Delete the indices for the pattern yyyy-mm-dd* with creation_date greater than x days.
# Source https://curator.readthedocs.io/en/latest/examples.html
index_list.filter_by_age(source='creation_date', direction='older', timestring='%Y-%m-%d', unit='days', unit_count=7)

print("Found %s indices to delete" % len(index_list.indices))

if index_list.indices:
curator.DeleteIndices(index_list).do_action()

print('Indices deleted successfully')

This example needs aws4auth, elasticsearch, curator modules installed. You can build these modules on using linux machine. I’ve used one of the ec2 instance that has amazon linux installed.
# Install Dependancies
yum -y install python-pip zip
pip install virtualenv

# Create the virtual environment
mkdir -p /var/es-cleanup && cd /var/es-cleanup
virtualenv /var/es-cleanup
cd /var/es-cleanup && source bin/activate
pip install requests_aws4auth -t .
pip install elasticsearch -t .
pip install elasticsearch-curator -t .

# Copy the code to current directory and set the file permission to execute mode
chmod 754 es-cleanup.py

# Package the lambda
zip -r /var/es-cleanup.zip *

# Send the package to S3 Bucket
# aws s3 cp /var/es-cleanup s3://BUCKET_NAME/

Hope you enjoyed the post.

Cheers

Ramasankar Molleti

LinkedIn

About Ramasankar

My name is Ramasankar Molleti, I have been working on various database and cloud systems (SQL Server, Oracle, PostgreSQL, DynamoDB, Redshift, Amazon, Azure) over 12 years, primarily focusing on Architect , Development, Administration and Business intelligence. I hold various Certifications such as Amazon Web services Solution Architect Associate, Microsoft certifications (Microsoft Certified Information Technology Professional, Microsoft Certified Technology Specialist, and Microsoft Certified Solution Associate).
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s