AWS Lambda Python Script to create Database Snapshots

The below script is used to create a database manual snapshots on all regions in an AWS account.  RDS support automatic snapshots which are very useful for point in time recovery but it does not protect you from accidental deletion of RDS instances. Here are the reasons why we need manual snapshots Accidental RDS InstanceContinue reading “AWS Lambda Python Script to create Database Snapshots”

AWS Lambda to generate CSV file from RDS PostgreSQL

One of the requirement was to generate csv file for set of queries from RDS PostgreSQL and upload the csv file to s3 bucket for power bi reporting. Powerbi connects to s3 url and generate report. There is no gateway to connect to PostgreSQL instance from power-bi, hence we need to have a mechanism toContinue reading “AWS Lambda to generate CSV file from RDS PostgreSQL”

Lambda to delete indices older than ‘x’ days on Elastic Search

This is simple example of how we can delete the indices older than ‘x’ days. This example needs aws4auth, elasticsearch, curator modules installed. You can build these modules on using linux machine. I’ve used one of the ec2 instance that has amazon linux installed. # Install Dependancies yum -y install python-pip zip pip install virtualenvContinue reading “Lambda to delete indices older than ‘x’ days on Elastic Search”

How To Create Jupyter Notebook using Amazon SageMaker

In this blog post, i would like to present how to create Jupyter Notebook  and run various commands using Amazon Sagemaker What is Jupyter Notebook? The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning andContinue reading “How To Create Jupyter Notebook using Amazon SageMaker”

Python Script – Create and Insert data into Dynamodb table

In this example, i would like to demonstrate how to create a AWS DynamoDB table using python. I’m taking the simple employee table which contains Id, FirstName, LastName, Dept and Sal columns. Also, i’m going to create a Partition key on id and sort key on Sal columns.  I will use boto3 to call theContinue reading “Python Script – Create and Insert data into Dynamodb table”

AWS S3—>AWS Lambda —>SSM—> SQL Server Job

Scenario: Application is running on java and the database is using PostgreSQL.  Application user uploads the data and this data needs to process to analytic server as well. There is no direct access to Analytic Database server that is running on SQL Server on different location. You need to architect a solution to process theseContinue reading “AWS S3—>AWS Lambda —>SSM—> SQL Server Job”

AWS Lambda to connect to PostgreSQL and execute a function/query using Python

It’s been long time since i wrote a blog post. In this post i would like to show an example of lambda to connect to PostgreSQL database and execute the query. Build a PostgreSQL psycopg module in order to connect to database. I’ve built a custom compiled python library to connect to latest PostgreSQL 9.6Continue reading “AWS Lambda to connect to PostgreSQL and execute a function/query using Python”

How to Backup SQL Server Backups to Amazon S3 Storage

It’s been a long time since i posted in my blog. One of the common request i have been asked by my colleagues, friends about “Is there a way to automate backups that are taken on the SQL Server database server running on EC2 instance to Amazon S3 storage?”  Today, I would like to showContinue reading “How to Backup SQL Server Backups to Amazon S3 Storage”

How to Migrate SQL Server Database running on EC2 instance to Amazon RDS Instance

Recently i worked on migrating sql server databases running on amazon EC2 instances to Amazon RDS instances. Starting from 27th Jul 2016 Amazon announced you can ship your databases from ec2 to rds using native backup and restore method. Let’s see how it works. Steps: Assume you have RDS Instance ready with default setting Create aContinue reading “How to Migrate SQL Server Database running on EC2 instance to Amazon RDS Instance”