Monthly Archives: August 2017

Accessing RDS from AWS Lambda

Servlerless architectures are justifiably focused on avoiding the use of server focused non-scalable resources are far as possible down the stack. In the Amazon world this typically translates to using stores like DynamoDB for data. In some instances it still does make sense to have a relational database in a serverless stack. Particularly for smaller more internal applications that won’t need to scale up to millions of concurrent users, where serverless is being used just as much to keep a consistent architecture and to keep prices down as to handle huge scaling events.

There are a number of different considerations to make with this kind of system, the details of how to do this are scattered across stack overflow and AWS documentation. In this short article we pull everything together into one place and document the considerations about how to do this in a way that is secure and performant.

While many of the principals are generalisable, there will be a specific focus on using Postgres and Node on AWS Lambda, setting up the lambda function to be callable via API gateway.

The two considerations that we will be focusing on are:

1) The general setup and configuration to allow Lambda to talk to RDS.
2) how to configure connection pooling to help make the lambda access the database performantly.

General Access

  1. ensure that the Lambda is deployed to a VPC with a security group.
  2. configure your RDS instance to allow access from the security group (specify an inbound rule in RDS to allow access from the source security group the lambda function has been deployed to.

Connection Pooling

Connection pooling is different to normal. Remember that each lambda instance is independant and we are in node js so each running lambda container will serve a single process. This means that connection pools should probably only have a single connection. The purpose of the connection pool will be to ensure that if there is a warm lambda instance it will use the connection from the pool.

To make the connection pooling work, we will need to allow the lambda to keep the connection alive when the response has been set. (see http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html#nodejs-prog-model-context-properties). We use the pg-pool library to do this: https://github.com/brianc/node-pg-pool

So we set up a connection pool outside of your Lambda handler function, with a max pool size of 1, a min pool size of 0, a big idleTimeoutMillis (5 minutes) and a small connectionTimeoutMillis (1 second). Then in the handler function set context.callbackWaitsForEmptyEventLoop = false and then use the connection pool as normal.

const Pool = require('pg-pool')
const pool = new Pool({
    host: 'something.REGION.rds.amazonaws.com',
    database: 'your-database',
    user: 'a_user',
    password: 'password',
    port: 5432,
    max: 1,
    min: 0,
    idleTimeoutMillis: 300000,
    connectionTimeoutMillis: 1000
});

Given that we will want to hook this into API gateway, it makes sense to make the json passed into the callback look like what the API Gateway will want for a seamless Lambda integration. The full example below shows this.

const Pool = require('pg-pool')
const pool = new Pool({
    host: 'something.REGION.rds.amazonaws.com',
    database: 'your-database',
    user: 'a_user',
    password: 'password',
    port: 5432,
    max: 1,
    min: 0,
    idleTimeoutMillis: 300000,
    connectionTimeoutMillis: 1000
});

module.exports.handler = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;

    let client;
    pool.connect().then(c => {
        client = c;
        return client.query("select 'stuff'");
    }).then(res => {
        client.release();
        let response =  {
            "isBase64Encoded": false,
            "statusCode": 200,
            "body": JSON.stringify(res.rows)
        }
        callback(null, response);
    }).catch(error => {
        console.log("ERROR", error);
        let response =  {
            "isBase64Encoded": false,
            "statusCode": 500,
            "body": JSON.stringify(error)
        }

        callback(null, response);
    });
};

The code above provides a great simple lambda function that queries a database, and provides the response in json for consumption by API Gateway. The use of a simple connection pool ensures that a warm lambda function will reuse connections, and perform well. API Gateway then provides the tools needed to enforce rate limiting and to help ensure that the underlying resources don’t get starved. With the above setup a performant API using API Gateway and Lambda that is backed by Amazon RDS can be created. Bigger picture a truly scalable architecture would involve NoSQL and dynamo without the constraint of a relational database.