Category Archives: Uncategorized

Accessing RDS from AWS Lambda

Servlerless architectures are justifiably focused on avoiding the use of server focused non-scalable resources are far as possible down the stack. In the Amazon world this typically translates to using stores like DynamoDB for data. In some instances it still does make sense to have a relational database in a serverless stack. Particularly for smaller more internal applications that won’t need to scale up to millions of concurrent users, where serverless is being used just as much to keep a consistent architecture and to keep prices down as to handle huge scaling events.

There are a number of different considerations to make with this kind of system, the details of how to do this are scattered across stack overflow and AWS documentation. In this short article we pull everything together into one place and document the considerations about how to do this in a way that is secure and performant.

While many of the principals are generalisable, there will be a specific focus on using Postgres and Node on AWS Lambda, setting up the lambda function to be callable via API gateway.

The two considerations that we will be focusing on are:

1) The general setup and configuration to allow Lambda to talk to RDS.
2) how to configure connection pooling to help make the lambda access the database performantly.

General Access

  1. ensure that the Lambda is deployed to a VPC with a security group.
  2. configure your RDS instance to allow access from the security group (specify an inbound rule in RDS to allow access from the source security group the lambda function has been deployed to.

Connection Pooling

Connection pooling is different to normal. Remember that each lambda instance is independant and we are in node js so each running lambda container will serve a single process. This means that connection pools should probably only have a single connection. The purpose of the connection pool will be to ensure that if there is a warm lambda instance it will use the connection from the pool.

To make the connection pooling work, we will need to allow the lambda to keep the connection alive when the response has been set. (see http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html#nodejs-prog-model-context-properties). We use the pg-pool library to do this: https://github.com/brianc/node-pg-pool

So we set up a connection pool outside of your Lambda handler function, with a max pool size of 1, a min pool size of 0, a big idleTimeoutMillis (5 minutes) and a small connectionTimeoutMillis (1 second). Then in the handler function set context.callbackWaitsForEmptyEventLoop = false and then use the connection pool as normal.

const Pool = require('pg-pool')
const pool = new Pool({
    host: 'something.REGION.rds.amazonaws.com',
    database: 'your-database',
    user: 'a_user',
    password: 'password',
    port: 5432,
    max: 1,
    min: 0,
    idleTimeoutMillis: 300000,
    connectionTimeoutMillis: 1000
});

Given that we will want to hook this into API gateway, it makes sense to make the json passed into the callback look like what the API Gateway will want for a seamless Lambda integration. The full example below shows this.

const Pool = require('pg-pool')
const pool = new Pool({
    host: 'something.REGION.rds.amazonaws.com',
    database: 'your-database',
    user: 'a_user',
    password: 'password',
    port: 5432,
    max: 1,
    min: 0,
    idleTimeoutMillis: 300000,
    connectionTimeoutMillis: 1000
});

module.exports.handler = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;

    let client;
    pool.connect().then(c => {
        client = c;
        return client.query("select 'stuff'");
    }).then(res => {
        client.release();
        let response =  {
            "isBase64Encoded": false,
            "statusCode": 200,
            "body": JSON.stringify(res.rows)
        }
        callback(null, response);
    }).catch(error => {
        console.log("ERROR", error);
        let response =  {
            "isBase64Encoded": false,
            "statusCode": 500,
            "body": JSON.stringify(error)
        }

        callback(null, response);
    });
};

The code above provides a great simple lambda function that queries a database, and provides the response in json for consumption by API Gateway. The use of a simple connection pool ensures that a warm lambda function will reuse connections, and perform well. API Gateway then provides the tools needed to enforce rate limiting and to help ensure that the underlying resources don’t get starved. With the above setup a performant API using API Gateway and Lambda that is backed by Amazon RDS can be created. Bigger picture a truly scalable architecture would involve NoSQL and dynamo without the constraint of a relational database.

JS sleep (setTimeout) using promises in one line of code.

As mentioned before, I like JavaScript promises.

One nice trick with promises is creating a simple sleep function using the standard setTimeout.

Here’s the line of code:

let sleep = (time) => new Promise((resolve) => setTimeout(resolve, time));

Then use it as:

sleep(500).then(() => {
    // Do something after the sleep!
})

A nice simple sleep function in JavaScript.

Creating a Deliverable HTML Email on AWS Lambda with SES

Creating deliverable rich html emails is a great goal for many web applications, communicating with your customers, and helping to send the messages that are beautiful and make your marketing/designers happy.

As discussed in this send grid article, there are quite a number of approaches for doing this, but one of the best options for ensuring that images in emails with images. The leaders today in 2017 are using cid and referencing attachments, and using data urls.

According to the comments in this Campaign Monitor blog post, the cid method is supported by all the major clients today. (ironically the post is talking about using the cooler approach of data urls for images).

With this background, the question is how to do this with AWS Lambda and SES.

Thankfully it’s really straight forward.

The simple steps are:

  • create a simple html email that references images using cid: as the protocol.
  • create a raw rfc822 email string that can be sent with the SES api.
  • use the ses.sendRawEmail method to send the email.

1 Create a simple html email

For example:

<html><body><p>Hello world</p><img src=”cid:world”></body></html>

Note that the source of the image is of the format cid:world, this cid will be what you specify when attaching the image to the blog post

2 Create a raw rfc822 email string

The mailcomposer  package a part of Nodemail(https://nodemailer.com/extras/mailcomposer/) provides a great simple easy to use api for creating rfc822 emails with attachments. When creating attachments you can specify cids to refer to them by, and you can specify the contents of the attachment with a local filename, a buffer, or even a http resource. It’s a great api.  Take a look at the npm page to see more. One example of using this package is:

let from = 'from@example.com';
let to = 'to@example.com';
let subject = 'Subject';
let htmlMessage = '<html><body><p>Hello world</p><img src="cid:world"></body></html>';
let mail = new MailComposer({
  from: from, to: to, subject: subject, html: htmlMessage,
  attachments: [{
    filename: 'hello-world.jpg',
    path: 'https://cdn.pixabay.com/photo/2015/10/23/10/55/business-man-1002781_960_720.jpg',
    cid: 'world'
  }]
});
mail.build(function(err, res) {console.log(res.toString())});

3 Send the email with SES

Take the buffer that you create and send it with SES.

let sesParams = {
  RawMessage: {
    Data: message
  },
};
ses.sendRawEmail(sesParams, function(err, res){console.log(err, res)});

Full example using promises

Let’s put it all together, and pull in some of the promise code that I talked about in an earlier blog post(http://www.rojotek.com/blog/2017/04/11/create-a-promise-wrapper-for-a-standand-node-callback-method/)

function createEmail(){
  let from = 'from@example.com';
  let to = 'to@example.com';
  let subject = 'Subject';
  let htmlMessage = '<html><body><p>Hello world</p><img src="cid:world"></body></html>';
  let mail = new MailComposer({
    from: from, to: to, subject: subject, html: htmlMessage,
    attachments: [{
      filename: 'hello-world.jpg',
      path: 'https://cdn.pixabay.com/photo/2015/10/23/10/55/business-man-1002781_960_720.jpg',
      cid: 'world'
    }]
  });

  return new Promise((resolve, reject) => {
    mail.build(function(err, res) {
      err ? reject(err) : resolve(res);
    });
  });
}
createEmail().then(message =>{
  let sesParams = {
    RawMessage: {
      Data: message
    },
  };
  return ses.sendRawEmail(sesParams).promise();
});

Creating emails that include attachments is really quite easy with node, lambda and ses. Doing this is a great step to delivering rich emails that look like what your designers want.

 

Report Writing for Occupational Therapists

Over the past 18 months I’ve been working on Report Hero, report writing software to help Paediatric Occupational Therapists write reports.  I’ve built it in partnership with one of the best Paediatric OT’s I know, and I’m happy to see it being used by a number of O.T’s.  If that’s your thing, take a look at the Report Hero website, and sign-up for a trial.

Open Badges

I’ve recently been looking into the Open Badges Framework, with a goal of being able to understand what it is from a high-level technical standpoint.

The Open Badge Specification provides a standard for issuing badges

The key participants in the open badge system are:

  • the badge issuer
  • badge displayers
  • badge earners
  • a badge backpack

A badge issuer will create a badge and issue it to a badge earner. The badge will consist of a number of cryptographically verifiable assertions about the badge. With the earners consent an issuer may publish the badge to a badge backpack.

There is a reference implementation of a badge backpack implemented by mozilla. This reference impementation is hosted out of the united states, and is probably the default way to publish badges. The source code for the reference implementation has also been made available for download and deployment (https://github.com/mozilla/openbadges-backpack).

In a healthy open badge ecosystem, there would be a small number of badge backpacks, a larger number of issuers, and an even larger number of earners.

Every organisation that wants to issue badges would need to be an issuer, but most organisations would (and should) be able to use a standard backpack. That said, when dealing with children, legal rules may lead to the creation of regional badge backpacks.