AWS-Lambdas - Automating function deploys

On this post, I talked about the ideas behind AWS-Lambdas computation service and how It works.The presented example shows how It can be be deployed and used. Even having an working example, there is an issue on the way I'm using It. All steps regarding the deploy process are manual. It just goes against agility. Manual deploys as such are error prone. More complex the application get, more expansive It will be to maintain.The side-effects when maintaining manual deploy steps are endless, so there should be an alternative to automate It and make AWS-Lambdas really cost effective as It promises to be.

Kappa seems to fill this gap. It is a command line tool that greatly simplifies the process of having lambdas deployed on the cloud. All steps described on the mentioned post can be automated. Now are talking!

Setup

Before start, be sure you have python (2.7.x)  and pip available on the command line.

Installing kappa: 

I strongly advice build It from sources as far there are important bug fixes that seems to be fixed recently:

git clone https://github.com/garnaat/kappa.git
cd kappa
pip install -r requirements.txt
python setup.py install

Instaling awscli:

sudo pip install awscli

Configuration:

First thing to do is create the the kappa configuration file. This is where I'm gonna tell It how to deploy my lambda function (config.yml).

---
profile: my-default-profile
region: us-west-2
iam:
  policy:
    name: AWSLambdaExecuteRole
  role:
    name: lambda_s3_exec_role
lambda:
  name: myLambdaFuncId
  zipfile_name: integrator.zip
  description: Somethhing that helps describe your lambda function
  path: src/
  handler: Integrator.handler
  runtime: nodejs
  memory_size: 128
  timeout: 3
  mode: event
  test_data: input.json
  event_sources:
    -
      arn: arn:aws:s3:::[set your bucket name]
      events:
        - s3:ObjectCreated:*

Lets see what is going on:

Line 2: There should be a profile that kappa will use to authenticate Itself on amazon and create the function in my behalf. We are gonna see it later on the awscli  configuration;
Line 4: The policies assigned to this lambda. In case they aren't there yet, kappa will create them for me.
Line 9 - 18: function Runtime configs.
Line 19: This is the file that contains an example request in order to test the function. It is useful once we want to be sure everything is working fine after the deploy is over.
Line 20: Here I'm setting from where events will come from. In this case, any changes on the given bucket, will trigger a call to my function.

Now It's time to configure aws-cli. The only configuration needed is the security profile. Kappa will use It as stated before:

Create the following file in case Isn't already there: ~/aws/credentials and put the following content

[my-default-profile]
aws_access_key_id=[YOUR KEY ID ]
aws_secret_access_key=[YOUR ACCESS KEY ]

Having it set, It's time to deploy it using kappa tasks:

kappa config.yml create
kappa config.yml add_event_source
kappa config.yml invoke
kappa config.yml status

It should be enough to see the function deployed on the aws-console. The previous commands in order did:

• create the function on the amazon
• Make it listen changes on a given bucket
• Test the deployed function using fake data (simulating an event)
• Check the status of the deployed function on amazon.

As far Kappa let me automate all deploy tasks,I'm able to create a smarter deploy process. I worked in an example about how could It be done here. I may forgot to mention some detail about the process of having It work, so in this case leave me message and I'll be glad to help.

AWS Lambda - Computation as Service

Less than one year ago, Amazon launched  a new computation service, the AWS-Lambda. It promises simplify the process of building applications, by hosting and running code for you.  All the infrastructure and some scalability and fail over aspects are Amazon's concerns. It also integrates pretty well with other Amazon's services like SQS, SNS, DynamoDB, S3, etc. The code hosted there can even be called externally by other applications using the aws-sdk.

Here, I'll show how to use this service by doing something very simple. It idea is implement some code that will listen to an event (PUT) in a given S3 bucket, apply some processing on the file content, and send It to a SQS Queue.

This service restricts the language and platform where the code is implemented. A NodeJS module needs to be exported and called after deployed into the Amazon infrastructure, So, if you are not familiarised with Javascript and Nodejs, I would advice you to step back and look some documentation first.

var AWS = require('aws-sdk');
var parser = require('xml2js').parseString;
var async = require('async');

var s3 = new AWS.S3();
var sqs = new AWS.SQS();

exports.handler =  function(event, context) {
 var bucketName = event.Records[0].s3.bucket.name;
 var fileName = event.Records[0].s3.object.key;

 async.waterfall([
  function download(next) {
   s3.getObject({Bucket: bucketName,  Key: fileName}, function (err, data) {
    next(err, data);
   })
  },
  function parseXml(response, next) {
   parser(response.Body.toString(), function(err, result) {
    next(err, result);
   })
  },
  function sendMessage(result, next) {
   var message = {
    MessageBody: JSON.stringify(result),
    QueueUrl: "[YOUR QUEUE URL: i.e: https://....]"
   };
   
   sqs.sendMessage(message, function(err, data) {
      if(err) {
         context.fail("Error: " + err);
       } else {
         context.succeed("Message sent succefully: " + data.MessageId);
       }
       context.done();
   });
  }
 ], function(err) {
  if (err) {
   context.fail("Error: " + err);
   throw err;
  }
 });

}

Lets see what is happening here:

• From line 1 to 3: Importing the modules needed on the implementation. All these modules needs to be packed when deploying the application, except by the aws-sdk, which is available by default in runtime.
• Line 9 and 10:  Getting information from the event. When listing to an event from a S3 bucket, what you receive is the event metadata. So, if you want to do something with the object that uploaded, you need to extract the event metadata and then get the uploaded object and do something with the content.
•  Line 12: The code from this point is a series of callbacks that depends from each other's results. So, to avoid the callback hell scenario, I used an external lib that make these functions dependencies a bit more clear to read.

In order to sure It's everything ok before deploying It, go to the console and perform a "npm install" command. It should check all code dependencies and put them into a specific directory.

Now It's time to set It up on Amazon infrastructure. The AWS-Lambda service let you upload a zip file with Its dependencies inside a Zip file. When using this option, be careful when creating the zip file. The Javascript file that will contains the code shown before needs to be at "/" on the zip file, otherwise It wont work. Worst than that, when running the code, the error message on the console is gonna show an error message that does not point on this direction.

Once is you have your code properly packed, go to the AWS console, access the "Lambda" option and and ask to create a new function. The  presented screen should look like this:

There, I'm putting basic information about what I'm uploading. The most relevant information are the Handler (Javascript file + the module name to be called) and Memory and Timeout values (Amazon will use this information billing). There is still the execution role. If you don't have one yet, create It using the options available on the combo box.  Once you managed to finish this step, the module is ready to be called. Now, the last step is just go to the bucket and I'm interested to monitor, and trigger this this function every time a new event happens by changing the bucket properties.

An additional and reasonable step would be test this deployed function in order be sure It's everything ok. In order to do that , go the console where all the functions are listed, select the function you want to test, press the "Actions" and select "Edit/Test". A new screen will be  presented. On the left side, there is a section called "Sample Events". It simulates some real use cases. To test this function, pick "S3 Put" option and  adapt the event setting valid bucket and file names. 

If everything went fine, you should able see a message looking like this on the Execution result area:

"Message sent successfully: 9431465....."

Some additional billing information should be displayed also and that is It. From this point you are sure that the function is properly deployed and ready to be used.
The working example can be found here.