AWS Lambda Functions with Examples (Best Tutorial 2019)

AWS Lambda Functions

What are the AWS Lambda Functions with Examples (Tutorial 2019)

Lambda projects are deployed to the service as functions. I will commonly refer to a project in its entirety as a Lambda function. Each function is designed to do just a single or a few common processes.  In this tutorial, we explain the complete process of AWS Lambda Functions with best Examples in 2019.

 

For example, a Lambda function may download an uploaded image and convert it to black and white. Unlike a traditional project, Lambda functions are usually quite small - most that I’ve written is only a few hundred lines of code (not including Node.js dependencies).

 

Each Lambda function can be treated as a separate project. Personally, I create a GitHub project for each; however, you may wish to use subfolders within a single repository if you’re limited by a number of repositories.

AWS Lambda Functions

 

The organizational structure is up to you; however, each Lambda function must be able to function independently (i.e. it cannot rely on dependencies within a different project unless they are copied to the project before deployment).

 

Lambda functions are uploaded to Lambda as a ZIP file. The ZIP file can be uploaded from the console, the command line, or placed on S3 (preferable for larger files).

 

AWS takes care of downloading the ZIP to the backend resources it manages. Each successive update to the code only requires you to re-upload another ZIP to replace the previous. Again, AWS will manage the deployment of the new code to its resources.

 

Resource Allocation

Resource Allocation

When uploading your Lambda function as a ZIP, you must decide how much computing power to allocate to the execution of your code. Of course, these allocations affect the performance of your function, as well as its cost. Be sure to review the Lambda pricing page to find a good balance of resources to allocate.

 

The allocated memory of your function is how much memory you want to allow your function to consume. It does not mean that every function execution will utilize all of the available memory, but it does allow the function to use up to that amount. Memory is allocated in increments of 128 MB, up to a (current) maximum of 1536 MB.

 

As an example, let’s say that you’ve created a function called “image processor” which downloads an image from S3 and performs an analysis on its metadata. Because image sizes can vary widely, you will need to test various allocations of memory on the performance of your function. Once done, you settle on allocating 256 MB of memory.

 

If the function executes 5 times, it may consume 30 MB, 100 MB, 70 MB, 200 MB, and 40 MB. Each of these executions is charged at the allocated rate (not the consumed rate, so it is important not to over-allocate resources).

 

If the function executes a sixth time and requires 280 MB to complete, it will fail with an error saying it ran out of memory. You would then be charged for execution at the 256 MB rate.

 

In the same way, that memory allocation will limit your function to consuming a certain amount of memory, the timeout value will limit your function to certain running time.

 

Timeout values range from 1 second to 300 seconds (at the time of publication). Setting a high timeout value does not mean that every function execution will be charged at the maximum rate, but rather it indicates that the function can execute for up to a specific amount of time.

 

In the above example, if I had set a timeout of 10 seconds for the first execution and the function executed in 2 seconds, I would be charged for only 2 seconds, not 10 seconds. These values are rounded to the nearest 100 milliseconds by AWS.

 

Getting Set Up

AWS console

In the next section, we will create the simplest of Lambda functions: one that only responds with “Hello World.” Before we can do that, it will be helpful to understand the Lambda console.

 

To get started, open the AWS console and navigate to “Lambda.” If it’s your first time using the service you’ll likely see a splash page that prompts you to create a new function.

 

Selecting “Get Started Now” will display a number of “blueprints.” AWS has provided a number of sample Lambda functions to help you get started. In the next blog, we will use the “hello-world” blueprint to create our function.

For now, read through the rest of this blog to gain an understanding of what the dashboard will look like once it is populated with functions.

 

On the main page of functions, the Lambda console lists the function name, a description, the base code size, the memory, and the timeout. You can click on the function to open additional details.

 

The function details page allows you to upload new code, configure the function’s entry point, event sources, and see some graphs of performance over the past few days.

Now, we’ll get started with a “Hello World” Lambda function.

Hello World

 

Just like almost every other programming tutorial, I will begin our exploration of Lambda with the simplest function: one that only echoes back “Hello World.” Even though Lambda allows you to edit code in-line directly from the Lambda console, only the simplest of functions can use this.

 

The rest, any functions with more than one file or dependencies, will need to be zipped and uploaded. Since almost every function you write will require this, we’ll use this format for creating our “Hello World” function.

 

Create a new file in your editor of choice named “index.js”. Inside this file, add the following contents:

exports.handler = function(event, context) {

context.succeed(“Hello world”);

};

That’s it! This is the simplest Lambda function you can create. While we’re here, let’s talk about each of its parts:

 

●exports.handler

This is the exported method that will be called by Lambda when the function executes. You can export multiple methods, but you will need to select one when creating your Lambda function in the console. While you don’t have to call your exported method “handler,” it’s the default used by Lambda and I recommend using it.

 

●(event, context)

Your exported method must take these two arguments. The “event” is the object that contains details about the executing event. For example, when S3 triggers a Lambda function, the event contains information about the bucket and key.

 

When the API Gateway triggers a Lambda function, the event contains details about the HTTP method, user input, and other web-based session information. The “context” object contains internal information about the function itself, as well as methods for ending the function (both successfully and unsuccessfully). We’ll explore these objects in-depth shortly.

 

●context.succeed

The “succeed” method of the context object allows the function author to tell Lambda that the function has completed its execution successfully. The argument passed to “succeed” (“Hello World” in this case) will be returned to the invoker.

 

Uploading the Function

Uploading the Function

Despite the fact that our function only contains a single file, create a ZIP file to wrap it. The name of the ZIP file doesn’t matter.

Log into the Lambda console and click “Create a Lambda function.” AWS provides a number of Lambda blueprints, but since we already have a ZIP, just click “Skip.” Give your function a useful name and description, and select “Node.js” as the runtime.

 

The next page allows you to either enter your code inline, upload it via a ZIP, or enter an S3 URL path to a ZIP hosted on S3. Select the second option and locate the ZIP you created.

 

Under the “Lambda function handler and role” section, you will need to define which file and exported method should be used to process events. If you followed the steps above, you can leave the default “index.handler.”

 

If you’ve chosen a different name, make sure that it follows the guidelines above (takes “event” and “context” as arguments, and ends with “context.succeed”).

 

Next, you’ll need to select a role for the execution of your function. We’ll discuss roles in more depth soon, but for now, understand that they behave just like EC2 IAM roles and allow your Lambda function to access resources within your account. At a minimum, your function will need permission to log its events to CloudWatch.

 

AWS has simplified the creation of roles by allowing you to select pre-defined roles from a drop-down list. For the “Hello World” function, you can select the basic execution role. A popup will prompt you to create the role within IAM.

 

The “Advanced settings” allow you to define the memory and time allocated to your function. In our case, echoing “Hello World” should only require the minimum memory and time.

 

You can review and then create your function on the next page. Once complete, you’ll be able to test it. To do this, click on your function from the main overview page and then choose “Actions” > “Configure sample event.”

 

You can just use an empty event object of “{}” because the “Hello World” function does not require any event properties.

 

Click “Submit” and you should see the results of your function: “Hello World.” You’ll also see a lot of extra information like logs, execution time, and memory consumption.

 

You’ll also notice that a log group and stream were created within CloudWatch. Every Lambda execution will log its events (and any in-app console.log statements) to the CloudWatch log group.

 

AWS Event-Driven Tasks

AWS Event-Driven

Event-driven tasks are the types of Lambda use cases that AWS had in mind when designing the Lambda service. In fact, some of the most common examples used in AWS documentation and Re: Invent presentations are event-driven tasks.

 

Tasks of this nature include essentially any form of AWS service event: objects uploaded to S3, DynamoDB table updates, CloudWatch Logs log deliveries, SNS topic entries, etc.

 

Because of the existing AWS development pattern, any changes to these services previously required a polling mechanism to detect for most use cases. For example, before Lambda, new objects uploaded to S3 could only trigger SNS or SQL events.

 

This required additional infrastructure to either continually poll for new SQS entries, or to listen to an SNS topic subscription; there was no way to directly invoke code immediately after an S3 event.

 

Users of S3 that had to process objects on S3 immediately after they were uploaded developed all kinds of hacks and workarounds, most involving code that continually polled either S3 or SQS for new events. This was inefficient not only for users but also for AWS who had to support the high API demand.

 

The story repeats itself for a variety of additional use cases throughout AWS: new posts to an SNS topic required listening servers, new CloudWatch logs could only be processed by polling the CloudWatch API, and so on.

 

As the Lambda service grows, AWS is continually adding new integrations; soon it will likely be possible to trigger Lambda functions from virtually any event within the AWS ecosystem: a new EC2 instance launches, a CloudFormation template updates, an ELB is created, an RDS instance consumes a certain percentage of storage space, a route is created in Route53, and much more.

 

Lambda solves the polling problem by creating an on-demand response to particular events. Developers no longer have to constantly check for new S3 objects, new CloudWatch logs, or DynamoDB updates, because AWS has developed an environment whereby those events are pushed to Lambda rather than pulled by traditional AWS resources accessing the API. This is truly game-changing for users who heavily utilize AWS for their projects.

 

Scheduled Events (Cron)

Scheduled Events

One very popular feature request during the early days of Lambda was for the ability to trigger a Lambda function at predefined intervals. Users were frustrated that an event-driven service couldn’t be executed by one of the most common events in computing: time.

 

Although there were some clever workarounds (a one-hour presentation at Re: Invent 2015 described how CloudWatch alarms, combined with an SNS topic, could create an oscillating 0/1 Lambda trigger), AWS finally announced a more supported option in October of 2015. As of the time of writing, a developer can now use an interval or cron-like definition to trigger Lambda functions.

 

As a developer, it would probably take me several hands to count the number of cron-based scripts I have running in my environments. I use these scripts to perform background work, task synchronization, resource cleanup, monitoring checks, and much more. In most cases, these scripts run on a variety of EC2 servers depending on the project.

 

While this is a valid approach, it does present several problems: the servers cost money to run, even when not in use, the server type must be allocated (and paid for) based on the maximum workload, IAM role permissions must be assigned to the instance based on the aggregate of all running scripts (if ten scripts access ten different S3 buckets.

 

The execution environment has excessive permissions for nine of their executions), script updates require SSH or complicated deployment mechanisms, a server that goes down will take down all of the scripts with it, and script failures are not easily detected. As you can see, there is a lot of room for improvement when running cron-based scripts on an EC2 server.

 

Lambda solves these problems with its scheduled execution model. As long as the script can execute in under five minutes (likely increasing in the future) and doesn’t rely on copious amounts of local resources, it can probably be replaced by Lambda.

 

If each script is replaced with a Lambda function, the permissions can be much more narrowly applied, failures are much more easily noticed, deployments can be easily triggered, logs are aggregated in one place, and the underlying server management is handled by AWS.

 

For a busy infrastructure administrator, this can mean the difference between spending hours ensuring hundreds of scripts execute across a fleet of EC2 servers and simply setting up CloudWatch alerts for failed Lambda executions.

 

Offloading Heavy Processing

Heavy Processing

Imagine that you have an application that accepts a series of image edits (crop, rotate, etc.) as well as a URL of an image to which those effects should be applied.

 

When the user makes a new request, your application downloads the image at the provided URL and then applies the necessary edits before saving the result to S3. If you’re working with traditional AWS resources, you may have a fleet of x-large EC2 servers in an autoscaling group behind an ELB.

 

For highly-trafficked applications, you may need a handful of servers to manage the load. Since image processing is a fairly CPU-intensive task, each new request consumes a good portion of available processing power.

 

This limits the maximum number of concurrent images your application can save and as more requests arrive your autoscaling group must increase, costing more money.

 

With Lambda, the processor-intensive image manipulation tasks can be offloaded from the EC2 server to Lambda. Instead of handling the request directly, the EC2 server can trigger a Lambda function to do the processing, wait for a response, and then respond directly to the user.

 

Now, your application can handle thousands of requests on a single server because the actual processing is being done in Lambda and not locally. While you may still need a few EC2 instances, it is highly possible that you can downsize to a much cheaper option, as well as decrease the total number of running instances.

 

API Endpoints

API Endpoints

When Lambda was first released, many developers quickly realized that the entire website could be created using Lambda functions. However, to do so, the permissions to invoke the function had to be exposed globally - not a security conscious thing to do.

 

Again, lots of intermediary hacks were developed to create “serverless” API endpoints, but just like in the case of scheduled Lambda events, these were quickly made obsolete by the official announcement of the AWS API Gateway service. The API Gateway made it possible to easily and securely configure a Lambda-based backend to replace an entire API web server.

 

The API Gateway is essentially treated as a first-class citizen in the world of Lambda. While other services such as S3 and Dynamo were updated to allow their events to be sent to Lambda, the API Gateway was built from the ground up with Lambda in mind. Because of this, the interaction between the two services is quite seamless:

 

API endpoints and Lambda functions can be tied together with just a few clicks, testing can be performed entirely in the console, and almost all of the API Gateway documentation involves Lambda in some way. blog 21 covers the API Gateway service, and its integration with Lambda in detail.

 

Infrequently Used Services

During my time working with AWS, I’ve seen scores of applications running on micro, small, or medium-sized instances that supported a very small amount of traffic. These applications included legacy APIs, internal dashboards, “glue” holding other applications together, and a variety of other use cases.

 

The common thread was the disparity between the amount of money and maintenance required to keep these applications running and their level of actual use. Some organizations spend hundreds of dollars a month to run a handful of applications which may be accessed only twice - a colossal waste of resources.

If you have multiple instances with less than 5% average CPU usage throughout the entire month, you’ve probably questioned this wastage as well.

 

The pay-per-execution pricing model of Lambda makes it an ideal candidate to replace these applications. You could have one hundred applications hosted on Lambda, each being accessed one hundred times per month and you wouldn’t even have to break out a nickel to pay for the costs.

 

Even if traffic to your application suddenly spiked, it would have to approach three million executions before becoming more expensive than a single t2.micro instance (at the current rate of $9/month).

 

Real-World Use Cases

Up until now, we’ve only explored hypothetical use cases for Lambda. In this blog, I’ll fix that by providing a number of real-world situations in which Lambda can be used.

 

While I don’t envision each of these being universally applicable, hopefully, you can find at least a few places in your current infrastructure that could benefit from a Lambda function inspired by the examples here.

 

S3 Image Processing

S3 Image Processing

I’ll begin with the example most commonly cited in AWS presentations: processing images uploaded to S3. This is helpful if your applications require several sizes of images: perhaps a 480px wide thumbnail is used on a multi-image layout, a 1000px wide rendition is used for the image details page, and the full-size image is available for download. Regardless of the application specifics, storing multiple sizes of an image is a great way to save bandwidth and costs.

 

As discussed previously, Lambda can respond almost instantly when a new object is uploaded to S3. A Lambda function will then be responsible for downloading the original image, resizing it, and uploading it to S3. blog 12 included an in-depth look at the multiple pieces required for a working S3 download function (S3 bucket, IAM role, event trigger).

 

To create our image processing function, we will extend this setup to not only download the object but also to process it and upload resized copies back to S3. Be sure that you have a valid S3 bucket and an IAM role that has permissions to “GetObject” and “PutObject” on the bucket

var async = require(‘async’);
var AWS = require(‘aws-sdk’);
var s3 = new AWS.S3();
var gm = require(‘gm’).subClass({imageMagick: true});
var widths = [480, 640, 1000];
exports.handler = function(event, context) {
var bucket = event.Records[0].http://s3.bucket.name;
var key = decodeURIComponent(
event.Records[0].s3.object.key).replace(/\+/g, ‘ ‘);
s3.getObject({
Bucket: bucket,
Key: key
}, function(s3Err, s3Data){
if (s3Err) return context.fail(s3Err);
var origImg = gm(s3Data.Body);
//
Loop through each width required to generate the resize async.each(widths, function(width, next){
origImg.resize(width)
.toBuffer(‘jpg’, resizeErr, resizedBuffer) {
if (resizeErr) return next(resizeErr);
//
Upload the resized data to S3 s3.putObject({
Bucket: bucket,
Key: ‘resized/’ + width + ‘-‘ + key,
Body: resizedBuffer,
ContentType: “image/jpg” }, function(s3UploadErr) {
next(s3UploadErr);
});
}
}, function(err){
if (err) return context.fail(err);
context.succeed(‘Completed ‘ +
widths.length + ‘ resizes’);
});
});
};

 

When creating the event trigger, be sure to utilize the prefix and suffix options so that the function is not triggered for uploads to the “resized” directory (this would cause a recursive loop that increases in size as Lambda launches functions in response to uploads from the previous function).

 

Shutting Down Untagged Instances

AWS tagging feature

Many organizations utilize the AWS tagging feature to keep track of costs, projects, or other parameters. Oftentimes, developers will forget to tag an instance launched for development purposes and then forget what its purpose was months later.

 

While I won’t delve into the various policies and processes that companies have to solve this problem, many have settled on requiring every instance to be tagged with specific variables and shutting down the ones that aren’t. Previously a cron script on an EC2 instance, this is a perfect job for Lambda.

 

Again, the IAM role for this function will need the proper permissions to work - “ ec2: DescribeInstances ” and “ec2:TerminateInstances” in this case.

var async = require(‘async’);

var AWS = require(‘aws-sdk’);
var ec2 = new AWS.EC2();
var REQ_TAG_KEY = ‘cost_center’;
exports.handler = function(event, context) { ec2.describeInstances({}, function(diErr, diData){
if (diErr) return context.fail(diErr);
if (!diData || !diData.Reservations ||
!diData.Reservations.length) {
return context.succeed(‘No instances found’);
}
var instsToTerm = [];
async.each(diData.Reservations, function(rsvtn, nxtRsvtn){ async.each(rsvtn.Instances, function(inst, nxtInst){
var found = false;
for (i in inst.Tags) {
if (inst.Tags[i].Key === REQ_TAG_KEY) {
found = true;
break;
}
}
if (!found) instsToTerm.push(inst.InstanceId);
nxtInst();
}, function(){
nxtRsvtn();
});
}, function(){
if (!instsToTerm.length) {
return context.succeed(‘No terminations’);
}
ec2.terminateInstances({
InstanceIds: instsToTerm
}, function(terminateErr, terminateData){
if (terminateErr) return context.fail(terminateErr);
context.succeed(‘Terminated the following ‘ +
‘instances:\n’ +
instsToTerm.join(‘\n’));
});
});
});
}

 

You can now schedule this function to run every hour, day, or other interval. You may want to test this function by adding “DryRun: true” to the first “ec2.terminateInstances” argument (right after “InstanceIds: instsToTerm”).

 

Triggering CodeDeploy with New S3 Uploads

Triggering CodeDeploy

CodeDeploy is a relatively new AWS service that uses an agent on EC2 instances to coordinate deployments across potentially thousands of servers. It has options for rolling back to previous deployments, performing deployments to multiple groups (such as an A/B stack), and many more options to help deploy new code without downtime.

 

I cannot elaborate on the CodeDeploy service much more, but if you use it or have looked into using it, you’re likely aware that it functions by deploying a ZIP of the application code from S3 to the EC2 servers. The usage pattern for CodeDeploy is: upload the new code ZIP to S3, copy the link, launch a new CodeDeploy deployment, paste the link, and then deploy.

 

These steps can become quite cumbersome if you deploy frequently. Fortunately, Lambda can be used to automatically trigger new CodeDeploy deployments when the application’s ZIP on S3 is updated.

 

We’ll begin with the same base setup of responding to an S3 “PutObject” event. However, instead of extracting and processing the ZIP, we’ll simply trigger a CodeDeploy deployment using the object’s location.

var AWS = require(‘aws-sdk’);

var codedeploy = new AWS.CodeDeploy();
exports.handler = function(event, context) {
var bucket = event.Records[0].http://s3.bucket.name;
var key = decodeURIComponent(
event.Records[0].s3.object.key).replace(/\+/g, ‘ ‘);
codedeploy.createDeployment({
applicationName: ‘My App’,
deploymentGroupName: ‘Blue Group’,
revision: {
s3Location: {
bucket: bucket,
bundleType: ‘zip’,
key: key
}
}
}, function(cdErr, cdData) {
if (err) return context.fail(cdErr);
context.succeed(‘Successfully triggered deployment: ‘ +
cdData.deploymentId);
});
};

 

As with each of these examples, the associated IAM role will require the correct permissions (“codedeploy:createDeployment” in this case). Additionally, you will need to create an event trigger in the Lambda console that fires when an object is uploaded to the S3 bucket. 

 

This isn’t quite a complete Jenkins replacement, but it does simplify the actual deployment of the application code once it is placed on S3. If your organization is already using CodeDeploy, this could make the service more automated.

 

Processing Inbound Email

Inbound Email

AWS recently updated the SES platform to allow inbound emails. Once an email is received by AWS on your behalf, it can be used to trigger a number of actions, including a Lambda function.

 

This function has access to the original email and can perform a variety of actions with it. One common example is removing attachments, uploading them to S3, and inserting a download link in the original email to save email server bandwidth and storage space.

 

Another potential use case is to copy the email contents to an S3 file for archiving purposes. You could even use Lambda as a rudimentary SPAM filter by maintaining a list of whitelisted sender addresses.

 

Since we’ve already covered several similar code examples above, I will not include a full code example for email processing. Instead, I’ll discuss the steps required to triggering a Lambda function from SES, since they differ from the others.

 

While most event sources can be configured through the Lambda console, some must be designed in the consoles of other AWS services. In the case of SES, triggering a Lambda function from an inbound email must be configured as part of the inbound email ruleset within the SES console.

 

Navigate to the SES console, and click on “Rule Sets” under “Email Receiving.” Create a new rule, enter a recipient email (such as test@domain.com) and click next. Note that you must have a verified domain or else you cannot receive mail. On the next page, select “Lambda” as the action for the rule.

 

Choose a Lambda function that you’ve uploaded and then select an invocation type. “Event” will simply trigger the event and complete, while “RequestResponse” will wait for the Lambda function to exit before completing.

 

You do not need to enter an “SNS topic.” Follow through the next steps to finish the creation of the rule. Be sure to accept the prompt asking whether you’d like to assign permissions to SES to invoke the function you’ve selected.

 

Once you begin receiving email at the specified address, AWS will trigger a Lambda function for each piece of inbound mail. If you constantly find yourself performing the same actions on inbound email or have scripts on your mail server, creating Lambda functions is one useful alternative.

 

AWS has some helpful guides for interacting with the SES inbound email event if this use case is appealing to you.

 

Enforcing Security Policies

Security Policies

AWS provides hundreds of helpful configuration options to secure accounts and the infrastructure within them.

 

However, until recently, there haven’t been many solutions for actually enforcing that these configurations were properly configured. For example, requiring multi-factor authentication (MFA) for users logging into the AWS console is a great security practice.

 

However, there isn’t currently a way to ensure that all existing users are configured to use MFA. A single developer with the right permissions could disable MFA for herself or even other users.

 

Just like almost all other aspects of AWS, the security configuration is accessible and configurable via the AWS API and SDKs. Using this, we can easily develop a periodic Lambda function that scans all users for the presence of MFA on their accounts.

 

If MFA is not enabled, the Lambda function could send an email to a security or compliance contact within the organization or even temporarily disable the user account.

var async = require(‘async’);

var AWS = require(‘aws-sdk’);
var iam = new AWS.IAM();
exports.handler = function(event, context) {
iam.listUsers({}, function(iamErr, iamData){
if (iamErr) return context.fail(iamErr);
if (!iamData ||
!iamData.Users ||
!iamData.Users.length) {
return context.fail(‘No users found’);
}
var good = [];
var bad = [];
async.eachLimit(iamData.Users, 50, function(user, cb){ if (!user.PasswordLastUsed) {
//
Skip users without passwords since
//
they won’t be logging into the console return cb();
}
iam.listMFADevices({
UserName: user.UserName
}, function(mfaErr, mfaData){
if (mfaErr) {
bad.push(‘User: ‘ + user.UserName +
‘ MFA check error: ‘ + mfaErr); return cb();
}
if (!mfaData ||
!mfaData.MFADevices ||
!mfaData.MFADevices.length) {
bad.push(‘User: ‘ + user.UserName +
‘ does not have an MFA device’); return cb();
}
good.push(‘User: ‘ + user.UserName +
‘ has an MFA device’);
cb();
});
}, function(){
var report = ‘Results for: ‘ + new Date() + ‘\n’ +
‘Warnings: \n’ +
bad.join(‘\n’) +
‘\nOkay: \n’ +
good.join(‘\n’);
//
Optional: Send an Email report context.succeed(report);
});
});
}

 

*This code is adapted from the open-source “CloudSploit” repository, of which I am the author.

** Ideally, you can include code to send email reports if there are warnings, but that requires configuring AWS SES as an authorized email sending for your domain which is out of the scope of this blog.

 

To tie everything together, give the function’s IAM role permissions to call “iam:ListUsers” and “iam:ListMFADevices” and then configure a scheduled event execution every few hours. 

 

MFA is just one example of a security policy that can be enforced using Lambda. Lambda can also scan for misconfigured security groups (allowing access to “0.0.0.0/0”), access keys that haven’t been rotated recently, and more.

 

If you’re comfortable with allowing it, Lambda could also implement remediations for any potential risks as well: disabling users with old keys, removing the offending security group rule, etc. However, this has the potential to impact production performance, so I recommend getting human input before taking automated steps.

 

Detecting Expiring Certificates

SSL

There are probably almost as many services to detect expiring SSL/TLS certificates as there are certificates (maybe a slight exaggeration). Regardless, having an expired certificate is a pretty terrible thing, especially when that certificate expires at 9:30pm on a Friday.

 

Let’s add another service into the mix by creating a simple Lambda function to query for expiring certificates in an AWS account.

 

Even if they aren’t being actively used, having expired certificates in IAM makes them readily available for attaching to ELBs and CloudFront distributions. Everyone understands the risks of that, so instead of paying a third-party to monitor all your sites, use Lambda for a more complete picture instead.

 

We’re going to create another scheduled function (the interval is up to you, but I recommend at least weekly), this time with “iam:ListServerCertificates” permissions.

 

var AWS = require(‘aws-sdk’);

var iam = new AWS.IAM();
exports.handler = function(event, context) {
iam.listServerCertificates(function(err, data){
if (err) return context.fail(err);
if (!data ||
!data.ServerCertificateMetadataList ||
!data.ServerCertificateMetadataList.length) {
return context.succeed(‘No certificates’);
}
var now = new Date();
var expired = [];
var warning = [];
var valid = [];
for (i in data.ServerCertificateMetadataList) {
var certData = data.ServerCertificateMetadataList[i];
if (certData.ServerCertificateName &&
certData.Expiration) {
var then = new Date(certData.Expiration);
//
number of days difference var difference = Math.floor(
(then - now) / 1000 / 60 / 60 / 24);
if (difference > 45) {
good.push(certData.ServerCertificateName);
} else if (difference > 0) { warning.push(certData.ServerCertificateName);
} else {
expired.push(certData.ServerCertificateName);
}
}
}
// Optional: Send an Email report
context.succeed(‘Certificate Report:\n’ +
‘Expired:\n’ +
expired.join(‘\n’) +
‘\nExpiring within 45 Days:\n’ +
warning.join(‘\n’) +
‘\nValid:\n’ +
valid.join(‘\n’));
});
}

 

Again, you will need to configure SES for outbound emails if you want to send email reports.

 

Utilizing the AWS API

AWS API

The examples in this blog provide just a small glimpse into what is possible with Lambda. Because Lambda executes using IAM roles and with full access to the AWS SDK, creating functions that interact with the AWS infrastructure is as simple as writing a script.

 

When access to the AWS API is combined with on-demand, event-driven, pay-per-execution computing, the possibilities are virtually limitless.

 

I challenge you to think of an existing monitoring, security, or infrastructure maintenance task that can be performed with Lambda and to implement it as a way of learning the platform.

 

Execution Environment

AWS has been fairly tight-lipped about the technical specifics of the environment running Lambda. Users have been left to guess how the service itself operates. For example, while AWS notes that it executes the Lambda function in a “container,” there have not been many technical details released about this setup.

 

Some users have hypothesized that Lambda is actually running in Docker containers; AWS has not confirmed or denied this. There is no documentation regarding the scaling policies, open ports on the instance, IP addressing, etc.

 

While the vagueness is most likely intentional in order to make the service as “set and forget” as possible, this will not appease the developers requiring more information for compliance or auditing requirements.

 

 From the documentation and presentations that AWS has released, we’ve been able to gain some insight into how the architecture behind Lambda is structured. In this blog, I will reveal what we do know, and how that affects the development of functions on Lambda.

 

The Code Pipeline

Code Pipeline

When the Lambda code is uploaded to the service as a ZIP file, AWS copies it to an S3 bucket under its control. According to the Lambda FAQ, the code is encrypted at rest, as well as passed through “additional integrity checks while your code is in use,” which I assume means that they verify the checksum of the code when it is downloaded to an instance before execution.

 

When the code is first uploaded and again when the code is updated, AWS downloads the code from S3 to a container allotted to execute it, based on memory requirements.

 

Everything outside of the event handler is initialized. In the case of Node.js, this means that all of the modules are loaded into memory, any external database connections are made, and the code is prepared for execution.

 

When the function is invoked, the event handler is called and the response is either returned to the caller (“response required”) or logged (“Event”).

 

If the function is not invoked after a period of time (AWS has not released the exact amount of time, but my testing indicates it is around 10-15 minutes), it is removed from the underlying container. If it is subsequently invoked, the code is re-downloaded and initialized.

 

Cold vs. Hot Execution

This pattern of initialization and decommissioning of code creates a dilemma for infrequently accessed functions that have low latency requirements. Because the code must be re-initialized after the ten-minute mark, there is an added period of latency before the code is ready for execution.

 

From my work with Lambda, I’ve seen this latency range from 100 to 800 ms for most Node.js functions under 10MB. However, some Java users on the AWS forums have claimed latencies as high as 3500 ms. Obviously, this is not feasible for applications with low latency requirements, but may be acceptable for other use cases.

 

These methods of starting a function are referred to by AWS as “hot” and “cold” executions. During a “hot” execution, the function has already been invoked and initialized in the past 10-15 minutes and simply re-executes the event handler.

 

Response times for hot executions are usually determined only by the time requirements of the code itself. “Cold” executions involve re-downloading and initializing the code.

 

Response times are composed of the ZIP download, extraction, code initialization, and then execution. Of course, larger files take longer to download and extract than their leaner counterparts and thus take longer to invoke from a cold state. Here is a helpful chart to illustrate this.

 

Although the relationship isn’t exactly linear, it is important to keep your ZIPs as small as possible. Removing README, help, documentation, unused module, and other files can help in reducing cold boot times.

 

If you are running into latency issues due to cold booting, there is a fairly simple solution: establish a scheduled task to invoke your function every nine minutes. Be sure to add an if-else clause to the main event handler to exit immediately in order to reduce total execution time.

 

Executing the function every nine minutes will only require about 4800 executions per month, well under the free tier limit. Even without the free tier, such a small number of executions will likely pay for itself by reducing cold boot times for the other executions.

 

 I encourage you to heavily test the impact of execution latency on your applications. When coming from an EC2-backed model, these latencies can introduce an unexpected wrench into the application.

 

What is Saved in Memory

Memory

As mentioned earlier, the application code outside of the event handler is only initialized once per host container. This means that if the function executes multiple times on the same container, all global variables and contents of memory will be accessible to each of those executions.

 

This requires important performance and security considerations. It cannot be guaranteed that anything outside of the event handler will be unique to the execution.

 

Take the following code samples using a hypothetical event object:

var ITEMS_TO_PROCESS = [];

exports.handler = function(event, context) {
for (i in event.items) {
ITEMS_TO_PROCESS.push(event.items[i]);
}
context.succeed();
};
exports.handler = function(event, context) {
var ITEMS_TO_PROCESS = [];
for (i in event.items) {
ITEMS_TO_PROCESS.push(event.items[i]);
}
context.succeed();
};

 

In the first snippet, the “ITEMS_TO_PROCESS” array can be read and modified from any Lambda function that is executing on the container. If the function uses this array for future dictation of events, one function may wind up processing events pushed to the array from another function that has executed on the same container.

 

In the second example, the array is kept private by its inclusion within the event handler. Every function invocation will create a separate array variable to which the event’s items can be added.

 

As you can likely see, this can lead to security issues if the function assumes that variables outside of the event handler are private. If the example above involved credit card numbers, it would certainly be a violation of regulatory requirements to make a list of credit card numbers accessible to additional copies of the same function being invoked.

 

To put this in a traditional, web server context, this would be akin to saving user data to global variables rather than variables private to the individual request.

 

Despite the negative issues associated with the sharing of globally initialized code, it can have beneficial use cases. One common example is caching. In the code samples above, “ITEMS_TO_PROCESS” could be replaced with an object cache.

 

Imagine that data is retrieved from a database. The following example utilizes caching to reduce database queries and improve the response time of the function.

var CACHE = {};

exports.handler = function(event, context) {
if (CACHE[event.item]) {
return context.succeed(CACHE[event.item]);
}
//
Lookup object in database db.find(event.item, function(err, item){
if (err) return context.fail(err);
CACHE[event.item] = item;
context.succeed(item);
});
};

 

Because the cache is outside of the event handler, subsequent executions will have access to the same cache shared among all other executions on the same container. Do keep in mind that you may want to clear the cache occasionally to reduce memory requirements depending on the frequency of the function’s invocations.

 

While caching can dramatically improve performance, keep in mind that the container running the Lambda function, and therefore storing the cache in memory, is not guaranteed to remain in place. Scaling events or increased times between execution can cause the cache to disappear without warning.

 

Scaling and Container Reuse

 

Container Reuse

AWS has not released many details about the scaling policies of the hosts and containers running Lambda. Throughout the Lambda documentation, AWS only reveals that it “handles the scaling requirements automatically” in the background and that no input is required from the developers.

 

For developers coming from EC2, this lack of control is a drastic contrast to the wide array of configuration options that can be used to create autoscaling policies.

 

Many applications have strict latency requirements, the metrics of which can be used to scale a group of servers as soon as a threshold is reached. With Lambda, this is not possible.

 

If the application must execute within 100 ms and Lambda is executing in 150 ms due to the load on the container, AWS may deem this acceptable and refuse to scale. Again, there is virtually no indication from AWS as to when these scaling events may occur, nor is there the ability to control how or when they do.

 

AWS will reuse the same container when executing simultaneous Lambda functions until the allotted memory requirements force additional containers to be launched. There is no indication that this occurs, other than the fact that executions will continue to complete without added latency because of resource constraints.

 

Perhaps in the future AWS will provide more configurations around scaling policies. Until then, be sure to stress test your functions heavily if you expect the load to increase considerably.

 

From Development to Deployment

Deployment

In the next sections, I will be focusing on Lambda as a production service. As with other production services, there needs to be a focus on application design, proper development patterns, testing, deployment, and continuous monitoring.

 

Unlike traditional, EC2-based projects, Lambda requires adaptations of thought for each of these processes, which I will explore here. As is also the case with software development, there are many very unique approaches, so do not assume that these are the only ways in which you can develop with Lambda.

 

Application Design

Application Design

Perhaps the biggest difference between Lambda and traditional applications is that Lambda functions are designed around specific tasks while applications tend to be much more monolithic.

 

If you’ve been following the recent trend of converting large applications into pieces, called “microservices,” then Lambda should be familiar. Lambda takes the idea of microservices a bit further by compartmentalizing individual workflows into separate functions.

 

To illustrate the design differences between traditional applications and Lambda, imagine a standard, web-based, e-commerce platform. At its core, the application is composed of interactions between a web front-end and a products database as well as payments and shipping systems.

 

Older, monolithic applications may include everything bundled together. The microservices approach may separate the product, payments, and shipping interactions into separate APIs.

 

Lambda may require a group of fifteen or twenty functions, several in each of the product, payment, and shipping focus areas. For example, one Lambda function could update the products database with new inventory counts after a purchase was made. Another may validate shipping address details. Still, another may process shipping info to calculate a shipping and handling charge.

 

There is no steadfast rule that dictates the separation of duties among Lambda functions. However, it is best to separate distinct duties into different functions.

 

The following diagrams illustrate the progression from a traditional, monolithic application running on EC2 servers in autoscaling groups to a microservices-based architecture running on ECS, to finally a complete Lambda replacement. Obviously, the example is quite simplified, but the principles remain.

  • Traditional, Monolithic Architecture with EC2 Instances
  • Microservices Architecture Running on ECS with Docker Containers

 

Lambda Architecture

Lambda Architecture

This separation may become cumbersome due to the number of separate projects (functions) you must maintain. However, it has the advantage of allowing you to update pieces of your application without impacting the remaining parts.

 

For example, you could update the logic used to calculate shipping costs without affecting the remaining functions that are updating the databases.

 

If you do decide to convert a large application to Lambda, I recommend diagramming out each of the parts and how they will interact. While most smaller applications can easily be implemented in a couple functions, larger ones will not be as easy to organize and maintain.

 

Development Patterns

Development Patterns

Just like microservices, most Lambda functions can (and should) be designed to be fairly project-agnostic. To continue our e-commerce example from earlier, the shipping cost calculation function is an ideal candidate for use in multiple projects. The e-commerce store, as well as internal tools and calculations, could all rely on the same function.

 

In most of our examples, the event object passed into the invocation of the function has been predetermined by AWS; the format is fairly standard across multiple services. When invoking the function directly, either from the command line or the SDK, the event object is arbitrary.

 

As a good development practice, I recommend standardizing on a common format for all of your events across all functions. Here is a quick example of a format I’ve used for almost all of the functions I’ve created.

{

“metadata”: {
// Information about the invoking resource, date, etc.
},
“data”: {
// Arbitrary data required for use in the function
}
}

 

The “metadata” object could contain the invoker’s IP address, hostname, user agent (if applicable), the current date, timezone, region, AWS account, IAM role ARN, and anything else that is relevant or required by your logs. Once the function receives this event, it can include this information in its logs for debugging, security, and auditing purposes later.

 

The “data” object contains the information necessary for the function to execute. To use a simple “Hello, User” example, this would be the user’s name.

 

This may seem like overkill for a simple function like “Hello, world.” But once you begin processing lots of data or writing Lambda functions that can handle a variety of inputs, I believe the above structure will become quite useful. Of course, you should adapt it to meet the needs of the majority of your projects.

 

Within the “controllers” directory, each action performed by the Lambda function is separated into new files. For example, if this function responded to S3 “PutObject” and S3 “DeleteObject” events within a bucket, there could be “objectAdded.js” and “objectRemoved.js” controllers.

 

Note that the line between creating multiple Lambda functions and creating multiple controllers within a single function can be blurry at times. As a rule of thumb, I create different Lambda functions if they touch different resources or I want to separate the updates to the functions from one another.

 

To determine which controller to use, the “index.js” file can handle the routing. For example, the following sample code would call the correct controller depending on the S3 event.

 

var objectAdded = require(__dirname + ‘/controllers/objectAdded.js’);
var objectRemoved = require(__dirname + ‘/controllers/objectRemoved.js’);
exports.handler = function(event, context) {
var action = event.Records[0].eventName;
if (action.indexOf(‘ObjectCreated’) > -1) {
objectAdded(event, context);
} else if (action.indexOf(‘ObjectRemoved’) > -1) { objectRemoved(event, context);
} else {
context.fail(‘Invalid event’);
}
};

 

Every event source will require a different algorithm for determining the correct route. The API Gateway event source could call a different controller based on the URL path.

 

Custom events could include a controller name in the metadata object. DynamoDB events could utilize a different controller based on the table name. Regardless of what you choose, I’ve found it is easiest to stick with one pattern per event type for all projects.

 

The “helpers” directory contains resources that can be used in any of the other files in the function. The “logger.js” file could implement custom logging as described in blog 10. This file could then be required in all the other files, providing access to common logging functionality across the function.

 

If you want to begin adding a timestamp to logs, the change could then be made in one place. The “responses.js” file could provide the same functionality for standardizing the format of responses. Below is a sample “responses.js” file that I use frequently.

var succeed = function(context, data) {
context.succeed({
status: 200,
code: 0,
data: data
});
};
var error = function(context, message, errors, status, code) { context.succeed({
status: status || 200,
code: code || 1,
message: message || ‘Error’,
errors: (Array.isArray(errors)) ? errors : [errors]
});
};
var fail = function(context, failureMessage) {
context.fail(failureMessage);
};
module.exports = {
succeed: succeed,
error: error,
fail: fail
};

 

You can modify the response format as needed, but by providing a common format, errors due to conflicting or unexpected response formats can be minimized.

 

In this blog, I’ve presented a number of patterns that I follow when developing Lambda functions. They are by no means complete or required. However, regardless of how you choose to develop your functions, I urge you to consider creating some kind of base project which you can fork when starting a new Lambda function.

 

Having a common starting point helps reduce development errors and standardize how you and your developers interact with Lambda in the application pipeline. Because of their “microservice” nature, you’ll likely have many more Lambda functions than monolithic applications, so deciding on good standards early on is a must.

 

Testing

Testing

I’ll cover specific items that you’ll want to test before launching a function into production. Most of these are difficult to detect when developing locally, so be sure to upload your function to a staging environment that mimics production.

 

●Does the function have the correct IAM role and permissions to send logs to CloudWatch?

Debugging Lambda functions is especially difficult without logs. If the function does not have permission to create a log group, create a log stream, and write to that log stream, you won’t find any logs.

 

●Does the function have the correct IAM role and permissions to access additional resources?

Just like EC2’s use of IAM roles, Lambda requires a role to access other resources in your account. If the function downloads an object from an S3 bucket and uploads it elsewhere, it will need permission to do both of this action.

 

●Is the function timeout set too low?

Even if you expect a function to execute in 1000 ms, cold boot times and potential external resource connection issues may mean the function occasionally takes 2000. For most functions, I recommend increasing the allotted time by a generous amount to prevent unexpected timeouts.

 

●Do resources accessed by your function limit access by IP address?

Developers have no control over the external IP address of the container executing the Lambda function. If your database limits connections by IP address, Lambda won’t be able to connect to it. In addition, don’t expect the IP to remain the same for an extended period of time; containers are rotated at various times.

 

●Does the function always exit using context.fail(), context.succeed(), or context.done()?

If a Lambda function does not call one of these methods, it will continue to consume billable time. To avoid unexpected behavior and increased costs, always exit cleanly. Don’t let Node.js callbacks prevent an exit either!

 

●Is memory shared securely across executions?

As discussed in the previous blog, any code outside of the event handler is only initialized once. It is then shared across every execution of Lambda on the same container. Do not store sensitive information outside of the event handler if you do not want it to be accessible by other executions.

 

Deployment

AWS CodeDeploy

The deployment of Lambda functions is perhaps one of my (many) favorite Lambda features. Gone are the days of blue-green deployments, DNS swaps, AWS CodeDeploy, SSH, downtime, traffic migrations, hot restarts, and other complicated mechanisms for deploying new functionality to production applications.

 

With Lambda, deploying is as simple as updating the function with a new ZIP through the console or API. AWS manages the distribution of that function to the underlying containers, as well as the traffic flow off of the old function. You can deploy hundreds of times per day without worrying about downtime or performance impacts.

 

Since we’ve already covered deploying a function through the console (simply uploading the ZIP), I will focus on ways to simplify the continuous integration pipeline with Lambda.

 

Like most developers, you likely use a source control service like GitHub to manage your code. Additionally, you likely have dependencies that need to be installed (i.e. Node modules) and unit tests to run.

 

While you could certainly run these locally, upload a ZIP of your local files to S3, and update the Lambda function using the AWS CLI, it will likely become tedious after a few deployments.

 

For that reason, I’ve been using Jenkins, a very popular continuous integration tool, to install dependencies, test, ZIP, and deploy Lambda functions. Combined with a Git webhook, it makes deployments repeatable and fast.

 

Jenkins provides a Lambda plugin that will update your function either directly or from S3. Personally, I prefer to keep a backup of files on S3 and then deploy from that, but either solution will work. To get started, simply search for “Lambda” in the Jenkins plugin console.

 

The access key used by Jenkins must have the following permissions:

{

“Action”: [
“lambda:GetFunction”,
“lambda:GetFunctionConfiguration”,
“lambda:ListFunctions”,
“lambda:UpdateFunctionCode”
],
“Effect”: “Allow”,
“Resource”: “arn:aws:lambda:*:01234567890:*”
}

 

If Jenkins will only be operating in one region, the “*” can be replaced with that region.

After applying the correct IAM permissions to the key and installing the plugin, new Lambda deployments can be initiated by adding a post-build action. The “Role” is the role of the Lambda function, not the Jenkins role (if you are using one). 

 

In this example, the “http://s3://my-bucket/function.zip” location could easily be replaced with a local file system ZIP location such as “/tmp/function.zip.”

 

One thing to keep in mind is that Jenkins may apply special OS-level permissions to the files within the ZIP. To work on Lambda, they will need the proper permissions, which, if it occurs on your Jenkins instance, can be fixed with a bash step to “chmod” the files before generating the ZIP.

 

If the deployment fails, the results of the calls made from Jenkins will be piped to the deployment logs, allowing further investigation. I won’t dive into the steps of setting up a Git webhook with Jenkins, but the process is simple and easily found with a quick web search.

 

Once connected, pushing to a Git branch should result in your function being built and updated by Jenkins. There are, of course, numerous other mechanisms of deploying Lambda functions. The simplest is to simply use the AWS CLI and run:

 style="margin:0;width:962px;height:141px">aws lambda update-function-code
—function-name <value>
[—zip-file <value>]
[—s3-bucket <value>]
[—s3-key <value>]
[—s3-object-version <value>]
[—publish | —no-publish]
[—cli-input-json <value>]
[—generate-cli-skeleton]

 

Be sure that any local files, especially sensitive ones, that you do not wish to deploy are not included in ZIPs built locally. I’ve focused on Jenkins because it is one of the most popular build tools.

 

However, any other build tool that allows shell scripts and the installation of the AWS CLI could easily replicate the above commands as a build step.

 

Monitoring

Monitoring

we briefly explored the Monitoring tab within the Lambda console for basic testing purposes. In this blog, I will dive into how CloudWatch alerts can be configured to continuously monitor for failed or throttled executions as well as above-average execution times.

  • Begin by creating a new alarm within the CloudWatch console. Select the function, resource, and metric name that you wish to monitor.

 

  • On the next page, specific information about what will trigger the alert can be added. For example, I can receive an SNS notification if the number of errors rises above two in a five-minute timeframe.

 

  • I recommend setting up alarms for errors, invocation duration, and throttled invocations for every production function that you launch. Since Lambda functions execute mostly independently of one another, obscure errors that would crash a traditional application can go unnoticed for some time.

 

Versioning and Aliasing

Versioning

When AWS first launched Lambda, updating a function would completely replace all future invocations with the new code. Since then, they have released an update which allows for both versioning and aliasing of Lambda functions, thus allowing an application to reference a specific version of a function which will continue to execute even as newer versions are uploaded.

 

A direct comparison of this functionality with that of versioned APIs can be made. For APIs that support numerous other applications, it is important that the functionality and response formats remain the same, even as new features are added and updates are made.

 

To create versions of a Lambda function, simply open the Lambda console, navigate to the function you’d like to update, and click “Actions” > “Publish new version.”

 

This will freeze the current code as a version, for which you can provide a description. This version will never change, even as new updates are made. It is now safe to reference this version directly in your application code.

var params = {
FunctionName: ‘arn:aws:lambda:us-east-1:1234567890:function:test-function:1’,
Payload: ‘{user:“Bob”}’
};
lambda.invoke(params, function(err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});

 

Note the “1” after “function:” in the ARN above. By specifying the “1,” this invocation will call the specific version “1.”

If you did not provide a “:1”, and simply invoked the function name or the base ARN, Lambda would execute the current, latest code. As you can imagine, referencing function versions only by a number can become quite confusing if you have numerous versions.

 

For that reason, AWS added the ability to create aliases. These human-readable names can allow you to replace the “1” with something like “:working_backend”.

 

It is important to know that aliases are simply pointers to versions (and not vice versa). Aliases can be modified to point to new versions at any time. For example, I could have an alias called “staging” which I point to new versions before they are released to production.

 

I could hard code the “staging” reference in my application code, but then change the function to which it is pointed at any time. This can all be done within the Lambda console.

 

Costs

 

Costs

Lambda is designed to be a low-cost alternative for EC2. That being said, the cost model can cause the monthly charges to fluctuate quite heavily depending on the volume of requests and their execution times. Generally, Lambda is less expensive than comparable EC2 instances until the requests approach several million per month at low to moderate execution durations.

 

For long-running processes with high memory requirements, that number can be reduced to just a few thousand before the costs would exceed the lowest EC2 offering.

 

Let’s look at a few examples to illustrate these points. All examples will be using the most recent pricing model at the time of publication as well as US dollars. In each case, I will calculate the number of executions allowed before the costs will exceed a t2.micro instance (currently $9/month). The free tier will not be included.

 

Short Executions

Lambda is quite suited to performing millions of small executions for a very small cost.

Allocated Memory

Execution Time (ms)
Number of Executions
(MB)
128
100
22,000,000
1024
500
1,055,000
1536
1000
360,000

 

As you can see, if your function only requires the minimum memory (128 MB) and execution time (100 ms), you could execute 22 million times in a month for about $9. If you use the maximum memory (1536 MB) and a one-second execution, that is reduced dramatically to 360,000 executions.

 

At this point, if you could run a t2. micro EC2 server (1 GB of memory) that could perform more than 1 million, 500 ms executions in a month, it would be more cost effective than running Lambda.

 

Long-Running Processes

 

Long-Running Processes

Currently, the maximum execution time of Lambda is five minutes. The following chart shows how long-running Lambda functions can wind up exceeding the cost of EC2’s t2.micro at one, two and a half, and five-minute durations.

Allocated Memory

Execution Time (ms)
Number of Executions
(MB)
128
60,000
72,000
1024
150,000
3,600
1536
300,000
1,200

 

The number of executions is now drastically lower. If you can spawn multiple processes on an EC2 server and run more than 3,600, 2.5-minute executions in a month, it would be more cost-effective.

 

High-Memory Applications

High-Memory Applications

If your application has memory requirements that exceed 1.5 GB, they can not run directly on Lambda (the maximum memory allocation is currently 1.5 GB). However, some developers have split these applications into pieces in order to fall under the memory limitation. While this could save money in some cases, I urge you to calculate the total costs versus comparable EC2 costs.

 

If you can launch an EC2 instance with 4 GB of memory for only $35/month, it may be able to complete far more executions in the same time period than $35 worth of Lambda executions. Of course, because of Lambda’s per-execution pricing model, this only applies if the total number of executions exceeds the break-even point.

 

Free Tier

As with most AWS services, Lambda includes a free tier that is actually quite generous. Every month, developers receive one million free executions and 400,000 GB-seconds of compute time. That’s enough to replace quite a few EC2 servers running cron scripts.

 

Calculating Pricing

Although AWS describes pricing in detail, I’ve found their examples to be a bit confusing at first. For that reason, I’ve developed a pricing calculator which I host online The JavaScript script for this calculator is below; you only need to enter the number of executions, memory, and average execution time.

var NUM_EXECUTIONS = 0;
var ALLOCATED_MEMORY = 128;
var EXECUTION_TIME = 1000;
var INCLUDE_FREE_TIER = true;
// MB
// ms
var executionsToCount =
INCLUDE_FREE_TIER ? (NUM_EXECUTIONS - 1000000) : NUM_EXECUTIONS; var requestCosts =
executionsToCount > 0 ? (executionsToCount / 1000000) * .20 : 0; var computeGBS =
(NUM_EXECUTIONS * (EXECUTION_TIME/1000)) * (ALLOCATED_MEMORY/1024); var totalCompute = INCLUDE_FREE_TIER ? (computeGBS - 400000) : computeGBS;
var executionCosts = totalCompute > 0 ? totalCompute * 0.00001667 : 0;
return ‘$’ + (requestCosts + executionCosts).toFixed(2);

 

CloudFormation

CloudFormation

One trend in infrastructure and operations has been the move towards describing infrastructure as code. By describing resources like EC2 servers, ELBs, and S3 buckets in a file as code, developers can version, check in, and comment on changes to the environment before they are made, reducing errors and maintaining a single source of truth for what is running.

 

The AWS answer to this trend is CloudFormation, a service that uses JSON templates describing AWS resources to launch the resources themselves into an AWS environment.

 

Although Lambda was not supported by CloudFormation at launch, it was added shortly afterward. It is now possible to describe Lambda functions, their memory configurations, code locations, and IAM permissions using CloudFormation.

 

In this blog, I will provide a few examples of complete CloudFormation templates that can be used as starting points for new function launches. All of these templates are available on my GitHub repository as well.

Reusable Template with Minimum Permissions

{
“AWSTemplateFormatVersion”: “2010-09-09”,
“Description”: “Launch a Lambda function with configurable memory and timeout”,
“Parameters”: {
“MemorySize”: {
“Type”: “Number”,
“Description”: “The memory size in 128 byte increments”,
“Default”: “128”,
“MinValue”: “128”,
“MaxValue”: “1536”,
“ConstraintDescription”: “Must be a valid number between 128 and 1536 in 128 byte increments.”
},
“Timeout”: {
“Type”: “Number”,
“Description”: “The maximum seconds to allow the function to run”,
“Default”: “5”,
“MinValue”: “1”,
“MaxValue”: “60”,
“ConstraintDescription”: “Must be a valid number between 1 and 60.”
},
“Runtime”: {
“Type”: “String”,
“Description”: “Which runtime to use”,
“Default”: “nodejs”,
“AllowedValues”: [“nodejs”, “java8”, “python2.7”]
},
“S3Bucket”: {
“Type”: “String”,
“Description”: “S3 bucket hosting the code”
},
“S3Key”: {
“Type”: “String”,
“Description”: “S3 key path to the code”
}
},
“Resources”: {
“LambdaRole”: {
“Type”: “AWS::IAM::Role”,
“Properties”: {
“AssumeRolePolicyDocument”: {
“Statement”: [{
“Effect”: “Allow”,
“Principal”: {
“Service”: [ “http://lambda.amazonaws.com” ]
},
“Action”: [ “sts:AssumeRole” ]
}]
}
}
},
“LambdaRolePolicies”: {
“Type”: “AWS::IAM::Policy”,
“Properties”: {
“PolicyName”: “IAMLambdaPolicy”,
“PolicyDocument”: {
“Statement”: [{
“Effect”: “Allow”,
“Action”: [
“logs:CreateLogGroup”,
“logs:CreateLogStream”,
“logs:DescribeLogGroups”,
“logs:DescribeLogStreams”,
“logs:GetLogEvents”,
“logs:PutLogEvents”,
“logs:PutRetentionPolicy”
],
“Resource”: [{
“Fn::Join”: [
””, [
“arn:aws:logs:”, {“Ref”: “AWS::Region”},
“:”, {“Ref”: “AWS::AccountId”},
“:log-group:/aws/lambda/*”
]
]
}]
}]
},
“Roles”: [{“Ref”: “LambdaRole”}]
}
},
“LambdaFunction”: {
“Type”: “AWS::Lambda::Function”,
“Properties”: {
“Code”: {
“S3Bucket”: {“Ref”: “S3Bucket”},
“S3Key”: {“Ref”: “S3Key”}
},
“Description”: “Lambda function”,
“Handler”: “index.handler”,
“MemorySize”: {“Ref”: “MemorySize”},
“Role”: {“Fn::GetAtt”: [“LambdaRole”, “Arn”]},
“Runtime”: “nodejs”,
“Timeout”: {“Ref”: “Timeout”}
}
}
}
}

 

This template can be launched for each new Lambda function. However, I recommend adapting it to handle several related functions with multiple parameters (simply copy additional parameter sets and function resources) instead of launching a single CloudFormation stack for every function.

 

Personally, I create one stack for each project and include Lambda functions directly with the other resources. You can use the template above as a starting point for this.

 

Cross-Account Access

If you prefer CloudFormation, the following template snippet will accomplish the same thing. Be sure to replace the account number and role name in the ARN as well as reference the correct Lambda function name.

 

By adding this resource to the CloudFormation template above, permission will be granted for the external AWS account to invoke this function.

“CrossAccountLambdaRolePermissions” : {
“Type”: “AWS::Lambda::Permission”,
“Properties”: {
“FunctionName” : {“Fn::GetAtt” : [“LambdaFunction”,“Arn”]},
“Action”: “lambda:InvokeFunction”,
“Principal”: “arn:aws:iam::0123456789012:role/role-name”
}
}

 

CloudWatch Alerts

CloudWatch Alerts

We created CloudWatch alerts within the CloudWatch console that can send a notification to an SNS topic if there are an elevated number of errors for a particular Lambda function.

 

If you prefer CloudFormation, the following snippet will create this alarm. Note that if the Lambda function and SNS topic are not also created in the same template, you will have to replace the references with the correct function and topic names.

“TestLambdaErrorsAlarmHigh”: {
“Type” : “AWS::CloudWatch::Alarm”,
“Properties” : {
“AlarmName” : “test-lambda-errors-high”,
“AlarmDescription” : “Alert if Lambda errors rise”,
“AlarmActions” : [ { “Ref” : “TestAlertsSNSTopic” } ],
“MetricName” : “Errors”,
“Namespace” : “AWS/Lambda”,
“Statistic” : “Sum”,
“Period” : “300”,
“EvaluationPeriods”: “1”,
“Threshold” : “0”,
“ComparisonOperator” : “GreaterThanThreshold”,
“Dimensions” : [ {
“Name” : “FunctionName”,
“Value” : {“Ref”: “LambdaFunction”}
}]
}
}

 

AWS API Gateway

AWS API Gateway

Even though an entire another blog could be written to cover the AWS API Gateway, I would be remiss if I didn’t at least mention it here.

As one of the services designed since the release of Lambda, the API Gateway is perfectly suited for integrating with the service. Without the API Gateway, Lambda functions cannot receive traffic directly from the Internet.

 

The API Gateway allows traditional HTTP traffic to be mapped from a client’s request, to a Lambda backend, and then back to the client. Let’s look at a very basic API Gateway setup that will take our “Hello, User” Lambda function and make it accessible as a web-based API.

 

API Gateway Event

Developing Lambda functions that are designed to respond to API Gateway events differ slightly from the ones we’ve explored so far. Since Lambda input events must be valid JavaScript objects, the API Gateway must convert all the relevant portions of an HTTP request to such an object.

 

For instance, the request headers, URL parameters, URL query strings, POST body, and additional client information are all accessible from the API Gateway. It then uses a configurable process called a “mapping” to turn that information into an event object which can be used in the Lambda invocation.

 

A complete list of mappings is available in the API Gateway documentation. However, some of the most relevant and useful include:

●$context.httpMethod The HTTP method of the original request. GET, POST, PUT, DELETE, etc.

●$context.resourcePath The URL path. Useful when making conditional decisions based on the URL.

●$input.json(‘$’) The body of the request (POST/PUT/etc. body).

●$input.params().header The request headers.

●$input.params().querystring The URL query string (ex: ?user=bob&version=1)

●$input.params().path

 

The URL path parameters. Not to be confused with the resource path. Whereas “$context.resourcePath” would look like “/hello/{user}”, the “$input.params().path” would look like {“user”:”bob”}.

 

This is only a small portion of what is made available. Additional fields include the remote user’s IP address and user agent, the AWS account hosting the API, identity information, and much more.

 

When setting up the API Gateway, it’s ultimately up to you which of the above items you pass to the Lambda function. Regardless of the ones you choose, it must be a valid object.

 

We will look at creating this event mapping within the API Gateway console shortly, but for now, just understand the event being passed to Lambda is configurable from the API Gateway and that this event differs from other AWS events like those created by S3 and DynamoDB.

 

Creating the Lambda Function

 

Lambda Function

Before we can configure a new API, we must create the Lambda function that will respond to requests. We’ll start out extremely simple and then build on it after the API is configured. Create a new Lambda function with 128 MB of memory, a ten-second timeout, and a basic execution IAM role. Paste the following code into the box:

exports.handler = function(event, context) { 
console.log(event);

context.succeed({code:0, data: “Hello, World”});

};

 

At this point, we are only concerned with viewing what data is available via the event object. Later, we will add functionality to obtain the user’s name from the HTTP request and send it as part of the response. Be sure to save the function.

Recommend