Integrate SQS and Lambda: serverless architecture for asynchronous workloads
The Gold Standard for modern cloud native applications is a serverless architecture. AWS Lambda allows you to implement scalable and fault tolerant applications without the need of a single virtual machine.
A serverless infrastructure based on AWS Lambda has two main advantages:
- You don’t need to manage a fleet of virtual machines anymore.
- Deploying new versions of your code can be fully controlled by API calls.
This article shows you how to process asynchronous tasks serverless. Possible use cases are: sending out massive amounts of emails, transcoding video files after upload, or analyzing user behavior. A SQS queue will be used to decouple your microservice from other parts of your system. You’ll learn how to implement the microservice with AWS Lambda.
A best practice when building scalable and highly available systems on AWS is to decouple your microservice by using one of the following options:
- ELB (Load Balancer) for synchronous decoupling: web application answering HTTPS requests from browsers
- SQS queue for asynchronous decoupling: sending out massive amounts of emails, transcoding video files after upload, or analyzing user behavior.
Amazon Simple Queue Service (SQS) is a managed message queuing service. SQS offers fault tolerant and scalable distributed message queues: simple to use but very powerful.
The following figure shows a typical architecture based on SQS. A producer sends tasks (also called messages) to a SQS queue. A fleet of consumers reads tasks from the SQS queue and does the actual computing.
The number of consumers can be scaled based on the number of tasks in the queue very easily. As the tasks are stored durable and are managed within SQS it is also very easy to build a fault tolerant system recovering from failed consumers without loosing any tasks automatically.
SQS offers a REST API so there are various options to integrate into producers and consumers: mobile or client-side web applications, applications running on EC2 instances, and AWS Lambda.
Amazon announced AWS Lambda in November 2014. Since then serverless computing has become one of the hottest topics within the public cloud market.
You are able to execute small computing tasks with the help of AWS Lambda. Currently your function needs to finish its work within 300 seconds.
AWS Lambda offers:
- Scalable and highly available computing capacity.
- Automated deployments allowing you to setup a continuous delivery pipeline.
- A fully managed and maintenance free service.
Currently Lambda supports Node.js, Java, and Python. You just have to upload your source code to S3. Afterwards you are able to execute your functions by calling the REST API or use one of the integrations: Kinesis, S3, DynamoDB, CloudTrail, and API Gateway.
Ath the moment there is no out-of-the-box integration for SQS. But you’ll learn how to integrate SQS with Lambda easily next.
The following figure shows all components needed to read and process messages from a SQS queue serverless:
- The SQS queue receives and stores tasks from other parts of the system.
- The CloudWatch Event Rule triggers the Lambda Consumer based on a schedule (e.g. every minute).
- The Lambda Consumer reads as many messages as possible from the SQS and executes a Lambda Worker for each message.
- The Lambda Worker performs the actual computing tasks and deletes the task from the SQS queue after the task was completed successfully.
Let’s dive a little bit deeper into implement and setup this infrastructure with the help of CloudFormation, my favourite Infrastructure as Code tool.
First you need to setup a SQS queue. The following CloudFormation snippet contains:
- A SQS queue for your tasks.
- A SQS queue acting as dead letter queue.
What is a dead letter queue? A dead letter queue is used to collect failed tasks. In this example a task is moved to the dead letter queue if the Lambda Consumer or Worker has failed to successfully complete the task within 60 seconds for 10 times. This mechanism allows your system to get rid of broken tasks without blocking the whole task queue.
Let’s proceed with the Lambda Consumer.
The following snippet contains all resources needed for the Lambda Consumer:
- An Events Rule called ScheduleRuleForConsumerLambda will trigger the Lambda Consumer every minute.
- A Lambda Permission named PermissionToInvokeConsumerLambda allows the Event Rule to invoke the Lambda Consumer.
- An IAM Role called ConsumerLambdaRole contains an IAM policy allowing the Lambda Consumer function to read messages from SQS and invoke the Lambda Worker.
- A Lambda function named ConsumerLambda creates and configures the Lambda Consumer.
The previous snippet can be used to setup the resources needed with the help of CloudFormation. What’s still missing is the source code for the Lambda Consumer executed on AWS Lambda. The Lambda function needs to:
- Read as many tasks as possible from the task queue.
- Invoke a Lambda Worker for each task.
Only one more part missing: the Worker Lambda.
The following snippet contains all resources needed to setup the Worker Lambda with the help of CloudFormation:
- An IAM Role called WorkerLambdaRole contains an IAM policy allowing the Lambda Worker function to delete messages from SQS.
- A Lambda function named WorkerLambda creates and configures the Lambda Worker.
Of course, you need to implement your own source code for the Lambda Worker. You can do whatever can be done within 300 seconds, the current execution timeout for Lambda functions. The Lambda Worker needs to delete the message from the SQS queue after processing the message successfully.
Asynchronous decoupling can be achieved with the help of SQS. It is possible to integrate the managed queuing system with Lambda. This allows you to build a serverless microservice consuming tasks from a queue, e.g.: sending out massive amounts of emails, transcoding video files after upload, or analyzing user behavior.
Published on and updated on