Serverless Big Data pipeline on AWS

Andreas Wittig – 14 Jul 2016

Lambda is a powerful tool when integrating different services on AWS. During the last months, I’ve successfully used serverless architectures to build Big Data pipelines. And I’d like to share my learnings with you.

The benefits of serverless pipeline are:

  • No need to manage a fleet of EC2 instances.
  • Highly scalable.
  • Paid per execution.

A Big Data pipeline is moving data between data sources and data targets. Often called an ETL process (extract, transform, load) as well. The following figure describes a typical serverless Big Data pipeline:

Big Data Pipeline with AWS Lambda

Use Cases

Using Lambda to implement your Big Data pipeline is especially useful if you need to transform or filter data during moving from data source to data target.

Typical use cases:

  • Load CloudFront and ELB logs from S3, transform and filter data, insert into Elasticsearch cluster.
  • Load business reports from S3, transform and filter data, insert into Redshift.
  • Load event data from Kinesis stream, transform and filter data, store on S3 for further processing.

Other use cases are possible as well. Changed data (S3 and DynamoDB), external events, or a schedule (CloudWatch Event Rule) are able to trigger a Lambda function. A Lambda can access data sources and targets connected to the Internet or VPC.

Andreas and Michael Wittig

Hej, Andreas & Michael here!

We launched the cloudonaut blog in 2015. Since then, we have published 325 articles: small tips and tricks, best practices, and service reviews. We enjoy writing about all things AWS a lot.

Do you like our blog posts and podcast episodes? Have you learned something new? Consider supporting us create in-depth and independent AWS content. Please help us with a monthly or one-time payment through GitHub Sponsors.

Start supporting us today!

Seems like there almost no limits?


Lambda is a powerful tool, but compared to an EC2 instance there are limitations as well. Limitations when building a serverless Big Data pipeline:

  • Maximum execution duration: 300 seconds
  • Maximum memory: 1536 MB
  • Ephemeral disk capacity: 512 MB

Real world example:

  1. Load CSV file from S3.
  2. Unzip data.
  3. Transform data.
  4. Zip data.
  5. Upload to S3.

About 800 MB of unzipped data. Implementing a Lambda function following the asynchronous model of Node.js is not possible as there is neither enough memory nor disk capacity to hold the unzipped data as well as the transformed data at once.

Solution: Data Streaming

Using streaming instead of linear execution allows you to extract, transform, and load data in chunks from the beginning to the end of the pipeline.

The following source code contains an example implementing a stream for the described scenario in Node.js:

  1. Load csv.tgz file from S3.
  2. Unzip data.
  3. Split at the end of the line.
  4. Transform data.
  5. Zip data.
  6. Upload file to S3.
var AWS = require("aws-sdk");
var zlib = require("zlib");
var split = require("split");
var transform = require("stream-transform");

var sourceBucket = "BUCKET_NAME";
var sourceKey = "KEY";
var targetBucket = "BUCKET_NAME";
var targetKey = "KEY";

var s3 = new AWS.S3();

var transformer = transform(function(record, callback) {
// TODO transform
callback(null, record);

var pipeline = s3.getObject({ // (1)
Bucket: sourceBucket,
Key: sourceKey
.pipe(zlib.createGunzip()) // (2)
.pipe(split()) // (3)
.pipe(transformer) // (4)
.pipe(zlib.createGzip()); // (5)

// (6)
s3.upload({"Bucket": targetBucket, "Key": targetKey, "Body": pipeline}, function(err) {
if (err) {

This approach allows you to process data without hitting the memory or disk space limitations. Of course, the maximum execution duration of 300 seconds is still limiting the maximum throughput of your serverless data pipeline. If you are hitting the limit, you need to split your data into smaller chunks.

Andreas Wittig

Andreas Wittig

I'm an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I'm writing this blog and all other projects together with my brother Michael.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

Feedback? Questions? Drop me a line: Email, Twitter, LinkedIn.

Briefcase icon
Hire me