Using DynamoDB Entity Store for cleaner TypeScript code

DynamoDB is a cloud-hosted NoSQL database from Amazon Web Services (AWS). DynamoDB is popular for two main reasons:

• It scales extremely effectively with little operational effort
• Since it is a serverless service it is also cheap, simple, and quick to run for lower throughput applications

I’ve worked with companies where the scaling behavior has been crucial, but most of the time what I like about DynamoDB is the second point: no complicated VPC networking, no clusters, no minimum monthly costs.

However there are also often at least two concerns with using DynamoDB:

• Getting DynamoDB table design correct is hard since it is very different from working with relational databases
• There are no “standard” libraries that provide a high level programming interface

There’s no getting around the first of these points - although Alex DeBrie is doing his best to educate the industry.

As for the second point, AWS does a great job at providing solid cloud services, with extensive APIs. However these APIs are fairly low-level. Even the AWS language-specific Software Development Kits - like the AWS JavaScript SDK - only do a small amount of work to abstract the underlying HTTP API semantics for an application developer.

This is where my new library - DynamoDB Entity Store - comes in. It provides a more developer-friendly interface to DynamoDB for TypeScript and JavaScript developers.

This article gives a brief introduction to DynamoDB Entity Store to show why you may want to consider it for your own projects. First though I show what life is like when you just use the AWS libraries.

Using the standard AWS SDK for DynamoDB

Let’s say you’re writing an application using TypeScript, and you’re using the AWS SDK V3 for your interactions with DynamoDB. And for the sake of this example let’s say you’re modeling animals on a farm.

Your business logic may have a domain type that looks like this:

interface Sheep {
  breed: string
  name: string
  ageInYears: number
}

And you may instantiate a particular object of this type like this:

const shaun: Sheep = {
  breed: 'merino',
  name: 'shaun',
  ageInYears: 3
}

Now you want to save this item to DynamoDB. If you’ve read Alex’s book I mentioned earlier you’ll know that you may want to use key attributes that are derived from the fields of the item you want to save, rather than being specific fields of that object. In other words to save shaun to DynamoDB using SDK V3 your code might look like this:

await documentClient.send(
  new PutCommand({
    TableName: 'AnimalsTable',
    Item: {
      PK: `SHEEP#BREED#${shaun.breed}`,
      SK: `NAME#${shaun.name}`,
      ...shaun
    }
  })
)

documentClient here is the interface to DynamoDB that’s been pre-configured for credentials, etc. AnimalsTable is the actual name in DynamoDB of the table you’re writing to. PK and SK are the Partition Key and Sort Key attributes of your table, which together form the unique key of each item.

Some time later you want to read shaun back from the database. To do that your code might look like this:

const shaunKey = {
  breed: 'merino',
  name: 'shaun'
}


const response = await documentClient.send(
  new GetCommand({
    TableName: 'AnimalsTable',
    Key: {
      PK: `SHEEP#BREED#${shaunKey.breed}`,
      SK: `NAME#${shaunKey.name}`,
    }
  })
)


const shaun: Sheep | undefined =
  response.Item?.breed !== undefined &&
  response.Item?.name !== undefined &&
  response.Item?.ageInYears !== undefined
    ? {
        breed: response.Item.breed,
        name: response.Item.name,
        ageInYears: response.Item.ageInYears
      }
    : undefined

The problems with using the standard AWS SDK

The code above will work, but it has some aspects that aren’t great:

• Operational concerns - like the actual table’s name and key attribute names - are mixed up with domain concerns - like the fields of your Sheep type
• The code for generating key attribute values from domain values is repeated, and is not clearly delineated
• Messy code to perform validation, and to provide a correctly typed object for use by subsequent logic

… and that’s just for a simple put and get operation. As soon as you start dealing with things like collection responses (from queries & scans), condition / update expressions, Time-to-Live fields, and more, then the mixing of DynamoDB concerns, domain logic concerns, and typing concerns, can make for spaghetti code.

Most projects of any size will end up dealing with this by introducing their own “helper code” to clean some of these issues up. That’s what I’ve done over the last few years on a few projects. But wouldn’t it be nice to be able to use a library that’s already written this helper code for you?

How DynamoDB Entity Store helps

There are at least a couple of other third party open source libraries that provide a higher abstraction for DynamoDB - OneTable, and DynamoDB Toolbox. When I was looking for something a couple of years ago neither of these called out to me, and so I wrote my own helper code.

I’ve evolved this code since then, and have wrapped it all up as a new library named DynamoDB Entity Store. With this library my domain code for working with sheep looks as follows.

Writing my sheep record looks like this:

await store.for(SHEEP_ENTITY).put(shaun)

And reading it looks like this:

const shaun: Sheep | undefined = await store
  .for(SHEEP_ENTITY)
  .getOrUndefined({ breed: 'merino', name: 'shaun'})

Isn’t that cleaner? Obviously there are a few things going on here.

First up, store is a wrapper around the SDK v3 Document Client, but it also includes operational configuration, like the table name and key attribute names, so that you don’t have to mess up domain code with those concerns.

Second - SHEEP_ENTITY is a specific object that defines the relationship between a domain object, and its persisted representation in DynamoDB. Keeping this code in a separate object means that you can perform several different types of operation on one Entity without having to repeat things like “how to calculate key attribute values”.

Your Entity code for SHEEP_ENTITY looks like this:

const SHEEP_ENTITY: Entity<Sheep, Pick<Sheep, 'breed'>, Pick<Sheep, 'name'>> = {
  type: 'sheep',
  parse: typePredicateParser(isSheep, 'sheep'),
  pk({ breed }: Pick<Sheep, 'breed'>) {
    return `SHEEP#BREED#${breed}`
  },
  sk({ name }: Pick<Sheep, 'name'>) {
    return `NAME#${name}`
  }
}


// TypeScript Type Predicate
function isSheep(x: DynamoDBValues): x is Sheep {
  const candidate = x as Sheep
  return candidate.breed !== undefined 
    && candidate.name !== undefined && candidate.ageInYears !== undefined
}

There’s some verbosity here for all of the type-related code, but at least you can get all of that cruft in one place.

This example so far has just shown a simple put and get, but DynamoDB Entity Store helps with pretty much all the operations you can perform with DynamoDB - updates, queries, scans, batch commands, transactions, and more. Here’s an example of a query, using the same SHEEP_ENTITY object:

const allMerinoSheep = await store
  .for(SHEEP_ENTITY)
  .queryAllByPk({ breed: 'merino' })

This code uses the same table configuration, key generation, validation, and parsing code as the put and get operations, allowing the specific elements of the query request to be kept clean.

While DynamoDB Entity Store defaults to a “single table design”, you can also use multiple tables, and if you’re using domain-object fields for key attributes - rather than generated attributes - then that’s supported too.

To see more examples checkout the project’s README, or the documentation.

I’ve written DynamoDB Entity Store because it fits my style of coding, hopefully it might work for some of you too. If it does, or doesn’t (!), then please let me know at mike@symphonia.io, or in the GitHub project issues.