🆕 Video Course Out Now: Rapid Docker on AWS

Introducing the Object Store: S3

Michael Wittig – 28 Mar 2016

Back in the old days, data was managed as files in a hierarchy consisting of folders and files. The file was the representation of the data. In an object store, data is stored as objects. Each object consists of a globally unique identifier, some metadata, and the data itself, as figure 1 illustrates. An object’s globally unique identifier is also known as its key; addressing the object from different devices and machines in a distributed system is possible with the globally unique identifier.

Figure1: Objects stored in object store have three parts: a unique ID, metadata describing the content, and the content itself (such as an image).

The separation of metadata and data allows clients to work only with the metadata for managing and querying data. You only have to load the data if you really need it. Metadata is also used to store access-control information and for other management tasks.

Amazon S3

The Amazon S3 object store is one of the oldest services on AWS. Amazon S3 is an acronym for Amazon Simple Storage Service. It’s a typical web service that lets you store and retrieve data in an object store via an API reachable over HTTPS.

The service offers unlimited storage space and stores your data in a highly available and durable way. You can store any kind of data, such as images, documents, and binaries, as long as the size of a single object doesn’t exceed 5 TB. You have to pay for every GB you store in S3, and you also incur minor costs for every request and transferred data. As figure 2 shows, you can access S3 via HTTPS using via the Management Console, the command-line interface (CLI), SDKs, and third-party tools, to upload and download objects.

Cover of Amazon Web Services in Action

Level up, strengthen your AWS skills.

Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!

Figure 2: Uploading and downloading an object to S3 via HTTPS

S3 uses buckets to group objects. A bucket is a container for objects with a globally unique name. By unique we really mean unique—you have to choose a bucket name that isn’t used by any other AWS customer in any other region, so we advise you to prefix the buckets with your domain name (such as com.mydomain.*) or your company name. Figure 3 shows the concept.

Figure 3: S3 uses buckets with a globally unique name to group objects.

Typical use cases are as follows:

  • Backing up and restoring files with S3 and the help of the AWS CLI
  • Archiving objects with Amazon Glacier to save money compared to Amazon S3
  • Integrating Amazon S3 into applications with the help of the AWS SDKs to store and fetch objects such as images
  • Hosting static web content that can be viewed by anyone with the help of S3
  • Building data pipelines with S3 Event Notifications and Lambda
Michael Wittig

Michael Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Andreas.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me