cloudonaut
Home Blog Podcast Books & Video Courses Support us
Subscribe

End-user monitoring of your website with CloudWatch Synthetics

Michael Wittig – 04 Jun 2020

There are countless reasons why your website is not working as your users expect. From a technical point of view, you can monitor your load balancers, your web servers, and your database. But what if that external script that you embed is breaking your site? Expired TLS certificate? Something wrong with DNS? How can you test that your website works for real users?

End-user perspective

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

Browsers can be used in an automated way controlled by a script. Wait for an element to become visible. Click on a link. Enter a form field. puppeteer allows you to remote control a headless Chrome browser in Node.js and is maintained by Google. We can monitor the user experience on our website if we can find a way to run a puppeteer script at regular intervals and record the results.

And that’s where Amazon CloudWatch Synthetics enters the stage. It allows you to create canaries to execute puppeteer scripts on a schedule. Each run creates detailed logs, screenshots, and a record of all network calls in HAR format uploaded to S3. All you need to do is provide a script that performs the test, or use one of the blueprints. The following Node.js script opens https://marbot.io, waits for an <h1> HTML element, checks the title and status code, and performs a screenshot.

const synthetics = require('Synthetics'); // CloudWatch Synthetics lib
exports.handler = async () => {
  const page = await synthetics.getPage();
  const response = await page.goto('https://marbot.io', {
  	waitUntil: 'domcontentloaded',
  	timeout: 30000
  });
  try {
    await page.waitFor('h1', {timeout: 15000}); // <h1> element expected
    const title = await page.title();
    if (!title.includes('marbot')) { // title must contain marbot
    	throw new Error('title not as expected');
    }
    if (response.status() !== 200) { // 200 status code expected
      throw(new Error('Failed to load page!'));
    }
  } finally {
    await synthetics.takeScreenshot('loaded', 'result'); // always create a screenshot
  }
};

The rest is taken care of by CloudWatch Synthetics and is presented like this:

CloudWatch Synthetics canary run

Looking for a new challenge?

Post a job ad
  • tecRacer

    Cloud Consultant

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    AWS only Infrastructure as Code EC2 Containers Serverless
  • tecRacer

    Cloud Migration Specialist

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    Lift&Shift Transformation EC2 RDS VPC

Want to get notified about failed canary runs? Create a CloudWatch Alarm that watches the metrics of the canary to alert you if things go wrong. I created a CloudFormation template to help you with the setup (don’t forget to resolve the TODOs in the template!).

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudWatch Synthetics website monitoring'
Resources:
  Topic:
    Type: 'AWS::SNS::Topic'
    Properties:
      Subscription:
      - Endpoint: 'mail@site.com'
        Protocol: email
  TopicPolicy:
    Type: 'AWS::SNS::TopicPolicy'
    Properties:
      PolicyDocument:
        Id: Id1
        Version: '2012-10-17'
        Statement:
        - Sid: Sid1
          Effect: Allow
          Principal:
            AWS: '*' # Allow CloudWatch Alarms
          Action: 'sns:Publish'
          Resource: !Ref Topic
          Condition:
            StringEquals:
              'AWS:SourceOwner': !Ref 'AWS::AccountId'
      Topics:
      - !Ref Topic
  CanaryBucket:
    Type: 'AWS::S3::Bucket'
    Properties: {}
  CanaryRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: 'lambda.amazonaws.com'
          Action: 'sts:AssumeRole'
      Policies:
      - PolicyName: execution
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action: 's3:ListAllMyBuckets'
            Resource: '*'
          - Effect: Allow
            Action: 's3:PutObject'
            Resource: !Sub '${CanaryBucket.Arn}/*'
          - Effect: Allow
            Action: 's3:GetBucketLocation'
            Resource: !GetAtt 'CanaryBucket.Arn'
          - Effect: Allow
            Action: 'cloudwatch:PutMetricData'
            Resource: '*'
            Condition:
              StringEquals:
                'cloudwatch:namespace': CloudWatchSynthetics
  CanaryLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/lambda/cwsyn-${Canary}-${Canary.Id}'
      RetentionInDays: 14
  CanaryPolicy:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyDocument:
        Statement:
        - Effect: Allow
          Action:
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: !GetAtt 'CanaryLogGroup.Arn'
      PolicyName: logs
      Roles:
      - !Ref CanaryRole
  Canary:
    Type: 'AWS::Synthetics::Canary'
    Properties:
      ArtifactS3Location: !Sub 's3://${CanaryBucket}'
      Code:
        Handler: 'index.handler'
        Script: |
          const synthetics = require('Synthetics');
          const log = require('SyntheticsLogger');
          exports.handler = async () => {
            const page = await synthetics.getPage();
            const response = await page.goto('https://site.com', {waitUntil: 'domcontentloaded', timeout: 30000}); // TODO replace with your URL
            try {
              await page.waitFor('h1', {timeout: 15000}); // TODO replace with an HTML element to look for
              const title = await page.title();
              if (!title.includes('REPLACE')) { // TODO replace with your important word in the title
                throw new Error('title not as expected!');
              }
              if (response.status() !== 200) {
                throw(new Error('Failed to load page!'));
              }
            } finally {
              await synthetics.takeScreenshot('loaded', 'result');
            }
          };
      ExecutionRoleArn: !GetAtt 'CanaryRole.Arn'
      FailureRetentionPeriod: 30
      Name: 'site-monitoring' # TODO replace with better name
      RunConfig:
        TimeoutInSeconds: 60
      RuntimeVersion: 'syn-1.0'
      Schedule:
        DurationInSeconds: '0' # run forever
        Expression: 'rate(15 minutes)'
      StartCanaryAfterCreation: true
      SuccessRetentionPeriod: 30
  SuccessPercentAlarm:
    DependsOn: TopicPolicy
    Type: 'AWS::CloudWatch::Alarm'
    Properties:
      AlarmActions:
      - !Ref Topic
      AlarmDescription: 'Canary is failing.'
      ComparisonOperator: LessThanThreshold
      Dimensions:
      - Name: CanaryName
        Value: !Ref Canary
      EvaluationPeriods: 1
      MetricName: SuccessPercent
      Namespace: CloudWatchSynthetics
      OKActions:
      - !Ref Topic
      Period: 300
      Statistic: Minimum
      Threshold: 90 # TODO replace or confirm threshold
      TreatMissingData: notBreaching

Don’t forget to check out the CloudWatch pricing details.

I also integrated CloudWatch Synthetics into marbot. You can set up your external website from Slack and receive alerts in Slack like this:

  1. Send a message to marbot on a channel and ask him to monitor your website.
  2. Select your preferred way of interacting with AWS (Management Console, CLI).
  3. Set the monitoring goal to Synthetics website.
  4. Select your AWS region.
  5. Follow marbot to deploy a CloudFormation stack to set up CloudWatch Synthetics, CloudWatch Alarms, and much more.

CloudWatch Synthetics setup

That’s it. Your website is now monitored from an end-user perspective. If things go wrong, you will receive a message in Slack.

CloudWatch Synthetics Alarm in Slack

Are you interested in marbot? Configure AWS monitoring, receive alerts, solve incidents from Slack.

Written by Michael Wittig on

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 345 articles, 45 podcast episodes, and 37 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Ross Mohan, Ross Mohan, sam onaga, Shawn Tolidano, Thorsten Hoeger, Todd Valentine, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.

Further reading