Setup an AWS SES Bounce dashboard monitor with daily email reports

This post will take you through some steps to setup an AWS SES (Amazon Web Services Simple Email Service) dashboard and daily reporting tool.  If you are moving any production systems over to SES you will need a method of keeping track of email bounces.  If you do not stay on top of it Amazon will suspend your account.

Overview

Essentially what we will be doing is:

1. An SQS queue is subscribed to an SNS topic
2. When emails are bounced the SNS subscription forwards the bounce to the SQS
3. CloudWatch checks the queue according to a schedule
4. When the CloudWatch schedule is invoked it runs a Lambda function
5. The Lambda function collects the details from the queue, clears the queue and then generates and HTML report.
6. The HTML report is uploaded to an S3 bucket.
7. The Lambda function copies the link from the html file in the bucket to a new email body and send it to the recipient in the functions environment variables list.

Setup an AWS SES Bounce dashboard monitor with daily email reports

SNS configuration

Open SNS Create a new topic giving it a name.

Setup an AWS SES Bounce dashboard monitor with daily email reports-1

SQS Configuration

Open SQS> Click Create New Queue> Choose Standard Queue> Click Configure Queue

Change Default Visibility Timeout to 5 minutes leave all other options as defaults.  Click create queue

Setup an AWS SES Bounce dashboard monitor with daily email reports-2

Setup an AWS SES Bounce dashboard monitor with daily email reports-3




Select the queue you just created and on the Queue Actions menu select Subscribe Queue to SNS Topic.

Setup an AWS SES Bounce dashboard monitor with daily email reports-4

Select the topic you just created and click Subscribe

Setup an AWS SES Bounce dashboard monitor with daily email reports-5

You should get a successful result

Setup an AWS SES Bounce dashboard monitor with daily email reports-6

S3 Bucket configuration

Open the S3 console> Click Create Bucket> Give the bucket a name> Select the region and click Create

 

Setup an AWS SES Bounce dashboard monitor with daily email reports-7

Allow ACL to create public objects by clicking on the bucket> go to permissions> public access settings> click edit

Uncheck all the boxes:

Setup an AWS SES Bounce dashboard monitor with daily email reports-8

Click Bucket policy
Specify the policy as below to restrict to your company’s IP address:

{
    "Id": "Policy",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SID1",
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Deny",
            "Resource": "arn:aws:s3:::ses-bounce/*",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "86.21.58.16"
                }
            },
            "Principal": "*"
        }
    ]
}

 

IAM Policy configuration

Open IAM> Policies> Create Policy

On the JSON tab paste in the below code, entering the S3 Bucket name and the SQS ARN.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSendEmail",
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "s3allow",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:getObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::ses-bounce/*"
            ]
        },
        {
            "Sid": "AllowQueuePermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:ChangeMessageVisibilityBatch",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage"
            ],
            "Resource": [
                "arn:aws:sqs:eu-west-1:123456789123:ses-bounce-queue"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

 

Click Review Policy

Setup an AWS SES Bounce dashboard monitor with daily email reports-9




Give the policy a name and click create policy

Setup an AWS SES Bounce dashboard monitor with daily email reports-10

Role configuration

Click roles> Create role> Select Lambada> Click Next Permissions

Setup an AWS SES Bounce dashboard monitor with daily email reports-11

Select the policy you just created click Next and Next: Review

Setup an AWS SES Bounce dashboard monitor with daily email reports-12

Setup an AWS SES Bounce dashboard monitor with daily email reports-13

Create a name for the role and click create role

Setup an AWS SES Bounce dashboard monitor with daily email reports-14




Apply the function to an SES object

Open the SES console> under identity management click email addresses. Select the email address that you want to receive bounce notifications from and click View Details.

Select the SNS topic we created in step one for Bounces and Complaints and click Save Config.

Setup an AWS SES Bounce dashboard monitor with daily email reports-115

Open the SQS console> Right click the queue you create> Click Purge queue (clears confirmation emails that cannot be parsed by Lambada)

Creating the Lambda function

Download the sesreport file from here

I actually ended up using this in the end instead of the above file, another AWS forum user made some nice modifications.

Open the Lambda console> Click functions> Create function> Author from scratch

Give the function a name> choose Node.js 6.10 for Runtime> Select the existing role you created in IAM

Setup an AWS SES Bounce dashboard monitor with daily email reports-16

Click Create Function

Under Function code select Upload a .ZIP file> Choose Upload and Upload sesreport.zip

KeyValueSample value
QueueURLThe URL of the Amazon SQS queue you created in the part where you created a Queue in Amazon Simple Queue Service.https://sqs.us-east-1.amazonaws.com/999623213###/sample-queue-name
RegionThe AWS Region in which you created the Amazon SNS topic in the part where you created a Topic in Amazon Simple Notification Service.us-east-1
ToAddrThe email address that will receive the bounce and complaint report.robert@example.com
SrcAddrThe email address that will send the bounce and complaint report.reports@example.com
BucketNameThe name of the Amazon S3 bucket you created earlier
sample-s3-bucket
BucketPrefix[Optional] If you want to save the dashboards in a folder in the Amazon S3 bucket, specify the path here. The path you specify must end with a forward slash (/).SES/reports/
Setup an AWS SES Bounce dashboard monitor with daily email reports-17

Under basic settings for Memory choose 512MB> For timeout select 5 min

Click test> choose an event name

Setup an AWS SES Bounce dashboard monitor with daily email reports-18

Click test

Then open the SES console> select a verified email address> click send a test email> in the To field type bounce@simulator.amazonses.com >click Send test email

Setup an AWS SES Bounce dashboard monitor with daily email reports-19




Do the same again but use complaint@simulator.amazonses.com in the To field

Open the SQS console you should see 2 messages available.

Open the Lambda console> choose functions> choose the function you created> click Test

Cloudwatch configuration

Open the Cloudwatch console> Select Events and Choose Rules

Choose create rule> Under Event Source choose Schedule

Under Targets choose add target

Set for a Fixed rate of 1 Day

For function choose the Lambada function created earlier

Setup an AWS SES Bounce dashboard monitor with daily email reports-20

Click Configure Details

Give the rule a name and click create Rule

Setup an AWS SES Bounce dashboard monitor with daily email reports-21

6 Replies to “Setup an AWS SES Bounce dashboard monitor with daily email reports”

  1. When I test the lambda function, I always get the same message: “Process exited before completing request”

    Please advise

    1. Wow, thanks for putting this together. There’s no way I would’ve been able to get it together based on AWS sparse documentation. Why is it not there by default?

  2. Hi,
    I am getting error,

    START RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52 Version: 1
    Unable to import module ‘index’: Error
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    END RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52
    REPORT RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52 Duration: 3.36 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 58 MB Init Duration: 1.20 ms

  3. Hello, Thanks for this post. can you please help me resolve this error when I run the ‘test’ on Lamda
    =================
    START RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c Version: $LATEST
    2020-06-17T08:33:26.059Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Reading from: https://sqs.ap-south-2.amazonaws.com/myaccountid/Stage-SQS-forSES
    2020-06-17T08:33:26.184Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Possible issue with SQS permissions or QueueURL wrong
    2020-06-17T08:33:26.185Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Reading queue, size = null
    2020-06-17T08:33:26.185Z 9b493548-09ae-4e07-a536-dedba3e1117c ERROR Uncaught Exception {“errorType”:”TypeError”,”errorMessage”:”Cannot read property ‘Attributes’ of null”,”code”:”TypeError”,”message”:”Cannot read property ‘Attributes’ of null”,”time”:”2020-06-17T08:33:26.185Z”,”stack”:[“TypeError: Cannot read property ‘Attributes’ of null”,” at Response.sqs.getQueueAttributes (/var/task/index.js:84:26)”,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:364:18)”,” at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)”,” at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)”,” at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)”,” at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)”,” at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)”,” at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10″,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)”,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)”]}
    END RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c
    REPORT RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c Duration: 182.71 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 32 MB
    Unknown application error occurred
    ===============================

Leave a Reply

Your email address will not be published. Required fields are marked *