This post will take you through some steps to setup an AWS SES (Amazon Web Services Simple Email Service) dashboard and daily reporting tool. If you are moving any production systems over to SES you will need a method of keeping track of email bounces. If you do not stay on top of it Amazon will suspend your account.
Overview
Essentially what we will be doing is:
1. An SQS queue is subscribed to an SNS topic
2. When emails are bounced the SNS subscription forwards the bounce to the SQS
3. CloudWatch checks the queue according to a schedule
4. When the CloudWatch schedule is invoked it runs a Lambda function
5. The Lambda function collects the details from the queue, clears the queue and then generates and HTML report.
6. The HTML report is uploaded to an S3 bucket.
7. The Lambda function copies the link from the html file in the bucket to a new email body and send it to the recipient in the functions environment variables list.
SNS configuration
Open SNS Create a new topic giving it a name.
SQS Configuration
Open SQS> Click Create New Queue> Choose Standard Queue> Click Configure Queue
Change Default Visibility Timeout to 5 minutes leave all other options as defaults. Click create queue
Select the queue you just created and on the Queue Actions menu select Subscribe Queue to SNS Topic.
Select the topic you just created and click Subscribe
You should get a successful result
S3 Bucket configuration
Open the S3 console> Click Create Bucket> Give the bucket a name> Select the region and click Create
Allow ACL to create public objects by clicking on the bucket> go to permissions> public access settings> click edit
Uncheck all the boxes:
Click Bucket policy
Specify the policy as below to restrict to your company’s IP address:
{ "Id": "Policy", "Version": "2012-10-17", "Statement": [ { "Sid": "SID1", "Action": [ "s3:GetObject" ], "Effect": "Deny", "Resource": "arn:aws:s3:::ses-bounce/*", "Condition": { "NotIpAddress": { "aws:SourceIp": "86.21.58.16" } }, "Principal": "*" } ] }
IAM Policy configuration
Open IAM> Policies> Create Policy
On the JSON tab paste in the below code, entering the S3 Bucket name and the SQS ARN.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSendEmail", "Effect": "Allow", "Action": [ "ses:SendEmail" ], "Resource": [ "*" ] }, { "Sid": "s3allow", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:getObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::ses-bounce/*" ] }, { "Sid": "AllowQueuePermissions", "Effect": "Allow", "Action": [ "sqs:ChangeMessageVisibility", "sqs:ChangeMessageVisibilityBatch", "sqs:DeleteMessage", "sqs:DeleteMessageBatch", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ReceiveMessage" ], "Resource": [ "arn:aws:sqs:eu-west-1:123456789123:ses-bounce-queue" ] }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] } ] }
Click Review Policy
Give the policy a name and click create policy
Role configuration
Click roles> Create role> Select Lambada> Click Next Permissions
Select the policy you just created click Next and Next: Review
Create a name for the role and click create role
Apply the function to an SES object
Open the SES console> under identity management click email addresses. Select the email address that you want to receive bounce notifications from and click View Details.
Select the SNS topic we created in step one for Bounces and Complaints and click Save Config.
Open the SQS console> Right click the queue you create> Click Purge queue (clears confirmation emails that cannot be parsed by Lambada)
Creating the Lambda function
Download the sesreport file from here
I actually ended up using this in the end instead of the above file, another AWS forum user made some nice modifications.
Open the Lambda console> Click functions> Create function> Author from scratch
Give the function a name> choose Node.js 6.10 for Runtime> Select the existing role you created in IAM
Click Create Function
Under Function code select Upload a .ZIP file> Choose Upload and Upload sesreport.zip
Key | Value | Sample value |
---|---|---|
QueueURL | The URL of the Amazon SQS queue you created in the part where you created a Queue in Amazon Simple Queue Service. | https://sqs.us-east-1.amazonaws.com/999623213###/sample-queue-name |
Region | The AWS Region in which you created the Amazon SNS topic in the part where you created a Topic in Amazon Simple Notification Service. | us-east-1 |
ToAddr | The email address that will receive the bounce and complaint report. | robert@example.com |
SrcAddr | The email address that will send the bounce and complaint report. | reports@example.com |
BucketName | The name of the Amazon S3 bucket you created earlier | sample-s3-bucket |
BucketPrefix | [Optional] If you want to save the dashboards in a folder in the Amazon S3 bucket, specify the path here. The path you specify must end with a forward slash (/). | SES/reports/ |
Under basic settings for Memory choose 512MB> For timeout select 5 min
Click test> choose an event name
Click test
Then open the SES console> select a verified email address> click send a test email> in the To field type bounce@simulator.amazonses.com >click Send test email
Do the same again but use complaint@simulator.amazonses.com in the To field
Open the SQS console you should see 2 messages available.
Open the Lambda console> choose functions> choose the function you created> click Test
Cloudwatch configuration
Open the Cloudwatch console> Select Events and Choose Rules
Choose create rule> Under Event Source choose Schedule
Under Targets choose add target
Set for a Fixed rate of 1 Day
For function choose the Lambada function created earlier
Click Configure Details
Give the rule a name and click create Rule
When I test the lambda function, I always get the same message: “Process exited before completing request”
Please advise
Wow, thanks for putting this together. There’s no way I would’ve been able to get it together based on AWS sparse documentation. Why is it not there by default?
Hi,
I am getting error,
START RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52 Version: 1
Unable to import module ‘index’: Error
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
END RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52
REPORT RequestId: b9bea36f-d2d4-458a-80f5-e673983acb52 Duration: 3.36 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 58 MB Init Duration: 1.20 ms
Do you have any tips for multiple domains?
Hello, Thanks for this post. can you please help me resolve this error when I run the ‘test’ on Lamda
=================
START RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c Version: $LATEST
2020-06-17T08:33:26.059Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Reading from: https://sqs.ap-south-2.amazonaws.com/myaccountid/Stage-SQS-forSES
2020-06-17T08:33:26.184Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Possible issue with SQS permissions or QueueURL wrong
2020-06-17T08:33:26.185Z 9b493548-09ae-4e07-a536-dedba3e1117c INFO Reading queue, size = null
2020-06-17T08:33:26.185Z 9b493548-09ae-4e07-a536-dedba3e1117c ERROR Uncaught Exception {“errorType”:”TypeError”,”errorMessage”:”Cannot read property ‘Attributes’ of null”,”code”:”TypeError”,”message”:”Cannot read property ‘Attributes’ of null”,”time”:”2020-06-17T08:33:26.185Z”,”stack”:[“TypeError: Cannot read property ‘Attributes’ of null”,” at Response.sqs.getQueueAttributes (/var/task/index.js:84:26)”,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:364:18)”,” at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)”,” at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)”,” at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)”,” at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)”,” at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)”,” at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10″,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)”,” at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)”]}
END RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c
REPORT RequestId: 9b493548-09ae-4e07-a536-dedba3e1117c Duration: 182.71 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 32 MB
Unknown application error occurred
===============================
the dashboard isnt generating, Just show me an empty queue.