AWS Logs to OpenSearch via Kinesis

Introduction

By default, the AWS Simple Email Service (SES) service does not offer us to be able to view logs of your actions / events, we can only see some metrics in Cloudwatch. In Geko we have encountered more than once the need to see SES logs, either to diagnose incidents or to better understand the status of the service. Well, to make this possible, we use AWS Kinesis. We are going to explain how we make it. Let’s begin!

Kinesis configuration

Kinesis is a data collector, processor and transmitter. The first step is to create a Delivery Stream. We access the Kinesis service, Delivery Streams and create a Delivery Stream

In “Source” we choose “Direct PUT” and in Destination “Amazon OpenSearch Service”. There are other destination options such as Redshift, S3, Dynatrace .. All available options appear in the drop-down list.

We add a name to the object “Delivery Stream” and add our OpenSearch. In our case, as we already have an OpenSearch created and operational by simply accessing “Browse”, it has appeared to select it.

We add the index name that we want it to create.

And by default it selects VPC, Subnet and Security Group based on our OpenSearch.

Finally we create the object “Delivery Stream”

SES Configuration

Now we access the SES service, we access “Configuration Sets” and “Create Set”

We add the Configuration Set name and create it.

Once the “Configuration Set” is created we go to “Event Destionation” and “Add Destination” to create one.

We select the type of event that we want and go to the next step

We select “Amazon Kinesis Data Firehose”, we put a name and we select the “Delivery Stream” that we have created previously.

 

Watch out! Now you have to create an “IAM Role” so that SES can write to Firehose, we have created it like this:

Con esta Policy

And this Trust Relationship

Finally we create the “Configuration Set”

 

Now, to test it, you can send a test email from SES assigning to our test the “Configuration Set” that we have created. This “Configuration Set” can be applied to any Identity that we have in SES. In fact we are going to do a test by sending a test email. We see the test log that we have generated in our OpenSearch:

And in the Kinesis Delivery monitoring:

 

From Geko we hope that if you have come this far, this entry is just what you were looking for! We also invite you to read other of our labs posts

Leave a Reply

Your email address will not be published. Required fields are marked *