AWS CloudTrail: Athena and CloudWatch Alerts
In our lab walkthrough series, we go through selected lab exercises on our INE Platform. Subscribe or sign up for a 7-day, risk-free trial with INE and access this lab and a robust library covering the latest in Cyber Security, Networking, Cloud, and Data Science!
Purpose: It is very important to have a service that allows you to observe all of the activities taking place in your AWS account. AWS CloudTrail is useful in this situation. It records the actions as CloudTrail events, which can be used to monitor your AWS account activities. In this article, we will be using Amazon Athena to process this data (events) and configure AWS CloudWatch alerts for CloudTrail.
Technical difficulty:
| Novice | Beginner | Competent | Proficient | Expert
What is AWS CloudTrail Service?
AWS CloudTrail is a service provided by AWS that enables operational and risk auditing, governance, and compliance for your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
CloudTrail is enabled on your AWS account when you create it. When activity occurs in your AWS account, it's recorded in a CloudTrail event. Go to Event history in the CloudTrail console to easily view recent events. You can use CloudTrail to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure. You can identify who or what took which action, what resources were acted upon, when the event occurred, and other details to help you analyze and respond to activity in your AWS account.
Let’s understand what are trails
A trail is a configuration that allows CloudTrail events to be delivered to an Amazon S3 bucket, CloudWatch Logs, or CloudWatch Events. You can use a trail to filter the CloudTrail events you want delivered, encrypt your CloudTrail event log files with an AWS KMS key, and configure Amazon SNS notifications for log file delivery.
What is Amazon Athena?
Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL. With a few actions in the AWS Management Console, you can point Athena at your data stored in Amazon S3 and begin using standard SQL to run ad-hoc queries and get results in seconds.
Athena is serverless, so there is no infrastructure to set up or manage, and you pay only for the queries you run. You can use Athena to generate reports or to explore data with business intelligence tools or SQL clients connected with a JDBC or an ODBC driver.
What is AWS CloudWatch?
Amazon CloudWatch continuously monitors your Amazon Web Services (AWS) resources and the applications you run on AWS. CloudWatch can be used to collect and track metrics, which are variables that can be measured for your resources and applications.
With CloudWatch, you can create alarms that watch metrics and send notifications or automatically make changes to the resources you are monitoring when a threshold is breached. You also get system-wide visibility into resource utilization, application performance, and operational health.
What are AWS CloudWatch Logs?
AWS CloudWatch Logs can be used to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, Route 53, and other sources. CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service. You can then easily view them, search them for specific error codes or patterns, filter them based on specific fields, or archive them securely for future analysis.
What is Amazon Simple Notification Service?
Amazon Simple Notification Service (Amazon SNS) is a managed service that provides message delivery from publishers to subscribers (also known as producers and consumers). Publishers communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. Clients can subscribe to the SNS topic and receive published messages using a supported endpoint type.
In this article, we will be using two different methods to send the notifications of the events generated by the cloud trail:
We will create CloudWatch logs metric filters to analyze log events that match the pattern, and we'll utilize the asterisk ("*") as a wildcard to match the text, so that it matches all created events. Then we'll set the alert threshold to 1 and the alarm condition to "more than or equal," which will generate an alarm for every matched event.
We will set up a lambda function triggered by Cloudwatch logs and parse the log using a python script and send a notification from the lambda function using the SNS service such that it will send a notification for all the events generated by CloudTrail.
Now that we have covered all the key terms for the lab, let's perform the lab itself.
Lab Scenario
We have set up the below scenario in our INE labs for our students to practice. The screenshots have been taken from our online lab environment.
Lab Link: AWS CloudTrail : Athena and CloudWatch Alerts
Objective
In this lab, you will process the data with Amazon Athena and configure AWS CloudWatch alerts for CloudTrail.
Solution
Step 1: Click the lab link button to get access credentials. Login to the AWS account with these credentials.
Step 2: Search for CloudTrail in the search bar and navigate to the CloudTrail dashboard.
Dashboard will list all the available trails.
“lab-account-managment-events” trail was created by the management account for the organization and hence cannot edit or delete this trail through this account
Step 3: Click on “Trails” from the navigation pane.
Step 4: Click on the “Create trail” button.
Step 5: Set trail name as “students-events” and choose “Create new S3 bucket” and use the default bucket name.
Disable Log file SSE-KMS encryption and Log file validation.
Step 6: Enable CloudWatch logs and set choose a new log group and IAM role. Set the role name as “CloudTrailRoleForCloudWatchLogs” and use the default group name.
CloudTrail sends only the events that match your trail settings. For example, if you configure your trail to log data events only, your trail sends data events only to your CloudWatch Logs log group. CloudTrail supports sending data, Insights, and management events to CloudWatch Logs.
Click on the “Next” button.
Step 7: Select Management events as well as Data events for event type.
Enable read and write operation API activity logs.
Select DynamoDB as data event type and set “Log all events” for log selector template.
Click on the “Next” button.
Review the trail configuration.
Click on the “Create trail” button.
Successfully created “student-events” trail. Click on “student-events”.
The details of the created trail will be available here.
Check out the CloudWatch log group name. The created trail will send events to your CloudWatch Logs log group, you can view the events in the CloudWatch console. CloudTrail typically delivers events to your log group within an average of about 15 minutes of an API call.
Step 8: Create or modify some resources to generate logs. Search for “DynamoDB” and navigate to the DynamoDB dashboard.
Step 9: Click on “Tables” from the navigation pane.
Click on the “Create table” button.
Step 10: Set table name as “Users” and partition key as “id”.
Click on the “Create table” button.
Successfully created the table named “Users”.
Step 11: Navigate back to CloudTrail dashboard and click on “Event history” from the navigation pane.
Step 12: Click on the “Create Athena table” button.
Set Athena to query these log files directly from Amazon S3 by specifying the location of log files.
Step 13: Choose the same S3 bucket which contains CloudTrail log files.
CloudTrail saves logs as JSON text files in compressed gzip format (*.json.gzip). The location of the log files depends on how you set up trails, the AWS Region or Regions in which you are logging, and other factors.
Click on the “Create table” button.
The table is created with a default name that includes the name of the Amazon S3 bucket. Navigate to the Athena dashboard in the new tab using the hyperlink
Step 14: Set the query result location. Click on “Query editor” from the navigation pane.
From the query editor, click on “Settings”.
Click on the “Manage” button.
Step 15: Choose a bucket for the query results.
Click on the “Save” button.
Successfully set the query result location.
Amazon Athena automatically stores query results and metadata information for each query that runs in a query result location that you can specify in Amazon S3. If necessary, you can access the files in this location to work with them. You can also download query result files directly from the Athena console.
Step 16: Copy and paste the query and click on “RUN” to get the ‘UpdateTable’ and ‘CreateTable’ event details.
Query:
SELECT
eventtime,
eventsource,
useridentity.arn,
sourceipaddress
FROM cloudtrail_logs_aws_cloudtrail_logs_664289593040_add4f277
WHERE eventname = 'UpdateTable'
OR eventname = 'CreateTable'
Successfully got the query result showing the events.
Step 17: Now list a few details from all the CloudTrail logs and create a new table from the result.
Query:
SELECT
eventtype,
eventtime,
useridentity.arn,
sourceipaddress
FROM cloudtrail_logs_aws_cloudtrail_logs_664289593040_add4f277
Successfully got the query result showing all the logs matching the query.
Step 18: Click on “Table from query” under the “Create” button.
Step 19: Set table name as “UserActions”.
Step 20: Select “Choose an existing database” for Database configuration.
Step 21: Append “/UserAction” to the S3 URI to create a new directory and set it as the input data location.
Choose CSV as the format.
Click on the “Create table” button.
It will generate a query similar to the following. Click on the “RUN” button.
Query:
CREATE TABLE "default"."UserActions" WITH (
format = 'TEXTFILE',
external_location = 's3://aws-athena-query-results-664289593040-us-east-1/UserActions'
) AS
SELECT eventtype,
eventtime,
useridentity.arn,
sourceipaddress
FROM cloudtrail_logs_aws_cloudtrail_logs_664289593040_add4f277
Successfully saved query results to S3 bucket.
Step 22: Navigate to the S3 bucket location.
Click on any object and download or open it to view the log details as CSV format.
Step 23: Search for SNS in the search bar and navigate to the SNS dashboard.
Step 24: Set topic name as “CloudTrailEventsAlertTopic”.
Choose type as “Standard”.
Click on the “Create topic” button.
Step 25: Now create a subscription for the created topic. Click on the “Create subscription” button.
Set protocol as “Email” and enter your email address in the endpoint field.
Click on the “Create subscription” button.
A subscription confirmation email will be available at the provided email address. Click on the “Confirm subscription” link to confirm the email address.
Successfully confirmed the subscription.
Step 26: Search for CloudWatch in the search bar and navigate to the CloudWatch dashboard.
Step 27: Click on “Log groups” from the navigation pane.
This will list the log group which is created while creating the CloudTrail trail. A log group is a group of log streams that share the same retention, monitoring, and access control settings.
Step 28: Select the log group and click on “Create metric filter” under “Actions”.
Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs. CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics that you can graph or set an alarm on.
Step 29: Copy and paste the following as a filter pattern.
Filter patterns make up the syntax that metric filters use to match terms in log events. Terms can be words, exact phrases, or numeric values. Here will use the asterisk ("*") as a wildcard to match text such that it will match all the generated events.
Pattern: { $.eventType = "*" }
Click on the “Next” button.
Step 30: Set filter name as “CloudTrailEventMetric”.
Step 31: Again set name and namespace as “CloudTrailEventMetric”.Set Metric value as 1.
When your metric filter finds a match in your log events, it increments your metric's count by your metric's value. If your metric filter doesn't find a match, CloudWatch reports the metric's default value. For example, your log group publishes two records every minute, the metric value is 1, and the default value is 0. If your metric filter finds matches in both log records within the first minute, the metric value for that minute is 2.
Click on the “Next” button.
Review the metric filter configuration.
Click on the “Create metric filter” button.
Step 32: Click on “All metrics” under the Metrics section from the navigation pane.
Search for “CloudTrailEventMetric” and select the metrics.
Navigate to “Graphed metrics” and select “CloudTrailEventMetric”.
Click on the bell icon to create an alarm.
CloudWatch Alarms feature allows you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure.
Step 33: Set metric name as “CloudTrailEventMetric” and statistic to “Average”. Set period as 1 minute.
Step 34: Set threshold type as “Static” and alarm condition as “Greater/Equal”. Set threshold value as 1.
A CloudWatch Alarm is always in one of three states: OK, ALARM, or INSUFFICIENT_DATA. When the metric is within the range that you have defined as acceptable, the Monitor is in the OK state. When it breaches a threshold it transitions to the ALARM state.
Step 35: Choose “In alarm” as an alarm state trigger. Set “Select an existing SNS topic” for SNS topic and select the created topic name.
Click on the “Next” button.
Set the alarm name as “CloudTrailEventsAlarm”.
Click on the “Next” button.
Review the alarm configuration and click on the “Create alarm” button.
Step 36: Create or modify some resources to generate events. Navigate back to the DynamoDB dashboard and delete the created table.
Confirm the action by typing “delete” in the box. Then, click on the “Delete table” button.
You will receive a notification generated by CloudWatch alarm. Deleting a table action triggered the alarm because of the threshold value.
Now we will configure the alerts through the second method using a lambda function.
Step 37: Search for Lambda in the search bar and navigate to Lambda dashboard.
Step 38: Click on the “Create function” button.
Step 39: Set the function name as “CloudTrailAlertTrigger” and runtime as “Python 3.8”.
Step 40: Choose the execution role as “Create a new role from AWS policy templates”. Set role name as “CloudTrailAlertTriggerRole”. Select “Amazon SNS publish policy” from policy templates.
Every Lambda function has an IAM role called an execution role. In this role, you can attach a policy that defines the permissions that your function needs to access other AWS services and resources. At a minimum, your function needs access to Amazon CloudWatch Logs for log streaming. Here we will add “Amazon SNS publish policy” for publishing notifications from the lambda function.
Click on “Create function”
Step 41: Copy and replace the python code in lambda_function.py file. Replace the SNS topic ARN with the created ARN and click on deploy.
This code will parse AWS logs and send SNS notifications for every log generated in CloudWatch logs group.
Code:
import json
import boto3
import gzip
import base64
import os
sns_client = boto3.client('sns')
def lambda_handler(event, context):
decoded_event = json.loads(gzip.decompress(base64.b64decode(event['awslogs']['data'])))
body = '''
LogGroup: {loggroup}
Logstream: {logstream}
Filter Match: {filtermatch}
'''.format(
loggroup=decoded_event['logGroup'],
logstream=decoded_event['logStream'],
filtermatch=decoded_event['logEvents'][0]['message'],
)
def send_message(body):
sns = sns_client.publish(
TopicArn = 'arn:aws:sns:us-east-1:809795150143:CloudTrailAlertsTopic',
Message = body,
)
send_message(body)
Click on the “Add trigger” button.
Step 42: Set “CloudWatch Logs” as trigger and select the log group created by CloudTrail. Set the filter name as “LambdaLogTrigger”.
Click on the “Add” button.
Step 43: Navigate back to the DynamoDB dashboard and create a table again to make a log entry.
Set table name as “Users” and partition key as “id” with the data type as “Number”.
Click on the “Create table” button.
Successfully created a table.
Step 44: Navigate to the inbox of the provided email. Check out the email with the same format provided in the lambda function.
This email is triggered by the lambda function when a log is added to the CloudWatch log group corresponding to the “CreateTable” event.
References:
AWS Athena and CloudTrail Logs (https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html)
Conclusion
In this article, we saw how one can use Amazon Athena to process the data (events) and configure AWS CloudWatch alerts for CloudTrail.
Try out AWS CloudTrial hands-on in our lab! Subscribe or sign up for a 7-day, risk-free trial with INE to access this lab and a robust library covering the latest in Cyber Security, Networking, Cloud, and Data Science!
