Accept
This website is using cookies. More details

Yohan Beschi
Developer, Cloud Architect and DevOps Advocate

Understanding AWS CloudTrail Logs

Understanding AWS CloudTrail Logs

AWS CloudTrail is an unvaluable service for governance, compliance and auditing. AWS CloudTrail Insights, Amazon GuardDuty and AWS Security Hub backed by AWS Config already help us a great deal in these areas. However, it happens from time to time that we need to dig into AWS CloudTrail logs and it far from an easy task.

With CloudTrail we can:

  • monitor API call errors
  • track user activity
  • enforce the Principle Of Least Privilege
  • …

From AWS Console we can explore events history over the past 90 days. To go back further we need to create a Trail and store logs into an AWS S3 Bucket. These logs can then be loaded into Amazon Athena (for more information see Querying AWS CloudTrail Logs and Analyze Security, Compliance, and Operational Activity Using AWS CloudTrail and Amazon Athena).

CloudTrail produces a lot of logs. Therefore, we first need to understand the logs format before exploiting them effectively.

Note from the author:

To write this article billion of AWS CloudTrail logs coming from 15 different AWS Accounts have been analyzed. Unfortunately, not all AWS services being used, we won’t be able to cover all the use cases.

AWS CloudTrail logs in S3

AWS CloudTrail logs can be stored in AWS CloudWatch Logs and AWS S3. To save costs and for easy processing AWS S3 is recommended as CloudTrail produce a lot of logs and a regular project with multiple AWS Accounts can have hundreds of GB of logs within a year.

AWS CloudTrail logs are stored in S3 as JSON text file compressed in gzip with the following name:

1

<bucket_name>/<prefix_name>/AWSLogs/<aws_account_id>/CloudTrail/<region>/YYYY/MM/DD/file_name.json.gz

Furthermore, every hour CloudTrail produce a digest file, containing the names of the log files that were delivered to the S3 bucket with the following name:

1

<bucket_name>/<prefix_name>/AWSLogs/<aws_account_id>/CloudTrail-Digest/<region>/YYYY/MM/DD/file_name.json.gz

For me information see Finding Your CloudTrail Log Files & CloudTrail Digest File Structure

A log file contains one or more records. A record being an event (an action made in an AWS Account - e.g. AssumeRole, StartInstances, CreateLogGroup).

1
2
3
4
5
6
7

{
  "Records": [
    {
      // Event
    }
  ]
}

The structure of a record is partially documented (see CloudTrail Record Contents & CloudTrail userIdentity Element). Unfortunately, with these pages only we won’t be able to do much.

AWS CloudTrail logs format

Let’s go through the most important properties and see how they can help us in our different tasks.

eventSource & eventName

The eventSource is the AWS Service against which the API call has been made (e.g. route53.amazonaws.com, ec2.amazonaws.com).

The eventName is an Action done through the AWS APIs (e.g. AddTags, DeleteUser, CreateAutoScalingGroup). As we will see later, this is not completely true, some Events don’t have any API call equivalent.

Of course the same Event Name can be used by multiple Event Sources. Here few examples:

  • "ListApplications" : [ "codedeploy.amazonaws.com", "application-insights.amazonaws.com", "kinesisanalytics.amazonaws.com" ]
  • "CreateWebACL" : [ "wafv2.amazonaws.com", "waf.amazonaws.com" ]
  • "DeleteRepository" : [ "ecr.amazonaws.com", "codecommit.amazonaws.com" ]
  • "ListTags" : [ "cloudtrail.amazonaws.com", "es.amazonaws.com", "backup.amazonaws.com" ]

With the eventSource and the eventName we can easily recreate the AWS IAM permission required to do this action. For example, if "eventSource" : "ssm.amazonaws.com" and "eventName" : "PutParameter" the IAM permission is ssm:PutParameter. Some eventNames are suffixed with a date (e.g. PublishLayerVersion20181031) or a date and version (e.g. TagResource20170331v2).

Being able to associate a CloudTrail event with an IAM permission can help us find a missing permission or to gather all the permissions actually used.

Programmatically, a one liner can do the job.

Let’s see an example in Python:

1
2
3
4
5
6
7
8
9
10

import re

eventSource = 'lambda.amazonaws.com'
eventName = 'PublishLayerVersion20181031'

iam_permission = eventSource.split('.')[0] \
      + ':' \
      + re.sub(r'^([a-zA-Z0-9]+)([0-9]{8})(v[0-9])?$', r'\1', eventName)

print(iam_permission)

and another one in Java:

1
2
3
4
5
6
7
8

var eventSource = 'lambda.amazonaws.com';
var eventName = 'PublishLayerVersion20181031';

var iamPermission = eventSource.split(".")[0]
      + ":"
      + eventName.replace("^([a-zA-Z0-9]+)([0-9]{8})(v[0-9])?$", "$1");

System.out.println(iamPermission);

It is important to note that this convention has some exceptions. For example, the eventSource for Amazon CloudWatch is monitoring.amazonaws.com while the IAM permission is cloudwatch:*.

Furthermore, some Event Names do not match the AWS Action name and therefore not the IAM permissions either. As described in the S3 documentation, some translations are required:

SOAP API name API event name used in CloudTrail log
ListAllMyBuckets ListBuckets
CreateBucket CreateBucket
DeleteBucket DeleteBucket
GetBucketAccessControlPolicy GetBucketAcl
SetBucketAccessControlPolicy PutBucketAcl
GetBucketLoggingStatus GetBucketLogging
SetBucketLoggingStatus PutBucketLogging

Finally, the Event Source signin.amazonaws.com does not have any matching permission in IAM. Therefore, the following permissions does not exist:

  • signin:ConsoleLogin
  • signin:CheckMfa
  • signin:PasswordRecoveryCompleted
  • signin:PasswordRecoveryRequested
  • signin:RenewRole
  • signin:ExitRole
  • signin:SwitchRole

But some of these Events (e.g. SwitchRole) are associated with Events that have IAM permissions (e.g. sts:AssumeRole)

eventType

There are three types of Event:

  • AwsConsoleSignIn - Used with the Event Source signin.amazonaws.com and the Events describes previously
  • AwsServiceEvent - Event generated by a service (RotationSucceeded, DeleteKey, ProcessBackupPlanSelection, BackupJobCompleted, BackupDeleted, NewClientConnection, SharedSnapshotVolumeCreated, etc.)
  • AwsApiCall - An API was called.

Author’s note: in my logs I’ve found a fourth Event Type: AwsConsoleAction associated with the SetIAMAccessPreference event.

sourceIPAddress

The sourceIPAddress can have the following values:

  • An IP from which the call has been made
  • A DNS (e.g. cloudformation.amazonaws.com, elasticbeanstalk.amazonaws.com), the service doing the action on our behalf
  • AWS Internal

userAgent

The userAgent is like the HTTP User-Agent header and can have multiple values:

  • A Browser Agent
  • A DNS (e.g. cloudformation.amazonaws.com, elasticbeanstalk.amazonaws.com), the service doing the action on our behalf
  • The information of the SDK used (e.g. aws-sdk-go/1.28.12 (go1.13.15; linux; amd64), [aws-sdk-java/1.11.792 Linux/4.9.217-0.1.ac.205.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/Oracle_Corporation])
  • The information of the SDK used through an AWS Toolkit (e.g. AWS-Toolkit-For-JetBrains/1.7 IntelliJ-IDEA-Ultimate-Edition/2019.1.2, aws-sdk-java/2.6.5 Linux/4.15.0-108-generic OpenJDK_64-Bit_Server_VM/25.202-b49 Java/1.8.0_202-release kotlin vendor/JetBrains_s.r.o io/sync http/UNKNOWN PAGINATED/2.6.5)
  • The information of the AWS CLI used (e.g. aws-cli/2.0.57 Python/3.7.3 Linux/5.4.0-56-generic exe/x86_64.ubuntu.20 command/cloudformation.deploy)
  • AWS Internal

errorCode & errorMessage

If an error occurs (included if an action is not permitted - Access Denied) it is logged into CloudTrail.

Most of the time we have the errorCode and the errorMessage, but sometimes only one of them.

userIdentity

For user activity tracking or enforcing the Principle Of Least Privilege, the userIdentity is a very important piece of the puzzle.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

{
  "userIdentity" : {
    "type" : "AssumedRole",
    "principalId" : "<value>:botocore-session-1612455092",
    "arn" : "arn:aws:sts::<account_id>:assumed-role/codebuild-deploy-assumable-role/botocore-session-1612455092",
    "accountId" : "<account_id>",
    "sessionContext" : {
      "sessionIssuer" : {
        "type" : "Role",
        "principalId" : "<value>",
        "arn" : "arn:aws:iam::<account_id>:role/codebuild-deploy-assumable-role",
        "accountId" : "<account_id>",
        "userName" : "codebuild-deploy-assumable-role"
      },
      "webIdFederationData" : { },
      "attributes" : {
        "mfaAuthenticated" : "false",
        "creationDate" : "2021-02-04T16:11:32Z"
      }
    },
    "invokedBy" : "cloudformation.amazonaws.com"
  }
}

Let’s start with the attribute type, which identify which entity has made the action. It can have the following values:

  • AssumedRole
  • AWSAccount
  • AWSService
  • IAMUser
  • Root
  • FederatedUser
  • SAMLUser
  • Unknown
  • <novalue> - the attribute is absent

The attribute userName has a value only for certain types:

  • For the types IAMUser, FederatedUser and SAMLUser the value is the user real name (the one used to login)
  • For the type Root, the value is an AWS Account ID or account alias if defined
  • For all the other types the userName attribute is absent

The arn attribute has the following value depending on the type:

  • Root: arn:aws:iam::<aws_account_id>:root
  • IAMUser: arn:aws:iam::<aws_account_id>:user/<username>. For the Event Type AwsConsoleSignIn the arn does not always have a value (e.g. CheckMfa event)
  • AssumedRole: arn:aws:sts::<aws_account_id>:assumed-role/<role_name>/<session_name> or arn:aws:sts::<aws_account_id>:assumed-role/<role_name>/<user_name>
  • For all the other types, the arn attribute is absent

Going a little further, the type AssumedRole has the attribute sessionContext.sessionIssuer.arn where the arn’s value is the actual ARN of the role used to do the action (e.g. arn:aws:iam::<account_id>:role/codebuild-deploy-assumable-role")

The accountId is the ID of the AWS Account in which the action has been made. For the types SAMLUser, FederatedUser and AWSService, the attribute is not set. For the type AWSAccount the value can either be an AWS account ID or ANONYMOUS_PRINCIPAL.

AssumeRole Event

In order to gather the activity from each user, we first need to understand the workflow in term of CloudTrail Events.

A user can login through the console, or using an SSO, then either stay in the AWS account or switch role to access another Account (It is why a centralized logging S3 Bucket is mandatory for a project using multiple AWS Accounts. It makes exploiting CloudTrail logs much easier.).

Let’s start with a Login from the AWS Console Event (for clarity, only the useful attributes are shown):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "IAMUser",
    "arn" : "arn:aws:iam::A_0123456789:user/Yohan.Beschi@arhs-spikeseed.com",
    "accountId" : "A_0123456789",
    "userName" : "Yohan.Beschi@arhs-spikeseed.com"
  },
  "eventTime" : "2021-02-11T07:56:54Z",
  "eventSource" : "signin.amazonaws.com",
  "eventName" : "ConsoleLogin",
  "responseElements" : {
    "ConsoleLogin" : "Success"
  },
  "recipientAccountId" : "A_0123456789"
}

The user Yohan.Beschi@arhs-spikeseed.com logged in successfully into the AWS Account with the ID: A_0123456789. If the login failed, we would have the same information but the reponseElements would be: "ConsoleLogin" : "Failure".

Now the connected user decides to switch from the current AWS Account (A_0123456789 for this example) to another AWS Account (B_9876543210) using the SwitchRole/AssumeRole feature. Here things start to get a little crazy.

In the account A_0123456789 a SwitchRole Event with userIdentity.type = IAMUser is generated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "IAMUser",
    "arn" : "arn:aws:iam::A_0123456789:user/Yohan.Beschi@arhs-spikeseed.com",
    "accountId" : "A_0123456789",
    "userName" : "Yohan.Beschi@arhs-spikeseed.com"
  },
  "eventTime" : "2021-02-11T07:57:32Z",
  "eventSource" : "signin.amazonaws.com",
  "eventName" : "SwitchRole",
  "responseElements" : {
    "SwitchRole" : "Success"
  },
  "additionalEventData" : {
    "SwitchTo" : "arn:aws:iam::B_9876543210:role/cloudadmin-console-assumable-role"
  },
  "recipientAccountId" : "A_0123456789"
}

In the account B_9876543210 a SwitchRole Event with userIdentity.type = AssumedRole is generated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "AssumedRole",
    "arn" : "arn:aws:sts::B_9876543210:assumed-role/cloudadmin-console-assumable-role/Yohan.Beschi@arhs-spikeseed.com",
    "accountId" : "B_9876543210"
  },
  "eventTime" : "2021-02-11T07:57:32Z",
  "eventSource" : "signin.amazonaws.com",
  "eventName" : "SwitchRole",
  "responseElements" : {
    "SwitchRole" : "Success"
  },
  "recipientAccountId" : "B_9876543210"
}

Furthermore, in the account A_0123456789 a AssumeRole Event with userIdentity.type = IAMUser is generated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "IAMUser",
    "arn" : "arn:aws:iam::A_0123456789:user/Yohan.Beschi@arhs-spikeseed.com",
    "accountId" : "A_0123456789",
    "userName" : "Yohan.Beschi@arhs-spikeseed.com",
  },
  "eventTime" : "2021-02-11T07:57:32Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRole",
  "requestParameters" : {
    "roleArn" : "arn:aws:iam::B_9876543210:role/cloudadmin-console-assumable-role",
    "roleSessionName" : "Yohan.Beschi@arhs-spikeseed.com"
  },
  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::B_9876543210:assumed-role/cloudadmin-console-assumable-role/Yohan.Beschi@arhs-spikeseed.com"
    }
  },
  "recipientAccountId" : "A_0123456789",
  "sharedEventID" : "73bbd83b-9a8f-4a3f-8e54-5c214c65692e"
}

And finally, in the account B_9876543210 a AssumeRole Event with userIdentity.type = AWSAccount is generated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "AWSAccount",
    "accountId" : "A_0123456789"
  },
  "eventTime" : "2021-02-11T07:57:32Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRole",
  "requestParameters" : {
    "roleArn" : "arn:aws:iam::B_9876543210:role/cloudadmin-console-assumable-role",
    "roleSessionName" : "Yohan.Beschi@arhs-spikeseed.com"
  },
  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::B_9876543210:assumed-role/cloudadmin-console-assumable-role/Yohan.Beschi@arhs-spikeseed.com"
    }
  },
  "recipientAccountId" : "B_9876543210",
  "sharedEventID" : "73bbd83b-9a8f-4a3f-8e54-5c214c65692e"
}

While the SwitchRole Events are pretty useless, the AssumeRole ones are not.

First, we have a sharedEventID attribute that links the two events, which have been generated in two different AWS Accounts, together. And second, both events have the same requestParameters and responseElements.assumedRoleUser.arn arn:aws:sts::B_9876543210:assumed-role/cloudadmin-console-assumable-role/Yohan.Beschi@arhs-spikeseed.com

Let’s see few other examples with AssumeRole and other userIdentity.type values without going into a detailed explanation:

userIdentity.type = SAMLUser

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "userIdentity" : {
    "type" : "SAMLUser",
    "userName" : "Yohan.Beschi@arhs-spikeseed.com",
  },
  "eventTime" : "2021-02-09T12:00:43Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRoleWithSAML",
  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::<account_id>:assumed-role/AWSReservedSSO_AdministratorAccess_<id>/Yohan.Beschi@arhs-spikeseed.com"
    }
}

userIdentity.type = AWSAccount

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "AWSAccount",
    "accountId" : "<account_id>"
  },
  "eventTime" : "2021-02-02T11:31:20Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRole",
  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::<another_account_id>:assumed-role/cloudadmin-pa-assumable-role/botocore-session-1612265479"
    }
  }
}

userIdentity.type = AWSService

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "AWSService",
    "invokedBy" : "trustedadvisor.amazonaws.com"
  },
  "eventTime" : "2020-11-24T16:51:08Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRole",
  "requestParameters" : {
    "roleArn" : "arn:aws:iam::<account_id>:role/aws-service-role/trustedadvisor.amazonaws.com/AWSServiceRoleForTrustedAdvisor",
    "roleSessionName" : "TrustedAdvisor_<account_id>_1a6601d4-3a39-4fbe-a820-98d23299927f"
  },

  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::<account_id>:assumed-role/AWSServiceRoleForTrustedAdvisor/TrustedAdvisor_<account_id>_57b9d37f-975b-4787-b0ee-cc7b5d714025"
    }
  }
}

userIdentity.type = AssumedRole and userIdentity.sessionContext.sessionIssuer.type = Role

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

{
  "eventVersion" : "1.08",
  "userIdentity" : {
    "type" : "AssumedRole",
    "arn" : "arn:aws:sts::<account_id_a>:assumed-role/codebuild-deploy-test-app-role/AWSCodeBuild-5923ff10-3773-4c14-a807-3f4185513106",
    "accountId" : "<account_id_a>",
    "accessKeyId" : "[...]",
    "sessionContext" : {
      "sessionIssuer" : {
        "type" : "Role",
        "principalId" : "[...]",
        "arn" : "arn:aws:iam::<account_id_a>:role/codebuild-deploy-test-app-role",
        "accountId" : "<account_id_a>",
        "userName" : "codebuild-deploy-test-app-role"
      },
      "webIdFederationData" : { },
      "attributes" : {
        "mfaAuthenticated" : "false",
        "creationDate" : "2021-02-04T15:11:52Z"
      }
    }
  },
  "eventTime" : "2021-02-04T15:20:52Z",
  "eventSource" : "sts.amazonaws.com",
  "eventName" : "AssumeRole",
  "awsRegion" : "us-east-1",
  "sourceIPAddress" : "34.250.63.250",
  "userAgent" : "Boto3/1.17.1 Python/3.8.3 Linux/4.14.203-116.332.amzn1.x86_64 exec-env/AWS_ECS_EC2 Botocore/1.20.1",
  "requestParameters" : {
    "roleArn" : "arn:aws:iam::<account_id_b>:role/codebuild-deploy-assumable-role",
    "roleSessionName" : "botocore-session-1612452052"
  },
  "responseElements" : {
    "assumedRoleUser" : {
      "arn" : "arn:aws:sts::<account_id_b>:assumed-role/codebuild-deploy-assumable-role/botocore-session-1612452052"
    }
  }
}

Conclusion

Understanding CloudTrail Logs is not easy. But it is mandatory to be able to exploit them seamlessly when needed.

Furthermore, this will help us for an upcoming article, when we will investigate ways to implement the Principle of Least Privilege on IAM Roles.

Now it’s your turn!

Schedule a 1-on-1 with an ARHS Cloud Expert today!

+352 26 11 02 1
Schedule by email