Import Amazon S3 Bucket Access Logs
This solution uses AWS's "Assume Role" functionality for authentication. For the documentation of the older access key authentication click here.
This Solution describes how to import Amazon S3 bucket access logs for analysis in Scalyr.
Note that S3 makes logs available in batches. Thus, there can be a delay of minutes to hours before log messages show up in Scalyr. Amazon's documentation specifies that "most log records will be delivered within a few hours of the time that they were recorded".
Prerequisites
1. Enable access logs for your S3 bucket(s). See http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html. It's best to create a new S3 bucket just for these logs.
2. Create an SQS queue, and configure your S3 bucket to publish new-object notifications to the queue.
3. Use Amazon's IAM (Identity and Access Management) tools to create an IAM role account which can only be used to read this bucket and queue. For instructions, see the section Create IAM Role.
Steps
Scalyr uses "monitors" to fetch data from other services. These steps will guide you through creating a monitor to fetch your S3 bucket access logs.
1. From the navigation bar, click , and select .
2. Click to open the monitors configuration file.
3. Find the monitors section of the configuration file. If you have never edited this file before, the monitors section will look like this:
monitors: [
// {
// type: \"http\",
// url: \"http://www.example.com/foo?bar=1\"
// },
// {
// type: \"http\",
// url: \"http://www.example.com/foo?bar=1\"
// }
]4. Add a stanza for the SQS queue you created earlier. The section might now look like this:
monitors: [
{
type: "s3Bucket",
region: "us-east-1",
roleToAssume: "XXX",
queueUrl: "https://sqs.us-east-1.amazonaws.com/nnnnnnnnnnnn/scalyr-s3-bucket-log"
fileFormat: "s3_bucket_access",
hostname: "s3_bucket_access"
}
]Fill in the appropriate values for each field:
| Field | Value |
|---|---|
| type | Always s3Bucket. |
| region | The AWS region in which your SQS queue is located, e.g. us-east-1. |
| s3Region | The AWS region in which your S3 bucket instance is located, e.g. us-east-1. You can omit this unless it is different than region. |
| roleToAssume | The ARN of the IAM role you created. |
| queueUrl | The name of the SQS queue to which your bucket sends new-object notices. |
| fileFormat | Always s3_bucket_access. |
| objectKeyFilter | Optional. If you specify a value, then S3 objects are ignored unless their name (object key) contains this substring. If you have multiple logs being published to the same S3 bucket, use this option to select the appropriate subset. |
| hostname | The server name under which your bucket access logs will appear in the Overview page. Defaults to "S3Bucket". |
| logfile | The file name under which your bucket access log will appear in the Overview page. Defaults to "S3Bucket". |
| logAttributes | Specifies extra fields to attach to the messages imported from this log. Optional. |
Always use an IAM role with limited permissions. If you haven't already done so, follow the Create IAM Role instructions to create a special role which only has access to the S3 bucket and SQS queue.
5. Click to save your changes. Scalyr will begin checking for new data batches once per minute.
6. Wait for the initial batch of log data to be retrieved. It may take minutes to hours for Amazon to publish the first batch.
7. In the top navigation bar, click Overview. In the list of servers, you should see an entry named according to the hostname you specified in the monitor configuration. To the right will be a link to your bucket access logs.
Troubleshooting
If your logs don't appear, make sure you've waited at least a few hours since saving your changes to the Monitors configuration (i.e. since clicking Update File). Also verify that there is fresh activity in your S3 bucket(s). Then return to the Scalyr Overview page and refresh your browser.
If the logs still don't appear, you may have a configuration error which is preventing the Scalyr monitor from retrieving your logs. To check for error messages, in Scalyr's top navigation bar, click . In the box, type tag='S3BucketMonitor' and click the button. Click to jump to the most recent log messages, and click on an individual message to see details for that message. If the details page includes an "errorMessage" field, then AWS returned an error when Scalyr attempted to retrieve your logs. Some common error messages:
| Cause | errorMessage |
|---|---|
| Incorrect Role configuration | Status Code: 403, AWS Service: AmazonSQS, AWS Request ID: xxx-xxx-xxx-xxx, AWS Error Code: AccessDenied, AWS Error Message: Access to the resource https://sqs.us-east-1.amazonaws.com/nnnnnnnnnnnn/queue-name is denied. |
| Incorrect Role configuration or incorrect Role ARN | Status Code: 403, AWS Service: AWSSecurityTokenService, AWS Request ID: xxx-xxx-xxx-xxx, AWS Error Code: AccessDenied, AWS Error Message: User: arn:aws:iam::913057016266:user/user is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::nnnnnnnnnnnn:role/RoleName |
Further Reading
To learn how to work with your imported logs, see the Search overview page, and the Query Language page.
Appendix: Create IAM Role
You can use Amazon IAM to create a role which can only be used to read your S3 bucket access logs. This allows you to grant Scalyr the ability to import the logs, without opening up any other access to your AWS resources. Create the IAM role as follows:
- Make a note of your AWS account ID (a 12-digit number). You can find it near the top of the AWS My Account page.
- Log into the Amazon AWS console. From the Services menu, choose "IAM".
- Go to the Roles list.
- Click "Create Role".
- Under "Select type of trusted entity" select "Another AWS account".
- For "Account ID" enter "913057016266".
- Under options check "Require external ID" and enter the value "(Log in to view External Id.)".
- Click "Next: Permissions", then "Create policy", this will open in a new tab.
- Select the following values:
Effect: Allow
AWS Service: Amazon S3
Actions: check GetObject
Amazon Resource Name: arn:aws:s3:::bucket-name/*
Replace bucket-name with the name of the S3 bucket you specified when setting up bucket access logging. - Click "Add additional permissions".
- Update the form with the following values:
Effect: Allow
AWS Service: Amazon SQS
Actions: check GetQueueAttributes, DeleteMessage, and ReceiveMessage
Amazon Resource Name: arn:aws:sqs:us-east-1:account-id:queue-name
Replace account-id with your 12-digit AWS account ID, without hyphens. Replace bucket-name with the name of the SQS queue you subscribed to the S3 bucket. - Note: If the contents of your S3 bucket are encrypted you will need to also add "KMS" permissions to this policy.
- Click "Review policy", name it, then click "Create policy".
- Return to the create role tab and select your newly created policy and hit "Next".
- Skip past adding tags and give your role a name, then hit "Create role".