Import Amazon Fargate Container Logs

Fargate Logs

Amazon Fargate is a platform for running Docker containers. This solution enables you to stream logs from your containers in Fargate into Scalyr using AWS Firelens and fluentd.

Setup

The idea behind this configuration is to add a container to your task definition that will act as a log router, in this case using fluentd with the Scalyr output plugin installed. The `logConfiguration` section of any other container in the task definition can then be set to use the `awsfirelens` log driver and configured to use the Scalyr fluentd output plugin. Please take note of the user defined fields below (enclosed by \<your \>).

Example Task Definition

This is a simple example task definition that can be modified to suit your use case.

{
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "containerDefinitions": [
        {
            "image": "scalyr/fluend:latest",
            "name": "log_router",
            "memory": 100,
            "essential": true,
            "firelensConfiguration": {
                "type": "fluentd",
                "options": {
                    "enable-ecs-log-metadata": "true"
                }
            },
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-ecs-fargate",
                    "awslogs-region": "us-west-1",
                    "awslogs-stream-prefix": "awslogs-ecs-fargate"
                }
            }
        },
  		{
            "essential": true,
            "image": "<your Docker image>",
            "name": "<your app name>",
            "logConfiguration": {
                "logDriver":"awsfirelens",
                "options": {
                    "@type":"scalyr",
                    "api_write_token": "<your write API key>"
                }
            },
            "memory": 100
  		}
    ],
    "family": "firelens-fargate",
    "cpu": "256",
    "executionRoleArn": "arn:aws:iam::<your account id>:role/ecsTaskExecutionRole",
    "memory": "512",
    "volumes": [],
    "placementConstraints": [],
    "taskRoleArn": "arn:aws:iam::<your account id>:role/ecs_task_iam_role",
    "networkMode": "awsvpc"
}

The first container defined here is the log router that will be running fluentd with the Scalyr output plugin. There is not much to configure here, but you may wish to omit the "logConfiguration" section if you don't want the logs the log router itself generates to be sent to CloudWatch.

        {
            "image": "scalyr/fluend:latest",  # This is the Docker image for fluentd with the Scalyr output plugin installed that will run on this container
            "name": "log_router",
            "memory": 100,
            "essential": true,  # To avoid missing out on logs it makes sense to mark this as "essential"
            "firelensConfiguration": {  # The "firelensConfiguration" section lets Fargate know to send logs from "awsfirelens" logdrivers here
                "type": "fluentd",  # Scalyr output plugin is a fluentd plugin so we use that instead of fluentbit
                "options": {
                    "enable-ecs-log-metadata": "true"  # Optionally add metadata to logs that go through this log router
                }
            },
            "logConfiguration": {  # This section is optional and if present will send the logs for the log router itself to CloudWatch, useful for debugging if there are any issues getting your logs into Scalyr
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-ecs-fargate",  # The name of an existing log group these logs should be sent to
                    "awslogs-region": "us-west-1",  # AWS region you want to send these logs to
                    "awslogs-stream-prefix": "awslogs-ecs-fargate"
                }
            }
        }

The second container will be running your application. You can have multiple such sections if you plan to run more than one application in the same task definition. The log driver in this section is configured identically to the Scalyr fluentd plugin. Please see this page for more details.

        {
            "essential": true,
            "image": "<your Docker image>",  # The Docker image of the app you want to run and collect the logs of
            "name": "<your app name>",
            "logConfiguration": {
                "logDriver":"awsfirelens",  # This time our logDriver is "awsfirelens" which will route the stdout from this container through our logrouter container, and up to Scalyr from there
                "options": {  # The options for this logDriver get translated into key value pairs in the fluentd output configuration, this is a minimal configuration
                    "@type":"scalyr",  # "@type" is always "scalyr" to let fluentd know to use the Scalyr output plugin
                    "api_write_token": "<your write API key>"  # A valid write API key for Scalyr
                }
            },
            "memory": 100
        }