Docker Monitor
The Docker monitor automatically detects all Docker containers running on a given host, and streams stdout and stderr for each container to Scalyr
Note: If you're using Docker, you generally don't need to configure this monitor yourself. The Scalyr Agent Docker container will automatically import logs from other containers. See the Scalyr Agent Docker installation instructions for more details.
Note: If you are not running the docker version of the Agent you need to have the `docker` python library installed on the machine. You can install it with this command: `pip install docker==4.1.0`
An agent monitor plugin is a component of the Scalyr Agent. To use a plugin, simply add it to the monitors section of the Scalyr Agent configuration file (/etc/scalyr-agent-2/agent.json). For more information, see Agent Plugins.
Sample Configuration
Here is a typical configuration fragment:
monitors: [
{
module: "scalyr_agent.builtin_monitors.docker_monitor",
log_mode: "syslog"
}, {
module: "scalyr_agent.builtin_monitors.syslog_monitor",
mode: "docker"
}
]
This runs the Docker plugin configured to collect metrics from all containers running on the same host. It will use the recommended syslog mode to collect logs from the other containers. Since it is using syslog to import logs, you must also configure the syslog_monitor as well.
Note, in addition to use this configuration, you must also configure the other containers on the hosts to send their logs to the Scalyr Agent container via the Docker syslog logging plugin. See the Scalyr Agent Docker installation instructions for more details.
Configuration Reference
The Docker monitor has several additional configuration options you can use to change its behavior.
Option | Usage |
---|---|
module | Always scalyr_agent.builtin_monitors.docker_monitor |
log_mode | Optional (defaults to docker_api). Determines which method is used to gather logs from local containers. If docker_api, then the agent will query container logs over the Docker api. If syslog, then the agent expects the other containers to push logs to this one using the Docker syslog logging driver. This option is to protect legacy behavior; the preferred method is to leave this value set to default (docker_api) and then set docker_raw_logs to True. |
docker_percpu_metrics | Optional (defaults to False). If True, the agent emits CPU usage metrics per core. Note: this is disabled by default because it can lead to an excessive amount of metric data on CPUs with a large number of cores. |
docker_raw_logs | Optional (defaults to True). If True, the docker monitor will use the raw log files on disk to read logs. The location of the raw log file is obtained by querying the path from the Docker API. If False, the logs will be streamed over the Docker API. We recommend this be set to True. (Prior to release 2.0.50, this defaulted to False to preserve legacy behavior.) |
container_name | Optional (defaults to None). Defines a regular expression that matches the name given to the container running the scalyr-agent. If this is None, the Scalyr agent will look for a container running /usr/sbin/scalyr-agent-2 as the main process. |
container_check_interval | Optional (defaults to 5). How often (in seconds) to check for new containers. |
api_socket | Optional (defaults to /var/scalyr/docker.sock). Defines the unix socket used to communicate with the docker API. WARNING, if you have `mode` set to `syslog`, you must also set the `docker_api_socket` configuration option in the syslog monitor to this same value. Note: You need to map the host's /var/run/docker.sock to the same value as specified here, using the -v parameter, e.g. docker run -v /run/docker.sock:/var/scalyr/docker.sock ... |
docker_api_version | Optional (defaults to 'auto'). The version of the Docker API to use. WARNING, if you have log_mode set to syslog, you must also be running the syslog monitor and must set the docker_api_version configuration option in the syslog monitor to this same value. |
docker_log_prefix | Optional (defaults to docker). Prefix added to the start of all docker log file names. Only applies if docker_raw_logs if False. |
max_previous_lines | Optional (defaults to 5000). The maximum number of lines to read backwards from the end of the stdout/stderr logs when starting to log a container's stdout/stderr. Used to find the last line that was sent to Scalyr. This is only used if the``log_mode`` is not docker_api and docker_raw_logs is False. |
readback_buffer_size | Optional (defaults to 5k). The maximum number of bytes to read backwards from the end of any log files on disk when starting to log a container's stdout/stderr. Used to find the last line that was sent to Scalyr. This is only used if the``log_mode`` is not docker_api and docker_raw_logs is False. |
metrics_only | Optional (defaults to False). If True, the docker monitor will only log docker metrics and not any other logs from running containers. If set to True, this value overrides the config variable 'report_container_metrics'. |
container_globs | Optional (defaults to None). A whitelist of container name glob patterns to monitor. Only containers whose name matches one of the glob patterns will be monitored. If None, all container names are matched. This value is applied before container_globs_exclude. |
container_globs_exclude | Optional (defaults to None). A blacklist of container name glob patterns to exclude from monitoring. Any container whose name matches one of the glob patterns will not be monitored. If `None`, all container names matched by `container_globs` are monitored. This value is applied after `container_globs`. |
report_container_metrics | Optional (defaults to True). If True, metrics will be collected from the container and reported to Scalyr. |
use_labels_for_log_config | Optional (defaults to True). If True, the docker monitor will check each container for any labels that begin with com.scalyr.config.log. and use those labels (minus the prefix) as fields in the containers log_config. Keys that contain hyphens will automatically be converted to underscores. |
labels_as_attributes | Optional (defaults to False). If True, the docker monitor will add any labels found on the container as log attributes, after applying label_include_globs and label_exclude_globs. |
label_include_globs | Optional (defaults to ['*']). If labels_as_attributes is True then this option is a list of glob strings used to include labels that should be uploaded as log attributes. The docker monitor first gets all container labels that match any glob in this list and then filters out any labels that match `label_exclude_globs`, and the final list is then uploaded as log attributes. |
label_exclude_globs | Optional (defaults to ['com.scalyr.config.*']). If labels_as_attributes is True, then this is a list of glob strings used to exclude labels from being uploaded as log attributes. Any label whose key matches any glob on this list will not be added as a log attribute. Note: the globs in this list are applied after label_include_globs |
label_prefix | Optional (defaults to ""). If labels_as_attributes is True, then append this prefix to the start of each label before adding it to the log attributes. |
Container metrics.
Below is a description of all metrics collected from the running containers by the Scalyr Agent.
All metrics will have a field called id which will contain the name of the container from which the metric was collected.
Network metrics
Metric | Description |
---|---|
docker.net.rx_bytes | Total received bytes on the network interface |
docker.net.rx_dropped | Total receive packets dropped on the network interface |
docker.net.rx_errors | Total receive errors on the network interface |
docker.net.rx_packets | Total received packets on the network interface |
docker.net.tx_bytes | Total transmitted bytes on the network interface |
docker.net.tx_dropped | Total transmitted packets dropped on the network interface |
docker.net.tx_errors | Total transmission errors on the network interface |
docker.net.tx_packets | Total packets transmitted on the network interface |
Memory metrics
Metric | Description |
---|---|
docker.mem.stat.active_anon | The number of bytes of active memory backed by anonymous pages, excluding sub-cgroups. |
docker.mem.stat.active_file | The number of bytes of active memory backed by files, excluding sub-cgroups. |
docker.mem.stat.cache | The number of bytes used for the cache, excluding sub-cgroups. |
docker.mem.stat.hierarchical_memory_limit | The memory limit in bytes for the container. |
docker.mem.stat.inactive_anon | The number of bytes of inactive memory in anonymous pages, excluding sub-cgroups. |
docker.mem.stat.inactive_file | The number of bytes of inactive memory in file pages, excluding sub-cgroups. |
docker.mem.stat.mapped_file | The number of bytes of mapped files, excluding sub-groups |
docker.mem.stat.pgfault | The total number of page faults, excluding sub-cgroups. |
docker.mem.stat.pgmajfault | The number of major page faults, excluding sub-cgroups |
docker.mem.stat.pgpgin | The number of charging events, excluding sub-cgroups |
docker.mem.stat.pgpgout | The number of uncharging events, excluding sub-groups |
docker.mem.stat.rss | The number of bytes of anonymous and swap cache memory (includes transparent hugepages), excluding sub-cgroups |
docker.mem.stat.rss_huge | The number of bytes of anonymous transparent hugepages, excluding sub-cgroups |
docker.mem.stat.unevictable | The number of bytes of memory that cannot be reclaimed (mlocked etc), excluding sub-cgroups |
docker.mem.stat.writeback | The number of bytes being written back to disk, excluding sub-cgroups |
docker.mem.stat.total_active_anon | The number of bytes of active memory backed by anonymous pages, including sub-cgroups. |
docker.mem.stat.total_active_file | The number of bytes of active memory backed by files, including sub-cgroups. |
docker.mem.stat.total_cache | The number of bytes used for the cache, including sub-cgroups. |
docker.mem.stat.total_inactive_anon | The number of bytes of inactive memory in anonymous pages, including sub-cgroups. |
docker.mem.stat.total_inactive_file | The number of bytes of inactive memory in file pages, including sub-cgroups. |
docker.mem.stat.total_mapped_file | The number of bytes of mapped files, including sub-groups |
docker.mem.stat.total_pgfault | The total number of page faults, including sub-cgroups. |
docker.mem.stat.total_pgmajfault | The number of major page faults, including sub-cgroups |
docker.mem.stat.total_pgpgin | The number of charging events, including sub-cgroups |
docker.mem.stat.total_pgpgout | The number of uncharging events, including sub-groups |
docker.mem.stat.total_rss | The number of bytes of anonymous and swap cache memory (includes transparent hugepages), including sub-cgroups |
docker.mem.stat.total_rss_huge | The number of bytes of anonymous transparent hugepages, including sub-cgroups |
docker.mem.stat.total_unevictable | The number of bytes of memory that cannot be reclaimed (mlocked etc), including sub-cgroups |
docker.mem.stat.total_writeback | The number of bytes being written back to disk, including sub-cgroups |
docker.mem.max_usage | The max amount of memory used by container in bytes. |
docker.mem.usage | The current number of bytes used for memory including cache. |
docker.mem.fail_cnt | The number of times the container hit its memory limit |
docker.mem.limit | The memory limit for the container in bytes. |
CPU metrics
Metric | Description |
---|---|
docker.cpu.usage | Total CPU consumed by container in nanoseconds |
docker.cpu.system_cpu_usage | Total CPU consumed by container in kernel mode in nanoseconds |
docker.cpu.usage_in_usermode | Total CPU consumed by tasks of the cgroup in user mode in nanoseconds |
docker.cpu.total_usage | Total CPU consumed by tasks of the cgroup in nanoseconds |
docker.cpu.usage_in_kernelmode | Total CPU consumed by tasks of the cgroup in kernel mode in nanoseconds |
docker.cpu.throttling.periods | The number of of periods with throttling active. |
docker.cpu.throttling.throttled_periods | The number of periods where the container hit its throttling limit |
docker.cpu.throttling.throttled_time | The aggregate amount of time the container was throttled in nanoseconds |