Environment-Aware Variables (Scalyr Agent)

Environment-aware variables are Scalyr Agent configuration variables that are mapped to system environment variables. If not defined in the agent.json file (or agent.d fragments), the agent falls back to the value set in the environment.

An environment-aware variable usually maps to a system environment variable of the same name, but prefixed with SCALYR_. Both upper-case and lower-case versions are recognized by the agent.

E.g., the global api_key environment-aware config variable maps to the SCALYR_API_KEY environment variable.

For aesthetics, legacy reasons, or proper association, some environment variables do not follow the SCALYR prefix format. For example:

  • The global scalyr_server config variable maps to just SCALYR_SERVER (no extra prefix).
  • The Kubernetes event monitor leader_node has an additional K8S prefix (maps to SCALYR_K8S_LEADER_NODE).

Commonly Modified

The following environment-aware variables are required by the Scalyr Agent and are usually modified by customers. (Note: <prefix format> is short-form for the `SCALYR_` prefix described above.)

Context Config variable Environment variable Default value Description
Global api_key <prefix format> <placeholder> The write logs access API key
Global scalyr_server SCALYR_SERVER agent.scalyr.com Scalyr backend server
Docker monitor docker_raw_logs <prefix format> true If true, the docker monitor will use the raw log files on disk to read logs. The location of the raw log file is obtained by querying the path from the Docker API. If false, the logs will be streamed over the Docker API. (This value defaults to false to preserve legacy behavior. It is recommended to set this to true for all new deployments.)

Occasionally Modified

You may need to modify the following environment-aware variables:

Context Config variable Environment variable Default value Description
Global max_line_size <prefix format> 9900 Maximum size for a line when reading from a log file. The Scalyr Agent will truncate lines longer than this limit when possible.
Global read_page_size <prefix format> 64*1024 Number log data bytes to be buffered at a time. Must always be greater than max_line_size.
Global log_rotation_backup_count <prefix format> 2 Number of log rotations
Global log_rotation_max_bytes <prefix format> 20*1024*1024 Size of each log rotation file.
Global debug_level <prefix format> 0 The debug level must be between 0 and 5 inclusive. Level 0 messages go into agent.log and are uploaded. Level 1 and above are finer grained and are not.
Global ca_cert_path <prefix format> TODO
Global verify_server_certificate <prefix format> true
Global http_proxy <prefix format> None
Global https_proxy <prefix format> None
Global k8s_ignore_namespaces <prefix format> kube-system List of namespaces whose pods are to be ignored.
Global k8s_api_url <prefix format> https://kubernetes.default
Global k8s_verify_api_queries <prefix format> true
Global k8s_cache_expiry_secs <prefix format> 30
Global k8s_cache_purge_secs <prefix format> 300
Global k8s_events_disable <prefix format> false If true, will disable collection of Kubernetes events.
Global k8s_cache_start_fuzz_secs <prefix format> 0 Random delay (in seconds) when k8s monitor starts up to prevent all agents in the cluster from querying the API master simultaneously. (Typically only needed for very large clusters.)
Global k8s_cache_expiry_fuzz_secs <prefix format> 0 Periodic random delay (in seconds) designed to further distribute multiple-agent queries to the API master that occur every `k8s_cache_expiry_secs` seconds. (Typically only needed for very large clusters.)
Global disable_send_requests <prefix format> false
Global config_change_check_interval <prefix format> 30
Global user_agent_refresh_interval <prefix format> 60
Global garbage_collect_interval <prefix format> 300
Global implicit_agent_log_collection <prefix format> true
Global implicit_metric_monitor <prefix format> true min_request_spacing_interval
Global implicit_agent_process_metrics_monitor <prefix format> true
K8s monitor container_globs <prefix format> None Comma-separated list of glob patterns of container names to monitor.
K8s monitor container_check_interval <prefix format> 5 Polling interval for started or stopped containers.
K8s monitor docker_max_parallel_stats <prefix format> 20 Maximum stats requests to issue in parallel when retrieving container metrics using the Docker API.
K8s monitor report_container_metrics <prefix format> true If true, also collect metrics from containers whose logs are monitored.
K8s monitor report_k8s_metrics <prefix format> true If true and report_container_metrics is also true, then collect Kubernetes system metrics.
K8s monitor k8s_ignore_pod_sandboxes <prefix format> true If true, exclude logs from containers where label io.kubernetes.docker.type == podsandbox
K8s monitor k8s_include_all_containers <prefix format> true Controls the default behavior for inclusion/exclusion of container logs. See documentation on Kubernetes Monitor annotations for more details.
K8s events check_labels SCALYR_K8S_CHECK_LABELS false
K8s events log_flush_delay SCALYR_K8S_LOG_FLUSH_DELAY 1.0
K8s events leader_check_interval SCALYR_K8S_LEADER_CHECK_INTERVAL 60
K8s events leader_node SCALYR_K8S_LEADER_NODE None
K8s events max_log_size SCALYR_K8S_MAX_LOG_SIZE None
K8s events max_log_rotations SCALYR_K8S_MAX_LOG_ROTATIONS None
Docker monitor container_globs_exclude <prefix format> None A blacklist of container name glob patterns to exclude from monitoring. Any container whose name matches one of the glob patterns will not be monitored. If `None`, all container names matched by `container_globs` are monitored. This value is applied after `container_globs`.
Docker monitor container_check_interval <prefix format> 5 How often (in seconds) to check for new containers.
Docker monitor report_container_metrics <prefix format> true If true, metrics will be collected from the container and reported to Scalyr.
Docker monitor labels_as_attributes <prefix format> false If true, the docker monitor will add any labels found on the container as log attributes, after applying label_include_globs and label_exclude_globs.
Docker monitor label_include_globs <prefix format> ['*'] If labels_as_attributes is true then this option is a list of glob strings used to include labels that should be uploaded as log attributes. The docker monitor first gets all container labels that match any glob in this list and then filters out any labels that match label_exclude_globs, and the final list is then uploaded as log attributes.
Docker monitor label_exclude_globs <prefix format> ['com.scalyr.config.*'] If labels_as_attributes is true, then this is a list of glob strings used to exclude labels from being uploaded as log attributes. Any label whose key matches any glob on this list will not be added as a log attribute. Note: the globs in this list are applied after label_include_globs.

Rarely Modified

The following environment-aware variables are rarely modified and usually by power users or under guidance from Scalyr support.

Context Config variable Environment variable Default value Description
Global allow_http <prefix format> false
Global agent_log_path <prefix format> /var/log/scalyr-agent-2 Agent log files dir
Global agent_data_path <prefix format> /var/lib/scalyr-agent-2 Agent metadata dir (for checkpoints etc.)
Global config_directory <prefix format> agent.d Agent config dir (located in same parent directory as agent.json file)
Global debug_init <prefix format> false
Global additional_monitor_module_paths <prefix format> <empty string>
Global compression_type <prefix format> bz2 Compression type for data uploaded to Scalyr. One of bz2 or deflate (which uses zlib compress).
Global compression_level <prefix format> 9
Global check_remote_if_no_tty <prefix format> true
Global close_old_files_duration_in_seconds <prefix format> 3600
Global copying_thread_profile_interval <prefix format> 0 min_allowed_request_size
Global copying_thread_profile_output_path <prefix format> /tmp/copying_thread_profiles_
Global copy_staleness_threshold <prefix format> 900
Global full_checkpoint_interval_in_seconds <prefix format> 60
Global global_monitor_sample_interval <prefix format> 30
Global max_allowed_request_size <prefix format> 1024*1024
Global min_allowed_request_size <prefix format> 100*1024
Global min_request_spacing_interval <prefix format> 1
Global max_request_spacing_interval <prefix format> 5 Max amount of time to wait between sending requests
Global max_error_request_spacing_interval <prefix format> 30
Global minimum_scan_interval <prefix format> None
Global low_water_bytes_sent <prefix format> 20*1024
Global high_water_bytes_sent <prefix format> 100*1024
Global low_water_request_spacing_adjustment <prefix format> 1.5
Global high_water_request_spacing_adjustment <prefix format> 0.6
Global failure_request_spacing_adjustment <prefix format> 1.5
Global request_too_large_adjustment <prefix format> 0.5
Global log_deletion_delay <prefix format> 600 Min seconds the agent waits for file to reappear on a file system after removal before it's considered deleted.
Global line_completion_wait_time <prefix format> 5 Max number of seconds the agent waits before returning partial lines. (A partial line is the last log line in a file not ending in a new line.)
Global max_new_log_detection_time <prefix format> 60
Global max_log_offset_size <prefix format> 5*1024*1024 How far to search back when getting a new log
Global max_existing_log_offset_size <prefix format> 100*1024*1024 How far to search back when getting an existing log
Global max_sequence_number <prefix format> 1024^4
Global pipeline_threshold <prefix format> 1.1 Size required for a message to trigger pipelining the next addEvents request, expressed as a multiple of max_allowed_request_size.
Global pidfile_advanced_reuse_guard <prefix format> false
Global request_deadline <prefix format> 60.0
Global strip_domain_from_default_server_host <prefix format> false
Global use_requests_lib <prefix format> false
Global use_unsafe_debugging <prefix format> false
Global enable_profiling <prefix format> false If true, the agent will log performance profiling data about itself into a log file.
Global profile_log_name <prefix format> 'agent.callgrind' Profile data log file name.
Global profile_duration_minutes <prefix format> 2 Duration (in minutes) to gather profiling information.
Global max_profile_interval_minutes <prefix format> 60 Frequency (in minutes) between consecutive runs of the profiler.
Global profile_clock <prefix format> 'random' The clock used for measuring profiling performance. One of ['random', 'wall', 'cpu'].
K8s monitor docker_percpu_metrics <prefix format> false If true, the agent emits CPU usage metrics per core. Note: this is disabled by default because it can lead to an excessive amount of metric data on CPUs with a large number of cores.
K8s monitor k8s_parse_format <prefix format> 'auto'
K8s monitor k8s_always_use_cri <prefix format> false
K8s monitor k8s_cri_query_filesystem <prefix format> false
K8s events message_log SCALYR_K8S_MESSAGE_LOG kubernetes_events.log
K8s events event_object_filter SCALYR_K8S_EVENT_OBJECT_FILTER 'CronJob', 'DaemonSet', 'Deployment', 'Job', 'Node', 'Pod', 'ReplicaSet', 'ReplicationController', 'StatefulSet'
K8s events ignore_master SCALYR_K8S_IGNORE_MASTER true
Docker monitor docker_percpu_metrics <prefix format> false If true, the agent emits CPU usage metrics per core. Note: this is disabled by default because it can lead to an excessive amount of metric data on CPUs with a large number of cores.
Docker monitor log_mode SCALYR_DOCKER_LOG_MODE 'docker_api' Determines which method is used to gather logs from local containers. If docker_api, then the agent will query container logs over the Docker api. If syslog, then the agent expects the other containers to push logs to this one using the Docker syslog logging driver. This option is to protect legacy behavior; the preferred method is to leave this value set to default (docker_api) and then set docker_raw_logs to true.
Docker monitor docker_api_version <prefix format> 'auto' The version of the Docker API to use. WARNING, if you have log_mode set to syslog, you must also be running the syslog monitor and must set the docker_api_version configuration option in the syslog monitor to this same value.
Docker monitor docker_log_prefix <prefix format> 'docker' Prefix added to the start of all docker log file names. Only applies if docker_raw_logs if false.
Docker monitor metrics_only SCALYR_DOCKER_METRICS_ONLY false If true, the docker monitor will only log docker metrics and not any other logs from running containers. If set to true, this value overrides the config variable 'report_container_metrics'.
Docker monitor container_globs <prefix format> None A whitelist of container name glob patterns to monitor. Only containers whose name matches one of the glob patterns will be monitored. If None, all container names are matched. This value is applied before container_globs_exclude.
Docker monitor label_prefix <prefix format> <empty string> If labels_as_attributes is true, then append this prefix to the start of each label before adding it to the log attributes.
Docker monitor use_labels_for_log_config <prefix format> true If true, the docker monitor will check each container for any labels that begin with com.scalyr.config.log. and use those labels (minus the prefix) as fields in the containers log_config. Keys that contain hyphens will automatically be converted to underscores.