Environment-Aware Variables (Scalyr Agent)
Environment-aware variables are Scalyr Agent configuration variables that are mapped to system environment variables. If not defined in the agent.json file (or agent.d fragments), the agent falls back to the value set in the environment.
An environment-aware variable usually maps to a system environment variable of the same name, but prefixed with SCALYR_. Both upper-case and lower-case versions are recognized by the agent.
E.g., the global api_key environment-aware config variable maps to the SCALYR_API_KEY environment variable.
For aesthetics, legacy reasons, or proper association, some environment variables do not follow the SCALYR prefix format. For example:
- The global scalyr_server config variable maps to just SCALYR_SERVER (no extra prefix).
- The Kubernetes event monitor leader_node has an additional K8S prefix (maps to SCALYR_K8S_LEADER_NODE).
Commonly Modified
The following environment-aware variables are required by the Scalyr Agent and are usually modified by customers. (Note: <prefix format> is short-form for the `SCALYR_` prefix described above.)
Context | Config variable | Environment variable | Default value | Description |
---|---|---|---|---|
Global | api_key | <prefix format> | <placeholder> | The write logs access API key |
Global | scalyr_server | SCALYR_SERVER | agent.scalyr.com | Scalyr backend server |
Docker monitor | docker_raw_logs | <prefix format> | true | If true, the docker monitor will use the raw log files on disk to read logs. The location of the raw log file is obtained by querying the path from the Docker API. If false, the logs will be streamed over the Docker API. (This value defaults to false to preserve legacy behavior. It is recommended to set this to true for all new deployments.) |
Occasionally Modified
You may need to modify the following environment-aware variables:
Context | Config variable | Environment variable | Default value | Description |
---|---|---|---|---|
Global | max_line_size | <prefix format> | 9900 | Maximum size for a line when reading from a log file. The Scalyr Agent will truncate lines longer than this limit when possible. |
Global | read_page_size | <prefix format> | 64*1024 | Number log data bytes to be buffered at a time. Must always be greater than max_line_size. |
Global | log_rotation_backup_count | <prefix format> | 2 | Number of log rotations |
Global | log_rotation_max_bytes | <prefix format> | 20*1024*1024 | Size of each log rotation file. |
Global | debug_level | <prefix format> | 0 | The debug level must be between 0 and 5 inclusive. Level 0 messages go into agent.log and are uploaded. Level 1 and above are finer grained and are not. |
Global | ca_cert_path | <prefix format> | TODO | |
Global | verify_server_certificate | <prefix format> | true | |
Global | http_proxy | <prefix format> | None | |
Global | https_proxy | <prefix format> | None | |
Global | k8s_ignore_namespaces | <prefix format> | kube-system | List of namespaces whose pods are to be ignored. |
Global | k8s_api_url | <prefix format> | https://kubernetes.default | |
Global | k8s_verify_api_queries | <prefix format> | true | |
Global | k8s_cache_expiry_secs | <prefix format> | 30 | |
Global | k8s_cache_purge_secs | <prefix format> | 300 | |
Global | k8s_events_disable | <prefix format> | false | If true, will disable collection of Kubernetes events. |
Global | k8s_cache_start_fuzz_secs | <prefix format> | 0 | Random delay (in seconds) when k8s monitor starts up to prevent all agents in the cluster from querying the API master simultaneously. (Typically only needed for very large clusters.) |
Global | k8s_cache_expiry_fuzz_secs | <prefix format> | 0 | Periodic random delay (in seconds) designed to further distribute multiple-agent queries to the API master that occur every `k8s_cache_expiry_secs` seconds. (Typically only needed for very large clusters.) |
Global | disable_send_requests | <prefix format> | false | |
Global | config_change_check_interval | <prefix format> | 30 | |
Global | user_agent_refresh_interval | <prefix format> | 60 | |
Global | garbage_collect_interval | <prefix format> | 300 | |
Global | implicit_agent_log_collection | <prefix format> | true | |
Global | implicit_metric_monitor | <prefix format> | true | min_request_spacing_interval |
Global | implicit_agent_process_metrics_monitor | <prefix format> | true | |
Global | default_sessions_per_worker | <prefix format> | 1 | The default number of sessions per worker. See Multi-sessions configuration |
Global | use_multiprocess_workers | <prefix format> | false | Start each session in a separate process. See Multi-session configuration |
K8s monitor | container_globs | <prefix format> | None | Comma-separated list of glob patterns of container names to monitor. |
K8s monitor | container_check_interval | <prefix format> | 5 | Polling interval for started or stopped containers. |
K8s monitor | docker_max_parallel_stats | <prefix format> | 20 | Maximum stats requests to issue in parallel when retrieving container metrics using the Docker API. |
K8s monitor | report_container_metrics | <prefix format> | true | If true, also collect metrics from containers whose logs are monitored. |
K8s monitor | report_k8s_metrics | <prefix format> | true | If true and report_container_metrics is also true, then collect Kubernetes system metrics. |
K8s monitor | k8s_ignore_pod_sandboxes | <prefix format> | true | If true, exclude logs from containers where label io.kubernetes.docker.type == podsandbox |
K8s monitor | k8s_include_all_containers | <prefix format> | true | Controls the default behavior for inclusion/exclusion of container logs. See documentation on Kubernetes Monitor annotations for more details. |
K8s events | check_labels | SCALYR_K8S_CHECK_LABELS | false | |
K8s events | log_flush_delay | SCALYR_K8S_LOG_FLUSH_DELAY | 1.0 | |
K8s events | leader_check_interval | SCALYR_K8S_LEADER_CHECK_INTERVAL | 60 | |
K8s events | leader_node | SCALYR_K8S_LEADER_NODE | None | |
K8s events | max_log_size | SCALYR_K8S_MAX_LOG_SIZE | None | |
K8s events | max_log_rotations | SCALYR_K8S_MAX_LOG_ROTATIONS | None | |
Docker monitor | container_globs_exclude | <prefix format> | None | A blacklist of container name glob patterns to exclude from monitoring. Any container whose name matches one of the glob patterns will not be monitored. If `None`, all container names matched by `container_globs` are monitored. This value is applied after `container_globs`. |
Docker monitor | container_check_interval | <prefix format> | 5 | How often (in seconds) to check for new containers. |
Docker monitor | report_container_metrics | <prefix format> | true | If true, metrics will be collected from the container and reported to Scalyr. |
Docker monitor | labels_as_attributes | <prefix format> | false | If true, the docker monitor will add any labels found on the container as log attributes, after applying label_include_globs and label_exclude_globs. |
Docker monitor | label_include_globs | <prefix format> | ['*'] | If labels_as_attributes is true then this option is a list of glob strings used to include labels that should be uploaded as log attributes. The docker monitor first gets all container labels that match any glob in this list and then filters out any labels that match label_exclude_globs, and the final list is then uploaded as log attributes. |
Docker monitor | label_exclude_globs | <prefix format> | ['com.scalyr.config.*'] | If labels_as_attributes is true, then this is a list of glob strings used to exclude labels from being uploaded as log attributes. Any label whose key matches any glob on this list will not be added as a log attribute. Note: the globs in this list are applied after label_include_globs. |
Rarely Modified
The following environment-aware variables are rarely modified and usually by power users or under guidance from Scalyr support.
Context | Config variable | Environment variable | Default value | Description |
---|---|---|---|---|
Global | allow_http | <prefix format> | false | |
Global | agent_log_path | <prefix format> | /var/log/scalyr-agent-2 | Agent log files dir |
Global | agent_data_path | <prefix format> | /var/lib/scalyr-agent-2 | Agent metadata dir (for checkpoints etc.) |
Global | config_directory | <prefix format> | agent.d | Agent config dir (located in same parent directory as agent.json file) |
Global | debug_init | <prefix format> | false | |
Global | additional_monitor_module_paths | <prefix format> | <empty string> | |
Global | compression_type | <prefix format> | bz2 | Compression type for data uploaded to Scalyr. One of bz2 or deflate (which uses zlib compress). |
Global | compression_level | <prefix format> | 9 | |
Global | check_remote_if_no_tty | <prefix format> | true | |
Global | close_old_files_duration_in_seconds | <prefix format> | 3600 | |
Global | copying_thread_profile_interval | <prefix format> | 0 | min_allowed_request_size |
Global | copying_thread_profile_output_path | <prefix format> | /tmp/copying_thread_profiles_ | |
Global | copy_staleness_threshold | <prefix format> | 900 | |
Global | full_checkpoint_interval_in_seconds | <prefix format> | 60 | |
Global | global_monitor_sample_interval | <prefix format> | 30 | |
Global | max_allowed_request_size | <prefix format> | 1024*1024 | |
Global | min_allowed_request_size | <prefix format> | 100*1024 | |
Global | min_request_spacing_interval | <prefix format> | 1 | |
Global | max_request_spacing_interval | <prefix format> | 5 | Max amount of time to wait between sending requests |
Global | max_error_request_spacing_interval | <prefix format> | 30 | |
Global | minimum_scan_interval | <prefix format> | None | |
Global | low_water_bytes_sent | <prefix format> | 20*1024 | |
Global | high_water_bytes_sent | <prefix format> | 100*1024 | |
Global | low_water_request_spacing_adjustment | <prefix format> | 1.5 | |
Global | high_water_request_spacing_adjustment | <prefix format> | 0.6 | |
Global | failure_request_spacing_adjustment | <prefix format> | 1.5 | |
Global | request_too_large_adjustment | <prefix format> | 0.5 | |
Global | log_deletion_delay | <prefix format> | 600 | Min seconds the agent waits for file to reappear on a file system after removal before it's considered deleted. |
Global | line_completion_wait_time | <prefix format> | 5 | Max number of seconds the agent waits before returning partial lines. (A partial line is the last log line in a file not ending in a new line.) |
Global | max_new_log_detection_time | <prefix format> | 60 | |
Global | max_log_offset_size | <prefix format> | 5*1024*1024 | How far to search back when getting a new log |
Global | max_existing_log_offset_size | <prefix format> | 100*1024*1024 | How far to search back when getting an existing log |
Global | max_sequence_number | <prefix format> | 1024^4 | |
Global | pipeline_threshold | <prefix format> | 1.1 | Size required for a message to trigger pipelining the next addEvents request, expressed as a multiple of max_allowed_request_size. |
Global | pidfile_advanced_reuse_guard | <prefix format> | false | |
Global | request_deadline | <prefix format> | 60.0 | |
Global | strip_domain_from_default_server_host | <prefix format> | false | |
Global | use_requests_lib | <prefix format> | false | |
Global | use_unsafe_debugging | <prefix format> | false | |
Global | enable_profiling | <prefix format> | false | If true, the agent will log performance profiling data about itself into a log file. |
Global | profile_log_name | <prefix format> | 'agent.callgrind' | Profile data log file name. |
Global | profile_duration_minutes | <prefix format> | 2 | Duration (in minutes) to gather profiling information. |
Global | max_profile_interval_minutes | <prefix format> | 60 | Frequency (in minutes) between consecutive runs of the profiler. |
Global | profile_clock | <prefix format> | 'random' | The clock used for measuring profiling performance. One of ['random', 'wall', 'cpu']. |
K8s monitor | docker_percpu_metrics | <prefix format> | false | If true, the agent emits CPU usage metrics per core. Note: this is disabled by default because it can lead to an excessive amount of metric data on CPUs with a large number of cores. |
K8s monitor | k8s_parse_format | <prefix format> | 'auto' | |
K8s monitor | k8s_always_use_cri | <prefix format> | false | |
K8s monitor | k8s_cri_query_filesystem | <prefix format> | false | |
K8s events | message_log | SCALYR_K8S_MESSAGE_LOG | kubernetes_events.log | |
K8s events | event_object_filter | SCALYR_K8S_EVENT_OBJECT_FILTER | 'CronJob', 'DaemonSet', 'Deployment', 'Job', 'Node', 'Pod', 'ReplicaSet', 'ReplicationController', 'StatefulSet' | |
K8s events | ignore_master | SCALYR_K8S_IGNORE_MASTER | true | |
Docker monitor | docker_percpu_metrics | <prefix format> | false | If true, the agent emits CPU usage metrics per core. Note: this is disabled by default because it can lead to an excessive amount of metric data on CPUs with a large number of cores. |
Docker monitor | log_mode | SCALYR_DOCKER_LOG_MODE | 'docker_api' | Determines which method is used to gather logs from local containers. If docker_api, then the agent will query container logs over the Docker api. If syslog, then the agent expects the other containers to push logs to this one using the Docker syslog logging driver. This option is to protect legacy behavior; the preferred method is to leave this value set to default (docker_api) and then set docker_raw_logs to true. |
Docker monitor | docker_api_version | <prefix format> | 'auto' | The version of the Docker API to use. WARNING, if you have log_mode set to syslog, you must also be running the syslog monitor and must set the docker_api_version configuration option in the syslog monitor to this same value. |
Docker monitor | docker_log_prefix | <prefix format> | 'docker' | Prefix added to the start of all docker log file names. Only applies if docker_raw_logs if false. |
Docker monitor | metrics_only | SCALYR_DOCKER_METRICS_ONLY | false | If true, the docker monitor will only log docker metrics and not any other logs from running containers. If set to true, this value overrides the config variable 'report_container_metrics'. |
Docker monitor | container_globs | <prefix format> | None | A whitelist of container name glob patterns to monitor. Only containers whose name matches one of the glob patterns will be monitored. If None, all container names are matched. This value is applied before container_globs_exclude. |
Docker monitor | label_prefix | <prefix format> | <empty string> | If labels_as_attributes is true, then append this prefix to the start of each label before adding it to the log attributes. |
Docker monitor | use_labels_for_log_config | <prefix format> | true | If true, the docker monitor will check each container for any labels that begin with com.scalyr.config.log. and use those labels (minus the prefix) as fields in the containers log_config. Keys that contain hyphens will automatically be converted to underscores. |