The Scalyr query language is used to select a set of events from your log data. It is used in data views, alert triggers, report definitions, and elsewhere. Examples:
All log messages containing the word "error":
Text containing spaces, digits, or punctuation must be enclosed in quotes:
Requests in the access log for "/index.html", for which at least 5000 bytes were returned:
$logfile contains 'access_log' uriPath = '/index.html' bytes >= 5000
Log messages containing the phrase "deadline exceeded", from servers tagged as part of a database tier:
"deadline exceeded" $serverTier = "database"
A query can contain any number of terms. To select events matching all of the terms (an "AND" query), you can simply enter the terms next to one another:
$logfile contains 'access_log' uriPath = '/index.html'
You can also use explicit AND, OR, and NOT keywords to combine terms:
$logfile contains 'access_log' and not (uriPath = '/home' or path = "/away")
The operators &&, ||, and ! can be used as synonyms for AND, OR, and NOT:
$logfile contains 'access_log' && !(uriPath = '/home' || uriPath = "/away")
To search for a word, simply type that word. That forms a search term, which can be combined with other search terms as described above. For example, to search for all events containing the word "hello" and at least one of "sir" or "madam":
hello (sir || madam)
To search for a more complex string, enclose it in single or double quotes:
"cache miss" '***critical error***'
You can also search using regular expressions. Enclose the expression in double quotes, preceeded by a $:
All of these terms search in the "message" field of an event. For logs uploaded by the Scalyr Agent, this field contains the complete text of the log message. However, you can also search in other fields. To perform a string match, use the "contains" keyword:
$logfile contains 'access_log'
For a regular expression match, use "matches":
uriPath matches '\\.png$'
All text search is case-insensitive.
The most powerful searches rely on event fields. (For a review of fields, refer back to the Getting Started page.) You can select events that do or do not have a particular value, using the == and != operators. = can be used as a synonym for ==:
uriPath = '/index.html' status == 404 client != "localhost"
You can also use the <, <=, >, and >= operators to compare values. For instance, this query matches all requests with a status in the range 400-499:
status >= 400 status <= 499
For field comparison operators, strings are treated as case sensitive.
When using a field which is associated with a server or log file, place a $ before the field name:
$serverHost = 'frontend-1'
For fields that come directly from the log event, the $ is optional.
If a field name contains spaces or punctuation, use a backslash to escape those characters. For instance, to display events with field service-name equal to "memcache":
service\-name == "memcache"
Finally, you can compare a field with the special value * to match events with any value in that field:
error == *
Scalyr Generated Fields
In addition to the fields from event parsing or server configuration, there are a few other special fields that Scalyr generates:
|k8s-cron-job||The CronJob Kubernetes workload resource that generated the log|
|k8s-daemon-set||The DaemonSet Kubernetes workload resource that generated the log|
|k8s-deployment||The Deployment Kubernetes workload resource that generated the log|
|k8s-job||The Job Kubernetes workload resource that generated the log|
|k8s-replica-set||The ReplicaSet Kubernetes workload resource that generated the log|
|k8s-replication-controller||The ReplicationController Kubernetes workload resource that generated the log|
|k8s-stateful-set||The StatefulSet Kubernetes workload resource that generated the log|
|sca:ingestTime||For events that arrive at Scalyr more than 30 minutes after their actual event time, we add the sca:ingestTime field which contains the time, in seconds-since-1970, at which the event was ingested (whereas the event's timestamp will reflect the actual event time, of course). This can help debug issues related to late-arriving data.|
Searching For Multiple Values
You can use the in operator to search for several values at once:
userid in ("aaa", "bbb", "ccc")
The in operator can be used to match numeric, text, or boolean (true / false) values, but can not match null / missing fields.
The graph view differs from the search view in that it displays numerical values rather than events. By default, the graph view will show a rate graph tracking the number of matching events per second over the search's time range. You can graph a different set of values by using a function that is applied either to the rate of matching events, or to the values of a particular field:
|count||The number of matching events|
|sum(value)||Sum of values. In a bar chart, the height of each bar corresponds to the sum of all values in the time period covered by that bar. In a line or area chart, if you select "Smooth curves", the time period you choose for smoothing determines the time period for sums. (For instance, if you smooth using 5 minute intervals, then each each point on the graph represents the sum of all values in a 5-minute period.) Otherwise, Scalyr automatically divides the time span of the graph into several hundred intervals and computes a sum for each interval, which is less useful. For line or area charts, you might consider `sumPerSecond` (next) as an alternative.|
|sumPerSecond(value)||Sum of values, divided by the number of seconds in the time period from which the values were obtained. For instance, if you have a field responseSize which records the number of bytes returned by some operation, then sumPerSecond(responsesize) will graph the bandwidth consumed by this operation, in bytes per second.|
|median(value)||The median (50th percentile) value.|
|p10(value)||The 10th percentile value.|
|p50(value)||The 50th percentile value.|
|p90(value)||The 90th percentile value.|
|p95(value)||The 95th percentile value.|
|p99(value)||The 99th percentile value.|
|p999(value)||The 99.9th percentile value.|
|p(value, n)||The Nth percentile value. For instance, p(value, 80) gives the 80th percentile.|
|fraction(expr)||The fraction (from 0 to 1) of events which match the given expression. For instance, fraction(status >= 500 status <= 599) is the fraction of requests which have a status in the 5xx range. You can use any query expression, as documented in the earlier sections of this page.|
Ratios and Complex Expressions
A graph can combine multiple numeric queries into a single expression. For example, this expression computes the ratio of log messages containing "error" to messages containing "success":
count(message contains "error") / count(message contains "success")
In this type of expression, you can use any number of numeric queries, combining them with the standard arithmetic operators +, -, *, and /, as well as parentheses. You can also use simple numbers. For instance, the following expression shows free disk space in gigabytes, converting from the kilobyte units reported by the Scalyr Agent:
mean(value where $serverHost='frontend-1' metric='df.1kblocks.free' mount='/') / (1024 * 1024)
The general form of a numeric query is as follows:
FUNCTION([FIELD where] FILTER)
FUNCTION is one of the functions listed above. FIELD is the name of a numeric event field; if omitted, "value" is used. FILTER is a query. Examples:
count(message contains "error")
The number of log messages that contain the word "error".
mean(bytes where uriPath == "/home")
The average number of bytes returned in requests for /home.
p(time where uriPath == "/home")
95th percentile latency of requests for /home.