Scalyr ingests all your data, from system metrics to logs to monitoring probes. Each log message or monitoring report becomes an event, which is further parsed into fields.
Scalyr query language allows you to retrieve events that meet your search criteria. Expressions constructed in this language form the basis for Graphs, Dashboards, Alerts, and PowerQueries. For more elaborate queries, where you can pipe (multiple) search results into a set of commands for transforming, manipulating, grouping, and summarizing your data, see PowerQueries
All events uploaded by the Scalyr Agent have a message field, containing the unparsed text of the event. You can search the full text of the event, or you can search the fields parsed from the event. A search that does not reference a field implicitly searches the message field. For example, the following search:
is equivalent to the following:
message contains 'error'
Both of these queries will retrieve events containing the text "error" in the message field.
When searching for something other than a sequence of letters and numbers, such as text containing spaces, digits, or punctuation, enclose it in quotes. The contains and matches operators, and sometimes in, expect quoted terms:
"production-database" logfile contains 'access_log'
Searching a specific field is faster than searching the entire message, and allows you to match on specific parts of the message. You can also work with numeric values. For example, to retrieve requests in the access log for "/index.html", for which at least 5000 bytes were returned:
logfile contains 'access_log' uriPath = '/index.html' bytes >= 5000
This query searches events for which the logfile field contains 'access_log', the uriPath field equals '/index.html', and the bytes field is greater than or equal to 5000.
If a field name contains spaces or punctuation, use a backslash to escape those characters. For instance, to display events with field service-name equal to "memcache":
service\-name == "memcache"
Please note that prior to March 2021 searching fields associated with a server or logfile required a $ prefixed to the field name (e.g., $serverHost). This $ is no longer needed. Feel free to remove it from any saved queries, etc.
And, Or and Not
A query can contain any number of terms. To select events matching all of the terms (an "AND" query), you can simply enter the terms next to one another:
logfile contains 'access_log' uriPath = '/index.html'
You can also use explicit AND, OR, and NOT keywords to combine terms:
logfile contains 'access_log' and not (uriPath = '/home' or path = "/away")
The operators &&, ||, and ! can be used as synonyms for AND, OR, and NOT:
logfile contains 'access_log' && !(uriPath = '/home' || uriPath = "/away")
The contains operator retrieves events containing the search term, and has the following syntax:
fieldname contains 'search-term' (can use single or double quotes)
Multiple search-terms are supported:
fieldname contains ('search-term-1', 'search-term-2', 'search-term-3')
All text search is case-insensitive by default. For case sensitivity, append ":matchcase" to the operator:
fieldname contains:matchcase 'Search-term' fieldname contains:matchcase ('Search-term-1', 'search-term-2', 'Search-term-3')
Please note that contains only searches for text/string matches. The following search will retrieve all events containing the text '200' in the message field:
message contains '200'
Whereas the following search will fail because an httpStatus of "200" is parsed as a number:
httpStatus contains '200'
To search fields containing numeric values, use the comparison operators > / >= / == / <= / <, described below.
Regex: $ and Matches
You can also search text using regular expressions. To search in the message field of an event, you can use the shorthand $, followed by the expression in double quotes:
Alternatively you can use the matches operator to search a field for a regex match. matches has the following syntax:
fieldname matches 'regex' (can use single or double quotes)
Note that unlike the $"..." syntax explained above, double escaping regex elements is required when using matches. In fact, double escaping regex elements is required almost everywhere at Scalyr. See Regex for more information.
uriPath matches '\\.png$'
Multiple regex are supported:
message matches ( 'regex-1', 'regex-2', 'regex-3')
For case sensitivity, append ":matchcase" to the operator:
message matches:matchcase 'Regex-1', message matches:matchcase ( 'regex-1', 'Regex-2', 'regex-3')
Please note that matches only searches for text/string matches. To search fields containing numeric values, use the comparison operators > / >= / = / <= / <, described below.
Comparison Operators: > / >= / == / <= / <
You can select events that do or do not have a particular value, using the == and != operators. = can be used as a synonym for ==:
uriPath = '/index.html' status == 404 client != "localhost"
You can also use the <, <=, >, and >= operators to compare values. For instance, this query matches all requests with a status in the range 400-499:
status >= 400 status <= 499
For field comparison operators, strings are treated as case sensitive.
Finally, you can compare a field with the wildcard character * to match events with any value in that field:
error == *
The In Operator: Search for Multiple Values in a Field
You can use the in operator to search for several values in a field at once:
userid in ("aaa", "bbb", "ccc")
The in operator can be used to match numeric, text, or boolean (true / false) values, but can not match null / missing fields. Quote text values. Do not quote numeric and boolean values.
in is case-sensitive by default; use in:anycase for case-insensitive matches.
Functions Available in Graph View
Graph view differs from Search view in that it displays a graph of numerical values, rather than events matching your search. There are many ways to Access Graph view. The simplest is to click "Expand Graph" for the bar chart of matching events on the Search view page.
While in Graph view, you can enter Graph Functions into the Search box and visualize the results. You can even enter and visualize Complex Expressions, explained below. These aspects of Scalyr query language are utilized in Graphs, Dashboards, Alerts, and PowerQueries.
By default, Graph view will show a rate graph tracking the number of matching events per second over the search's time range. You can graph a different set of values by using a function that is applied either to the rate of matching events, or to the values of a particular field:
|count||The number of matching events|
|sum(value)||Sum of values. In a bar chart, the height of each bar corresponds to the sum of all values in the time period covered by that bar. In a line or area chart, if you select "Smooth curves", the time period you choose for smoothing determines the time period for sums. (For instance, if you smooth using 5 minute intervals, then each each point on the graph represents the sum of all values in a 5-minute period.) Otherwise, Scalyr automatically divides the time span of the graph into several hundred intervals and computes a sum for each interval, which is less useful. For line or area charts, you might consider `sumPerSecond` (next) as an alternative.|
|sumPerSecond(value)||The sum of all values per second. For instance, if you have a field responseSize which records the number of bytes returned by some operation, then sumPerSecond(responsesize) will graph the bandwidth consumed by this operation, in bytes per second. (We divide the time period of your graph into a number of time spans, sum all values per time span, and then divide by the time span in seconds to get an average sum per second, per time span.)|
|median(value)||The median (50th percentile) value.|
|p10(value)||The 10th percentile value.|
|p50(value)||The 50th percentile value.|
|p90(value)||The 90th percentile value.|
|p95(value)||The 95th percentile value.|
|p99(value)||The 99th percentile value.|
|p999(value)||The 99.9th percentile value.|
|p(value, n)||The Nth percentile value. For instance, p(value, 80) gives the 80th percentile.|
|fraction(expr)||The fraction (from 0 to 1) of events which match the given expression. For instance, fraction(status >= 500 status <= 599) is the fraction of requests which have a status in the 5xx range. You can use any query expression, as documented in the earlier sections of this page.|
Ratios and Complex Expressions
A graph can combine multiple numeric queries into a single expression. For example, this expression computes the ratio of log messages containing "error" to messages containing "success":
count(message contains "error") / count(message contains "success")
In this type of expression, you can use any number of numeric queries, combining them with the standard arithmetic operators +, -, *, and /, as well as parentheses. You can also use simple numbers. For instance, the following expression shows free disk space in gigabytes, converting from the kilobyte units reported by the Scalyr Agent:
mean(value where serverHost='frontend-1' metric='df.1kblocks.free' mount='/') / (1024 * 1024)
The general form of a numeric query is as follows:
FUNCTION([FIELD where] FILTER)
FUNCTION is one of the functions listed above. FIELD is the name of a numeric event field; if omitted, "value" is used. FILTER is a query. Examples:
count(message contains "error")
The number of log messages that contain the word "error".
mean(bytes where uriPath == "/home")
The average number of bytes returned in requests for /home.
p(time where uriPath == "/home")
95th percentile latency of requests for /home.
Scalyr Generated Fields
In addition to the fields from event parsing or server configuration, there are a few other special fields that Scalyr generates:
|k8s-cron-job||The CronJob Kubernetes workload resource that generated the log|
|k8s-daemon-set||The DaemonSet Kubernetes workload resource that generated the log|
|k8s-deployment||The Deployment Kubernetes workload resource that generated the log|
|k8s-job||The Job Kubernetes workload resource that generated the log|
|k8s-replica-set||The ReplicaSet Kubernetes workload resource that generated the log|
|k8s-replication-controller||The ReplicationController Kubernetes workload resource that generated the log|
|k8s-stateful-set||The StatefulSet Kubernetes workload resource that generated the log|
|message||All events uploaded by the Scalyr Agent have a message field, containing the unparsed text of the event.|
|metric||Scalyr places the values of built-in metrics in a value field, and the corresponding names in a metric field.|
|sca:ingestTime||For events that arrive at Scalyr more than 30 minutes after their actual event time, we add the sca:ingestTime field which contains the time, in seconds-since-1970, at which the event was ingested (whereas the event's timestamp will reflect the actual event time, of course). This can help debug issues related to late-arriving data.|
|value||Scalyr places the values of built-in metrics in a value field, and the corresponding names in a metric field.|