Search

This section provides a detailed guide to Search view.

Scalyr ingests all your data, from system metrics to logs to monitoring probes. Each log message or monitoring report becomes an event, which is further parsed into fields.

You can search the entire text of the event or you can search by referencing fields. Hint: Search is faster when you reference fields.

For more elaborate queries, where you can pipe (multiple) search results into a set of commands for transforming, manipulating, grouping, and summarizing your data, see PowerQueries

(1) To search for a specific word or phrase, type it here. Numbers, punctuation, or phrases must be enclosed in quotes. For example, error, "503", and "customer 1309".

You can reference a field by just typing it, for example serverHost == 'appserver-4', or status>=500 status<=599.

As you type, your text is parsed and presented in an easy-to-read form. Fields, operators, and values are differentiated via highlighting. The Search Helper, an enhanced autocomplete feature, provides suggestions for field names, field values, functions, keywords, and operators.

See Query Language for a full description of the Scalyr query language.

(2) This is the time range of your search. By default, the last four hours are displayed. You can customize this default.

Click the button to change the time range:

You can select a preset to quickly search a range, or enter a Custom range via the Start and End boxes. Hint: it's faster to build and refine your query over a short time range, then apply it to the full range of interest.

You can enter a time (e.g. 14:30 or 5:05 AM), a date (May 23), or date and time (5/14/2016 2:00 PM), using a wide variety of formats. Shortcuts like 5d/ 5h/ 5m/ 5s indicate five days/hours/minutes/seconds. The End time assumes NOW, so entering 5m for the Start time and hitting Enter will search the last five minutes. Using the + shortcut for the End time, for example +24h or +1d, will search from the Start time to one-day later.

See the Date/Time Reference for a complete list of options.

(3) Use these fields to search a specific server or log file. If you're using Kubernetes these will allow you to search cluster and controller name, respectively. You can use * as a wildcard to represent zero or more characters, wherever they appear.

(4) The Field list displays all parsed fields in the events matching your search. They are alphabetically arranged in a scrollable window, with a Filter box for searching. Fields referenced by your query in (1) and in (3) are bold. As you move your cursor over the list the selected field is highlighted. You can also use the up/down arrows on your keyboard to navigate the list.

The number next to each field indicates how many distinct values appear in that field. (If there are more than a few hundred distinct values, the number shown will be an estimate.)

Click on a field to bring up a list of its most common values:

(4a) Depending on the type of data, various graphing options appear as buttons:

  • Graph Values graphs the selected field over time.
  • #Matches graphs matching events per second, broken down by the selected field.
  • Distribution graphs a distribution of the selected field.

For more information on Scalyr's graphing tools, see Graphs.

(4b) You can click on a value to restrict your events to only that value. You can also use the </ ==/ !=/ > operators to restrict events to values greater than, equal to, not equal to, or lesser than the value. This is often done as a precursor to graphing specific values of interest.

(4c) The bars provide a visual indication of how often each value appears, while the numbers provide more precision. Note that these are often estimates due to sampling (see (4d)).

(4d) Information concerning estimated values is located here. (We sample to achieve a statistically valid number, using a two-pass method. In the first pass (first bullet point), we query a 1% subsample to estimate the total number of matches. In the second pass (second bullet point), we sample the data with a rate based on information obtained in the first pass.)

(4e) Up to 30 distinct values are displayed in the scrollable window. Click "Check for More" to show up to a maximum of 200 values.

(5) The bar chart graphs the number of events matching your search, over the specified time range. You can use it to look for spikes in volume. The dotted line shows a smoothed distribution of actual log volume over the time period.

To the upper-left is information concerning the number of events matching your search, and the time span of each bar in the graph. To the upper-right is information concerning the time range of the graph /search, and the time zone.

As you move your cursor over the graph the info to the upper-right is replaced with point information for your selection. Clicking the selection will display your filtered events at that point in time (see (6)). You can also click and select a time range, and the graph/search will re-plot based on your selection.

Click Expand Graph to enlarge and enter Graph view, where you have access to the complete set of graphing tools.

(6) This area shows the events matching your search. You can scroll horizontally to view long messages, and vertically to move through your selected time range. To jump to a specific point in time, click the time you wish to go to in the bar chart (see (5)). You can also use the Start/End buttons (see (10)).

The bullet to the left of the event message is color-coded according to the severity field, a special field ranging from 0 to 6. For levels 0 through 3 it is light grey; for level 4 (warning) it is yellow; and for levels 5 (error) and 6 (critical or fatal) it is red.

Scalyr supports linking from recognized portions of log lines in the search results. For example, you may want to have the value of a field called userId link to that user's page in your company directory. See SmartLinks to enable this feature.

Selecting text will bring up additional options:

  • Issue a new search for your selection.
  • Add or Exclude your selection from the existing search.

(7) Clicking on a line brings up the Inspect Log Line pane to the right:

(7a) At the top of the pane are some options:

  • Click See In Original Log to view the raw log file where this message originated.
  • Click See In Thread Log to view log messages from the specific server thread that generated this message. (This works only for messages reported using Scalyr's Java API library.)
  • Click Edit Parser to manage the parsing rules used for this log file. See Log Parsers for more on parsing logs.

(7b) The full text of the message is shown here. Click the clipboard to copy the text.

(7c) If the parser was able to identify a timestamp in the message, that value is used. Otherwise, the timestamp is assigned according to the time that the message was received by Scalyr's servers.

(7d) All parsed fields are listed in the Inspect Fields list. Server-specific fields are grouped and listed below the others.

(8) The Display button opens a dialog allowing you to tailor the display of your events in (6):

You can choose whether to view search results as log lines, wrapped log lines, or in a table. If you choose to view them in a table, you will need to select which fields to show.

A set of checkboxes let you control what information is included before each log message in the Matching Events list:

  • Date: The date assigned to the event by Scalyr. If the parser identifies a timestamp in the event, that value is used. Otherwise, the time the event is received by Scalyr's servers is used.
  • Time: The time assigned to the event. Works like the Date field, described above.
  • Source/Cluster: The name of the server, k8s cluster, or other source from which the event originated.
  • Logfile/Deployment: The name of the log file or k8s deployment from which the event originated.
  • Raw log: The full, unparsed text of the event.

Use the Add and All buttons to move fields from the Available Fields list over to the Fields to Show list. The Up, Down and Remove buttons let you navigate and remove fields in the Fields to Show list.

If you make changes and click OK, those settings will remain in effect for the life of the current account session. Once you log out or change teams, the changes will be lost. Use the Save As Default button if you find yourself frequently applying the same changes across teams and you want the settings to persist. Default settings can then be temporarily overridden by a later change to the dialog or url parameters.

(9) The Live Tail button allows you to continuously view new messages that match your search. You can Pause and Resume the feed, and you can modify your query in the Search box. After an uninterrupted 10 minutes the feed will pause; a "Live Tail will pause" message appears 30 seconds beforehand.

Click the X to return to Search view.

(10) Clicking the Start and End buttons displays events at the beginning or end of your time range.

(11) Click the Save button to display the following Save actions for your current search:

  • Save Search: Opens a dialog box that lets you save the active query to either your personal or team's list of saved searches; your list is selected by default. Saved searches are available in the Search main navigation menu.
  • Save as Alert: Create a new alerting rule, which will trigger if the number of matches to your current search goes above or below a level you specify.
  • Save to Dashboard: Add this search to an existing dashboard, or start a new dashboard with this search.
  • Download: Download the current search results as a text file.
  • Export to S3: Save search results to an S3 bucket.

(12) Click the Share button to display the following Share actions for your current search:

  • Copy Link: Copies a link for this search to your clipboard with relative time references replaced by absolute (e.g., instead of the searching the previous hour, it would search 8 a.m. to 9 a.m.).
  • Save to Shared Searches: Opens a dialog box that lets you save the active search query to either your personal or team's list of saved searches; the team list is selected by default. Saved searches are available from the Search dropdown.