Create an alarm

Alarms watch workspace metrics and change state when an expression is true for enough datapoints in an evaluation window. Use them for operational signals that need attention, such as high latency, missing throughput, queue growth, or error spikes.

An alarm belongs to the current workspace. It can use metrics from that workspace and notify workspace notification channels.

What alarms are for

Use alarms when a metric condition should produce an operational signal, not just a chart.

Good alarm rules are specific and actionable. For example:

• API latency is above a threshold for several datapoints.
• Error rate is higher than expected while traffic is present.
• Queue depth keeps growing.
• A periodic job stops reporting samples.

Dashboards help you inspect data. Alarms help you notice when the data needs action.

Create an alarm

Open Alarms from the app sidebar, then select New Alarm.

Enter a name that describes the condition, such as checkout-latency-high or worker-queue-depth. Alarm names are used in URLs and API calls, so prefer stable names that are easy to recognize.

Add metric variables

An alarm expression uses one or more variables. Choose a metric source or a numeric expression, as with dashboard chart variables.

For each metric variable:

• choose a variable name, such as var1, latency, or errors;
• select the metric to read;
• select the aggregation, such as avg, max, count, percentile(95), or trimmed_mean(10;10);
• set percentile or trimmed-mean bounds when the aggregation needs parameters;
• add dimension filters when the alarm should only evaluate a subset of the metric.

Use Add Variable when the expression needs multiple signals. For example, one variable can represent errors and another can represent requests.

Add expression variables

Choose Expression as the variable type for calculations. For example, define errorRate as errors / requests * 100, then use errorRate > 5 as the alarm condition. Define netIncoming as incoming - outgoing to compare netIncoming > 100.

Expression variables support numbers, named variables, +, -, *, /, unary minus, and parentheses. They may reference other expression variables, including variables listed later; circular dependencies are rejected. They also support diff and increase. Repeated unary minus (--a) is not supported. Multiplication and division take precedence over addition and subtraction.

Each metric variable uses its selected aggregation. Calculations combine values from the same time bucket. Missing operands, division by zero, and nonfinite intermediate results produce missing data, which feeds the alarm's configured missing-data handling. Values are not replaced with zero or borrowed from another bucket.

Use Show in preview to display the variables you want to inspect. You can hide errors and requests while showing errorRate; hidden inputs still participate in calculations. The preview plots the derived error rate with a threshold line at 5.

In API requests, variables have type: METRIC or type: EXPRESSION, and visible controls preview visibility. Expression variables specify expression; metric variables specify metricName, aggregation, and optional dimension filters. Omitted type means METRIC, and omitted visible means true, including existing alarms.

Write the condition

The final alarm condition supports comparisons between variables and numeric constants, combined with and, or, not, and parentheses. Signed numeric literals are allowed. Put arithmetic, functions, and variable negation in expression variables, not in the final condition.

latency > 500
errors > 10 and requests > 100
errorRate > 5
netIncoming > 100
temperature < -5

The UI validates the condition and draws threshold reference lines for variable-to-constant comparisons.

Use functions

A function takes an argument in parentheses and produces a numeric value for each resolution bucket. Write the function name followed by its argument, such as diff(A) or increase(A); spaces are allowed, including diff (A). Define an expression variable such as growth = increase(A), then use growth > 10 as the alarm condition.

Both functions compare adjacent buckets, not the first and last values of the whole evaluation window. At a one-minute resolution, they measure the change from the preceding minute. They do not convert that change to a per-second rate. The argument can be a metric variable or a numeric expression, such as A + B.

Define change = diff(A) and use change > 10 to alarm on the change from the immediately preceding resolution bucket, after the variable's selected aggregation. diff(A) is the current value minus the preceding value, including negative changes. If either bucket is missing, the difference is missing and the alarm uses its existing missing-data policy. Gaps are not skipped or replaced with zero.

You can use arithmetic and nested differences, such as diff(A + B) or diff(diff(A)) in expression variables. diff(5) is zero. Preceding buckets are fetched for the first evaluated point but do not count toward Datapoints to Alarm. The preview can show the expression variable and its threshold; the threshold is not drawn on its source metrics.

Use an expression variable such as growth = increase(A) with the condition growth > 10 for counters that can reset. It subtracts the preceding bucket from the current bucket after the selected aggregation. When the value decreases, it assumes the counter reset to zero and returns the current value. For example, a change from 100 to 4 returns 4, representing growth since the assumed reset.

Both functions take exactly one argument. Missing buckets stay missing. Arithmetic, nesting, and lookback work as for diff, including diff(increase(A)). Increase thresholds are not drawn on the source metrics in the preview.

For the same two adjacent values of A, the functions produce:

Choose diff when a decrease is a meaningful negative change. Choose increase when a decrease should be interpreted as a counter reset to zero.

Choose evaluation settings

The Evaluation Period is the time window, in minutes, used to decide the alarm state.

Datapoints to Alarm is how many datapoints inside that window must breach the expression before the alarm changes to an alarm state.

Use more than one datapoint for noisy metrics so a single spike does not page the team. Use a shorter period and fewer datapoints for conditions that need fast reaction.

The Resolution controls the metric rollup interval used by the alarm. Match it to the expected reporting frequency of the metric. A service that reports every minute usually works well with 1m; high-frequency metrics may use shorter resolutions.

See Metric rollups and resolution for how alarm buckets affect aggregations and sensitivity.

Treat missing data

Use Treat Missing Data As to decide what happens when the alarm cannot evaluate enough datapoints in the current window.

This can happen when a metric has not reported yet, when some buckets in the window are empty, or when one side of a variable-to-variable expression is missing data. For example, if Datapoints to Alarm is 5 but only 3 datapoints are present in the window, the alarm does not have enough data to make a normal OK or IN_ALARM decision.

Choose one of these options:

• Show insufficient data stores the alarm as INSUFFICIENT_DATA.
• Treat as breaching stores the alarm as IN_ALARM.
• Treat as not breaching stores the alarm as OK.

In the API, this setting is named treatMissingDataAs. The allowed values are MISSING, BREACHING, and NOT_BREACHING.

Notify channels

Use Notification Channels to choose which channels should receive alarm state changes.

The alarm form loads the available notification channels in the current workspace. Check each channel that should receive notifications. Each row shows the channel name, type, and target email or webhook URL.

If no channels are available, create one from Notification Channels first, then return to the alarm.

Create and verify notification channels before selecting them for an alarm. The form may list an unverified email channel, but the API rejects an alarm that references it.

Review and save

The preview uses the visible metric and expression variables and the last hour of data. Use it to check that the selected aggregation, filters, and threshold make sense before saving.

Select Create Alarm when the rule is ready. The app opens the alarm detail page after creation.

Manage alarms

Open Alarms to review configured alarms and their current status.

Open an alarm to inspect its configuration. Select Edit Alarm to change the rule, or Delete to remove it.

Permissions

Creating and managing alarms requires a role that allows the relevant alarm actions in the current workspace.

See Create and use roles for custom role policies, and Create an API token if automation should create or manage alarms through the API.