Set up alerts
Alerts tell you when a cron keeps failing, when it recovers, and when a queue job fails. Every alert appears on the Alerts tab. Whether it also reaches Slack, Discord or email depends on where it comes from:
| Alert | Alerts tab | Slack or Discord | |
|---|---|---|---|
| Cron: warning, failure, recovery | Yes | Yes, if that’s the team channel | No, never |
| Queue job: warning, failure | Yes | Yes, if that’s the team channel | Yes, to every team member |
| Usage limit warning | Yes | No | No |
So if you leave the channel on Email, cron alerts only show up in the app. To get cron alerts outside Delay, pick Slack or Discord.
Choose the team channel
- Open Settings > Notifications (account menu > Settings). The setting applies to the team selected in the header, and only team admins can save it.
- Under Notification channel, pick Email, Slack or Discord.
- For Slack or Discord, paste the incoming webhook URL into Webhook URL (how to get one is below).
- Click Save Preferences.
- Click Send Test Notification. The button appears once a Slack or Discord URL is saved.
Good to know:
- A team has one channel and one webhook URL. For different channels, use separate teams.
- The saved URL is never shown again. Leave Webhook URL blank to keep it, or paste a new one to replace it.
- “Test notification sent successfully.” means Delay sent the message. It doesn’t check what Slack or Discord answered, so look in the channel.
- Alert posts to Slack and Discord are not signed.
Get a Slack webhook URL
- Go to api.slack.com/apps and create an app, or open one you already have.
- Turn on Incoming Webhooks.
- Click Add New Webhook to Workspace and pick the channel.
- Copy the webhook URL. It looks like
https://hooks.slack.com/services/....
Get a Discord webhook URL
- In your Discord server, open Server Settings > Integrations > Webhooks.
- Click New Webhook and pick the channel.
- Copy the webhook URL. It looks like
https://discord.com/api/webhooks/....
Turn on cron alerts
Cron alerts are off until you pick a threshold. On the cron’s form, set Failure alerting to After 1 consecutive failure, After 2 consecutive failures or After 3 consecutive failures, then save.
With the API, send alert_threshold (1, 2 or 3). Leave it out and alerting stays off:
curl -X POST https://app.delay.run/api/crons \ -H "Authorization: Bearer $DELAY_TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -d '{"name": "Health check", "url": "https://api.yourapp.com/health", "cron_expression": "*/5 * * * *", "alert_threshold": 2}'All cron fields: POST /api/crons.
Here is what a threshold of 2 does over a few runs:
| Run | Result | Alert |
|---|---|---|
| 1 | Fails | Warning: Cron job "Health check" has experienced a failure. |
| 2 | Fails | Failure: Cron job "Health check" has failed 2 consecutive time(s). |
| 3 | Fails | None |
| 4 | Succeeds | Recovery: Cron job "Health check" has recovered and is working again. |
Warning and Failure messages end with the failure reason, for example Reason: HTTP Server Error (5xx).
The rules:
- Threshold 1: the first failure raises a Failure alert straight away, with no warning.
- Threshold 2 or 3: the first failure raises a Warning, and reaching the threshold raises a Failure. Warnings go to Slack or Discord like failures do.
- One Failure per streak. After it, the cron shows “Alerted” on its page and further failures stay quiet until a run succeeds.
- Recovery only follows a Failure. If the cron succeeds before reaching the threshold, the count resets and no recovery alert is sent.
- Crons don’t retry, so each failed run counts once. A run fails on any non-2xx response (redirects included), a timeout, a connection error or a failed assertion. See failure reasons.
- Runs blocked by the monthly limit don’t raise alerts.
Queue job alerts
Queue alerts are on by default, for queues you create and for queues the API creates. To change it, open Queues, click the queue, click Edit, and tick or untick Enable email alerts for job failures. Despite the label, it controls all of the queue’s alerts: unticked, the queue raises none, not even on the Alerts tab.
For each job that fails:
- Warning when its first attempt fails and a retry is scheduled.
- Failure when it has no retries left and moves to dead letters. On a queue with retries turned off, the first failure is the last, so you only get this one.
Each alert is emailed to every member of the team and also posted to Slack or Discord if that’s the channel. There’s no threshold, no grouping and no recovery alert: ten failing jobs mean ten sets of alerts. The job ID in a failure alert is the last attempt’s ID, because each retry is a new job (see retries).
Usage limit warning
When the team has used 80% of its monthly requests (800 of 1,000 requests a month on the free plan) or 90% on a paid plan, Delay adds one Limit warning to the Alerts tab, once per calendar month. It isn’t emailed or posted anywhere, so check the Usage tab now and then. What happens at the limit: how usage is counted.
Mute alerts during maintenance
A maintenance window stops Slack, Discord and email notifications for a time range, for example during a deploy. Your crons and jobs keep running, and alerts are still recorded on the Alerts tab.
- Open the Maintenance tab and click New window.
- Under Scope, pick All Resources (Team-wide), Specific cron job or Specific queue. For a specific one, choose it under Resource.
- Optionally fill in Reason (optional), for example “Deploying v2.1.0”.
- Set Start and End. They are in your browser’s local time.
- Click Create Window.
- Windows can’t be edited and don’t repeat. To change one, delete it (trash icon, no confirmation) and create a new one.
- If a cron reaches its threshold during a window, that Failure alert is muted, and the cron still counts as alerted. You won’t get a Failure for that streak after the window ends, so check the Alerts tab after maintenance. When the cron succeeds again, the recovery alert is sent as usual (or muted, if the window is still open).
Read and manage alerts
The Alerts tab lists every alert for the team. Filter with the tabs (All, Failure, Warning, Recovery, Limit warning) or by failure reason. Mark single alerts as read, use Mark all read, or delete alerts you don’t need. The bell in the header shows a dot while you have unread alerts.
Avoid alert noise
- Use threshold 2 or 3 for endpoints that sometimes blip. Keep 1 for health checks where a single failure matters.
- Turn off alerts on busy queues where a failed job now and then is normal, since every failure emails the whole team. Check the Dead letters tab instead.
- Add success assertions to catch a
200that carries an error, instead of waiting for an outage. - Create a maintenance window before planned deploys.
- To send results to your own monitoring, use callbacks.
Next steps
- Debug a failed run: find out why a run failed.
- Runs and failure reasons: what each failure reason means.
- Troubleshooting: alerts that don’t arrive, and other problems.