# Webhooks and notifications Kilby can tell you what happened in a workspace without you watching the dashboard: by **email** to the workspace's members, as messages in a **Slack channel**, and as **signed webhooks** to your own systems. All three are set up on the **Notifications** page. ## Events Webhooks, Slack channels and emails are about the same events. | Event | When it is sent | |---|---| | `deploy.started` | A deploy began rolling out. | | `deploy.succeeded` | A deploy is live. For a cron job: its new settings are in place. | | `deploy.failed` | A deploy did not go live. The event says why. | | `service.suspended` | A service or cron job was suspended, by someone in the workspace or by a platform administrator. | | `service.resumed` | A suspended service or cron job was started again. | | `cron.run.failed` | A run of a cron job exited with an error or ran too long. | | `database.available` | A database finished starting, or recovered, and accepts connections. | | `database.failed` | A database stopped being healthy and needs attention. | | `database.storage_grown` | Kilby made a database's disk bigger because it was nearly full. The event says from which size to which, and when the disk can grow again. | | `key_value.available` | A Key Value instance finished starting, or recovered, and accepts connections. | | `key_value.failed` | A Key Value instance stopped being healthy and needs attention. | | `log_stream.disabled` | A log stream was turned off automatically after its destination kept failing. Its address and credential are never in the event. | | `shell.opened` | Someone opened a shell into a running container of a service in this workspace. | Events about billing (`billing.invoice_ready`, `billing.payment_failed`, `billing.suspension_soon`, `billing.workspace_suspended`, `billing.workspace_resumed`, `billing.budget_alert`) exist too, and an endpoint can subscribe to them, but they are only sent once billing is live, which it is not yet. An event is delivered as JSON: ```json { "id": "b3a9c7e1-4d2f-4a85-8c6e-1f0d9b8a7c65", "type": "deploy.failed", "created_at": "2026-11-13T14:05:00Z", "workspace": {"id": "3d6f1c0e-5a9b-4c27-8e41-0b7a92d1f6c3", "name": "Acme"}, "data": { "service": { "id": "7cb800a8-7d0e-49ed-9df2-cb162d9fc2ea", "name": "my-api", "type": "web", "url": "https://my-api-x7k2.apps.kilby.neuramorphic.ai" }, "deploy": { "id": "5d1e2b7c-3a4f-4b8e-9c0d-1e2f3a4b5c6d", "title": "Instance type: Starter to Standard", "trigger": "update", "image": "ghcr.io/acme/api:1.5", "status": "failed", "reason": "probe", "message": "The service never accepted connections on port 8080.", "created_at": "2026-11-13T14:05:00Z", "finished_at": "2026-11-13T14:08:30Z" } } } ``` `type` is the event, `data.service` says which service, and `data.deploy` carries its `title` (what the deploy is called in a list: the commit, the image tag, what a settings change changed; see [Deploys](/docs/deploys.md#what-a-deploy-is-called)), the `reason` and a plain-English `message`. `data.service.disk` is `true` (and absent otherwise) for a service with a disk: its previous version is stopped before a new one starts, so after `deploy.failed` it is not running. The `id` is unique per event: use it to ignore a delivery you already handled. ## Email The Notifications page lists the events Kilby can email, and you tick the ones you want. By default Kilby emails every member of the workspace when a deploy fails, a cron job fails, a Kilby administrator suspends a service or a database becomes unavailable, and the owners and admins when someone opens a shell. A failing cron job sends at most one email an hour. ## Webhooks Add an endpoint on the Notifications page (or with `POST /api/v1/webhooks`): an `https` URL, an optional description and the events it receives. A workspace can have up to 5 endpoints. You need the `workspace` permission, which is an Admin or the Owner, or an Admin key. - Kilby sends a `POST` with the JSON body above and a `Kilby-Signature` header. The endpoint's **signing secret** (it starts with `whsec_`) is shown once when you create it. You can roll it later. - Answer with a 2xx status within 10 seconds. Anything else counts as a failure. - A failed delivery is tried up to 4 times in all: right away, then after 1, 5 and 30 minutes. A test event is tried once. - An endpoint that fails 20 attempts in a row is turned off, and says why. Fix it, send a test event and turn it back on. - The URL must be a public address. Kilby refuses to send to private or internal addresses (including names that resolve to one), does not follow redirects and reads only the start of the answer. - **Send test event** sends a `test` event to one endpoint. **Recent deliveries** show every attempt with the answer and how long it took. ### Check the signature Every delivery carries a header like this: ```text Kilby-Signature: t=1794524411,v1=aff0f5c2e1dfd0808335f42b64782bf5fe83b884517f2d73692d258345a8dfb6 ``` `t` is the time of the delivery in Unix seconds and `v1` is the hex HMAC-SHA256, with the signing secret as key, of the text `.`. To verify a delivery, compute it over the exact bytes you received, compare it in constant time, and reject a delivery whose `t` is more than a few minutes old so that a captured request cannot be replayed. The scheme is the same as Stripe's, so a library that checks Stripe signatures can be pointed at it. The API reference has the same steps with sample code in several languages. ## Slack A workspace can send the events above to a **Slack channel**: a message appears in the channel when a deploy fails, a cron job fails, a database stops being healthy or a shell is opened, and for any other event you choose. Set it up on the Notifications page under **Slack**, with the command line (`kilby slack add`) or with the API (`/api/v1/slack-destinations`). Kilby posts with an **incoming webhook**: a URL that Slack creates for one channel and that lets whoever holds it post there. It is the simple way. No Slack app is installed, there is no sign-in with Slack, and Kilby never reads anything from your Slack. A full Slack app (a Kilby bot you add to channels, with buttons that act from inside Slack) is a later step and does not exist yet. ### Connect a channel 1. In Slack, add the **Incoming Webhooks** app to your workspace and choose the channel the messages should go to. 2. Copy the webhook URL it shows you. It starts with `https://hooks.slack.com/services/`. 3. In Kilby, open Notifications, give the channel a name, paste the URL, choose the events and click **Connect channel**. Then click **Send a test message**. ```sh echo "$SLACK_WEBHOOK_URL" | kilby slack add deploys --channel "#deploys" --stdin kilby slack test deploys ``` The channel you type in Kilby is only a label for you: a message goes to the channel the webhook was created for. You need the `workspace` permission, which is an Admin or the Owner, or an Admin key. A workspace can have up to 5 channels. Everyone in the workspace can see which channels are connected and how they are doing; nobody can see the URL. The URL must be `https` and its host must be `hooks.slack.com`. This Kilby accepts no other host. Like every webhook, the URL has to point to a public address: Kilby refuses private and internal addresses (including names that resolve to one) and does not follow redirects. ### The URL is a secret Whoever has the webhook URL can post in your channel, so Kilby treats it like a password. It is checked, then encrypted before it is stored, and it is **never shown again**: not in the dashboard, not in the API, not in the activity log, not in an email and not in a log. After you save it, Kilby shows only its shape, `hooks.slack.com/services/T…/B…/****`. To use another webhook, paste the new URL; to stop using one, delete the channel in Kilby and remove the webhook in Slack. The CLI never takes the URL from a flag, which would leave it in your shell history: pipe it in with `--stdin`, set `KILBY_SLACK_WEBHOOK_URL`, or type it when asked. ### What a message holds Each message has a title (such as "Deploy failed: api"), one or two sentences on what happened, the name of the workspace, what the event is about, the time, and a **View in Kilby** button to the page that has the details. The sentences are the ones the email about the same event uses. For a failed deploy that includes what it means for the service: a service with a disk was stopped before the deploy started, so it is probably not running now, while a service without a disk keeps running its earlier deploy until a new deploy succeeds. A message never holds a secret, an environment variable, a line of a log or anything about your customers beyond names. Names are written so that they cannot change the message: Slack's special characters are escaped, so a workspace or service called `` is shown as text and notifies nobody. | Topic | Events | |---|---| | **Deploys** | Deploy started (`deploy.started`), Deploy succeeded (`deploy.succeeded`), Deploy failed (`deploy.failed`) | | **Services and cron jobs** | Service suspended (`service.suspended`), Service resumed (`service.resumed`), Cron run failed (`cron.run.failed`), Log stream turned off (`log_stream.disabled`) | | **Databases and Key Value** | Database available (`database.available`), Database failed (`database.failed`), Database disk grew (`database.storage_grown`), Key Value available (`key_value.available`), Key Value failed (`key_value.failed`) | | **Security** | Shell opened (`shell.opened`) | | **Billing** | Invoice ready (`billing.invoice_ready`), Payment failed (`billing.payment_failed`), Suspension soon (`billing.suspension_soon`), Workspace suspended (`billing.workspace_suspended`), Workspace resumed (`billing.workspace_resumed`), Budget alert (`billing.budget_alert`) | Billing events can be chosen too; like webhooks, they are only sent once billing is live, and they show invoice numbers and totals to everyone in the channel. ### Delivery - An event goes to the channels of the workspace it happened in that chose it, and to no other workspace's. - Messages to one channel are sent one at a time, a second apart, because Slack limits how fast a webhook may post. A cron job that keeps failing is announced once an hour per channel. - If Slack is busy (it answers 429 and says how long to wait, which Kilby honors) or does not answer, the message is tried up to 4 times in all: right away, then after 1, 5 and 30 minutes. An answer that says the webhook was removed, the channel is archived or the message is wrong is not tried again, because the same message would get the same answer. A test message is tried once and does not count against the channel. - A channel shows one of five states: **Healthy** (the last message got through), **Failing** (it did not, with the reason in Slack's words), **Disabled** (turned off, with the reason), **Sending** and **Not tested** (nothing was sent yet). - After 20 failed attempts in a row Kilby turns the channel off, says why on its page and emails the workspace's owners and admins. Fix the webhook, send a test message and turn it back on. ### An example message This is the exact message for a failed deploy. Paste it into Slack's Block Kit Builder to see it drawn. ```json { "text": "Deploy failed: my-api (Acme). The service never accepted connections on port 8080.", "blocks": [ { "type": "header", "text": { "type": "plain_text", "text": "Deploy failed: my-api", "emoji": false } }, { "type": "section", "text": { "type": "mrkdwn", "text": "The service never accepted connections on port 8080.\n\nIf an earlier deploy was live, it keeps running until a new deploy succeeds.", "verbatim": true } }, { "type": "context", "elements": [ { "type": "mrkdwn", "text": "Acme", "verbatim": true }, { "type": "mrkdwn", "text": "Image ghcr.io/acme/api:1.5", "verbatim": true }, { "type": "mrkdwn", "text": "", "verbatim": true } ] }, { "type": "actions", "elements": [ { "type": "button", "text": { "type": "plain_text", "text": "View in Kilby", "emoji": false }, "url": "https://onkilby.com/dashboard/services/7cb800a8-7d0e-49ed-9df2-cb162d9fc2ea/events", "action_id": "view_in_kilby" } ] } ] } ``` ## Deploy hooks (the other way round) A **deploy hook** is a secret URL that *you* call to start a deploy, for example from CI. It is not a webhook: nothing is sent to you. See [Deploys](/docs/deploys.md#deploy-from-ci-with-a-deploy-hook). ## Related - [Deploys, rollbacks and downtime](/docs/deploys.md): what a deploy event means. - [API keys and the API](/docs/api.md): manage endpoints and Slack channels over the API. - [CLI](/docs/cli.md): `kilby slack`. - [Troubleshooting](/docs/troubleshooting.md): the `reason` of a failed deploy. - [Logs, metrics and the shell](/docs/logs-metrics-shell.md): the shell-opened event.