On this page
Deploys, rollbacks and downtime
Every change to a running service is a deploy. For a service without a disk, Kilby starts the new version next to the old one, switches traffic when the new one is ready, and keeps the old one if the new one fails. A service with a disk cannot do that: it has a short downtime on every deploy, and a failed deploy leaves it not running (A service with a disk).
What starts a deploy#
| Trigger | When |
|---|---|
| First deploy | You create the service, by hand or from a template. |
| Settings change | You change the image, start command, port, health check, instance type, instances, disk or environment variables, or the variables or links of an environment group the service uses. |
| Manual deploy | You click Redeploy, or call POST /api/v1/services/<id>/deploys. |
| Resume | You resume a suspended service. |
| Rollback | You roll back to an earlier deploy. |
| Deploy hook | Your CI calls the service's deploy hook. |
A service built from a Git repository is deployed by its builds instead: each build that succeeds releases an image and starts a deploy, and the builds are listed under Deploys in the service. A push to a repository on GitHub builds each service whose folder the push changed. A public repository has no push webhook, so its builds start when you click Deploy latest commit, when your CI calls its build hook, or when the optional check for new commits finds one. See Deploy from Git.
The list is under Events in the service, and at GET /api/v1/services/<id>/deploys (the last 50). A deploy is queued, then deploying, then live or failed; while its pre-deploy command runs it is deploying. A deploy that a newer one replaced before it finished is superseded.
What a deploy is called#
Every deploy has a title, so a list says what happened without opening each deploy:
| Deploy | Title | Example |
|---|---|---|
| A deploy from Git | The first line of the commit message | Pin fastapi to 0.103 (#7) |
| An image | The image tag, or the start of its digest | v1.9, sha256:ab12cd34ef56… |
| A deploy hook | The tag it deployed, with the title you sent, if any | Fix login redirect (a1b2c3d) |
| A rollback | Rollback to the title of the deploy it went back to | Rollback to v1.8 |
| A settings change | What changed, never a value | Instance type: Starter to Standard, `` |
| A redeploy | Who started it | Manual redeploy by Ada Lovelace |
| A resume | Who started it | Resumed by Ada Lovelace |
The Overview, the Events tab, the page of a deploy (click its title: it shows the status, how long it took, what started it, the commit or the image, the instance type, and the variables Kilby set for that deploy), the REST API (title in each deploy), the deploy.started, deploy.succeeded and deploy.failed webhooks, the failed-deploy email and the MCP tool get_deploy all show the same one. A title is plain text of at most 120 characters. It is kept with the deploy when the deploy is queued, so it does not change later, and it never contains the value of a variable or the text of a start command: a settings change says Variables changed, not what they are.
How a rolling deploy works#
This is how a service without a disk deploys.
- Kilby starts one new instance with the new settings, next to the running ones. At most 1 extra instance runs at a time, and 0 instances are taken away before a new one is ready.
- The new instance must pass its check. Without a health check path, Kilby waits for your program to accept connections on its port (up to 5 minutes). With one, the path must answer 2xx or 3xx (up to 5 minutes). See Services.
- When the new instance is ready it gets traffic, and an old instance is removed from the service. The old instance waits 5 seconds after that, so a request that was already on its way to it is still answered. Then it receives
SIGTERMand has 30 seconds to finish what it is doing before it is killed. HandleSIGTERMin your program: a request that takes longer than that, or a connection your program keeps open, is cut when the instance is killed. - This repeats until every instance runs the new version. The deploy is then
live.
Without a disk, if the new version never becomes ready, the old one is still running and still serving, because it was not stopped. After 7 minutes without progress the deploy is marked failed, sooner when the cause is clear: a crash, running out of memory, an image that cannot be pulled. The service page says why.
The extra instance of a rolling deploy does not count against your workspace limits.
A worker that must run as a single copy#
During a rolling deploy the old and the new version run together for a few seconds. For most programs that is what you want. A background worker or a private service that must never run twice at once, such as a queue consumer that is not safe to run in parallel or a scheduler, can turn on Stop the old instance before starting the new one under Settings, in the Deploys row (or send "stop_first": true to PATCH /api/v1/services/{id}).
With it on, every deploy stops the old instances first and then starts the new ones, the way a service with a disk deploys: the two versions never overlap, but nothing runs in between, and if the new version fails to start, the service stays down until you roll back or deploy again. It is off by default, and web services always deploy the rolling way, so their visitors are not turned away.
A service with a disk#
A disk attaches to one machine at a time, so Kilby cannot start the new instance next to the old one. A service with a disk deploys with the strategy Recreate instead of RollingUpdate: the old instance receives SIGTERM and stops, the disk is released, and only then the new instance starts, attaches the disk and must pass its check.
- Every deploy has a short downtime: a few seconds, up to a minute when the disk moves to another machine. This is true of every change that restarts the service: new settings, environment variables, the variables of a group it uses, a redeploy, a rollback, a resume.
- A failed deploy leaves the service not running. The old version was stopped before the new one started, so nothing answers until you roll back to the last deploy that worked, or fix the problem and deploy again. The service page says "Not running" and the header badge says Failed. The one exception is a deploy that fails in its pre-deploy command: that runs before the old instance is stopped, so the service keeps answering on its old version, disk or not.
- A service with a disk runs one instance, so there is no second instance to keep it up.
If a short downtime is not acceptable, keep the files in a Postgres database instead of on a disk. See Disks and scaling.
Pre-deploy command#
A service can have a pre-deploy command: a command that runs once before each new version gets traffic, usually your database migrations (python manage.py migrate, bundle exec rails db:migrate, npx prisma migrate deploy, alembic upgrade head). Set it in Settings, under Run, when you create the service (under Advanced), with pre_deploy_command in the API, --pre-deploy in the CLI, or preDeployCommand in a blueprint. Cron jobs have none.
Before anything of the new version reaches the cluster, Kilby runs the command once, in a one-off container with:
- the new image and the new environment (the service's variables and those of its environment groups),
- the same instance type as the service, which counts against your workspace's resource limit while it runs, and
- the service's network identity and the same isolation as the service, so it can reach the workspace's databases and nothing else.
It runs with /bin/sh -c, so the image needs a shell. It cannot use the service's disk: a disk attaches to one machine, and the running instance holds it.
Kilby waits for it to exit:
- Exit code 0: the deploy goes on and the rollout starts as described above.
- Any other exit code, or no end within the time limit: the deploy fails with the reason
predeploy, before the new version starts. If it exits with an error or runs past the time limit, the deploy fails before the new version starts, and a version that is already running is not changed. This is also true of a service with a disk: its old instance stops only when the rollout begins, and a failed pre-deploy never gets there. The deploy shows Pre-deploy command failed, the exit code, and the last 200 lines the command printed (up to 32 KB). Kilby AI can explain it like any other failed deploy.
The time limit is 600 seconds unless you set another, from 10 to 1800 seconds. It counts from the start of the container, so it includes starting a machine and pulling the image.
The command may run more than once for the same version (a retry after Kilby restarts, a redeploy), so it must be safe to repeat: migrations that can run twice, CREATE ... IF NOT EXISTS, or a framework migrator that records what it ran.
The command runs for every deploy of a service that has one: the first deploy, a settings change, a manual redeploy, a deploy hook and a Git build. A rollback does not run it. A rollback goes back to a version that already ran, so a command that is broken now, or a migration the earlier code does not know, cannot block the way back. The deploy shows the stage as skipped with the reason "Rollback: the pre-deploy command is not run." If you need the command to run for the earlier version, redeploy after the rollback.
A resume runs it only for a version it has not run for. A resume starts the service again, usually with the version it ran before it was suspended. When the image and the pre-deploy command are exactly those of the last deploy whose command succeeded or was skipped, the command is not run again, and the deploy shows the stage as skipped with the reason "Resume: the pre-deploy command is not run. The image and the command are the same as in the last deploy whose command succeeded or was skipped." When either changed while the service was suspended (a new image, a Git build, a new command), or the command never succeeded for that version, it runs as for any deploy. An image tagged latest, or without a tag, always runs it: Kilby cannot know whether the tag points to the image that ran before. Changing only environment variables while the service is suspended does not run it; redeploy after the resume if it needs to run.
To redeploy once without it, tick Skip the pre-deploy command next to Redeploy (or send {"skip_pre_deploy": true} to POST /api/v1/services/<id>/deploys). You need the permission to deploy, the deploy shows the stage as skipped, and the Activity records who skipped it.
A newer deploy cancels the pre-deploy command of an older one that is still running, and a service never has two running at once. If kilbyd restarts while one runs, it carries on with the same container instead of starting another; a container lost with its machine is started once more, and no more than twice for a deploy.
What the command prints is stored with the deploy after Kilby removes the values of the service's variables that look like secrets, in the forms programs print them. That is a safety net, not a barrier: do not print secrets.
What the dashboard says#
The service page, the services list and the Overview say the same thing about a service, in the same words:
- Live (green): the latest deploy is running.
- Running (amber) with Latest deploy failed: the latest deploy failed, and the version before it still answers. This happens only to a service without a disk, because only a rolling deploy leaves the old version running.
- Failed (red): the latest deploy failed and nothing answers. For a service with a disk this is what every failed deploy looks like.
The status of the REST API is the stored status and does not change: failed means the latest deploy failed. Read instances_running with it: above 0 for a service without a disk means the previous version still serves.
With two or more instances and no disk, Kilby also keeps at least one instance up while a machine is replaced or maintained.
Tags: use a new one for every release#
A node does not pull an image again if it already has that tag, except for the tag latest (or no tag). Give every release its own tag, such as a version or a commit SHA, and deploy that tag. Then a redeploy, a rollback and a new node always run exactly what you meant.
When a deploy fails#
A failed deploy has a one-sentence reason, shown on the service page, in the deploy list and in the deploy.failed webhook. The reason codes are:
| Reason | Meaning | What to change |
|---|---|---|
image | The image could not be pulled: no such name or tag, a private image without credentials, rejected credentials, no linux/amd64 build, or a rate limit. | Check the image and tag, add or fix registry credentials, build for linux/amd64. |
probe | The service never accepted connections on its port, or never passed its health check. | Listen on 0.0.0.0 and on the PORT variable, check the health check path. |
crash | The container keeps exiting. The reason names the exit code. | Read the logs. Check the start command: exit code 127 means the command was not found, 126 that it is not executable. |
oom | The service used more memory than its instance type has and was stopped. | Choose a larger instance type, or lower the memory use. |
quota | The workspace's limits would be exceeded. | Suspend or delete something else, or choose a smaller instance type. |
timeout | The new version was not ready in time and nothing clearer was found. | Read the logs and the health check path. |
predeploy | The pre-deploy command exited with an error, ran past its time limit, could not start (its image could not be pulled, or the workspace had no room for its container), or its container was lost twice. The new version never started. | Read the command's output on the deploy, fix the command or what it needs (the database address, a missing migration), and deploy again. |
More on each is in Troubleshooting. A failed deploy keeps the last lines the service printed. Kilby AI, when a workspace has turned it on, can explain a failed deploy in more words (Kilby AI).
Roll back#
Open the service's More actions menu and choose Roll back to the previous deploy, or choose Roll back next to any earlier successful deploy in the Events tab. Either one opens a page that shows what the rollback would change before anything happens:
- Every setting that differs: the image (the exact one it will run), the start command, the port or schedule, the instance type and the health check path. The number of instances is kept. A disk is never changed or restored, because it holds your data: the earlier version starts with the files that are on it now.
- The environment variables that would be added, removed or changed, by name. Values are hidden. A role that can see secrets can choose Show values to see the current and the restored value side by side.
- What else happens: that the service restarts, that a service with a disk is down for a short time while it does, and what a deploy hook does afterwards.
Confirm with Roll back to this deploy. A rollback is a new deploy that runs what the earlier one ran. Only a deploy that reached live can be restored, and the latest deploy cannot be, because the service is already set to it. If the service changed while the page was open, Kilby asks you to look at the changes again instead of applying what you saw.
Variables#
Each deploy remembers the environment variables it ran with, sealed like the service's own variables and never shown in clear. Kilby keeps them for the last 10 deploys of a service and deletes them with the service. On the preview page, Also restore the variables of this deploy is off until you tick it, and it appears only when the variables differ (when they are the same, the page says so and there is nothing to choose). Rolling the code back should not silently put old variables back: a service that gained a required variable since would break. Next to the box a line counts what differs, by name only and never a value, for example "3 variables differ: 1 added since, 1 removed since, 1 changed". "Added since" means set now and not set by that deploy, so restoring removes it; "removed since" means that deploy had it and the service no longer does, so restoring brings it back.
- On: the variables become the service's saved variables in the same new deploy, together with the image. They stay that way. A later redeploy, a call to the deploy hook with a newer image, or a change of settings uses them too, and nothing goes back to the newer values unless you save them again on the Environment tab. The variables you had are not lost: the deploy that used them keeps its copy.
- Off: the variables saved now stay, and the earlier version runs with them.
- Secrets and generated values are variables like any other: Kilby keeps every variable of a service in one sealed set and marks none of them as secret or generated. With the box on they are restored with the rest, so one you changed since goes back to its old value, which may no longer be valid where it came from (a rotated key, for example), and one added since is removed.
- A deploy from before Kilby recorded variables, or one older than the last deploys kept, did not record them. The page says so, and the variables stay as they are now.
- Variables of environment groups are not part of the copy: the service keeps receiving the values its groups have now.
- If the copy cannot be read, nothing changes at all, and the page says so. You can still roll back without the variables.
Rolling back does not turn off the deploy hook. A deploy it starts later brings the image it names and runs it with the variables that are saved at that time.
With the API, preview a rollback first. The answer lists the names of the variables that would change and never a value:
curl -sS https://onkilby.com/api/v1/services/<id>/deploys/<deploy>/rollback-preview \
-H "Authorization: Bearer $KILBY_API_KEY"
curl -sS -X POST https://onkilby.com/api/v1/services/<id>/rollback \
-H "Authorization: Bearer $KILBY_API_KEY" -H "Content-Type: application/json" \
-d '{"deploy_id": "<id of an earlier live deploy>", "restore_env": true}'Unlike the page, a call that leaves restore_env out restores the variables: it is true when the deploy recorded them and false when it did not. Send false to keep the variables saved now. Both calls need a key that can deploy. The same is available from the command line:
kilby rollback api --preview
kilby rollback api --env=restore --yesDeploy from CI with a deploy hook#
A service's Settings tab can create a deploy hook: a secret URL that deploys the service when it receives a POST request. Store it as a secret in your CI and call it after you push an image:
- name: Deploy
run: curl -fsS -X POST "${{ secrets.KILBY_DEPLOY_HOOK }}?image=ghcr.io/acme/api:${{ github.sha }}"With ?image= the hook deploys a new tag or digest of the service's own image and nothing else, so a leaked URL cannot run someone else's image. Without it, the hook redeploys the current image. Add title to say what is being deployed, such as the commit's subject: as a query parameter (?image=ghcr.io/acme/api:3f2a9c1&title=Fix+login+redirect) or as the title field of a JSON body ({"title": "Fix login redirect"}). It is plain text of at most 120 characters, only its first line is kept, and the deploy is listed as the title followed by the tag, for example Fix login redirect (3f2a9c1). A longer title is refused with 422 and nothing is deployed. A hook is limited to 10 deploys a minute. The answer is 202 with the deploy. Regenerate or revoke the hook in Settings and the old URL stops working at once. The hook works as long as the person who created it can still deploy in the workspace.
Related#
- Services: ports and health checks.
- Troubleshooting: the failures Kilby recognises, and how to fix each.
- Webhooks and notifications: get told when a deploy fails.
- Disks and scaling: instances, autoscaling and disks.