Skip to the page
KilbyDocs
Help View as Markdown
On this page

Troubleshooting

When something fails, Kilby tries to say why in one sentence you can act on. The messages below are the ones it gives, grouped by what you see. Find yours, then follow the fix.

Where to look first: the service page shows the status and the message of the latest deploy; the Logs tab shows what your program printed; the Events tab lists every deploy with its reason. If a workspace has turned on Kilby AI, Analyse on a failed deploy explains it in more words (Kilby AI). If Kilby itself seems to be having trouble, look at /status.

The image cannot be pulled#

What Kilby saysWhat to do
Image not found: Docker Hub has no image acme/api:1.5. Check the name and the tag.Check the spelling of the image and the tag in Settings.
Private image: ghcr.io requires credentials to pull ghcr.io/acme/api:1.4. Add registry credentials for ghcr.io and select them in Settings.Create registry credentials (Registry credentials in the dashboard) for that registry and select them in the service's Settings.
ghcr.io rejected the registry credentials "my registry login" for ghcr.io/acme/api:1.4. Check the username and the token, and that the token can read this image.Check the username and the token, and that the token can read this image.
The image ghcr.io/acme/api:1.4 has no linux/amd64 version, which is what Kilby runs. Build it for linux/amd64, for example with docker buildx build --platform linux/amd64.Kilby runs linux/amd64. Build and push a linux/amd64 image. An Apple silicon laptop builds arm64 by default.
Docker Hub is limiting anonymous pulls right now. Add registry credentials for docker.io to pull with your own account, or deploy again later.Add credentials for the registry (a Docker Hub account raises the limit), or deploy again later.
Could not reach the registry ghcr.io: dial tcp: lookup ghcr.io: no such hostThe registry did not answer. Check its status, and deploy again.

Kilby checks the image when you type it and when you save the service (The image check), so most of these are caught before a deploy starts. Images must come from a registry on the internet. Private Amazon ECR is not supported yet; ECR Public works.

The service starts but never goes live#

What Kilby saysWhat to do
The service does not accept connections on port 8080. Make sure it listens on 0.0.0.0:8080 (the PORT variable), not only on localhost or another port.Listen on 0.0.0.0 and read the port from the PORT variable. A server bound to localhost or 127.0.0.1 cannot be reached.
No connections accepted on port 8080 yet. Loading a large model can take minutes; if it should be up by now, check that it listens on 0.0.0.0:8080 (the PORT variable).A model that loads for minutes is fine: Kilby waits up to 5 minutes. If it should be up by now, check the logs.
The health check at /healthz never succeeded: it answered HTTP 503. Check the logs, or the health check path in Settings.Open the path in the service and see what it answers. It must return a 2xx or 3xx status. Fix the path under Health check path, or fix the app.
The container was restarted because the health check at /healthz failed: it answered HTTP 500. Check the logs.The service worked and then stopped answering. Read the logs from before the restart.

The health check sends the service's public host name in the Host header. An app that only answers for other host names answers the check with an error.

The container crashes#

What Kilby saysWhat to do
The start command was not found in the image (exit code 127). Check the start command and that the image contains it.The command is not in the image. Check the spelling, and that it is on the PATH.
The start command could not be run: it is not executable (exit code 126). Check the start command.The file exists but cannot be run: it is not executable.
The container's command finished (exit code 0) instead of staying up. A service must keep running and listen on port 8080.A service has to stay up. A command that finishes is a job: use a cron job for it, or start a server.
The container's command finished (exit code 0) instead of staying up. A background worker must keep running; for work that runs and stops, use a cron job.Make the worker loop, or use a cron job.
The container was killed (exit code 137) shortly after starting. Check the logs.Something killed it. If it ran out of memory, the out-of-memory message below is shown instead.
The container crashed with a segmentation fault (exit code 139). Check the logs.A native crash in your program or one of its libraries.
The container exits right after starting (exit code 1). Check the logs.The logs have the error. The start command runs with /bin/sh -c.
The service ran out of memory: it used more than the 512 MB of memory of its instance type and was stopped. Choose a larger instance type.Choose a larger instance type under Settings, or lower the memory use.

A running service restarts#

A container of a service that is Live can die and be started again: it ran out of memory, crashed, or was killed. The service's status does not change when it happens, because the service kept answering or came back. Kilby looks at the containers of running services about once a minute and writes each restart down: on the Activity page and the service's Events tab as "Kilby recorded a restart of an instance of" the service, with the reason, and as a marker on the Metrics tab. It sends no email and no webhook for it.

The service's Overview and the services list add a muted line while there were restarts in the last 24 hours, for example Restarted 2 times in the last 24 hours. Last: ran out of memory (limit 2 GB).

What it saysWhat Kilby knowsWhat to do
Ran out of memory (limit 2 GB)The container's status says it was killed for running out of memory, and the limit it had.Choose a larger instance type in Settings. If the workspace's limits do not allow the next one, the line says to use less memory or to ask for a higher limit.
Was restarted because its health check failedThe service has a health check path, and the container was stopped from outside after it had been running for a while.Open the path and see what it answers, and read the logs from before the restart.
Crashed (exit code N)The container exited with a code other than 0, 137 and 143.The logs have the error.
Was killed (exit code 137 or 143)Something stopped the container from outside, and nothing in its status says why. Memory is possible, but Kilby says so only when the container's status does.Read the logs, and look at the memory chart on the Metrics tab.
Exited (exit code 0)The program finished, and the cluster started it again.A service has to stay up. Make it keep running, or use a cron job.

An event covers every restart since the last one, up to one event per service per minute, and says how many. Replacing instances in a deploy, rolling back, or stopping an instance because the service has fewer of them is not a restart. A restart is only known while the cluster still reports it: an instance that was deleted takes its history with it.

The pre-deploy command fails#

What Kilby saysWhat to do
Pre-deploy command failed (exit code N). The new version was not started and the running version was not changed.Read the command's output under the failed deploy, in Events or on the Overview card, fix the command or what it needs (the database address, a migration that cannot apply), and deploy again. What was running keeps running.
Pre-deploy command failed (exit code N). The new version was not started. No previous version was running.The same, but nothing of the service is running: this was its first deploy, it was resumed after being suspended, or no instance was up. Fix the command and deploy again, or redeploy with Skip the pre-deploy command if the version does not need it.
Pre-deploy command failed: it did not finish within N seconds.Make the command faster, or raise its time limit in Settings. The limit includes starting a machine and pulling the image.
Pre-deploy command could not start. (an image message)The same causes as The image cannot be pulled: the pre-deploy runs with the new image.
Pre-deploy command failed: the machine it ran on went away, twice.Deploy again. If it keeps happening, contact support.

The command may have run partly. Fix what it printed, and deploy again: it must be safe to repeat (Deploys). To deploy once without it, tick Skip the pre-deploy command next to Redeploy.

The deploy cannot start#

What Kilby saysWhat to do
This workspace has reached its resource limit. Suspend or delete another service and deploy again.The cluster refused the new instance because of the workspace quota. See Limits and quotas.
The new version did not become ready in time. Check the logs.Nothing clearer was found within 7 minutes. Read the logs.
Waiting for a machine to run on…Kilby is adding capacity. It can take a minute.
Pulling the image and starting the container…A large image takes a while to pull.
The disk could not be prepared for writing. Try a redeploy; if it keeps failing, contact support.A problem with the service's disk.
Network isolation for this service could not be confirmed yet; retrying.Kilby retries on its own.

Without a disk, the previous version keeps running while a deploy fails. With a disk it does not: the old instance was stopped before the new one started, so the service is not running until you roll back or deploy a fix. See Deploys.

Not a deploy problem#

  • I changed a variable and nothing changed. Saving the Environment tab starts a new deploy. Wait for it to be Live. A cron job reads variables at its next run.
  • A change to an image tag did not take effect. A node keeps an image it already has under the same tag. Use a new tag for every release.
  • The shell says it is not available. The service must be running, and your role must be Developer or higher. The image needs a shell.
  • A cron job did not run at the time I expected. The schedule is in UTC. If the previous run was still going, the new one was skipped.
  • The API answers 403. The key's role does not allow it, or the person who made the key lost access. See API keys and the API.
  • The API answers 422. A field is wrong or a limit was reached. The message says which.
  • A template did not deploy. It is refused whole, never half made. The message says why: a name already in use ("This workspace already has a service named <name>. Choose another name."), a setting that is not valid, a template that needs a GPU (not available yet), or a limit the template would go over. See Templates and Limits and quotas.
  • A variable from an environment group does not show up, or has the wrong value. The service restarts once when the group changes; wait for it to be Live. A variable defined on the service itself wins over the same name in a group, and between groups the first one listed wins. The Effective variables table on the Environment tab says where each value comes from. See Environment groups.
  • A domain is stuck on Pending DNS. See Custom domains: every message is there.
  • A database does not become available. Give it a minute. If its status message names a problem, read it on the database page.