# Troubleshooting When something fails, Kilby tries to say why in one sentence you can act on. The messages below are the ones it gives, grouped by what you see. Find yours, then follow the fix. Where to look first: the service page shows the status and the message of the latest deploy; the **Logs** tab shows what your program printed; the **Events** tab lists every deploy with its reason. If a workspace has turned on Kilby AI, **Analyse** on a failed deploy explains it in more words ([Kilby AI](/docs/kilby-ai.md)). If Kilby itself seems to be having trouble, look at [/status](/status). ## The image cannot be pulled | What Kilby says | What to do | |---|---| | Image not found: Docker Hub has no image acme/api:1.5. Check the name and the tag. | Check the spelling of the image and the tag in **Settings**. | | Private image: ghcr.io requires credentials to pull ghcr.io/acme/api:1.4. Add registry credentials for ghcr.io and select them in Settings. | Create registry credentials (**Registry credentials** in the dashboard) for that registry and select them in the service's Settings. | | ghcr.io rejected the registry credentials "my registry login" for ghcr.io/acme/api:1.4. Check the username and the token, and that the token can read this image. | Check the username and the token, and that the token can read this image. | | The image ghcr.io/acme/api:1.4 has no linux/amd64 version, which is what Kilby runs. Build it for linux/amd64, for example with docker buildx build --platform linux/amd64. | Kilby runs `linux/amd64`. Build and push a `linux/amd64` image. An Apple silicon laptop builds `arm64` by default. | | Docker Hub is limiting anonymous pulls right now. Add registry credentials for docker.io to pull with your own account, or deploy again later. | Add credentials for the registry (a Docker Hub account raises the limit), or deploy again later. | | Could not reach the registry ghcr.io: dial tcp: lookup ghcr.io: no such host | The registry did not answer. Check its status, and deploy again. | Kilby checks the image when you type it and when you save the service ([The image check](/docs/services.md)), so most of these are caught before a deploy starts. Images must come from a registry on the internet. Private Amazon ECR is not supported yet; ECR Public works. ## The service starts but never goes live | What Kilby says | What to do | |---|---| | The service does not accept connections on port 8080. Make sure it listens on 0.0.0.0:8080 (the PORT variable), not only on localhost or another port. | Listen on `0.0.0.0` and read the port from the `PORT` variable. A server bound to `localhost` or `127.0.0.1` cannot be reached. | | No connections accepted on port 8080 yet. Loading a large model can take minutes; if it should be up by now, check that it listens on 0.0.0.0:8080 (the PORT variable). | A model that loads for minutes is fine: Kilby waits up to 5 minutes. If it should be up by now, check the logs. | | The health check at /healthz never succeeded: it answered HTTP 503. Check the logs, or the health check path in Settings. | Open the path in the service and see what it answers. It must return a 2xx or 3xx status. Fix the path under **Health check path**, or fix the app. | | The container was restarted because the health check at /healthz failed: it answered HTTP 500. Check the logs. | The service worked and then stopped answering. Read the logs from before the restart. | The health check sends the service's public host name in the `Host` header. An app that only answers for other host names answers the check with an error. ## The container crashes | What Kilby says | What to do | |---|---| | The start command was not found in the image (exit code 127). Check the start command and that the image contains it. | The command is not in the image. Check the spelling, and that it is on the `PATH`. | | The start command could not be run: it is not executable (exit code 126). Check the start command. | The file exists but cannot be run: it is not executable. | | The container's command finished (exit code 0) instead of staying up. A service must keep running and listen on port 8080. | A service has to stay up. A command that finishes is a job: use a cron job for it, or start a server. | | The container's command finished (exit code 0) instead of staying up. A background worker must keep running; for work that runs and stops, use a cron job. | Make the worker loop, or use a cron job. | | The container was killed (exit code 137) shortly after starting. Check the logs. | Something killed it. If it ran out of memory, the out-of-memory message below is shown instead. | | The container crashed with a segmentation fault (exit code 139). Check the logs. | A native crash in your program or one of its libraries. | | The container exits right after starting (exit code 1). Check the logs. | The logs have the error. The start command runs with `/bin/sh -c`. | | The service ran out of memory: it used more than the 512 MB of memory of its instance type and was stopped. Choose a larger instance type. | Choose a larger instance type under **Settings**, or lower the memory use. | ## A running service restarts A container of a service that is Live can die and be started again: it ran out of memory, crashed, or was killed. The service's status does not change when it happens, because the service kept answering or came back. Kilby looks at the containers of running services about once a minute and writes each restart down: on the **Activity** page and the service's **Events** tab as "Kilby recorded a restart of an instance of" the service, with the reason, and as a marker on the **Metrics** tab. It sends no email and no webhook for it. The service's **Overview** and the services list add a muted line while there were restarts in the last 24 hours, for example *Restarted 2 times in the last 24 hours. Last: ran out of memory (limit 2 GB).* | What it says | What Kilby knows | What to do | |---|---|---| | Ran out of memory (limit 2 GB) | The container's status says it was killed for running out of memory, and the limit it had. | Choose a larger instance type in **Settings**. If the workspace's limits do not allow the next one, the line says to use less memory or to ask for a higher limit. | | Was restarted because its health check failed | The service has a health check path, and the container was stopped from outside after it had been running for a while. | Open the path and see what it answers, and read the logs from before the restart. | | Crashed (exit code N) | The container exited with a code other than 0, 137 and 143. | The logs have the error. | | Was killed (exit code 137 or 143) | Something stopped the container from outside, and nothing in its status says why. Memory is possible, but Kilby says so only when the container's status does. | Read the logs, and look at the memory chart on the **Metrics** tab. | | Exited (exit code 0) | The program finished, and the cluster started it again. | A service has to stay up. Make it keep running, or use a cron job. | An event covers every restart since the last one, up to one event per service per minute, and says how many. Replacing instances in a deploy, rolling back, or stopping an instance because the service has fewer of them is not a restart. A restart is only known while the cluster still reports it: an instance that was deleted takes its history with it. ## The pre-deploy command fails | What Kilby says | What to do | |---|---| | Pre-deploy command failed (exit code N). The new version was not started and the running version was not changed. | Read the command's output under the failed deploy, in **Events** or on the Overview card, fix the command or what it needs (the database address, a migration that cannot apply), and deploy again. What was running keeps running. | | Pre-deploy command failed (exit code N). The new version was not started. No previous version was running. | The same, but nothing of the service is running: this was its first deploy, it was resumed after being suspended, or no instance was up. Fix the command and deploy again, or redeploy with **Skip the pre-deploy command** if the version does not need it. | | Pre-deploy command failed: it did not finish within N seconds. | Make the command faster, or raise its time limit in **Settings**. The limit includes starting a machine and pulling the image. | | Pre-deploy command could not start. (an image message) | The same causes as [The image cannot be pulled](#the-image-cannot-be-pulled): the pre-deploy runs with the new image. | | Pre-deploy command failed: the machine it ran on went away, twice. | Deploy again. If it keeps happening, contact support. | The command may have run partly. Fix what it printed, and deploy again: it must be safe to repeat ([Deploys](/docs/deploys.md#pre-deploy-command)). To deploy once without it, tick **Skip the pre-deploy command** next to **Redeploy**. ## The deploy cannot start | What Kilby says | What to do | |---|---| | This workspace has reached its resource limit. Suspend or delete another service and deploy again. | The cluster refused the new instance because of the workspace quota. See [Limits and quotas](/docs/limits.md). | | The new version did not become ready in time. Check the logs. | Nothing clearer was found within 7 minutes. Read the logs. | | Waiting for a machine to run on… | Kilby is adding capacity. It can take a minute. | | Pulling the image and starting the container… | A large image takes a while to pull. | | The disk could not be prepared for writing. Try a redeploy; if it keeps failing, contact support. | A problem with the service's disk. | | Network isolation for this service could not be confirmed yet; retrying. | Kilby retries on its own. | Without a disk, the previous version keeps running while a deploy fails. With a disk it does not: the old instance was stopped before the new one started, so the service is not running until you roll back or deploy a fix. See [Deploys](/docs/deploys.md). ## Not a deploy problem - **I changed a variable and nothing changed.** Saving the Environment tab starts a new deploy. Wait for it to be Live. A cron job reads variables at its next run. - **A change to an image tag did not take effect.** A node keeps an image it already has under the same tag. Use a new tag for every release. - **The shell says it is not available.** The service must be running, and your role must be Developer or higher. The image needs a shell. - **A cron job did not run at the time I expected.** The schedule is in UTC. If the previous run was still going, the new one was skipped. - **The API answers 403.** The key's role does not allow it, or the person who made the key lost access. See [API keys and the API](/docs/api.md). - **The API answers 422.** A field is wrong or a limit was reached. The `message` says which. - **A template did not deploy.** It is refused whole, never half made. The message says why: a name already in use ("This workspace already has a service named . Choose another name."), a setting that is not valid, a template that needs a GPU (not available yet), or a limit the template would go over. See [Templates](/docs/templates.md) and [Limits and quotas](/docs/limits.md). - **A variable from an environment group does not show up, or has the wrong value.** The service restarts once when the group changes; wait for it to be Live. A variable defined on the service itself wins over the same name in a group, and between groups the first one listed wins. The **Effective variables** table on the Environment tab says where each value comes from. See [Environment groups](/docs/environment-groups.md). - **A domain is stuck on Pending DNS.** See [Custom domains](/docs/domains.md): every message is there. - **A database does not become available.** Give it a minute. If its status message names a problem, read it on the database page. ## Related - [Deploys, rollbacks and downtime](/docs/deploys.md): how a deploy runs and how to roll back. - [Logs, metrics and the shell](/docs/logs-metrics-shell.md): where to read what happened. - [Custom domains](/docs/domains.md): DNS and certificate messages. - [Limits and quotas](/docs/limits.md): the limits a change can hit. - [Status page](/docs/status.md): whether the platform itself is having trouble.