Skip to the page
KilbyDocs
Help View as Markdown
On this page

Status page

The status page says whether Kilby is up. It is public: you do not need an account, and it says nothing about any customer or about how the platform is built.

  • /status: one sentence for the whole platform, a state and a 90-day bar for every component, open and recent incidents, and scheduled maintenance.
  • /status.json: the same data as JSON, for a script or an agent. Public, read-only and cached for 30 seconds. Any website may read it.

While an incident or a maintenance window is going on, a slim bar at the top of every signed-in dashboard page links to the status page. From 72 hours before a scheduled maintenance window, the bar announces it as Upcoming maintenance, with the start in your time zone and in UTC. Close it with the cross and it stays closed for that window, for you only; a window you have not closed shows again, and so does a new one. An incident or a window in progress takes the place of the announcement.

Components#

ComponentId in the JSONWhat it coversHow
Dashboard and APIdashboardThe web dashboard, the REST API and the command line tool.Measured every minute
Deploys and rolloutsdeploysCreating, updating and restarting services.Measured every minute
DatabasesdatabasesManaged Postgres.Measured every minute
Apps routingappsThe public gateway that carries traffic to services.Measured every minute
Email deliveryemailVerification, notification and alert emails.Measured every minute
BuildsbuildsBuilding services from a Git repository.Measured every minute

Each component has one of these states: operational, degraded, outage, maintenance, or unknown. unknown is shown as "No recent data": there is no recent measurement and nobody said anything by hand. It is never counted as up. A component that nothing measures is "Reported by hand": it shows Operational until an incident says otherwise, and the page says so.

How a state is decided#

Once a minute, one kilbyd replica measures each component and writes down what it found. A state changes only after 2 bad minutes in a row, and recovers only after 3 good minutes in a row, so one bad reading does not flap the page. These are the measurements and the values at which they change a state:

ComponentWhat is measuredDegradedOutage
Dashboard and APIA round trip to the platform's databasefrom 250 msfrom 2 s
Deploys and rolloutsTime since the engine last finished a pass over every servicefrom 90 secondsfrom 5 minutes
Deploys and rolloutsA round trip to the Kubernetes APIfrom 1.5 sfrom 8 s
DatabasesThe pods of the database operator that are not readyEvery pod not readyNever on its own
DatabasesThe share of managed databases that are failed (only with 5 or more databases)from 25%from 50%
Apps routingThe proxy pods of the public gateway that are not readyAny pod not readyEvery pod not ready
Email deliveryThe share of the last emails that failed (the last 30 minutes, only with 3 or more)from 20%from 50%
BuildsHow long the oldest build that could start has waitedfrom 10 minutesfrom 30 minutes

In order of precedence, a component's state is:

  1. A state a Kilby administrator pinned by hand, if there is one.
  2. Maintenance, while a maintenance window that covers the component is going on.
  3. Otherwise the worse of what was measured and the impact of the component's open incidents. An incident can make a component look worse, never better.

The state of the whole platform comes from its components: an outage anywhere makes it an outage, then degraded, then maintenance. A measurement that cannot be taken at all (the database does not answer, the cluster's API times out) is an outage by itself.

Uptime, and time nobody measured#

Under each component there are 90 cells, one per UTC day, oldest first, ending today. A day counts as operational only if it was measured and nothing bad happened. A day with an outage, a degraded day and a maintenance day are marked as such, and a day with no measurement, or fewer than half of its minutes measured, is grey: unknown, never operational.

The uptime figure is the share of the minutes Kilby measured in which the component was operational. Minutes in maintenance and minutes nobody measured are not counted on either side, the figure is cut rather than rounded (99.996 is 99.99, so a day with an outage never reads 100), and it is not shown at all until at least an hour has been measured.

A component with no sample in the last 5 minutes shows "No recent data".

Incidents and maintenance#

An incident has a title, a status (investigating, identified, monitoring, resolved), an impact (none, degraded or outage), the components it affects and a timeline of updates, newest first. Open incidents and the ones of the last 30 days are shown. A maintenance window has a start, an end and a state (scheduled, in_progress, completed). They are written by Kilby administrators in plain text. Kilby does not open incidents by itself: the page shows what the measurements say, and an administrator reports what happened.

What it cannot tell you#

  • The page is served by kilbyd itself, from the same database as the dashboard. If every kilbyd replica is down, or its database cannot be read, /status answers 503 with "We could not load the current status", and the JSON says unknown. It does not claim anything, but it is not an independent page: it cannot report an outage of the platform it runs on. An outage of the platform's own database cannot be recorded as an outage either, because the measurements are stored in it: it shows as a grey gap in the bar. For a status that survives an outage of Kilby, keep your own check, for example one that reads /status.json.
  • "Apps routing" is measured from the gateway's proxy pods only: no request is made through the public gateway. It says the gateway is up, not that one particular service is.
  • It is about the platform, not your service. Your service's own state is on its page. A component in "operational" does not mean your service answers.
  • Email delivery counts every failed send, including one to an address that does not exist, so it only judges once there are 3 or more emails in the last 30 minutes.
  • There are no email or RSS subscriptions to the page yet. Poll /status.json.

Use the JSON#

sh
curl -sS https://onkilby.com/status.json

The document has schema (1), generated_at, status (state and a one-sentence description), components, incidents and maintenance. Each component has id, name, description, state, monitoring (automatic, or manual for "Reported by hand"), uptime_90d (or null) and days (90 entries, oldest first, each with date, state, degraded_minutes, outage_minutes, maintenance_minutes and uptime). Each incident has id, title, status, impact, components, started_at, resolved_at (or null) and updates. Fields are only ever added within a schema version. The texts are plain text: escape them before you put them in a page.

A check an agent can run before it deploys:

sh
curl -sS https://onkilby.com/status.json | jq -r '.status.state'