Skip to the page
KilbyDocs
Build and run View as Markdown
On this page

Log streams

Log streams forward the logs of your services to a provider you already use. They are not available yet on this Kilby: the dashboard offers the page, the API answers enabled: false and nothing is sent anywhere. This page describes how it works once it is turned on.

What you can send to#

ProviderWhat you enterWhat Kilby sends
DatadogDatadog site (required), API key (required), TagsJSON array to the log intake of your region, with the key in the DD-API-KEY header
HTTPS endpointURL (required), Token, Header name, Body, Compression, CertificatePOST of newline-delimited JSON (or a JSON array), optionally gzip
Syslog over TLSHost and port (required), Token, Facility, CertificateRFC 5424 messages with octet-counting framing over TLS
Grafana LokiPush URL (required), User name, Token or password, CertificateLoki push API, JSON, gzip; labels source, service, level and instance
Better StackIngesting URL (required), Source token (required)POST of a JSON array with a bearer token

Each log stream has a name, a provider, the services it sends (all of them unless you pick some) and the lowest level it sends. The level is guessed from each line, the same way the Logs tab does it. A workspace can have up to 3 log streams.

  • Datadog: choose the site your organisation is in (us1, us3, us5, eu1 and ap1) and paste an API key (not an application key). Lines arrive with the source kilby, the service name, the instance as the host and the level as the status. Tags you enter are added to every line.
  • HTTPS endpoint: a URL you control. Kilby sends a POST with one JSON object per line (Content-Type: application/x-ndjson), or one JSON array if you choose that. A token is sent as Authorization: Bearer ..., or in the header you name. Compression is optional.
  • Syslog over TLS: host:port of your collector. Kilby writes RFC 5424 messages, each framed with its length (RFC 5425), so a line with a line break stays one message. The facility is one of user, daemon, local0, local1, local2, local3, local4, local5, local6 and local7. A token is sent in the structured data of every message as [auth@32473 token="..."]. Syslog has no acknowledgement: a successful delivery means the TLS connection was verified and the bytes were written.
  • Grafana Loki: the push URL of your Loki. Kilby sends the push API's JSON, compressed, with the labels source, service, level and instance. With a user name the token is the password (basic authentication); without one it is a bearer token.
  • Better Stack: the ingesting URL of your source and its source token. Kilby sends a JSON array with dt and message and the same fields as the HTTPS endpoint.

A JSON line looks like this:

json
{"timestamp":"2026-11-14T10:00:00.123456789Z","service":"api","instance":"x2k4p","level":"error","message":"request failed path=/v1/chat err=\"context deadline exceeded\""}

Add one#

The dashboard page is at /dashboard/settings/log-streams, and the command line has kilby log-stream list, add, test and rm. Until log streams are available they say so and change nothing.

Credentials#

The API key or token is the only secret of a log stream. Kilby stores it encrypted and never shows it again: the dashboard and the API show only whether one is set and its last four characters. It is never in a page, an API answer, an email, the Activity page or Kilby's own logs. A credential has to be at least 12 characters, so that its last four characters do not give much of it away.

  • A URL cannot contain a user name, a password or a key in its query string. Put credentials in the token field.
  • The CLI never takes a token as an argument, because it would end up in your shell history: pipe it in with --token-stdin, set KILBY_LOG_STREAM_TOKEN, or type it when asked.
  • If you change where a credential is sent (the address, the Datadog site, the header or the Loki user), you have to enter it again. Nobody can read it back, so Kilby does not move a stored credential to a new address.
  • Deleting a log stream deletes its credential.

Where Kilby connects#

Kilby connects only to public addresses, over TLS (https://, or TLS for syslog). It checks the address a connection really goes to after the name is resolved, so a name that points to a private, link-local or internal address is refused, and so is one that is moved there later. Redirects are not followed. A syslog host that resolves to a private address is refused the same way. The same rules protect webhooks.

A destination with its own certificate can be set to not check the certificate. The connection stays encrypted, but Kilby cannot tell whether it reaches the right server. Use it only for a collector you run yourself.

Slow, down or too busy#

Forwarding never slows your services or the storage of their logs. Each log stream has its own small buffer in memory:

  • Lines go out in batches, every 2 seconds or sooner when a batch is full. Datadog, Loki and, if you choose it, an HTTPS endpoint get gzip.
  • A batch that fails with a network error, a 5xx, 408 or 429 is tried again with a growing pause and some randomness, and for as long as the provider's Retry-After says. Up to 6 tries.
  • Any other 4xx (a wrong key, a wrong address, a batch that is too large) is not retried: repeating it would only repeat the refusal.
  • When a destination is slow or down, its buffer fills (it holds 4 MB at most) and further lines are dropped. Dropped lines are counted, shown on the log stream's page, and announced inside the stream itself by a line that starts with kilby: and says how many were not forwarded.
  • Each log stream is also limited to 500 lines and 512 KB a second. Lines over a limit are dropped and counted the same way. The limits are per log stream, not per service.

If a log stream keeps failing, Kilby turns it off: after 10 failed batches in a row that went on for at least 0 minutes. The owners and admins of the workspace get an email, the page says Turned off and why, and nothing more is sent to it. Send a test line and turn it back on when the destination works again. A short outage never turns anything off.

A log stream shows one of four states: Healthy (the last delivery worked), Failing (deliveries fail and are being retried), Turned off (by you, or by Kilby after sustained failure) and Waiting for logs (on, nothing delivered yet).

What to expect#

  • Forwarding is best effort. Lines are held in memory for a moment before they are sent. A restart of Kilby, or a hand-over between its replicas, can repeat a few seconds of lines or lose the ones that were waiting. If you need every line, read them from the stored logs.
  • Forwarding does not depend on log storage limits. A service that prints more than storage keeps is still forwarded, up to the limits of the log stream.
  • A destination only ever receives the lines of its own workspace, and only of the services you chose.
  • Changes apply within about 15 seconds, not at once.

Who can do what#

ActionPermission
See each log stream's name, provider and stateview (every member)
See its address, services, level, last four characters of the credential, delivery state and counterssecrets (Developer, Admin, Owner)
Add, change, test, turn on or off and deleteworkspace (Admin, Owner); with the API, an admin key

Every change is in the workspace's Activity page, without any credential: only the name of the log stream and the names of the settings that changed.