Skip to the page
KilbyDocs
Build and run View as Markdown
On this page

AI Gateway

The AI Gateway is not available yet on this Kilby. This page describes how it works once it is.

The AI Gateway is an OpenAI-compatible API in front of AI models. Your app sends chat requests with a gateway key; Kilby checks the key's limits, sends the request to a model, meters the tokens and the cost, and answers in the same format as OpenAI's Chat Completions API. Any OpenAI SDK works by changing two settings: the base URL and the key.

Model names#

You call Kilby's own names, not a vendor's:

NameFor
kilby-autoKilby's router reads each prompt and sends it to a fast, a balanced or the strongest model.
kilby-fastFast and inexpensive: classification, extraction, short answers, chat.
kilby-smartMore capable: code, longer reasoning, writing.

GET https://<the gateway's address>/v1/models lists them with their price per million input and output tokens. You pay the price of the model that answered. If a model is busy or down, the gateway retries it and, when the name has another model of the same kind, tries that one; it changes model only before the first word of an answer is sent.

Keys#

In the dashboard, AI Gateway → Create key. Admins and owners create and revoke keys. Each key:

  • starts with kai_, is shown once, and is stored only as a fingerprint;
  • can only call models: it cannot read or change anything else in the workspace, so it can live in an app's environment variables;
  • has its own limits: requests a minute, tokens a day (optional), dollars a month (optional), and the model names it may use (optional).

Create one key per app or per customer of yours, so each has its own limits and its own line in the usage.

Limits and the monthly cap#

Every request is checked before it reaches a model, against the key's limits and the workspace's monthly cap: the most all its keys may spend in a calendar month (UTC). A request that would go past a limit gets a 429 that names it.

PlanMonthly capMost an admin can set
Hobby$1 (a trial amount)$1
Pro$25$500
Scale$250$5000

The cap is set when the workspace creates its first key; admins change it on the AI Gateway page.

Quickstart#

python
from openai import OpenAI

client = OpenAI(base_url="https://<the gateway's address>/v1", api_key="kai_…")
reply = client.chat.completions.create(
    model="kilby-auto",
    messages=[{"role": "user", "content": "Write a haiku about deploys."}],
)
print(reply.choices[0].message.content)
sh
curl https://<the gateway's address>/v1/chat/completions \
  -H "Authorization: Bearer kai_…" \
  -H "Content-Type: application/json" \
  -d '{"model": "kilby-fast", "messages": [{"role": "user", "content": "Hello"}]}'

Streaming ("stream": true) works as in OpenAI's API. Each answer carries X-Kilby-Request-Id; answers to kilby-auto also say which kind of model served them (fast, balanced or strong).

Usage and billing#

The AI Gateway page shows, for the last 30 days, each day's requests, input and output tokens and cost, by key and model name. On the invoice the gateway is one line: the sum of what each request cost, at the gateway's list prices, rounded once to the cent. During the beta nothing is charged.

What Kilby keeps#

Kilby counts tokens, time and money for each request. It never stores or logs a prompt or an answer. The models run at a provider Kilby contracts with (Subprocessors), which processes the request to answer it.

  • Kilby AI: the assistant in the dashboard, a different thing from this gateway.
  • Limits: what each plan includes.
  • API keys: keys for the Kilby API, which manage the workspace; gateway keys only call models.