Skip to the page
KilbyDocs
Build and run View as Markdown
On this page

Neuravision

Neuravision analyzes photos of retail shelves. Send the photos of one shelf, left to right; the answer lists each product it found with its shelf level, its position from the left, the facings it counted and how sure it is. It runs on Neuramorphic's analysis engine, behind the same host and keys as the AI Gateway.

What it does not do: it recognizes the products of your catalog, which you build and train here. It is not a general object detector, and it does not recognize products that are not in your catalog (it reports them as unidentified). When it is not sure, it says so (needs_review, count_uncertain) instead of guessing: treat as firm only what has both set to false, and have a person check the rest.

The steps: activate a plan, create a catalog, add your products with their photos, train the catalog, check its numbers and publish the version, then send shelf photos with a key. Your app can do the catalog's steps by itself, through the API for your app; the Neuravision page then shows how the service is doing: photos analyzed and not, why not, how long analyses take, by key and by day, and what the month costs.

Activate a plan#

In the dashboard, Neuravision → choose Starter or Growth → Activate. Only the workspace's admins and owners can activate, change or deactivate a plan. The plan is active at once.

StarterGrowthEnterprise
Price$99 a month$399 a monthTalk to us
Catalogs13by agreement
Products (SKUs) per catalog500500by agreement
Photos included a month2,50015,000by agreement
Trainings1 a month1 a week per catalogby agreement
Each photo over the included ones$0.04$0.03by agreement
Detector fine-tuning on requestnoyesyes

The fee is prorated to the second, like every monthly item of Kilby: a plan active for half of a month costs half the fee that month, and includes half the photos. A photo counts when the engine analyzed it (the job is done); failed, refused and expired analyses do not count. On the invoice, Neuravision is one line per plan ("Neuravision Starter", "Neuravision Growth") and one line for the photos over the plan.

During the beta nothing is charged: the Neuravision page and the Billing page show what it would cost.

Deactivate (same page, admins and owners) stops analyses and trainings at once, and the fee stops counting from that second. Your catalogs, products and versions stay until you ask us to delete them; activating again finds them as you left them.

Catalogs#

A catalog is a set of products a shelf photo is read against. Each analysis uses one catalog. Create it on the Neuravision page with a name (you can change it later) and a slug: 1 to 40 lowercase letters, digits and dashes, such as personal-care. The slug never changes: it is how the API names the catalog. Deleting a catalog deletes its products, photos and versions.

Products and their photos#

Each product has a SKU (up to 64 characters), a name, and optionally a brand and a category, with 1 to 5 photos of its package: JPEG, PNG and WebP, up to 10 MB each, told by their content, not their name. A photo of the front of the package, well lit and filling the frame, is what helps most; add other sides as further photos.

Your app can keep them through the API for your app. On the dashboard, add products one at a time on the catalog's page (under Products and photos), or import many at once:

  • A table, CSV (UTF-8, comma or semicolon) or XLSX (its first sheet), whose first row names the columns sku, name, brand, category, image. sku and name are required. image names a photo file in the ZIP; several can be separated by ;, or a product can take several rows, one photo each. Up to 5,000 rows.
  • A ZIP of the photos the table names, found by their file name (folders inside the ZIP do not matter). Up to 50 MB: import a large catalog in several parts, each table with its own ZIP.
csv
sku,name,brand,category,image
7801234000011,Shampoo Example 400 ml,Example,Hair care,7801234000011-front.jpg;7801234000011-side.jpg
7801234000028,Conditioner Example 400 ml,Example,Hair care,7801234000028-front.jpg
7801234000035,Body lotion Example 250 ml,Example,Skin care,7801234000035.png

An import adds new SKUs and updates the ones the catalog has: a row's photos replace that product's photos; a product without photos in the import keeps the ones it had. Rows that cannot be used are skipped and listed with the reason (no name, a photo missing from the ZIP or not a photo, a SKU over the plan's limit), and the rest are imported.

Confirmed crops (optional)#

If a person checked analyses you ran through Neuravision, upload what they confirmed as a JSON Lines file: one object per line, with the job id of the analysis, the photo's index in it, the product's item_index in the result, and the right SKU.

json
{"job_id": "vjob_…", "image_index": 0, "item_index": 3, "sku": "7801234000011"}

The next training learns from them. Uploading a file replaces the catalog's confirmed crops. Lines of jobs that are not this workspace's analyzed jobs, or with a SKU the catalog does not have, are skipped and counted. Up to 20,000 lines.

Train a catalog#

Train on the catalog's page builds a new version of it from its products, photos and confirmed crops. The catalog needs at least 2 products with a photo. Builds run on Neuramorphic's engine and take minutes; the page shows the status as it goes. A training that fails does not count against the plan, and its error says what to fix:

ErrorWhat to do
no_productsThe catalog has no products: add them.
too_few_imagesFewer than 2 products have a photo the engine could use: add photos. A product without a usable photo is left out and counted.
bad_manifestKilby sent something the engine refused. Try again; if it fails again, write to us.
internalThe engine failed. Try again later.

When it is done, the page shows the new version's numbers on the engine's evaluation photos: top-1, how often the right product is the engine's first answer, and top-3, how often it is among its first three, next to the published version's numbers on the same photos.

Versions#

Exactly one version of each catalog is published: the one analyses use. A new version is published by itself when the catalog has none published yet, or when its top-1 is at least the published version's. Otherwise it stays a candidate: publish it by hand if you prefer it. Publish on any version of the history also takes you back to an older one; the next analysis uses it.

Versions are built and kept by the engine, in Neuramorphic's AWS account. No model file ever leaves Neuramorphic: neither you nor Kilby can download a version. You get results only.

Keys#

In the dashboard, Neuravision → Create key. Admins and owners create and revoke keys. A key:

  • starts with kai_, is shown once, and is stored only as a fingerprint;
  • sends photos for analysis and reads their results and crops (and, where the AI Gateway is offered and you allow it when you create it, calls its models);
  • with Manage catalogs and trainings, an option when it is created, also manages the workspace's catalogs through the API for your app: products, photos, confirmed crops, trainings and published versions. The Neuravision page lists which keys can;
  • cannot read or change anything else in the workspace;
  • has its own limits: requests a minute, photos a day (optional) and photos a month (optional).

Send photos#

POST https://ai.onkilby.com/v1/neuravision/analyze, as multipart/form-data:

Field
images1 to 6 photos of one shelf, left to right, in one part each. JPEG, PNG, WebP and HEIC, told by their content, not their name; up to 9 MB in all. The engine reduces photos to 3,000 pixels on their longest side, so reduce larger ones before sending.
catalogThe slug of the catalog to read the photos against. Required when the workspace has more than one catalog; with one, it is that one. Also as ?catalog=.
store_id, gondola_idRequired. The store and the shelf: 1 to 40 letters, digits, dots, dashes or underscores. The same pair always means the same shelf.
nameOptional. A name for the shelf, up to 100 characters.
linear_metersOptional. The shelf's width in meters, for the per-meter figures.
waitOptional. Seconds to wait for the result, from 0 to 25 (also as ?wait=).

The photos are read against the catalog's published version at the time of the request. The answer is a job, 202 Accepted:

json
{"id": "vjob_…", "object": "neuravision.job", "status": "queued", "photos": 2, "catalog": "personal-care", "created_at": "…", "finished_at": null, "billed": false, "poll": "/v1/neuravision/jobs/vjob_…"}

With wait, the request waits up to that many seconds and answers 200 with the finished job when it ended in time; otherwise 202 as above.

Get the result#

GET https://ai.onkilby.com/v1/neuravision/jobs/{id} returns the job. Its status is:

StatusMeaningCounted
queued, runningThe engine has it. Ask again in a few seconds.Not yet
doneAnalyzed: result holds what the engine found.Yes, each photo (billed is true)
failedThe engine could not analyze it; error says why (analysis_rejected: no shelf of your catalog in the photos, such as a blurred or distant photo; analysis_error: the engine failed, try again).No
expiredThe engine did not finish it in time, or no longer has it.No

An analysis takes from several seconds to a few minutes: it depends on the number of photos and on the analyses queued before it. GET https://ai.onkilby.com/v1/neuravision/jobs lists the workspace's latest jobs (?limit= up to 100), without their results.

Kilby does not keep results: each time you ask for a done job, Kilby reads it from the engine. The engine keeps a job for 7 days; after that result_available is false. Store what you need on your side.

The result#

result has these fields: total_facings, shelf_levels_detected, empty_space_pct, needs_review, count_uncertain, categories, linear_meters, facings_per_linear_meter, vertical_meters, display_area_m2, total_units_estimate, compliance, standards, possible_missing (products whose absence the engine suspects but does not state), and products.

Each entry of products is a product found, or a gap, ordered by shelf level and position, with these fields when the engine has them: product, brand, laboratorio, category, sku, facings, shelf_level, position_index, match_status, confidence, source, needs_review, count_uncertain, out_of_stock, estimated_depth, units_estimate, planogram_status, planogram_tag, ocr_text, match_score, image_index, item_index.

The main ones:

FieldMeaning
product, brand, skuThe product of your catalog it matched (or espacio vacío for a gap).
shelf_level, position_indexThe shelf level (1 is the top) and the position from the left.
facingsFacings counted.
match_statusmatch, unmatched (text read, but no product of the catalog fits), unreadable (no legible text), or gap.
confidenceFrom 0 to 1; under 0.60 the product is marked needs_review.
needs_reviewThe engine is not sure which product this is.
count_uncertainThe product is sure, but its facings may change once a doubtful neighbor is reviewed.
image_index, item_indexWhere the product is: the photo, and its place in that photo. A confirmed crop names these.

compliance and standards are objects the engine computes; they are passed as the engine gives them.

Review crops#

GET https://ai.onkilby.com/v1/neuravision/jobs/{id}/crops/{image_index}/{item_index}.jpg returns the JPEG crop of one product of a done analysis, named by the image_index and item_index of its entry in products: what a person looks at on a review screen before confirming it (a confirmed crop names the same job and indexes).

  • Any key that may call Neuravision reads the crops of its workspace's jobs; the dataset permission is not needed.
  • A crop is not an analysis: it never counts against the plan and is never billed.
  • Each key may read 4 times its requests a minute in crops, up to 600 a minute, apart from its analyses (429 rate_limit past it, with Retry-After).
  • The answer is image/jpeg with Cache-Control: private, max-age=86400: your app may keep it for a day.
  • Crops only: Kilby never returns where the products are in your photos (boxes or coordinates).
  • 404 not_found for a job of another workspace, one that does not exist or was not analyzed, an index the result does not have, or a job the engine no longer keeps (7 days); 409 job_not_done while the job is queued or running.
sh
curl -o crop.jpg https://ai.onkilby.com/v1/neuravision/jobs/vjob_…/crops/0/3.jpg -H "Authorization: Bearer kai_…"

Limits and errors#

Every request is checked before its photos are sent: the workspace's plan, the key's requests a minute, the photos a day and a month of the key, and the workspace's photos a month. Photos still being analyzed count against the caps.

The workspace's monthly cap guards your bill; it is not the included amount: photos over the included ones are analyzed and counted at the extra price, up to the cap.

PlanMonthly cap when you activateMost an admin can set
Starter5,000 photos25,000 photos
Growth30,000 photos150,000 photos

Admins change the cap on the Neuravision page. For more, write to peter@neuramorphic.ai.

Statuserror.code
400invalid_requestA field is missing or wrong; the message says which (also: catalog is needed when the workspace has several).
401missing_key, invalid_key, expired_key
403product_not_allowedThe key may not call Neuravision.
403vision_not_enabledThe workspace has no active plan.
404catalog_not_foundThe workspace has no catalog with that slug.
409catalog_not_readyThe catalog has no published version yet (or the workspace has no catalog).
413photo_too_large, request_too_large
415unsupported_photoA file that is not a JPEG, PNG, WebP or HEIC photo.
422rejected_by_engineThe engine did not take these photos or this store_id and gondola_id. Not counted.
429rate_limit, key_daily_photos, key_monthly_photos, workspace_monthly_photosA limit; the message names it.
502, 503, 504engine_unavailable, busy, capacity, timeoutTry again later. Not counted.

Quickstart#

python
import time, requests

URL = "https://ai.onkilby.com/v1"
H = {"Authorization": "Bearer kai_…"}

with open("shelf.jpg", "rb") as photo:
    r = requests.post(f"{URL}/neuravision/analyze", headers=H,
                      data={"catalog": "personal-care", "store_id": "S001", "gondola_id": "G01"},
                      files=[("images", ("shelf.jpg", photo, "image/jpeg"))])
r.raise_for_status()
job = r.json()
while job["status"] in ("queued", "running"):
    time.sleep(5)
    job = requests.get(f"{URL}/neuravision/jobs/{job['id']}", headers=H).json()

if job["status"] == "done":
    for p in job["result"]["products"]:
        print(p["shelf_level"], p["position_index"], p["product"], p["facings"])
else:
    print(job["error"]["code"], job["error"]["message"])
sh
curl https://ai.onkilby.com/v1/neuravision/analyze \
  -H "Authorization: Bearer kai_…" \
  -F catalog=personal-care -F store_id=S001 -F gondola_id=G01 \
  -F images=@shelf-left.jpg -F images=@shelf-right.jpg

API for your app#

Your app can keep a catalog up to date by itself: push its products and their photos when they change, upload confirmed crops, start a training, follow it and publish the version. Kilby never calls your app; it answers it.

It uses a key created with Manage catalogs and trainings on the Neuravision page (only admins and owners create keys); any other key gets 403 dataset_not_allowed. Everything a key changes is recorded in the workspace's activity log, with the key's name. The endpoints are on the same host as the analyses, under https://ai.onkilby.com/v1/neuravision, and each request counts against the key's requests a minute, apart from its analyses. Bodies are JSON unless said otherwise; a list answers {"object": "list", "data": [...]}. Creating a catalog, adding or changing products, uploading confirmed crops and starting a training need an active plan (403 vision_not_enabled without one); reading, deleting a product and publishing a version do not.

RequestWhat it does
GET /catalogsThe workspace's catalogs: slug, name, products and photos (its product list), published_version (version, top1, top3, products, seeded; null when none), trainings_left (what the plan still allows now), last_training (id, status, version, created_at; null when none) and created_at.
POST /catalogs{"slug": "personal-care", "name": "Personal care"} creates a catalog: 201 with it. The plan's number of catalogs applies. Without slug, it is made from the name.
GET /catalogs/{slug}/productsA page of the product list, in the order of the SKUs' bytes: sku, name, brand, category, photos (how many) and updated_at. ?limit= from 1 to 500 (100 when left out); next_cursor is the ?cursor= of the next page, and null on the last one (has_more says the same).
PUT /catalogs/{slug}/products/{sku}Adds the product or changes it: 201 when the SKU is new, 200 otherwise, with the product. As JSON, {"name": …, "brand": …, "category": …} changes its text and keeps its photos (a new product starts without photos, and a training leaves it out until it has one). As multipart/form-data, the fields name, brand and category with 1 to 5 photos files (JPEG, PNG and WebP, up to 10 MB each) also replace its photos. name is required; brand and category are optional, and one left out is stored empty. Sending the same request again gives the same product (as JSON, not even updated_at changes). Percent-encode the SKU in the path; a SKU with a slash (/) cannot be sent in a path: manage such a product on the catalog's page.
DELETE /catalogs/{slug}/products/{sku}Deletes the product and its photos: 204, or 404 when the catalog has no such SKU.
POST /catalogs/{slug}/labelsThe body is the confirmed crops as JSON Lines, one object per line; they replace the catalog's. Answers accepted (lines stored), skipped (lines that are not such objects), usable (stored lines the next training uses: of this workspace's analyzed jobs, with a SKU of the catalog) and problems (the first 20 skipped lines: line, reason). Up to 8 MB.
POST /catalogs/{slug}/trainingsStarts a training of the catalog: 202 with the training. The plan's quota applies, and the catalog needs at least 2 products with a photo.
GET /catalogs/{slug}/trainings/{id}The training: status (queued, running, done or failed), version, and once done metrics (top1, top3, skus) and baseline (the published version's top1 and top3 on the same photos; null when it could not be compared), published (analyses use its version now) and error (code, message) when it failed.
GET /catalogs/{slug}/versionsThe catalog's versions, newest first: version, created_at, top1, top3, products (how many it recognizes), seeded (registered from an existing catalog) and published. A number Kilby does not know is null.
POST /catalogs/{slug}/versions/{version}/publishPublishes that version, such as v3-202611181200 (an older one too: a rollback); the next analysis uses it. Answers the version.

The product list is what the next training uses. The published version recognizes the products it was built with: a catalog registered from an existing one has a published version and may start with an empty list. Fill the list before training again.

A sync recipe#

  1. When a product is added or changed in your app, PUT it: as JSON when only its text changed, as multipart with its photos when they changed. When one is removed, DELETE it. For a full sync, read the list page by page and delete the SKUs your app no longer has.
  2. Optionally, when people confirmed crops in your app, POST them as labels, the whole set each time (it replaces the catalog's).
  3. When the list is ready and trainings_left of GET /catalogs is above 0, POST a training.
  4. Ask for it every minute or so until its status is done or failed; a build takes minutes. A failed training does not count against the plan; its error.code is one of the training errors.
  5. When it is done and published is true, analyses use the new version. When published is false, the version is a candidate: its top-1 was below the published version's on the same photos (compare metrics with baseline). Publish it if you prefer it; GET /versions lists them all.
python
import time, requests

URL = "https://ai.onkilby.com/v1/neuravision"
H = {"Authorization": "Bearer kai_…"}  # a key with "Manage catalogs and trainings"
CATALOG = "personal-care"

def push(sku, name, brand="", category="", photos=None):
    """photos: None keeps the product's photos; a list of file paths replaces them."""
    url = f"{URL}/catalogs/{CATALOG}/products/{requests.utils.quote(sku, safe='')}"
    if photos is None:
        r = requests.put(url, headers=H, json={"name": name, "brand": brand, "category": category})
    else:
        files = [("photos", (p, open(p, "rb"))) for p in photos]
        r = requests.put(url, headers=H, data={"name": name, "brand": brand, "category": category}, files=files)
    r.raise_for_status()
    return r.json()

push("7801234000011", "Shampoo Example 400 ml", "Example", "Hair care", ["7801234000011-front.jpg"])
push("7801234000028", "Conditioner Example 400 ml", "Example", "Hair care", ["7801234000028-front.jpg"])
push("7801234000011", "Shampoo Example 400 ml (new label)")  # text only: the photos stay

r = requests.post(f"{URL}/catalogs/{CATALOG}/trainings", headers=H)
if r.status_code == 429:
    print("no training left:", r.json()["error"]["message"], "retry after", r.json()["error"].get("retry_after"), "s")
else:
    r.raise_for_status()
    t = r.json()
    while t["status"] in ("queued", "running"):
        time.sleep(60)
        t = requests.get(f"{URL}/catalogs/{CATALOG}/trainings/{t['id']}", headers=H).json()
    if t["status"] == "failed":
        print(t["error"]["code"], t["error"]["message"])
    elif not t["published"]:
        print("candidate", t["version"], t["metrics"], "published version on the same photos:", t["baseline"])
        # requests.post(f"{URL}/catalogs/{CATALOG}/versions/{t['version']}/publish", headers=H)
sh
curl -X PUT https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/products/7801234000011   -H "Authorization: Bearer kai_…"   -F name="Shampoo Example 400 ml" -F brand=Example -F category="Hair care"   -F photos=@7801234000011-front.jpg -F photos=@7801234000011-side.jpg

curl -X PUT https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/products/7801234000011   -H "Authorization: Bearer kai_…" -H "Content-Type: application/json"   -d '{"name": "Shampoo Example 400 ml", "brand": "Example", "category": "Hair care"}'

curl -X POST https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/labels   -H "Authorization: Bearer kai_…" -H "Content-Type: application/x-ndjson" --data-binary @labels.jsonl

curl -X POST https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/trainings -H "Authorization: Bearer kai_…"
curl https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/trainings/<id> -H "Authorization: Bearer kai_…"
curl https://ai.onkilby.com/v1/neuravision/catalogs/personal-care/versions -H "Authorization: Bearer kai_…"

Errors come as {"error": {"type": …, "code": …, "message": …}}, with param when a field is wrong:

Statuserror.code
400invalid_requestA field is missing or wrong; param names it.
401missing_key, invalid_key, expired_key
403product_not_allowedThe key may not call Neuravision.
403dataset_not_allowedThe key may not manage catalogs and trainings.
403vision_not_enabledThe workspace has no active plan.
404catalog_not_foundThe workspace has no catalog with that slug.
404not_foundThe catalog has no such product, training or version, or there is no such endpoint.
409catalog_existsThe workspace has a catalog with that slug.
409catalog_limit, product_limitThe plan's catalogs, or its products a catalog, are reached.
409training_runningA training of the catalog is running: wait for it to end.
413photo_too_large, request_too_large
415unsupported_photoA file that is not a JPEG, PNG or WebP photo.
422too_few_productsFewer than 2 products of the catalog have a photo.
429training_quotaThe plan's trainings are used: retry_after (and the Retry-After header) is how many seconds until the next one is allowed.
429rate_limitThe key's requests a minute; Retry-After says when to try again.
500, 503internal, busy, unavailableTry again in a few seconds.

Detector fine-tuning#

The engine first finds the products on the shelf, then recognizes each one against your catalog. If your shelves are unusual (very small packages, hanging displays, fridges), the first step can be fine-tuned for you. On Growth, Request detector fine-tuning on the Neuravision page records your request; Neuramorphic runs it and writes to you. It is not self-service.

What Kilby stores, and where#

  • Your product photos and the manifest of each training: in a private S3 bucket of Neuramorphic in AWS us-east-1 (Virginia), next to the engine, encrypted at rest. Kilby writes them; the engine reads them to build your versions. A training's manifest (the list of products and photo addresses it was built from) is deleted 90 days after the training. Product photos stay until you delete the product or the catalog.
  • Your catalog versions: written and kept by the engine in the same bucket; they never leave Neuramorphic.
  • Review crops: read from the engine each time your app asks for one; Kilby does not store them.
  • Your shelf photos: Kilby does not store them. The gateway holds them in memory only while it sends them to the engine, and Kilby's database keeps only each photo's size, its status and the time the engine took. The engine keeps each job and its result for 7 days so you can read it. To have anything deleted from the engine, write to peter@neuramorphic.ai.
  • Your products' names, SKUs, brands and categories, and your confirmed crops: in Kilby's database.
  • AI Gateway: the same host and keys, for AI models.
  • Limits: what each Kilby plan includes.