Vision API: analyse, segment, read text and track

Developers and integrations · Intermediate · 13 min read · Updated

Describe images, cut objects out with text or clicks, read text, estimate depth, upscale and follow objects through video, with the exact request and response shapes.

The AI Models tab filtered to FOTOhub's own IDA Q models, with price per unit and the endpoint of each.

What the vision API covers

Vision turns pixels into information or into masks. Most calls take an image_url (a public URL), bill a small fixed amount in USD and answer on the same request. The answer either carries the result inline (labels, boxes, text, base64 masks) or a URL to a finished file.

TaskEndpointReturns
Describe an imagePOST /v1/ai/analyze/imagelabels, colours, faces, objects, text, safety flags
Find objects by a phrasePOST /v1/vision/segmenta box and a mask for every match
Select by clickingPOST /v1/vision/segment-clickone mask from foreground and background points
Soft edges for hair and furPOST /v1/vision/mattean alpha matte
Remove an areaPOST /v1/vision/inpaintthe finished image
EnlargePOST /v1/vision/upscalethe enlarged image
Depth mapPOST /v1/vision/deptha depth image
Read textPOST /v1/vision/ocrtext blocks with positions
Score labels you choosePOST /v1/vision/taga score per label
Video: track, remove, smooth/v1/vision/track, /remove-object, /interpolatea job to poll at GET /v1/vision/jobs/{id}

Every call needs Authorization: Bearer $FOTOHUB_API_KEY and goes to https://apis.fotohub.app. Successful answers share a tail: cost_usd, currency, a billing block and, on the vision routes, processing_time_ms. Inputs are public URLs; outputs are inline data, file links or a job id.

Which route do I need

You want toUse
File or search photos by contentPOST /v1/ai/analyze/image
Rank a fixed list of labels you chosePOST /v1/vision/tag
Cut out something you can namePOST /v1/vision/segment
Cut out what a user pointed atPOST /v1/vision/segment-click
Keep hair and fur softPOST /v1/vision/matte
Take the main subject out in one stepPOST /v1/images/remove-background
Delete an object and repaintPOST /v1/vision/inpaint
Read text from a photoPOST /v1/vision/ocr
Read a PDF, a form or a receipt/v1/ai/document/*
Anything on a video/v1/vision/track, /remove-object, /interpolate

Your first call, step by step

  1. Pick an image the internet can reach FOTOhub downloads the image itself, so image_url must be a public https address. A link behind a login, a localhost URL or a private network address is refused before any charge.
  2. Send the request with your key Every route here is a POST with a JSON body and Authorization: Bearer $FOTOHUB_API_KEY, except the job status, which is a GET. The snippet above shows analyse, segment and tag against the same fashion photo.
  3. Read the cost in the answer Every answer ends with cost_usd, currency, a billing block with your remaining balance_usd, and on the vision routes processing_time_ms. Log these three; they are your audit trail.

Price shows up in four places: GET /v1/models, GET /v1/pricing, a quote from POST /v1/billing/estimate before you spend, and the cost_usd of the response itself. The figures in this guide are examples from real calls, not a price list.

Analyse an image

POST /v1/ai/analyze/image takes image_url and a list of features: labels, objects, faces, nsfw, ocr and colors. The default is labels and objects. An unknown feature name is a 400 that lists the allowed ones, so a typo never costs money. Optional: language for text, max_labels (1 to 50) and min_confidence (0 to 1).

SRC="https://static.fotohub.app/home/hero/fashion-coral-photo-520.webp"

# Labels and dominant colours.
curl -s -X POST https://apis.fotohub.app/v1/ai/analyze/image \
  -H "Authorization: Bearer $FOTOHUB_API_KEY" -H "Content-Type: application/json" \
  -d "{\"image_url\": \"$SRC\", \"features\": [\"labels\", \"colors\"]}" \
  | jq '{cost_usd, labels: [.labels[] | .name], dominant: .colors.dominant[0].hex}'

# Find every object that matches a phrase, with a box and a mask each.
curl -s -X POST https://apis.fotohub.app/v1/vision/segment \
  -H "Authorization: Bearer $FOTOHUB_API_KEY" -H "Content-Type: application/json" \
  -d "{\"image_url\": \"$SRC\", \"prompt\": \"coat\"}" | tee segment.json \
  | jq '{count, cost_usd, objects: [.objects[] | {score, box}]}'

# The mask of the first match is a grayscale PNG inside the JSON: decode it to a file.
jq -r '.objects[0].mask_png_base64' segment.json | base64 -d > coat-mask.png
file coat-mask.png

# Zero-shot tags: how well does each label fit?
curl -s -X POST https://apis.fotohub.app/v1/vision/tag \
  -H "Authorization: Bearer $FOTOHUB_API_KEY" -H "Content-Type: application/json" \
  -d "{\"image_url\": \"$SRC\", \"labels\": [\"coat\", \"dress\", \"street\"]}" | jq '{labels, cost_usd}'
{
  "cost_usd": 0.003,
  "features_analyzed": ["labels", "colors"],
  "auto_tags": ["winter", "hood", "freezing", "walking", "parka"],
  "labels": [
    { "name": "winter", "confidence": 0.785 },
    { "name": "hood", "confidence": 0.752 }
  ],
  "colors": { "dominant": [{ "hex": "#ce554b", "score": 0.237, "percentage": 3.1 }] }
}

A real analyse answer, trimmed. The call cost a fraction of a cent.

auto_tags is a ready list of the strongest labels, handy for filing a photo. Label names are always English. Each requested feature adds its own key to the answer, and only the features you asked for appear in features_analyzed:

FeatureKey in the answerWhat it holds
labelslabelsname and confidence, strongest first
objectsobjectsname, confidence and a bounding_box for each object
facesfacesA bounding_box, a confidence and likelihood buckets for joy, sorrow, anger, surprise, headwear and blur. No age and no gender.
nsfwnsfwis_safe plus likelihoods for adult, violence, racy, medical and spoof
ocrocrtext as one string and blocks with a box each
colorscolorsdominant: hex, score and percentage

Four more names are accepted: landmarks, logos, text (an alias of ocr) and safe_search (an alias of nsfw). The 400 for an unknown name lists the full set.

Retrying without paying twice

Send the same X-Idempotency-Key on a retry and the API answers with the first result instead of running and charging again. A replay carries the header Idempotent-Replay: true. A key is remembered for 24 hours. If the first call is still running you get a 409; if you reuse a key with a different body you get a 422, never a silent first result. Generate one fresh key per logical request, for example a UUID, and keep it across retries.

SRC="https://static.fotohub.app/home/hero/fashion-coral-photo-520.webp"
KEY=$(uuidgen)

# Run it twice with the same key: the second call is a replay and is not charged.
for i in 1 2; do
  curl -si -X POST https://apis.fotohub.app/v1/ai/analyze/image \
    -H "Authorization: Bearer $FOTOHUB_API_KEY" \
    -H "X-Idempotency-Key: $KEY" -H "Content-Type: application/json" \
    -d "{\"image_url\": \"$SRC\", \"features\": [\"labels\"]}" \
    | grep -iE "^HTTP|^idempotent-replay"
done

The replay header is how you tell a cached answer from a new run.

Tag a product photo for a catalogue

Goal. Get search keywords and the main colour of a photo in one request.

Input you can copy.

`features: ["labels", "colors"]` on the fashion photo.
The photo analysed in the vision examples.

Why it works. The call returned five auto_tags (winter, hood, freezing, walking, parka), the top label winter at 0.785 confidence, and a dominant colour of #ce554b. It cost 0.003 USD in this real call. Store auto_tags as your search index and the first colour as a swatch. Labels describe the scene, not the garment alone, so treat them as search hints and keep a human-written title next to them.

Segmentation and masks

Find objects by a phrase

segment takes a short noun phrase in prompt (up to 120 characters, for example coat or person in red), a min_score (0 to 1, default 0.4) and max_objects (1 to 200). Each match carries a score, a box ([x1, y1, x2, y2] in pixels) and a mask_png_base64: a grayscale PNG the size of the image, white where the object is. An empty list is a valid answer meaning the phrase was not found, and image_size gives the dimensions.

Write the prompt in English as a plain noun with at most a colour or a position: coat, red bag, person on the left. A phrase that is a whole sentence matches worse than the object name. Lower min_score to catch faint matches, raise it to drop doubtful ones, and set max_objects to what your interface can show.

Turn the base64 mask into a file

The mask is an ordinary 8-bit grayscale PNG, base64-encoded. Decoding needs nothing but the standard library:

import base64
import json

body = json.load(open("segment.json"))      # the saved /v1/vision/segment answer
best = max(body["objects"], key=lambda o: o["score"])

with open("coat-mask.png", "wb") as f:
    f.write(base64.b64decode(best["mask_png_base64"]))

print(best["score"], best["box"], body["image_size"])

Pick the highest score, write the PNG, no imaging library needed.

To cut the object out, use the mask as the alpha channel of the original with Pillow. Resize the mask only if its size differs from the photo you downloaded:

import base64
import io
import json

import requests
from PIL import Image

URL = "https://static.fotohub.app/home/hero/fashion-coral-photo-520.webp"
body = json.load(open("segment.json"))
best = max(body["objects"], key=lambda o: o["score"])

photo = Image.open(io.BytesIO(requests.get(URL, timeout=30).content)).convert("RGB")
mask = Image.open(io.BytesIO(base64.b64decode(best["mask_png_base64"]))).convert("L")
if mask.size != photo.size:
    mask = mask.resize(photo.size)

photo.putalpha(mask)                        # white = opaque, black = transparent
photo.save("coat-cutout.png")

A transparent cut-out of the coat. Needs pip install pillow requests.

Cut a coat out of a street photo

Goal. Get a mask of just the coat to recolour it in your own editor.

Input you can copy.

`prompt: "coat"` on a 416 by 520 fashion photo.
The grayscale mask returned for the phrase coat: white is the coat.

Why it works. One match came back with a score of 0.91, a box of roughly 105, 67, 300, 423 and a 416 by 520 grayscale mask. Decoding the base64 gives a PNG you can use as an alpha channel or pass as mask_url to the editing calls. This real call cost about 0.11 USD and took 34 seconds, so budget for a slow first request when you set your client timeout.

Select by clicking

segment-click takes points, each with x and y as fractions from 0 to 1 and a label (1 for foreground, 0 for background), up to 32 points. It returns one mask_png_base64, a score and image_size. Use it behind a click-to-select UI.

The two routes combine well. Let segment find a candidate from a phrase, then let the user correct it with clicks. Convert pixel coordinates to fractions by dividing by the image size: a click at pixel 203, 245 on a 416 by 520 image is x = 0.488, y = 0.471. A foreground point inside the object adds to the mask, a background point (label: 0) outside it trims the mask back.

{
  "image_url": "https://static.fotohub.app/home/hero/fashion-coral-photo-520.webp",
  "points": [
    { "x": 0.488, "y": 0.471, "label": 1 },
    { "x": 0.12, "y": 0.5, "label": 0 }
  ]
}

A request body for POST /v1/vision/segment-click: one click on the coat, one on the empty area to its left.

Matte, inpaint, upscale, depth

  • matte returns alpha_png_base64, a soft alpha mask for hair and fur where a hard mask fails.
  • inpaint takes image_url and mask_url (a PNG whose white area is removed and repainted) and returns an output_url. The mask is passed by URL, so host a mask you made yourself first; segment returns base64, not a URL.
  • upscale takes variant x2, x4 or general (gentler for graphics and text) and quality standard or pro; pro is a 4K service for the Professional plan, with its own price and limits.
  • depth returns an output_url to a depth image, useful for parallax and lighting that follows the scene. The related POST /v1/images/depth-map accepts output_type grayscale or colored; raw is refused with a 400 before anything is charged.
  • ocr returns text, blocks and count. tag takes labels (1 to 100 words) and returns a score for each, sorted from best to worst, a quick zero-shot classifier. Scores are small numbers, so compare them against each other rather than against a fixed threshold.

Score your own labels with tag

Goal. Decide which of three categories a photo belongs to.

Input you can copy.

`labels: ["coat", "dress", "street"]` on the fashion photo.

Why it works. The real answer ranked coat at 0.0088, dress at 0.0001 and street at 0. None of the numbers is near 1, yet the ranking is clear: coat wins by two orders of magnitude. Pick the top label, or require it to beat the runner-up by a factor you choose, rather than comparing to 0.5. This call cost about 0.05 USD and took 13 seconds.

Read text from a photo that has none

Goal. Know what an OCR call returns when nothing is written in the image.

Input you can copy.

`POST /v1/vision/ocr` with the fashion photo and the default `mode`.

Why it works. The real answer was blocks: [], count: 0, text: "", image_size: [416, 520] and mode: "text". That is a successful, billed run, about 0.01 USD in this call, not an error. Check count before you use text, and send image_url as a public address or as a data:image/...;base64 string. The document mode, which returns reading-order text without boxes, is access-gated and can be refused for an account that does not have it.

Video: track, remove, smooth

Three routes work on video and run as jobs. track follows up to 8 phrases through a clip (video_url, prompts, start_frame, max_frames up to 3000, min_score). remove-object erases an object using per-frame masks (masks_url, a ZIP of PNGs) or, with version: "v2" on the Professional plan, one static_mask_url for every frame with a frame window (start_frame, end_frame) and dilation (0 to 24 pixels). interpolate multiplies the frame count by multi (2 to 8) for smoother motion.

  • Submit, then poll GET /v1/vision/jobs/{job_id}. Polling is free.
  • If the engine fails after the charge, you get a 424 that says you were not charged. If a poll itself fails, you get a 502, 503 or 504: just retry the poll.
  • A job id belongs to the account that submitted it; another account gets a 404.
Poll answerMeaningWhat you do
status other than done or failed, with progressStill runningWait a few seconds and poll again
done with output_url and expires_inA finished video; the link is signed and lasts 3600 secondsDownload it now and keep your own copy
done with frames, detected, frame_countThe result of track: the masks come back in the bodyRead them straight away; there is no file
failed with refundedThe job did not produce a result; refunded says whether the charge was returnedDo not poll again; resubmit if you still need it
import os
import time

import requests

BASE = "https://apis.fotohub.app"
HEAD = {"Authorization": f"Bearer {os.environ['FOTOHUB_API_KEY']}"}


def wait_for_job(job_id, every=5, limit=900):
    """Poll a vision job until it is done or failed. Polling costs nothing."""
    waited = 0
    while waited < limit:
        r = requests.get(f"{BASE}/v1/vision/jobs/{job_id}", headers=HEAD, timeout=30)
        if r.status_code not in (502, 503, 504):     # those three mean: ask again
            r.raise_for_status()
            job = r.json()
            if job["status"] in ("done", "failed"):
                return job
        time.sleep(every)
        waited += every
    raise TimeoutError(f"job {job_id} did not finish in {limit} s")

A polling loop that retries the retryable statuses and stops on a terminal one.

Documents, files and limits

  • The document routes (/v1/ai/document/detect-text, /analyze, /analyze-expense) take a base64 document and power the Document Intelligence page. At the time of writing they accept the signed-in web session but answer 401 Invalid or expired token to an API key, so they are not part of the key-based API yet. See the Document Intelligence guide for the page itself.
  • Behind those routes: PDF, JPEG, PNG and TIFF up to 10 MB, and for analyze the features TABLES, FORMS, SIGNATURES and LAYOUT. Do not plan an integration on them until key access is announced.
  • Inputs are fetched by FOTOhub, so image_url must be reachable from the internet. Private or internal addresses are refused before anything is charged, with a 400 whose body names the field and says charged: false.
  • Results that are files (output_url) from images are publicly readable links with random names. Copy sensitive results to your own storage. Video results are signed links that expire after an hour.
  • Prices differ by operation and can change. Read them from GET /v1/pricing or quote with POST /v1/billing/estimate; the exact charge is always in the response.
The AI Models tab filtered to FOTOhub's own IDA Q models, with price per unit and the endpoint of each.
AI Models in the console: check the unit price and availability of an operation before you build on it.
API Usage: requests, spend, utilisation of the rate limits and the traffic chart.
API Usage in the console: the per-endpoint breakdown shows what each vision route has cost you.

Errors and what to do

StatusWhenWhat to do
400A bad URL (private or not https), an unknown features name, an unreadable image. The text says your wallet was not charged.Fix the input and send it again. Nothing was spent.
401A bad key, or a document route called with an API keyCheck the key; document routes need a web session
409 / 422An idempotency key whose first call is still running (409), or reused with a different body (422). Body validation also answers 422, for example a prompt over 120 characters or more than 32 points.Wait and retry with the same key, or use a new key for a new body
413OCR can refuse an image larger than 25 MBDownscale the image first
424The engine failed after the charge. The message states you were not charged.Retry after a short pause; no money is lost
429OCR document mode is busy; the answer carries Retry-AfterWait for the given number of seconds
404 / 410A job id that is not yours or does not exist (404); a finished result that was already collected and has expired (410)Check the id and the account; resubmit after a 410
502 / 503 / 504A job poll could not reach the engine or the job registryPoll again; polling is free and the job is not affected
{
  "error": "url_blocked",
  "field": "image_url",
  "charged": false,
  "message": "image_url must be a public https URL. Your wallet was not charged for this request."
}

The shape of the 400 for an address FOTOhub refuses to fetch. The detail key wraps it in the HTTP answer.

For the general error envelope, rate limits and the exact wording of wallet errors, read the errors and limits guide. Webhooks can tell you when a vision call finishes without polling; see the webhooks guide.

Production checklist

  • Keep the key on your server. A browser that holds it can spend your wallet.
  • Set client timeouts of two minutes or more on synchronous vision routes, and 30 seconds on job polls.
  • Send an X-Idempotency-Key on analyze, and keep one job_id per submitted video job so you never submit twice.
  • Log cost_usd, processing_time_ms and the x-request-id header of every call, so a support question has an exact reference.
  • Download and store results you need to keep: image files are public links, video links expire after an hour.
  • Do not describe these routes to your own users as EU-only processing. Re-read the note near the top of this guide.

Troubleshooting and questions

How do I turn a mask into a cut-out?
Decode mask_png_base64, use it as the alpha channel of your original with any imaging library, or send it as mask_url to an editing call that accepts one. The Pillow snippet above does exactly this in four lines.
Why is tag giving tiny scores?
It scores each label independently as a similarity, not a probability that sums to one. Rank the labels; do not expect 0.9 for the right one.
A route answered 404 for my job id.
Job ids are private to the account. Check you poll with the same account and key owner that submitted it.
segment returned an empty objects list. Was I charged?
An empty list is a normal, completed answer, so it is billed like any other run. Try a simpler noun, lower min_score, or check the image actually contains the object.
My signed or storage URL is refused.
The address must be reachable from the public internet over https and must not have expired. A link that needs cookies or a VPN fails, and so does http://localhost.

Continue in the images guide for background removal and generation, the video guide for generated clips, and the errors and limits guide for rate limits and the full error format.

Open AI Models

Open Vision API in FOTOhub