Images

How it works#

Add a top-level image field to any decision request. Wity reads the image and the state together as one piece of evidence, and every question in the request sees both. You don't point a question at the image. It is simply part of what the questions are about, like a photo attached to a case file.

The same field works on /v1/generate, for reading text or details off an image. The answer types don't change: a choice about an image is still a choice with a probability for every option.

Example#

A semiconductor fab tests every die on a wafer and draws the results as a wafer map. The shape of the failures points to the cause. A ring at the edge suggests a process problem at the rim. A line suggests a scratch from handling. A cluster in the middle points to something else again. Engineers classify these maps by eye. With an image, the same classification is one choice question.

• state explains how to read the map: what a square is, what the colours mean, and that the grey rim is not tested. Without it, a map is just coloured squares.
• image is the rendered wafer map as a data URL.
• criteria describe each pattern by its geometry, which is what can actually be seen. none covers a clean wafer, so it isn't forced into one of the defect patterns.

{
  "state": "This is a wafer map from final electrical test. Each small square is one die. Green die passed; red die failed; grey squares at the rim are partial die and are not tested.",
  "image": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
  "questions": {
    "pattern": {
      "type": "choice",
      "instructions": "Which defect pattern do the red failing die form on this wafer?",
      "criteria": {
        "center": "Failures clustered in the middle of the wafer",
        "edge_ring": "A ring of failures around the whole edge",
        "scratch": "A thin, roughly straight line of failures across the wafer",
        "random": "Failures scattered with no spatial pattern",
        "none": "Almost no failures at all"
      }
    }
  },
  "reasoning": "auto"
}

For a map with a band of red squares running around the rim, the answer is clear. scratch gets a small share because a curved line along one side can look like part of a ring:

In production, the engineering system would file this wafer under an edge-ring investigation straight away and send anything below its threshold to an engineer, along with the probabilities.

Sending an image#

The image goes in as a data URL: data:, the media type, ;base64,, then the base64-encoded bytes. PNG, JPEG, WebP, GIF and BMP all work. Wity reads the format from the file itself, so a generic media type (application/octet-stream), bare base64, line breaks or URL-safe base64 are all fine. Remote URLs are not fetched: send the image itself. Most languages turn a file into one in a few lines:

import base64, mimetypes
 
def data_url(path):
    mime = mimetypes.guess_type(path)[0] or "image/png"
    with open(path, "rb") as f:
        return f"data:{mime};base64," + base64.b64encode(f.read()).decode()
 
payload = {"state": "...", "image": data_url("wafer.png"), "questions": {...}}

Large photos should be made smaller first. Phone cameras produce images many times bigger than a decision needs, and they count towards the 12 MB request limit. Scaling the long side to about 1,500 pixels keeps the detail that matters and cuts the upload:

from PIL import Image
import io, base64
 
img = Image.open("site-photo.jpg")
img.thumbnail((1500, 1500))                # keep aspect ratio, long side at most 1500 px
buf = io.BytesIO()
img.convert("RGB").save(buf, "JPEG", quality=85)
image = "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()

If only part of the image matters (one gauge on a panel, one field on a form), cropping to that part helps more than any amount of resolution, because there is less for Wity to look past.

Tell Wity what it is looking at#

An image without context is ambiguous. Is the red on a chart good or bad? Is the photo from before the delivery or after? Should a missing sticker be there? The state is where you answer those questions, the way you would brief a colleague before handing them a photo:

"state": "See image."

"state": {
  "what": "Photo of a delivered parcel, taken by the courier at the door.",
  "look_for": "Tears, crushed corners, water marks, or tape that has been cut and re-applied.",
  "known": { "order": "#88213", "packed_by": "warehouse B", "fragile": true }
}

The clear version says what the image is, who took it and when, what to look for, and the facts that are already known. "Fragile: true" changes how a small dent should be read. None of that can be seen in the picture.

• Explain colours, symbols and axes on charts and maps: "green passed, red failed".
• Put numbers you already have in the state instead of asking Wity to read them off a chart. A sensor value from your database is exact; a value read off a line chart is an estimate.
• Say what "normal" looks like when the question is whether something is wrong.

Patterns that work well#

Classify what the image shows

Defect patterns, document types, damage categories, the state of a machine display. A choice or score question with criteria that describe what can be seen, like the wafer example above.

Check a condition

Is the seal intact? Is the label present and straight? Is the safety guard in place? These are nouls, and several of them can check the same photo in one request, like an inspection checklist.

Decide an agent's next step from a screenshot

A browser or desktop agent takes a screenshot after every step and asks what to do next. The state carries the goal and what has been done so far; the screenshot shows where things stand now. A choice picks the next action, and a noul checks the page has loaded before the agent acts:

{
  "state": {
    "goal": "Book the cheapest direct train from Lisbon to Porto on 14 October.",
    "done_so_far": ["Opened cp.pt", "Typed Lisbon into From"],
    "page": "Screenshot of the current browser tab, 1280x800."
  },
  "image": "data:image/png;base64,…",
  "questions": {
    "next": {
      "type": "choice",
      "instructions": "What should the agent do next?",
      "criteria": {
        "type_destination": "The To field is empty or wrong",
        "pick_date": "Stations are set but the date is not 14 October",
        "search": "Stations and date are set; the search button has not been pressed",
        "select_result": "Results are showing",
        "dismiss_popup": "A cookie banner or popup is covering the page",
        "stuck": "An error page, a captcha, or anything the steps above can't handle"
      }
    },
    "loaded": { "type": "noul", "instructions": "Has the page finished loading?" }
  },
  "reasoning": "auto"
}

Each action is described by what would be visible when it is the right move. stuck gives the agent a clean way out when the page is something it wasn't built for. The agent acts only when loaded is high and the top action is clear, and otherwise takes another screenshot. When the next step needs text, like typing "Porto" into the To field, generate writes it from the same screenshot.

Read text off a scan

For values rather than decisions, send the image to /v1/generate with a shape. pattern keeps the output in the format your system expects. A legible flag lets Wity say when the scan can't be read, so it doesn't have to invent a value:

{
  "state": "Scan of the front page of a home insurance certificate.",
  "image": "data:image/jpeg;base64,…",
  "instructions": "Read the policy number and the insured address off the certificate.",
  "shape": {
    "type": "object",
    "properties": {
      "policy_number": { "type": "string", "pattern": "^HC-[0-9]{5}$" },
      "postcode":      { "type": "string", "maxLength": 10 },
      "legible":       { "type": "boolean" }
    },
    "required": ["policy_number", "postcode", "legible"]
  },
  "max_tokens": 60
}

Route anything with legible: false, or a policy number that isn't in your system, to a person.

Limits#

• One image per request. For several photos of the same case, ask about each in its own request and combine the answers in code.
• The whole request body is limited to 12 MB, so keep images to a few megabytes. Detail beyond about 1,500 pixels on the long side rarely helps a decision.
• Images add some latency: typically a few hundred milliseconds to about a second on top of a text decision.
• An image that cannot be decoded is rejected with a 400 that says why (not base64, unknown format, or a truncated file).