Probabilities & confidence

What a probability means#

When Wity says an option has probability 0.9, it aims for that to hold in the plain sense: across many answers at 0.9, about 9 in 10 should be right. That is what makes the numbers usable as thresholds. We measure this on held-out benchmark items and keep improving it; treat it as well-behaved on average, not as a guarantee for any single answer, and check it on your own data (below).

The full distribution is the point. A single label says what Wity picked; the probabilities say how the evidence split. Take a payment dispute routed between three queues:

The pick is refunds_queue, but almost a third of the probability sits on fraud review. That tells you the message has features of both. Maybe a genuine refund request, maybe a chargeback pattern. Acting on the label alone would lose that. The right move here is a person, not an automatic refund.

Where the numbers are#

Each answer type puts its probabilities in a slightly different place:

# choice: the pick, and a probability for every option
p_top  = a["route"]["probabilities"][a["route"]["choice"]]
p_any  = a["route"]["probabilities"]["fraud_review"]          # any option, not just the top one
 
# noul: one number, the probability of yes
p_yes  = a["phishing"]["noul"]
p_no   = 1 - p_yes
 
# score: a probability per level, plus the expected level
p_high = a["urgency"]["probabilities"]["2"] + a["urgency"]["probabilities"]["3"]
level  = a["urgency"]["score"]

• Choice probabilities cover every option you offered and add up to 1. You can read any of them, not only the winner. "Send to fraud review if fraud_review ≥ 0.25, whatever won" is a perfectly good rule.
• Noul is a single number. There is no label to compare against; your threshold creates it.
• Score levels can be added: the chance the urgency is "today or worse" is the sum of those levels.score is the expected level, good for sorting.

Acting on probabilities#

A common pattern is three zones per action: act on your own above a high threshold, ask for a quick confirmation in the middle, and send the rest for full review. Where the lines go depends on what a mistake costs:

ans = r.json()["answers"]["route"]
p = ans["probabilities"][ans["choice"]]
 
if ans["choice"] == "refunds_queue" and p >= 0.95:
    issue_refund()             # cheap to undo: act on your own
elif p >= 0.80:
    queue_for_one_click(ans)   # a person confirms with one click
else:
    send_to_human(ans)         # genuinely unsure: full review

Refunds here have their own, stricter bar because they cost money. Other routes only need 0.80 because a wrongly routed ticket is cheap to move.

Setting a threshold from cost

If you can put rough numbers on outcomes, the threshold follows from them. Say acting correctly saves 1 unit of work, and acting wrongly costs 9 (an angry customer, a clawback). Acting is worth it when the expected saving beats the expected cost:

So act above 0.9 and send the rest to a person. If a mistake only costs as much as a correct call saves, the bar drops to 0.5. The more lopsided the costs, the higher the threshold. This works because the probabilities are meant literally. With a raw score of unknown meaning it would be guesswork.

Setting a threshold from your own data

The most reliable way to set thresholds is to measure. Take a few hundred past cases where you know the right answer, run them through Wity once, and see what each cut-off would have done:

# past: a few hundred cases with the answer a person gave, run through Wity once
rows = []
for case in past:
    a = decide(case.state, QUESTIONS)["route"]
    rows.append((a["probabilities"][a["choice"]], a["choice"] == case.label))
 
for t in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95]:
    kept = [ok for p, ok in rows if p >= t]
    print(f"{t:.2f}  automated {len(kept) / len(rows):5.0%}   accuracy {sum(kept) / max(len(kept), 1):5.1%}")

0.50  automated  99%   accuracy  91.2%
0.60  automated  95%   accuracy  93.4%
0.70  automated  90%   accuracy  95.3%
0.80  automated  84%   accuracy  97.0%
0.90  automated  74%   accuracy  98.6%
0.95  automated  61%   accuracy  99.4%

Each row is a trade-off. At 0.80, 84% of cases would be handled automatically at 97% accuracy, and the other 16% would go to a person. At 0.95 accuracy rises to 99.4%, but people handle nearly 40% of the volume. Pick the row whose accuracy you can live with; the automation rate is what you get for it. Re-run the sweep when your inputs or options change.

Look at the runner-up

The top probability alone can hide a close call. 0.52 against 0.41 means something very different from 0.52 spread thinly against five others. When the top two options lead to very different actions, a rule on the margin catches the dangerous case:

top, second = sorted(ans["probabilities"].values(), reverse=True)[:2]
if top - second < 0.2:
    send_to_human(ans)   # two plausible readings: let a person pick

Combining answers

Several answers about the same state are not independent. They read the same evidence, so if one is wrong the others are more likely to be off too. Multiplying probabilities ("0.9 × 0.9 = 0.81 that both hold") assumes they are independent, which they rarely are, so the product is not a reliable probability that both hold. Rules that require each answer to pass its own threshold are simpler and easier to reason about:

if a["unused"]["noul"] >= 0.9 and a["wants"]["probabilities"]["refund"] >= 0.9:
    issue_refund()

confidence is not a probability#

Choice and score answers also include confidence, a number from 0 to 1 that says how peaked the distribution is: 1 minus its normalised entropy. It is 1 when all the probability sits on one option and 0 when it is spread evenly. It describes the shape of the answer, not the chance of it being right.

Two answers can pick the same option with the same probability and still have very different confidence. Both of these put 0.60 on billing:

The first has confidence 0.51: all the doubt is on one alternative, so it is a clear either-or. The second has 0.20: the doubt is spread across everything. The chance that billing is right is 0.60 in both. Use probabilities for decisions, and confidence for sorting a review queue or spotting inputs where Wity has no strong view at all.

Direct and thought answers#

When Wity thinks before answering (see Reasoning), the answer also includes direct_probabilities (or direct_noul): what it would have said without thinking. The main probabilities are the ones to act on. The direct ones show what thinking changed, which is useful when you debug a surprising answer or see how often thinking flips a result on your data.

"route": {
  "choice": "fraud_review",
  "probabilities":        { "refunds_queue": 0.28, "fraud_review": 0.68, "general": 0.04 },
  "direct_probabilities": { "refunds_queue": 0.55, "fraud_review": 0.41, "general": 0.04 },
  "reasoning": { "mode": "auto", "thought": true, "reason": "close_call", "forecast": false, "thought_tokens": 212 }
}

Here the direct read was a close call that leaned towards refunds. After thinking, fraud review is clearly ahead. Your code uses 0.68 on fraud review; the direct numbers explain why this ticket took a second longer.

Watching it in production#

Inputs drift: new products, new phrasing, a new kind of scam. Check now and then that the probabilities still mean what they should. Whenever a person confirms or corrects an answer, log the probability and whether it was right, then group by probability:

from collections import defaultdict
 
# log (probability of the pick, was it right?) whenever a person later confirms or corrects
buckets = defaultdict(lambda: [0, 0])
for p, ok in reviewed:
    b = min(int(p * 10), 9) / 10          # 0.0, 0.1, … 0.9
    buckets[b][0] += 1
    buckets[b][1] += ok
 
for b in sorted(buckets):
    n, right = buckets[b]
    print(f"{b:.1f}–{b + 0.1:.1f}  n={n:4d}  right {right / n:5.1%}")

In each bucket, the share that was right should be close to the bucket's probability. For example, around 85% right for answers between 0.8 and 0.9. A bucket that falls well short is your early warning. Tighten that threshold, improve the option descriptions, or add the missing facts to the state. Keep in mind that the cases people review are usually the uncertain ones, so sample some high-probability answers for review as well.