Forecast questions

Decisions versus forecasts#

"Which queue does this ticket belong to?" has one correct answer. It is already settled by the ticket, and the best response is a confident pick. "Will this lot fail inspection?" often does not. The outcome depends on something that hasn't happened yet, like which units get sampled. The honest answer is a probability, and it can be well below 50% even when the question is about failure.

The difference matters because the two go wrong in different ways. A decision goes wrong by picking the wrong answer. A forecast goes wrong by being confidently wrong about the odds. Asked "will this lot be rejected?", a quick read tends to lean towards yes simply because rejection is what the question is about. Getting the number right means working through the evidence: how many units, how many defective, how many drawn.

What Wity does#

In auto and always modes Wity checks whether a question asks for a forecast. When it does, it thinks, working through the rates, counts, sampling rules and conditions in the state, and returns the distribution it worked out. The answer's reasoning record says so: reason is forecast and forecast is true. In off mode there is no thinking, so the question is answered as a direct read.

Example: acceptance sampling#

A factory ships parts in lots of 80. Before a lot ships, an inspector draws 8 units at random and rejects the whole lot if any of them is defective. Lot LT-1791 is known to contain 2 defective units. Planning wants to know whether this lot will get through, because a rejected lot means a full re-inspection and a missed truck.

The state gives Wity everything the outcome depends on: the lot size, the known defects, and the inspection rule in plain words. The instructions ask for probabilities that reflect the evidence, which makes it clear this is a how-likely question.

{
  "state": {
    "lot": "LT-1791",
    "units_in_lot": 80,
    "known_defective_units": 2,
    "inspection": "8 units are drawn at random without replacement; the lot is rejected if any drawn unit is defective."
  },
  "questions": {
    "rejected": {
      "type": "noul",
      "instructions": "Will lot LT-1791 be rejected at inspection? Give probabilities that reflect the evidence.",
      "criteria": { "true": "At least one drawn unit is defective", "false": "No drawn unit is defective" }
    }
  },
  "reasoning": "auto"
}

The worked-out answer is 0.19, about a one-in-five chance of rejection:

"rejected": {
  "type": "noul",
  "noul": 0.19,
  "direct_noul": 0.81,
  "reasoning": { "mode": "auto", "thought": true, "reason": "forecast", "forecast": true, "thought_tokens": 296 }
}

That matches the exact odds. The lot passes only if all 8 draws come from the 78 good units, so the chance of rejection is 1 − C(78,8)/C(80,8) = 1 − (72 × 71)/(80 × 79) ≈ 0.191. direct_noul shows what a quick read would have said, 0.81, leaning heavily towards rejection. The two numbers lead to opposite plans. At 0.81 you would pull the lot for re-inspection. At 0.19 you ship it, and keep a small buffer in case it bounces.

Example: how late will it be?#

Forecasts aren't limited to yes or no. A score question with buckets as levels asks for the chance of each bucket. A freight planner wants to know how late a sea shipment is likely to be, and has the history of the lane:

{
  "state": {
    "shipment": "SH-5530, Rotterdam → Gdańsk, sea freight, departs 2026-10-02",
    "lane_history": "Last 200 shipments on this lane: 142 on time, 38 one to two days late, 14 three to seven days late, 6 more than a week late.",
    "notes": "No port or weather alerts for either end."
  },
  "questions": {
    "delay": {
      "type": "score",
      "instructions": "How late will SH-5530 arrive? Give probabilities that reflect the evidence.",
      "criteria": ["On time", "1–2 days late", "3–7 days late", "More than a week late"]
    }
  },
  "reasoning": "auto"
}

With no alerts in the notes, nothing suggests this shipment differs from the lane's record. The forecast is the base rate: 142, 38, 14 and 6 out of 200.

"delay": {
  "type": "score",
  "score": 0.42,
  "probabilities": { "0": 0.71, "1": 0.19, "2": 0.07, "3": 0.03 },
  "legend": { "0": "On time", "1": "1–2 days late", "2": "3–7 days late", "3": "More than a week late" },
  "confidence": 0.39,
  "reasoning": { "mode": "auto", "thought": true, "reason": "forecast", "forecast": true, "thought_tokens": 241 }
}

The expected level, 0.42, is useful for sorting shipments by risk. The bucket probabilities are what you plan with.

When conditions change the odds

Base rates are a starting point. When you know how the odds shift under a condition, and whether that condition holds, put both in the state:

"lane_history": "Last 200 shipments: 142 on time. When Gdańsk reports congestion, only half arrive on time.",
"notes": "Gdańsk port reported congestion this morning."

Now the relevant rate is the one under congestion, not the overall 71%, and the chance of arriving on time should drop to about 50%. Wity uses the evidence you give it. It doesn't know about today's port congestion unless you tell it, and it can't apply a conditional rate you don't provide.

Using forecasts#

A forecast's value is in the full distribution. Multiply each outcome's probability by what it would cost, and you get an expected cost you can compare with the price of doing something about it:

p = a["delay"]["probabilities"]
penalty = {"0": 0, "1": 200, "2": 800, "3": 2000}          # EUR owed to the customer per bucket
 
expected_penalty = sum(p[k] * penalty[k] for k in p)          # 0.19*200 + 0.07*800 + 0.03*2000 = 154
 
if expected_penalty > EXPRESS_UPGRADE_COST:                    # say, 120
    book_express_leg(shipment)

With the lane's base rates, late-delivery penalties are expected to cost about €154. That is more than a €120 express upgrade, so booking the upgrade is the cheaper bet. Under congestion the expected penalty would rise, and the case for upgrading gets stronger. A single "probably on time" label would have hidden all of this.

Getting good forecasts#

• Put the numbers in the state: counts, rates, base rates, and the rule that decides the outcome. Wity works the odds out from them; it doesn't look them up anywhere.
• Say what you want in the instructions, e.g. Give probabilities that reflect the evidence.
• Make the outcomes exhaustive and mutually exclusive, so the probabilities have something to add up to. For quantities, make the buckets cover the whole range, with an open-ended last bucket.
• State the time window and the conditions: "by Friday", "given no port alerts". A forecast without a window is ambiguous.
• Use auto or always. With off, the question is answered as a direct read.