The operations team says there is no rule for this, they just use judgement. How do you model that?

There is a rule; nobody has had to say it out loud

"We use judgement" is almost never true as stated. What it usually means is that the decision has never been written down, that the person making it cannot introspect on their own criteria, and that some residue of the decision really is discretionary. Those three things need entirely different treatment, and lumping them together produces one of two bad systems: an automation that encodes a guess as policy, or a screen that captures nothing but a click.

Asking for the rule does not work, and it is worth understanding why rather than trying harder. People asked to state their criteria produce a plausible reconstruction, weighted towards the reasons they consider respectable and towards the last memorable case. You will get a rule, it will be sincere, and it will not match what they do.

Elicit from decisions, not from criteria

The reliable technique is to work backwards from outcomes. Take thirty recent decisions of this type, real ones with identifiers, and go through them with the person who made them, asking only what they decided and why this one. Do not ask what they would generally do. The reasons attached to specific cases are far closer to the operative criteria, because a specific case cannot be answered from the procedure.

What emerges looks like this, and the pattern is usually visible by the twentieth row.

Two factors do most of the work here — the amount and the pattern of repetition — and both are already data you hold. A third is available in principle: case 4501 turns on the details disagreeing with what the customer told support, which is a fact that exists but lives in a ticketing system nobody has joined to this one, and which no field on either side currently expresses as a comparison. Treat that as a candidate factor with a cost attached rather than as discretion, because calling it judgement when it is really an unbuilt integration is how a factor stays manual for five years. One case turns on something nobody records at all, the renewal date, and one turns on a delegated limit that is a hard rule which had simply not been mentioned because it never feels like a decision. That distribution is typical: most of the volume goes on two or three factors you already have, one or two more are derivable if somebody pays for the join, a hard rule is hiding in plain sight, and a genuinely irreducible remainder rests on commercial context that only a person has.

The follow-up question that separates the categories is to hand back a decided case with one fact changed and ask whether the answer changes. Vary the amount until the answer flips and you have found a threshold the business did not know it had. Vary something they claimed mattered and watch the answer stay the same, and you have found a reason that is a justification rather than a cause.

Automate the ends, record the middle

The output of that exercise is not a rule engine. It is a boundary. There is a band of cases where the factors agree and any competent operator would decide the same way, and a band where they would not, and the design follows the split rather than trying to eliminate it.

flowchart TD
    A[Case arrives with the facts as they stand] --> B{Factors agree and inside the limit}
    B -->|yes| C[Decided by rule and stamped with the rule version]
    B -->|no| D[Routed to an operator with the recommendation shown]
    C --> F[Decision stored with inputs recommendation and reason]
    C -->|overridden| H[Override recorded against the rule version]
    D --> E[Operator decides and selects a reason from a controlled list]
    H --> F
    E --> F
    F --> G[Weekly review of disagreements moves the boundary]

Two edges are worth looking at. The loop from the bottom back to the branch says the boundary is a parameter the business adjusts from evidence, not a constant an engineer chose once. The short branch out of the rule-decided path is the override route, and it has to exist: an automated decision a person cannot overturn has no override rate to measure, and the awkward cases will find an unrecorded back door instead.

The decision record is the part candidates skip and the part that makes the rest work. It has to hold the inputs as they were at the moment of the decision rather than a foreign key to a mutable customer record, because a decision that cannot be reconstructed cannot be reviewed, defended or learned from. It holds the recommendation the system made, separately from the decision the human made, which is what lets you measure whether the two ever agree. It holds a reason from a controlled list, with free text in addition rather than instead, because free text is unqueryable and a reason nobody can count is a reason nobody will act on. And it holds who decided and under which delegated authority, since the same person may hold different limits in different months.

What you should refuse to automate, and how to say so

There is commercial pressure to push the boundary outwards, and the argument for resisting it has to be made in the business's terms rather than as engineering caution. Two grounds work.

The first is that the discretion is doing something the factors cannot see. The operator who approved case 4495 knew about a renewal, and that knowledge is worth more than the cost of the exception. Automating that case does not remove the judgement; it moves it to whoever complains afterwards, and it converts a decision somebody owned into an outcome nobody chose.

The second is asymmetry of cost. Where being wrong in one direction is cheap and reversible and the other is not, the expensive side stays with a person for longer than the accuracy figures alone would justify. That is a domain judgement rather than a modelling one and it should be recorded as such.

The failure this guards against is subtle and common: an engineer observes the current heuristic, implements it, and the heuristic becomes policy by deployment. Nobody approved it, it is now enforced more consistently than any human ever enforced it, and the discretion that used to absorb the awkward cases has gone — so the awkward cases turn into complaints, and the complaints are answered by a manual back door that records nothing. You have automated the easy half of the job and destroyed the mechanism that handled the hard half.

The signal that tells you the boundary is wrong

Because the design records both the recommendation and the decision, it produces its own feedback. If operators agree with the recommendation almost every time in some slice of cases, that slice belongs inside the automated band and you are wasting a person's attention. If they override frequently, either the factors are wrong or a factor is missing, and the reason codes on the overrides will usually name it. If they agree with each other rarely on the same kind of case, the problem is not the model at all — the business has no settled position and the modelling work has surfaced a decision that needs an owner.

Reviewing that regularly, with the operations lead rather than for them, is what turns "we use judgement" into a documented rule with an explicit discretion around it. That is the outcome to aim at, and note what it is not: it is not the elimination of judgement, it is the narrowing of it to the cases that deserve it.