Overview

A success-or-error model. A probabilistic feature succeeds partially, fails confidently, and fails differently on identical inputs -- none of which fits a two-state (correct / error) design, so the interface has no state for the feature's actual, dominant failure mode.

Confidently wrong, subtly wrong, refused, truncated, slow, and inconsistent across identical requests -- six distinct experiences, each requiring its own design response.

What a feature is told to be on first contact determines how its errors are read for the rest of the relationship -- an overpromise turns the first mistake into a broken promise; an honest framing absorbs it.

Over-trust (accepting output without scrutiny) and blanket distrust (abandoning the feature after one bad answer). Both are caused by the interface, not the user.

The better the system gets, the worse the scrutiny becomes -- users under-scrutinize output from a system that is usually right, and rising reliability worsens this effect rather than fixing it.

At the moment and location the user could still act on it -- a caveat the user only sees after acting (a footer, a settings page) carries no functional value regardless of its wording.

A numeric score is precise but routinely misread as a probability of correctness; a qualitative band is read more accurately but discards information. Every representation trades precision against being read correctly.

A confidence score typically measures how sure the model sounds to itself while generating a sequence of tokens, not whether the underlying factual claim is true -- a model can sound very sure while retrieving the wrong source entirely.

A means to verify the claim themselves -- converting "should I trust this?" into "let me check this," a strictly stronger and more reliable position than trusting an opaque number.

Structuring the output so checking it costs less than producing it -- verifiability is a property of the output's shape, designed in, not an afterthought.

The review step must be measurably faster than doing the underlying work unaided -- timed, not assumed.

A reviewer's accuracy decays with volume and with the system's own reliability. A review design that ignores this decay is theatre -- it looks like safety and is not, because scrutiny predictably collapses partway through a review session.

Its reversibility and its blast radius -- not a single uniform policy applied to every action regardless of consequence.

A field-level error (like a transposed digit) -- because it shows the exact value that will change, side by side with its source, rather than merely asking "do you want to do this?"

Undo's value compounds across every future error the feature will ever produce, while an accuracy improvement's value is capped and can erode with the next model change -- and undo is often cheaper to build.

A concrete, user-visible state for every way the model can fail to respond usefully -- unavailable, rate-limited, too slow, or returning unusable output -- rather than leaving the case undesigned.

Through what fills the wait -- a bare spinner, streaming output, or honest staged progress each produce a different perceived speed and a different level of user confidence that the system is working, independent of the actual duration.

An observable trigger condition that determines exactly when the system falls to it -- a vague or unmeasurable trigger is a rung that will not fire reliably when it should.

Because the user has no way to know they need to check it -- a wrong answer rendered identically to a right one removes the user's only chance to catch it before acting.

Scoping is available immediately and fully within a team's own control; model improvement is not guaranteed and depends on factors outside the product team's control.

A threshold, an interval, and a named acceptable worst case -- a single average pass-rate number alone is not a launch criterion.

Because an eval set, however good, does not contain the real, unpredictable production traffic that can still break a feature -- staged rollout watches pre-agreed guardrail metrics that only become observable once real users arrive.

Criteria decided during an incident are written by people under pressure to keep the feature live -- writing them down in advance, when nobody has a stake in the outcome yet, is what makes them genuinely enforceable.

It recovers the current user immediately, and it captures a structured record that feeds the next error-analysis pass -- a correction that is applied and discarded wastes the signal that would prevent the same failure recurring.

This is co-04's blanket-distrust quadrant: with no way to distinguish a trustworthy summary from an untrustworthy one, the rational user response to one visible mistake is to stop trusting all of them. The fix is co-06's selective uncertainty signal or co-09's provenance -- either gives users a case-by-case basis for trust instead of an all-or-nothing verdict.

co-05: rising accuracy causing falling review time is the predictable signature of automation bias, not simply efficiency. The improving headline number may be moving the eventual failure further out along a longer run of correct answers, eroding scrutiny in the meantime -- this needs a vigilance-decrement mitigation (co-12), not an unqualified celebration.

The numeric score's risk -- being misread as a probability of correctness -- is most severe here, because a user reading "87%" as "87% likely correct" on a health-related recommendation could make a consequential decision based on a false precision the score never actually claimed (co-08).

co-14's preview-and-diff-before-apply: a text description does not show the exact change, so it cannot catch an unrelated alteration the way an actual line-by-line diff would have.

Undo must be complete, not partial. Reversing the tag while leaving the notification's effect intact creates a false belief that the whole action was undone -- worse than no undo, because it produces unearned confidence.

A degradation design (co-16) with an observable trigger (co-18) -- a timeout-based detection rule that switches the timed-out half of requests to an explicit, honest state (a fallback rung or a labelled unavailable message), rather than leaving every request in an indistinguishable spinner state regardless of whether it will ever resolve.

A named acceptable worst case (co-21). An average-only threshold can hide a coherent failure cluster small enough not to move the headline number, which is exactly the gap a stated worst-case clause is designed to catch before launch.

The pre-agreed rollback criteria should be honored as written -- this is precisely the scenario co-23 anticipates: rollback decisions made in the heat of an incident are systematically biased toward staying live, which is why the criteria were written and signed off in advance, when nobody had a stake in the outcome yet.

Scoping was available immediately and entirely within the team's own control; model improvement depends on factors (a provider's roadmap, training data, research timelines) the team does not control and cannot guarantee. Reaching for the harder, less controllable lever first would have delayed a fix that was available on day one.

co-24's routing requirement: fixing the current user's view recovers that one user but, if the correction is not logged and routed into a systematic weekly review or the eval dataset, the signal that would have prevented recurrence is discarded every time.

After: an inline badge directly beside any answer flagged low-confidence, positioned before the user's next likely action.

Root cause: co-06 requires an uncertainty signal to sit between the answer and the user's next action. A footer disclaimer fails this regardless of its wording, because most users act on the answer before ever scrolling to see it.

After: "Nimbus's self-assessed certainty: High" -- relabelled to avoid the word "accurate," and calibrated against held-out eval accuracy per confidence bucket rather than displaying the raw model score directly.

Root cause: co-08 -- a raw confidence score measures how sure the model sounds to itself, not whether the claim is true. Labelling it "accurate" manufactures unearned trust in exactly the confidently-wrong case this course's worked scenarios trace repeatedly.

After: a friction table mapping each action's confirmation level to its own reversibility and blast radius -- no confirmation for viewing or saving; a full preview-and-diff plus explicit confirmation only for the irreversible, high-blast-radius external email.

Root cause: co-13 -- uniform friction trains users to click through the modal reflexively, which erases its protective value for the one action that genuinely needed it.

After: undo either reverses every effect (including sending a correction message to the notified colleague) or the interface explicitly states which specific effects it could not reverse.

Root cause: co-15 -- a partial undo is worse than none, because it creates a false belief that the entire action was reversed.

After: rung 2's answers carry a small "fast mode" label distinguishing them from rung 1's.

Root cause: co-19 -- a fallback hierarchy can still fail silently at any individual rung, even after the hierarchy itself is well-designed; each rung needs its own explicit visibility check.

After: "Ship if pass rate ≥ 95% with a 95% confidence interval no wider than ±3 points, AND no failing case is a confidently-wrong answer with no citation attached."

Root cause: co-21 -- a single average threshold with no interval and no named worst case can be cleared by a launch that hides a coherent, damaging failure cluster small enough not to move the headline number.

The model's stochasticity is a fixed property this course does not attempt to change. The interface is the layer a product team actually controls, and it is where a user's deterministic expectations (one question, one reliable answer) collide with the model's actual behaviour (partial success, confident wrong answers, inconsistency across identical requests). Locating the problem at the interface, not the model, is what makes this course's patterns actionable without requiring a better model first.

A silent failure is not primarily about how often the model is wrong -- it is about the interface rendering an emergent, uncertain output using the visual language of a deterministic, certain one. Even a highly accurate model produces silent failures if its rare wrong answers look identical to its common right ones; the fix (uncertainty signals, provenance, visible degradation states) is squarely an interface response to the determinism/emergence mismatch, not a model-accuracy fix.

Pursuing correctness (higher accuracy) has diminishing, uncertain, and non-compounding returns -- each percentage point is harder to gain than the last, and gains can erode with the next model change. Pragmatism (investing in undo) accepts that errors will keep happening and instead makes every future error cheap, a return that compounds across the feature's entire lifetime. The course consistently prefers the pragmatic, compounding investment over the harder, capped one.

The pragmatic frame asks "is this workflow recoverable," not "is this feature perfect." When neither scoping nor undo can bring a feature's risk down to something a workflow can absorb, shipping it anyway optimizes for having shipped something over the user's actual safety -- exactly the trade this course warns against. Recognizing and documenting that case is itself the pragmatic, correct professional move, not evidence the team failed to find a solution.

A confidence score is an abstraction over a much richer underlying reality -- the full distribution of possible outputs the model could have produced for that input, and the specific reasons any one of them might be wrong. Collapsing that distribution into a single number necessarily discards information, and the user has no way to see what was discarded -- they read the number as if it were the whole picture, which is exactly the abstraction's cost co-08 names directly.

A free-text paragraph is itself a lossy abstraction over the underlying evidence the model drew from -- checking it requires reconstructing which claims came from where. A structured, per-field, per-citation format reduces that abstraction's cost by making the mapping from claim to evidence explicit and cheap to traverse, rather than asking the user to reverse-engineer it from prose. The abstraction is not eliminated -- the output is still a summary -- but its verification cost is sharply reduced.