Overview

Because a single number hides exactly how much it could plausibly move on a different sample -- without the interval, a reader has no way to tell whether "83% pass rate" means "we are confident it is close to 83%" or "it could genuinely be anywhere from 70% to 95%." The interval is the part that actually carries the information a decision needs.

Because a proportion computed from n cases is an ESTIMATE of an unknown true rate, not the true rate itself -- a different sample of the same size, drawn from the same system, would give a slightly different number. Every technique in this topic (intervals, sample-size planning, significance tests) only makes sense once the pass rate is recognized as an estimate with its own uncertainty.

Sampling error is the spread you get from re-measuring the same true rate on a different sample -- it is quantifiable and expected, not a bug. Two runs of the same system against two different 40-case samples can legitimately land a few points apart with nothing wrong at all; the question sampling error answers is "how large a spread is expected by chance alone."

The range of true rates that are statistically consistent with what you observed, at a stated confidence level -- not "the true rate is definitely in this range," but "if you repeated this sampling procedure many times, an interval built this way would contain the true rate that proportion of the time."

The normal approximation assumes the sampling distribution of a proportion is symmetric and bell-shaped, which breaks down when n is small or the true rate sits near the boundary (0 or 1) -- it can even produce an interval that extends past 0 or 1, or one silently clipped to [0, 1] that hides rather than fixes the failure. The Wilson and Clopper-Pearson interval methods are built to stay well-behaved in exactly these conditions.

The effect size you need to be able to detect and the precision (interval half-width) you need to achieve -- "how many cases" is not a guess or a round number, it is the output of a calculation that takes those two inputs, and it should be computed and written down before data collection, not reverse-engineered afterward to justify whatever sample happened to be collected.

Because each strategy determines WHICH cases end up in the sample, and that selection process can itself bias the estimate independent of sample size -- a convenience sample (e.g., "the last 40 requests I happened to have logs for") is not a random draw from the true population, so no amount of increasing n fixes the bias baked into how the cases were chosen in the first place.

Because a random sample draws cases from a rare stratum in roughly proportion to its tiny population share -- at any feasible sample size, that stratum's own estimate would rest on too few cases to be meaningful. Deliberately oversampling the rare stratum, then reweighting its contribution back down to its true population share when computing the overall estimate, gets visibility on the rare mode without distorting the overall number.

Because on a task where one label dominates (say, 80% of items are genuinely "pass"), two raters could each independently guess "pass" most of the time and land near 85% raw agreement purely by chance, without either rater doing any real discrimination between pass and fail. Raw agreement does not subtract out how much agreement chance alone would produce.

(observed agreement - chance-expected agreement) / (1 - chance-expected agreement) -- it starts from raw agreement, subtracts what chance alone would produce given each rater's own marginal label distribution, then rescales so the result is 0 at chance-level agreement and 1 at perfect agreement.

Rater count (two raters vs. more than two -- Cohen's kappa vs. Fleiss' kappa), and label type (nominal vs. ordinal -- plain kappa vs. a weighted/ordinal variant that penalizes a near-miss less than a wild miss). Missing data is a third factor -- Krippendorff's alpha handles incomplete rating matrices that plain kappa cannot.

When one label's prevalence is extreme (near 0% or 100%), the chance-expected agreement term is already very high, which can shrink the coefficient toward zero or even negative even when raw agreement is high and the raters are behaving sensibly -- a coefficient near zero on a 90%-prevalence task does not necessarily mean the raters disagree badly. The fix is to always report label prevalence alongside the coefficient, not to abandon chance correction.

Because a coefficient is itself computed from a finite sample of rated items -- it is an estimate of an unknown true agreement level, subject to the same sampling error a pass rate is. Reporting a bare kappa of 0.60 without an interval hides whether that 0.60 is a precise estimate or one that could plausibly be anywhere from 0.30 to 0.85 on a small sample.

Because there is no single objective ground truth to score the judge's accuracy against -- both the judge and the human are raters, each producing their own fallible read of an item, and the question "do these two raters' verdicts agree, beyond what chance would produce" is exactly the chance-corrected-agreement problem this topic already built the tools for.

Because a judge's reliability genuinely differs by criterion -- an objective criterion like correctness is far easier for a judge to assess consistently than a subjective one like tone, and pooling both into one number would hide exactly the criterion where the judge should be trusted least.

The human-agreement ceiling is the chance-corrected agreement between two independent human raters on the same items -- it is the best agreement level realistically achievable on a task, because even two careful humans do not agree perfectly on a genuinely subjective criterion. Comparing a judge against perfect agreement (kappa = 1.0) sets an unreachable bar; comparing it against the measured human ceiling asks the fair question, "does the judge do about as well as a second human would?"

Because both A's and B's pass rates are themselves noisy estimates -- a difference of a few points between two runs could be the real effect of the change, or it could be sampling noise that would vanish or reverse on a re-run. A hypothesis test asks specifically whether the observed gap is larger than sampling noise alone would plausibly produce.

Because a paired test (like McNemar's) looks at how EACH SHARED CASE'S verdict changed from baseline to candidate, which cancels out the variance that comes from item-to-item difficulty -- an unpaired test throws that per-item information away and only compares two aggregate rates, which is much noisier and needs a larger gap or a larger n to detect the same real effect.

A huge eval set (n = 8000) can detect a genuinely real but tiny gap (0.85 vs. 0.862, p = 0.0015) that is statistically significant yet falls below a team's stated materiality threshold and is not worth shipping for. The reverse: a real, meaningful gap (0.70 vs. 0.78) can fail to reach significance on a too-small eval set (n = 25, p = 0.71) purely from insufficient power, even though it would clearly matter if properly measured -- the fix there is a larger n, computed via power analysis, not concluding "no effect."

It approximates the sampling distribution of almost any statistic -- a median, a kappa coefficient, a custom aggregate -- by resampling the observed data with replacement many times and recomputing the statistic each time, which works even when no closed-form formula for that statistic's standard error exists (as is the case for a median or for kappa).

Because each individual test carries some chance (typically 5%) of a false positive on its own, and testing many criteria multiplies that chance across the whole batch -- with enough criteria tested, finding at least one "significant" result by pure luck becomes likely even when nothing real changed anywhere. Corrections like Bonferroni or Benjamini-Hochberg adjust the significance threshold to account for how many tests were run.

The noise floor is the spread you would observe just from re-running an UNCHANGED, stochastic system repeatedly -- some amount of run-to-run variation exists even with zero real change. A regression bar smaller than this measured noise floor would flag ordinary noise as a regression on every run, producing false alarms regardless of whether anything actually changed.

Generation variance comes from re-running the SAME fixed cases and getting a different outcome each time (the system's own stochastic behavior); case-sampling variance comes from drawing a DIFFERENT set of cases from the same population each time. They are measured by holding the other source fixed in turn, and separating them matters because they suggest different fixes -- high generation variance points at the system's own instability, high case-sampling variance points at needing a bigger or more representative eval set.

The estimate, its interval, the sample size (n), and the method used to compute it. Any of the four missing turns a defensible statistic back into an unreportable bare number -- this is the single discipline every worked example and the whole capstone report in this topic enforces without exception.

No -- with 92% prevalence, a judge that always outputs "pass" regardless of the actual item would already agree with human raters roughly 92% of the time, MORE than the reported 88%. Raw percent agreement on a skewed task overstates how much real discrimination is happening (co-09); the team needs to report a chance-corrected coefficient (co-10) alongside the prevalence (co-12), not the raw agreement rate alone.

A significance test, not just two printed numbers (co-17) -- at n = 30, a gap this size could easily be sampling noise rather than a real effect, and if the same 30 cases were used for both runs (paired), a paired test like McNemar's (co-18) would be both more appropriate and more sensitive than treating the two runs as unrelated samples.

Testing 15 criteria at once (co-21) makes finding at least one nominally significant result by pure chance likely, even if nothing real changed on any criterion. The team needs a multiple-comparisons correction (Bonferroni or Benjamini-Hochberg) applied across all 15 tests before trusting that one "win" -- an uncorrected p = 0.041 out of 15 tests is exactly the kind of phantom result correction is designed to catch.

Concordance must be estimated per criterion, never pooled (co-15) -- even if the two numbers happen to coincide at 0.55, the underlying reliability, sample sizes, human ceilings, and confidence intervals for "factual-correctness" and "helpfulness-tone" could look completely different on inspection. A single "0.55 overall" number could paper over one criterion where the judge is excellent and another where it is barely above chance.

No -- 2 cases is far too few to support any claim about that stratum's own pass rate (co-08). This is exactly the case for deliberate stratified oversampling: draw a fixed minimum number of cases FROM that rare stratum specifically (say, 40-60), evaluate the candidate on those, then reweight that stratum's own estimate back down to its true 2% population share when computing any OVERALL rate -- naive random sampling at any feasible total n leaves the rare stratum invisible.

Not necessarily broken -- this spread could simply be the suite's own measured noise floor (co-22), the expected variation from re-running a stochastic system unchanged. The team should decompose that 5-point spread into its generation-variance and case-sampling-variance components (co-23) if the eval set itself also varies between runs, derive a regression bar from the measured spread, and only then judge whether a future candidate's gap against baseline clears that bar or falls inside ordinary noise.

Every claim in the report needs its estimate, interval, n, and method (co-24) -- "improved" with no supporting numbers is not a reportable claim at all under this topic's discipline. The reviewer should require, at minimum: the baseline and candidate point estimates with intervals, the sample size the comparison rests on, whether the comparison was paired, the test used and its p-value (or corrected p-value, if multiple criteria were tested), and whether the gap clears a stated materiality threshold -- not just statistical significance.

The honest response is to compute and report the interval before answering "is it good" at all (co-13) -- at n = 12, a bootstrap or exact interval around kappa = 0.71 will likely be very wide (the point estimate could plausibly reflect a true kappa anywhere from moderate to near-perfect agreement), and the team should say so rather than presenting 0.71 as a precise measurement. There is also no universal threshold for "good" kappa (this topic teaches the verbal scales as convention, not finding) -- the right response names both the interval's width and that any threshold needs justifying against the decision's own stakes, not a borrowed rule of thumb.

Because there is no coefficient that is both chance-corrected AND immune to prevalence's effect on its own scale -- the prevalence sensitivity is a mathematical consequence of correcting for chance in the first place, not a flaw specific to one formula. Reporting prevalence alongside the coefficient lets a reader interpret a low kappa correctly: "the coefficient is low partly because the label distribution is skewed, not necessarily because the raters disagree badly" -- context the number cannot carry by itself.

Both replace an unreachable or arbitrary absolute bar (perfect agreement; "any nonzero effect matters") with a bar DERIVED from the actual system or decision at hand (what two humans actually achieve; what this team has stated is worth acting on). In both cases, the absolute bar would set an unrealistic or irrelevant standard, while the derived bar answers the question that actually matters for the decision being made.

Computing from definition first forces genuine understanding of what the statistic IS, not just which function call produces it -- a reader who only ever calls cohen_kappa_score never sees why it subtracts a chance term. Verifying the two agree also catches a real class of error: passing arguments in the wrong order, using the wrong variant of a coefficient, or misreading a library's own default parameter (as this topic's own accuracy notes flag for proportion_confint's default method) -- errors that would otherwise silently produce a plausible-looking but wrong number.

Pairing is about HOW a single comparison is structured (using per-item information to cancel noise), while the multiple-comparisons problem is about HOW MANY separate comparisons are being tested at once (each one carrying its own chance of a false positive). A single paired test on one criterion is more sensitive than an unpaired test on that same criterion; running that same paired test independently across fifteen criteria still needs a correction, because pairing improves each individual test's power without changing how many independent chances at a false positive the batch of fifteen tests creates.

In both cases, treating the whole as undifferentiated hides a component that matters. A single overall sample size can be adequate for the dominant stratum while leaving a rare, high-stakes stratum invisible; a single overall noise-floor number can hide whether the spread comes mostly from the system's own stochastic generation (suggesting the system itself needs attention) or from which cases happened to get sampled (suggesting the eval set needs attention). Decomposing first is what lets the right fix be identified, rather than a generic "collect more data" response applied to a problem that a bigger sample alone would not actually solve.

Scoping by shared machinery would make this topic balloon into a general statistics course with no natural stopping point -- nearly any statistical technique COULD conceivably touch an eval pipeline somewhere. Scoping by "does this change whether an eval result can be trusted" keeps every technique here directly load-bearing for the one decision this topic exists to support, and cleanly hands off techniques that answer a genuinely different question (did a product change move a business metric) to analytics-and-experimentation, which shares the machinery but not the purpose.