The Sampling Ceiling: Why LLM-Authored Constraint Systems Cannot Exceed the Reliability of the Model That Authored Them

In plain language

The study asked whether better prompts or self‑checks can make LLM‑written rules arbitrarily reliable. It showed that a 40‑rule specification with a 2 % error per rule gives only about 45 % chance of no mistakes using prompts alone. This matters because any system that lets a stochastic generator write the standards it is judged by cannot be more reliable than the generator itself. It is like trying to bake a cake using a recipe that sometimes miswrites ingredients; even a good taste‑tester cannot fix a badly written recipe.

Restated for a non-expert reader; the original wording follows. 2 statement(s) without support in the text were removed. Still uses some technical language.

In the paper's words

Large language models (LLMs) are increasingly used to author the artifacts that other software systems must obey: specifications, system prompts, interface contracts, and validation rubrics. Practitioners currently assume that better prompting, self-critique, or meta-level instruction can raise the reliability of these authored constraint systems arbitrarily. This paper argues that assumption is false.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. We quantify the gap with explicit arithmetic: for a specification with $N=40$ constraints and a per-constraint error rate $\varepsilon=0.02$, a prompt-only pipeline achieves a clean-run probability of $0.98^{40}\approx 0.4457$, while a hybrid pipeline in which a formal verifier catches errors with probability $c=0.95$ achieves $(1-\varepsilon(1-c))^{40}\approx 0.9608$, a factor of $2.16$ improvement that no prompt change can match under the model.

   from Abstract

2. A complementary aggregate estimate $p_{\text{system}} = p \cdot v = 0.85 \times 0.54 = 0.4590$ under an assumed base correctness and the characterization yield reported in adaptive reliability qualification (Section 4).

   from 1. Introduction

3. The ceiling result also bears on adjacent problems that share the same structure: any pipeline in which a stochastic generator authors the standard against which its own output is judged, including automated evaluation and automated grading, inherits the same bound; and the formal-layer argument parallels the two-layer separation between human-auditable fragments and machine optimizers known in quantum circuit representation [12].

   from 1. Introduction

4. That work derives a per-chapter reliability threshold of $p^{*}\approx 0.958$ for a $50\%$ chance of a clean run and a per-constraint threshold of $q^{*}\approx 0.986$, and it reports that at a realistic adherence of $p=0.9$ the phase success probability is only $P_{\text{phase}}\approx 0.185$.

   from 2. Background and Related Work

5. The same study reports an automated-checking benefit — the per-constraint generation threshold drops to $0.979$ when automated checking is added — which is the hybrid-pattern signal this paper formalizes.

   from 2. Background and Related Work

6. The adaptive test-planning work [7] frames reliability qualification of semiconductor devices as a sequential decision problem and reports that over $5{,}000$ planning iterations an adaptive MCTS-based planner improves characterization yield from $20\%$ to over $54\%$, achieving $39\%$ cumulative yield while keeping damage fractions below $0.57$.

   from 2. Background and Related Work

How it is done

1. We model LLM generation as stochastic sampling from a fixed conditional distribution and constraint satisfaction as deductive verification, and we prove a ceiling result: any prompt-level transformation of the sampling process leaves the per-constraint error rate bounded below by a model-intrinsic minimum $\varepsilon_{\min} \gt 0$ on constraint types the model cannot self-verify, so no composition of prompts can close the reliability gap.

   from Abstract

2. We show that for constraint types on which the model's induced per-constraint error probability is $\varepsilon\gt 0$ and on which the model has no internal verification channel, no prompt can reduce the induced per-constraint error rate below a bound determined by the model; the bound is a property of the model, not of the prompt.

   from 1. Introduction

3. The combination — fine-grained factual errors that neither humans nor zero-shot LLMs reliably catch — is the operational form of the intrinsic error bound: errors of transcription and detail are exactly the constraint types the ceiling covers, and no prompt instructs them away because the model does not know which details are wrong.

   from 2. Background and Related Work

4. We model an LLM as a conditional distribution over strings.

   from 3.1 The model

What would falsify it

1. The claim is concrete and falsifiable.

   from 1. Introduction

2. A design principle for hybrid architectures — LLMs draft, formal tools enforce — with the smallest experiment that would falsify or de-risk the claim (Sections 6 and 7).

   from 1. Introduction

3. Its proposed falsification test is explicit: if a pilot yields per-chapter adherence below $0.958$ despite refined prompts, the hypothesis that prompting alone can raise adherence sufficiently is falsified.

   from 2. Background and Related Work

Where the claim stops

1. It is not a measurement of any deployed system, and it does not say that prompting is useless — Section 4.7 shows a projected best-case refinement halving $\varepsilon$ and lifting clean-run probability from $0.4457$ to $0.6690$, a real gain.

   from 6. Discussion

2. Section 4.1 also carries the whole-artifact product form $p_{\text{system}} = p \cdot v$ with $p = 0.85$ (assumption) and $v = 0.54$ (the lower bound of the characterization yield reported in [7]), giving $0.4590$.

   from 6. Discussion

3. This form is more pessimistic than the per-constraint form at the same operating point because it charges the full generation failure rather than only the residual error; we retain it as a bounding estimate under explicitly stated assumptions, per the convention declared in Section 1.

   from 6. Discussion

Where to read

1. Abstract214 words · 1 min
2. 1. Introduction714 words · 3 min
3. 2. Background and Related Work1,551 words · 7 min
4. 3. Methods731 words · 3 min
5. 4. Analysis910 words · 4 min
6. 5. Results395 words · 2 min
7. 6. Discussion935 words · 4 min
8. 7. Practical Implications and Engineering Pathways858 words · 4 min
9. 8. Conclusion239 words · 1 min
10. Provenance and AI Involvement240 words · 1 min