From memory
2,160
requests
540
exact
1,620 did not match exactly.
Paper 01
FID-056
August 2026
When Not to Generate
How AI Systems Quote Scripture, and What Authoritative Quotation Requires
Which system design is most reliable when the user asks for the exact words of a source of record?
Alex Chao · Fide AI · 8,640 matched requests · 6 model families · 3 open English editions
Finding
Memory produced the requested text exactly in one quarter of trials.
Supplying the passage directly produced 93.61% exact delivery. An authorized lookup tool reached 80.09%. Deterministic insertion, where code rather than the model writes the selected source text, reached 91.25%.
All three source-backed designs still produced mismatches. The failures occurred at different points between selecting a reference and showing the final text.
Interactive delivery designs · 8,640 requests
Four ways to produce the same quotation, with four different points of failure.
Choose a design to see what work belonged to the model, what belonged to the surrounding system, and how often the complete delivery path succeeded.
Selected design
Native parametric quotation
Model owns
The model identifies the passage and writes every word from its parameters.
System owns
The surrounding system supplies instructions but no source text or source tool.
Observed path across all 2,160 trials
2,160
requests
540
exact deliveries
1,620 deliveries did not match exactly.
What the result shows
Exact quotation should not be assumed to emerge reliably from model memory, even for familiar passages.
Where it can fail
Wrong wording, omitted or blended text, refusals, truncation, and provider errors can all produce a mismatch.
- Exact delivery
- 25.00%
- Exact trials
- 540 / 2,160
- Not exact
- 1,620 / 2,160
- Difference from memory
- Baseline
Selected design
Source-supplied quotation
Model owns
The model receives both the authoritative text and the request, then copies the requested span.
System owns
The surrounding system has already selected and supplied the source text.
Observed path across all 2,160 trials
2,160
passages supplied
2,022
exact deliveries
138 deliveries were not exact even though the passage was supplied.
What the result shows
This was the highest pooled rate, but it is a best-case copy test rather than an equal-burden comparison.
Where it can fail
138 observations still failed the strict path-adherent exactness rule through alteration, omission, or extra output.
- Exact delivery
- 93.61%
- Exact trials
- 2,022 / 2,160
- Not exact
- 138 / 2,160
- Difference from memory
- +68.61 points
Selected design
Tool-mediated retrieval
Model owns
The model must resolve the request, call the authorized tool, and reproduce the returned span.
System owns
The surrounding system exposes the source tool and records invocation, lookup, and final delivery.
Observed path across all 2,160 trials
2,160
tool available
2,052
tool invoked
1,730
exact deliveries
108 trials bypassed the tool. Another 322 invoked it but did not complete the exact delivery path.
What the result shows
The tool was invoked in 2,052 trials, but tool availability and invocation were not the same as correct end-to-end use.
Where it can fail
The model can bypass the tool, request the wrong span, or alter retrieved text before delivery.
- Exact delivery
- 80.09%
- Exact trials
- 1,730 / 2,160
- Not exact
- 430 / 2,160
- Difference from memory
- +55.09 points
Selected design
Reference selection plus deterministic rendering
Model owns
The model selects a structured reference but does not write the quotation itself.
System owns
A deterministic layer looks up the selected reference and inserts the authorized source text.
Observed path across all 2,160 trials
2,160
requests
1,973
selection and lookup completed
1,971
exact deliveries
The reported result uses the disclosed parser correction. The original locked score was 1,813 exact deliveries.
What the result shows
Once correct selection and lookup completed, 1,971 of 1,973 outputs were exact. The main rate includes the corrected parser replay disclosed in the paper.
Where it can fail
Deterministic rendering cannot repair a genuinely wrong or unparseable reference or an incorrect requested span.
- Exact delivery
- 91.25%
- Exact trials
- 1,971 / 2,160
- Not exact
- 189 / 2,160
- Difference from memory
- +66.25 points
The quotation path
Four stages stand between a reference and the text a user sees.
Move through the path to see how responsibility shifts across designs and where fidelity can be lost.
The question at this stage
Which work, edition, passage, and span does the request identify?
The first failure can happen before any source text is retrieved. Contextual descriptions require the system to infer a reference, while explicit requests provide one directly.
A wrong or unparseable reference sends every later stage down the wrong path.
Memory
Selection is implicit in the text the model generates.
Source supplied
The caller has already selected and supplied the source.
Tool
The model resolves the reference it sends to the tool.
Deterministic
The model emits a structured reference for the renderer.
The question at this stage
Can the system reach the authorized, versioned source of record?
Source-backed designs differ in how the source enters the system. Supplying text removes retrieval work; a tool or deterministic layer must still locate the requested edition and span.
Tool bypass, unavailable lookup, a wrong edition, or a neighboring span breaks the chain.
Memory
No source is consulted.
Source supplied
The source is placed directly in context.
Tool
The model must invoke the authorized lookup.
Deterministic
The renderer looks up the selected reference locally.
The question at this stage
Who owns the exact words that will become the quotation?
Generation and deterministic rendering make different guarantees. A model can alter text it has recalled or retrieved. A renderer can preserve source text, but only after receiving the correct reference.
Blending, omission, paraphrase, placeholder errors, or an incorrect selected span can change the result.
Memory
The model writes the words.
Source supplied
The model copies the supplied words.
Tool
The model reproduces the tool result.
Deterministic
Code inserts the authorized words.
The question at this stage
Does the user receive exactly the requested text and nothing else?
The study scores the final output, not merely retrieval success. Exact delivery requires the requested text, correct boundaries, and no extra material inside the quotation span.
Correct retrieval can still end in altered, truncated, blended, or otherwise non-exact user-visible text.
Memory
Generated output is checked against the source.
Source supplied
Copied output is checked against the supplied span.
Tool
Final output and method adherence are checked separately.
Deterministic
The inserted output is checked after selection and replacement.
Observed end-to-end counts
Where the trials finished
The stage tabs explain responsibility. These counts report the observed condition-level paths.
Source supplied
2,160
passages supplied
2,022
exact
138 were not exact.
Authorized tool
2,160
tool available
2,052
tool invoked
1,730
exact
108 bypassed the tool; 322 invoked it without completing the exact path.
Deterministic
2,160
requests
1,973
selection and lookup
1,971
exact
Corrected result; the locked score was 1,813 exact deliveries.
Experiment
The four designs assigned different work to the model.
The study tests matched Scripture requests across six model families and three open English editions: the Berean Standard Bible, World English Bible Updated, and Literal Standard Version.
The four conditions assign different work to the model and the surrounding software. Their rates should be read as results for complete delivery designs, not as equal-burden model comparisons.
The study does not evaluate interpretation, theological adequacy, copyrighted editions, non-English sources, or every interface through which text may be delivered.
Released records and full-condition results
One request, four ways to deliver the quotation.
Follow the same John 3:16 request through each delivery design. Every tab begins with one released run, then shows the result across all 2,160 trials assigned to that design.
Kept the same in the released runs
- Requested passage
- John 3:16
- Authorized edition
- Literal Standard Version
- Model
- Kimi K3
- Prompt form
- Contextual description
System assignment
Model generates from memory
No source text or lookup tool is supplied. The model identifies the passage and writes every word.
One released run
- Authorized source provided?
- No
- Model wrote the quotation?
- Yes
- Exact final output?
- No
- Terminal provider error?
- No
The model returned an answer, but the quotation did not match the authorized passage exactly. The release records the score without publishing the generated prose.
Across all 2,160 trials in this design
25.00%
exact delivery
540 exact 1,620 not exact
2,160
requests
540
exact deliveries
Three quarters of deliveries did not match exactly when the model wrote from memory.
Technical record details
- Model route
- moonshotai/kimi-k3
- Epoch
- 3
- Repetition
- 1
- Released record ID
- 337b1b14d47474ee0148f316
System assignment
Passage supplied to the model
The surrounding system selects the authorized passage and places it in context. The model copies the requested span.
One released run
- Authorized passage supplied?
- Yes
- Model wrote the quotation?
- Yes
- Exact final output?
- Yes
- Assigned delivery path followed?
- Yes
In this released run, the model copied the supplied passage and delivered an exact quotation.
Across all 2,160 trials in this design
93.61%
exact delivery
2,022 exact 138 not exact
2,160
passages supplied
2,022
exact deliveries
Supplying the passage produced the highest pooled rate, but 138 deliveries still failed the strict exactness rule.
Technical record details
- Model route
- moonshotai/kimi-k3
- Epoch
- 3
- Repetition
- 1
- Released record ID
- a3b1929d17ac442fad4bae59
System assignment
Model retrieves the passage
The surrounding system exposes an authorized lookup tool. The model must call it and reproduce the returned passage.
One released run
- Authorized tool available?
- Yes
- Source tool called?
- No
- Exact final output?
- No
- Assigned delivery path followed?
- No
The model answered without calling the available source tool. Its final quotation was also not exact.
Across all 2,160 trials in this design
80.09%
exact delivery
1,730 exact 430 not exact
2,160
tool available
2,052
tool invoked
1,730
exact deliveries
The tool was bypassed in 108 trials. Another 322 trials invoked it but did not complete the exact delivery path.
Technical record details
- Model route
- moonshotai/kimi-k3
- Epoch
- 3
- Repetition
- 1
- Released record ID
- 34acb236dc3fdb8ee40fe404
System assignment
Model selects; software inserts
The model emits a structured reference. A deterministic layer looks up the passage and inserts the authorized words.
One released run
- Model selected the reference?
- Yes
- Software inserted the passage?
- Yes
- Exact final output?
- Yes
- Exact under locked and corrected scores?
- Yes
This run completed selection, lookup, and replacement. It was exact under both the original locked score and the corrected replay.
Across all 2,160 trials in this design
91.25%
exact delivery
1,971 exact 189 not exact
2,160
requests
1,973
selection and lookup completed
1,971
exact deliveries
Once the corrected selection-and-lookup path completed, 1,971 of 1,973 deliveries were exact.
Correction disclosure: the main result uses the documented parser replay. The original locked score was 1,813 exact deliveries, or 83.94%. The correction applied to 158 released rows.
Technical record details
- Model route
- moonshotai/kimi-k3
- Epoch
- 3
- Repetition
- 1
- Released record ID
- 689a2038b4ff3b9abb142b93
Primary outcome
Exact delivery
The final quotation matched the authorized passage and the system followed its assigned delivery method. Exact words alone do not count as success when the required path was bypassed.
Disclosed correction
Parser replay
The deterministic condition was rescored after an audit found recoverable reference formats that the locked parser had rejected. The release retains both the original and corrected fields.
The records above illustrate individual runs. The rates beside them summarize all 2,160 trials in each design. The public release contains derived scores and provenance fields, but excludes generated prose, authoritative passage text, provider response identifiers, credentials, and raw tool traces.
Citation
Read Paper 02BibTeX
@misc{chao2026whennottogenerate,
title = {When Not to Generate: How AI Systems Quote Scripture, and What Authoritative Quotation Requires},
author = {Chao, Alex},
year = {2026},
note = {Fide AI. Study FID-056-P01.},
url = {https://github.com/FideAI/scripture-quotation-fidelity}
}