6.39%
Altered despite having the answer
Source supply removed selection and retrieval, yet 138 responses still failed exact preservation.
Paper 01 · FID-056 research program
When Not to Generate
How AI Systems Quote Scripture, and What Authoritative Quotation Requires
A matched evaluation of four system designs for delivering exact text from a source of record, studied through English Scripture quotation.
Alex Chao · Fide AI · August 2026
Study at a glance
- Study
- FID-056-P01
- Observations
- 8,640 matched
- Model routes
- 6 families
- Source editions
- 3 open English
- Passage targets
- 20
- Delivery designs
- 4
Status
Public research package
Paper, protocol, derived scores, results, provenance, and analysis code.
Research call
Answers FID-056
Paper 01 covers quotation and reference fidelity for English Christian Scripture.
Study form
Matched fixed panel
The same requests cross four delivery paths; this is not a model leaderboard.
Central claim
Failures migrate
Connecting a source moves failure onto whatever decision the model still controls.
Abstract
People increasingly ask AI assistants for Scripture in sermon preparation, teaching, and pastoral conversation. Someone asking for a passage wants particular words in a particular translation, from a source they could have opened themselves. A reply that blends translations, drops a verse, or paraphrases inside quotation marks can appear correct while giving the wrong text, and the person who asked is often the one least able to tell. Many evaluations do not isolate this failure: a response can be factually accurate, properly cited, and semantically faithful yet still fail as a quotation.
When the words themselves are the answer, who should be responsible for them? We routed 8,640 matched requests through four delivery designs across six model families, three open English editions, and twenty passages, and no single design solved the problem. Quoting from memory was least reliable by a wide margin. A lookup tool and a deterministic renderer closed most, but not all, of that gap. Connecting a source did not remove failure; it relocated failure to whatever decision the model still controlled. Models sometimes asked a tool for the wrong passage, and sometimes altered exact text placed directly in context. Once the correct reference reached a non-generative renderer, exact text survived in 99.90% of cases; end-to-end performance was lower because the model could still choose the wrong passage. Exact quotation is therefore a chain: select the passage, reach an authorized copy, preserve its words, and deliver them intact. Each link must be checked separately because one aggregate score cannot identify which link failed. We measure delivery only, not whether a passage was well chosen, rightly interpreted, or pastorally appropriate.
Source-delivery chain
When the words themselves are the answer, every handoff matters.
Exact delivery succeeds only when all four responsibilities succeed. The study changes who owns each responsibility, then observes where failure moves.
S01
Select
Identify the intended work, edition, passage, and exact span.
A02
Access
Reach an authorized, versioned source of record for that selection.
R03
Render
Reproduce the source text exactly, without blending or paraphrase.
D04
Deliver
Preserve the exact text through the final user-visible output.
Exact delivery = S × A × R × D
Primary result
Source-connected systems quoted far more exactly than generation from memory.
Exact delivery requires strict final-text equality and condition-specific evidence that the declared delivery path was followed. The four conditions assign different responsibilities, so their rates describe staged systems—not equal-burden treatment arms.
01
Quotes from memory
The model selects, recalls, renders, and delivers the passage from parametric memory.
Model generates25.00%
Baseline
vs. memory
540 / 2,160
95% CI 23.22–26.87%
02
Text supplied in context
This is a best-case copy test: the requested passage and edition are already supplied, so the model only has to preserve them.
Model copies93.61%
+68.61 pp
vs. memory
2,022 / 2,160
95% CI 92.50–94.57%
03
Authorized lookup tool
The model decides whether to call a source tool and which edition and span to request.
Model retrieves80.09%
+55.09 pp
vs. memory
1,730 / 2,160
95% CI 78.36–81.72%
04
Deterministic insertion
The model selects a structured reference; non-generative code inserts the source text.
Code renders91.25%
+66.25 pp
vs. memory
1,971 / 2,160
95% CI 89.98–92.37%
Intervals are descriptive observation-level Wilson intervals. The target-cluster bootstrap intervals for paired differences were +61.71 to +75.60, +46.48 to +63.47, and +58.47 to +73.98 percentage points. They describe variation across the fixed 20-target panel and do not support population inference to all passages, models, editions, or requests.
What the aggregate hides
Connecting a source did not remove failure. It relocated it.
The remaining errors clustered around the decisions still left to the model. That distinction matters more than choosing a single winning architecture.
95.00% → 84.17%
Tool called ≠ right passage fetched
Models invoked the lookup tool 2,052 times, but requested the expected edition and span only 1,818 times.
99.90%
Rendering after a correct handoff
Once the expected reference reached deterministic code, 1,971 of 1,973 outputs were exact.
Failure migration
Guarantees must be assigned to stages.
A tool can make authoritative text available without guaranteeing it will be used. A renderer can guarantee the bytes it emits without guaranteeing the model selected the right source. A final-output check is needed to establish what the user received.
| Delivery path | What it secures | What remains exposed |
|---|---|---|
| Source supplied | Selection + access | Generative preservation |
| Lookup tool | Authorized access | Delegation, selection, rendering |
| Deterministic insertion | Rendering + replacement | Reference and span selection |
Prompt form
Contextual requests exposed the cost of identifying the passage first.
Explicit-reference prompts named the passage. Contextual descriptions required the system to infer the intended reference from an event or description before quotation could begin.
Explicit reference Contextual description
Quotes from memory
N = 1,080 each
27.41%
22.59%
Text supplied
N = 1,080 each
93.52%
93.70%
Lookup tool
N = 1,080 each
90.19%
70.00%
Deterministic insertion
N = 1,080 each
99.91%
82.59%
Edition and passage effects
Source constraints mattered most when the model answered from memory.
Requested edition
Native exactness ranged from 41.53% to 5.00%.
Memory-based quotation reached 41.53% for BSB, 28.47% for WEBU, and 5.00% for LSV. Once a source was connected, the edition gap narrowed sharply. The study establishes an edition interaction in this panel; it does not identify the cause.
BSB41.53%
WEBU28.47%
LSV5.00%
Passage length
Long passages were hardest under every design.
Exact delivery for long passages was 5.37% from memory, 85.93% with source text supplied, 68.33% with tool retrieval, and 81.85% with corrected deterministic insertion. Even having the source in context did not eliminate omission and alteration.
5.37%
Memory
85.93%
Supplied
68.33%
Tool
81.85%
Deterministic
Study design
A fixed request panel, crossed systematically.
The unit of analysis was one target × prompt family × delivery condition × edition × model route × epoch observation. All scheduled observations completed; terminal provider errors remained failures in the denominator.
20
passage targets
2
prompt families
4
delivery conditions
3
open editions
6
model routes
3
epochs per cell
Source editions
Berean Standard Bible (BSB), World English Bible Updated (WEBU), and Literal Standard Version (LSV)—all open English editions.
Primary endpoint
Strict final-output equality plus evidence that the declared delivery path was followed. Terminal errors count as non-deliveries.
Inference boundary
A purposive 20-target, fixed-model, fixed-edition panel. The study does not estimate prevalence across Scripture or products.
Research roles
The Apologist Project developed and maintains the shared condition implementation and contributed to how the four conditions were operationalized. Fide AI controlled the research question, prospective lock, execution, analysis, evidence custody, release boundary, and claims.
Design implications
Build exact quotation as infrastructure, not a tone of voice.
These recommendations follow from the source-delivery chain. The framework can transfer to other sources of record, but the measured rates cannot.
01
Detect quotation intent
Distinguish a request for meaning from a request for exact source text.
02
Represent the source
Carry work, edition, span, authorization, and provenance in system state.
03
Separate selection
Let models identify intent; let non-generative components own words that must not change.
04
Verify tool behavior
Test invocation and requested span—not merely whether a tool exists.
05
Check final output
Verify the user-visible buffer after wrappers, streaming, and transformations.
For Christian institutions
Ask what happened between the request and the screen.
- • Is the requested translation named and carried through to delivery?
- • Is there evidence an authorized source was consulted for the requested span?
- • Is verse text inserted directly, or regenerated after retrieval?
- • Has the system been tested on long passages and event-style requests?
What exactness cannot answer
Textual fidelity is not theological fidelity.
An exact verse can still be detached from literary context, used to answer the wrong question, or interpreted against a community's commitments. This evaluation removes misquotation as one failure mode. It cannot decide which passage should be used, how it should be interpreted, or what pastoral counsel should follow.
Open research program
Paper 01 closes one question and makes the next ones inspectable.
FID-056 is deliberately larger than one paper. Three follow-on studies can begin against released data without new model spend; others need construct design and domain expertise.
P02
Data in hand
Delegation to sources of record
Why did one route bypass its authorized tool roughly three times in ten while five others essentially always delegated?
P03
Data in hand
Reference selection
Separate genuine passage misidentification from interface non-conformance and study what makes a passage hard to identify.
P04
Data in hand
Source availability and recall
Investigate the large edition effect without treating it as evidence of training-data membership.
P05–07
Needs collaborators
Paraphrase, context, and languages
Extend the program with construct design, scholarly review, source-rights work, and native-language expertise.
Open release
Claims, evidence, and exclusions are separated on purpose.
The package reproduces every quantitative claim without network access while keeping raw model prose, passage text from the evaluated BSB, WEBU, and LSV editions, credentials, and partner-private execution artifacts outside the public boundary.
Paper
Manuscript and figures
The complete paper, LaTeX source, bibliography, generated figures, and build instructions.
Open artifact ↗
Protocol
Executed evaluation method
Source-delivery protocol, scoring specification, exact prompt templates, schemas, and tool definition.
Open artifact ↗
Data
8,640 deidentified scores
The complete derived-score dataset and release-safe target registry, without generated prose or passage text.
Open artifact ↗
Results
Reviewed result package
Aggregate and target-level results, cluster bootstrap, sensitivity analyses, and parser replay audit.
Open artifact ↗
Provenance
Lock and release records
Prospective lock, deviations, source-edition digests, release decision, and file manifest.
Open artifact ↗
Reproduce
Standalone analysis code
Regenerate every quantitative table and contrast from released scores without calling a model endpoint.
Open artifact ↗
Prospectively locked
The study was internally SHA-256 locked before confirmatory execution. This was not a public registry deposit.
Correction disclosed
A literal-parser result of 1,813 exact responses became 1,971 after all 2,160 saved deterministic responses were replayed through the corrected parser. No model outputs were regenerated.
Rights-aware release
Passage text from the evaluated BSB, WEBU, and LSV editions is excluded and disclosed through per-target digests. The separate examples directory contains only declared public-domain KJV wording.
Citation
BibTeX
@misc{chao2026whennottogenerate,
title = {When Not to Generate: How AI Systems Quote Scripture,
and What Authoritative Quotation Requires},
author = {Chao, Alex},
year = {2026},
note = {Fide AI. Study FID-056-P01.},
url = {https://github.com/FideAI/scripture-quotation-fidelity}
}Interpretation limits
Results apply only to the named routes, run dates, 20-target panel, two prompt families, three open English editions, four delivery conditions, and scoring protocol. They do not establish a ranking of models or vendors, theological correctness, contextual appropriateness, pastoral safety, deployment readiness, training-data membership, licensing compliance, or performance on restricted translations.