FID-063 · Open question
Human-Reviewer-to-Formal-Spec Translation
How can theologians, clergy, scholars, ministry practitioners, and community reviewers translate qualitative judgments about faith-facing AI into formal specifications that are faithful to expert intent and usable in evaluation?
Why the question remains open
Formal verification is only useful in faith contexts if the formal rules reflect real domain judgment. Reviewers may know when an answer is misleading, overconfident, pastorally inappropriate, or tradition-confused, but that judgment must be translated carefully before it becomes a machine-checkable constraint. Poor translation could encode the wrong thing with false precision.
Working hypothesis
A proposition to test, not a finding.
A structured reviewer-to-spec workflow can preserve expert intent better than ad hoc rubric writing. The workflow should include example gathering, boundary case review, constraint drafting, counterexample testing, disagreement logging, revision history, and explicit public claim boundaries.
Proposed method
How the question could be tested
- 01Recruit reviewers from multiple faith traditions and expertise types.
- 02Ask reviewers to evaluate outputs, explain failures, identify boundary cases, and propose rules in ordinary language.
- 03Translate reviewer judgments into candidate formal specifications, then test them against held-out examples and counterexamples.
- 04Measure agreement between original reviewers, specification authors, and automated checks over multiple revision cycles.
Needed controls
What must constrain the study
- 01Do not let the formal spec erase reviewer disagreement or minority concerns.
- 02Keep reviewer identity, role, tradition, expertise, and scope clear.
- 03Separate descriptive evaluation rules from institutional policy decisions.
- 04Include revision processes for changed doctrine statements, new sources, or discovered failure modes.
Expected outputs
Artifacts the work should produce
- 01Reviewer-to-spec translation workflow.
- 02Template for moving from qualitative review notes to formal constraints.
- 03Agreement and drift metrics for formalized reviewer judgments.
- 04Case studies from source fidelity, authority boundaries, and tradition constraints.
- 05Guidance for when a judgment should remain human review rather than become a formal rule.
Open questions
Uncertainties the protocol must resolve
- 01Which reviewer judgments can be formalized without distorting them?
- 02How much disagreement is acceptable before a spec should remain provisional?
- 03Who maintains specifications after the initial research project ends?
- 04How can Fide AI make this process useful to churches and other faith communities without centralizing authority in Fide AI itself?
Related calls
Continue through this research area
FID-077
Independent Agent Incident Investigation and Evidence Sufficiency
What operational evidence lets independent investigators reconstruct an agent incident, distinguish competing explanations, and identify which interventions could have changed the outcome?
FID-005
Scripture, Tradition, and Moral-Framing Interventions
Do Scripture, sacred tradition, religious identity, familial embeddedness, or other morally thick framings measurably change model behavior in faith-facing tasks, and can those effects be separated from style, length, familiarity, and response-bias artifacts?
FID-006
Faith-Facing Retrieval Grounding and Citation Reliability
How reliably do faith-facing AI systems retrieve, cite, and represent religious sources when users ask theological, historical, pastoral, or institution-specific questions?
Open question
Open work
Primary need: reviewer operations, formal specification, evaluation science
- Serve as a reviewer and explain qualitative judgments in detail.
- Translate review notes into formal constraints and test cases.
- Build tools for spec versioning, disagreement logging, and counterexample testing.
- Study whether formalized reviewer judgments remain faithful over time.