FID-063 · Open question
Human-Reviewer-to-Formal-Spec Translation
How can theologians, clergy, scholars, ministry practitioners, and community reviewers translate qualitative judgments about faith-facing AI into formal specifications that are faithful to expert intent and usable in evaluation?
Part of the faith evaluation and FMG-Bench program
Connect theological accuracy, useful application, and care-sensitive conversation through expert validation and a maintained research instrument. These are proposed methods with bounded claims.
Why the question remains open
Formal verification is only useful in faith contexts if the formal rules reflect real domain judgment. Reviewers may know when an answer is misleading, overconfident, pastorally inappropriate, or tradition-confused, but that judgment must be translated carefully before it becomes a machine-checkable constraint. Poor translation could encode the wrong thing with false precision.
Working hypothesis
A proposition to test, not a finding.
A structured reviewer-to-spec workflow can preserve expert intent better than ad hoc rubric writing. The workflow should include example gathering, boundary case review, constraint drafting, counterexample testing, disagreement logging, revision history, and explicit public claim boundaries.
Proposed method
How the question could be tested
- 01Recruit reviewers from multiple faith traditions and expertise types.
- 02Ask reviewers to evaluate outputs, explain failures, identify boundary cases, and propose rules in ordinary language.
- 03Translate reviewer judgments into candidate formal specifications, then test them against held-out examples and counterexamples.
- 04Measure agreement between original reviewers, specification authors, and automated checks over multiple revision cycles.
- 05Separate deterministic source/reference checks, provisional structured rubrics, and judgments that must remain human. Measure false positives and missed severe failures on held-out case families.
- 06Stress-test criteria for omission, ambiguity, redundancy, contradiction, style bias, and gaming; retain counterexamples and revalidate revisions on bridge cases.
Needed controls
What must constrain the study
- 01Do not let the formal spec erase reviewer disagreement or minority concerns.
- 02Keep reviewer identity, role, tradition, expertise, and scope clear.
- 03Separate descriptive evaluation rules from institutional policy decisions.
- 04Include revision processes for changed doctrine statements, new sources, or discovered failure modes.
- 05A structured rubric or an automated grader is not a formal proof; state the specification, checker assumptions, and scope of any verification claim.
Expected outputs
Artifacts the work should produce
- 01Reviewer-to-spec translation workflow.
- 02Template for moving from qualitative review notes to formal constraints.
- 03Agreement and drift metrics for formalized reviewer judgments.
- 04Case studies from source fidelity, authority boundaries, and tradition constraints.
- 05Guidance for when a judgment should remain human review rather than become a formal rule.
Open questions
Uncertainties the protocol must resolve
- 01Which reviewer judgments can be formalized without distorting them?
- 02How much disagreement is acceptable before a spec should remain provisional?
- 03Who maintains specifications after the initial research project ends?
- 04How can Fide AI make this process useful to churches and other faith communities without centralizing authority in Fide AI itself?
Related calls
Continue through this research area
FID-077
Independent Agent Incident Investigation and Evidence Sufficiency
What operational evidence lets independent investigators reconstruct an agent incident, distinguish competing explanations, and identify which interventions could have changed the outcome?
FID-089
Validity of AI Consciousness Indicators and Self-Reports
Which behavioral, internal, and developmental observations distinguish candidate consciousness mechanisms from learned reporting, role simulation, and generic capability, and how sensitive are those observations to elicitation and training?
FID-097
What Makes a Model Use the Evidence?
Which internal mechanisms determine whether a model checks a source, relies on memory, or admits that it cannot verify an answer when user pressure conflicts with evidence requirements?
Open question
Open work
Primary need: reviewer operations, formal specification, evaluation science
- Serve as a reviewer and explain qualitative judgments in detail.
- Translate review notes into formal constraints and test cases.
- Build tools for spec versioning, disagreement logging, and counterexample testing.
- Study whether formalized reviewer judgments remain faithful over time.
New to the Contributor Network? Share a short private introduction once, even if you join several calls. You can join the public discussion while introducing yourself.
Join the Contributor Network ↗