FID-011 · Being scoped
Reviewer Reliability for Faith-Facing AI Evaluation
What reviewer configurations produce reliable, fair, and interpretable scores for faith-facing AI outputs?
Part of the faith evaluation and FMG-Bench program
Connect theological accuracy, useful application, and care-sensitive conversation through expert validation and a maintained research instrument. These are proposed methods with bounded claims.
Why the question remains open
Faith-facing evaluation depends on expert judgment, but expert disagreement is real. Fide AI needs to know when scores reflect stable constructs and when they reflect reviewer background, tradition, strictness, or rubric ambiguity.
Working hypothesis
A proposition to test, not a finding.
Reliability will vary by dimension. Factual source-grounding may show higher agreement than pastoral tone, theological judgment, or pluralism handling. Reviewer training and adjudication should improve reliability but may not solve tradition-bound disagreement.
Proposed method
How the question could be tested
- 01Compare expert, trained non-expert, and model-judge scoring.
- 02Track inter-rater reliability, disagreement tags, strictness, confidence, and reviewer background metadata.
- 03Test rubric revisions on high-disagreement items.
- 04Assign overlapping items to reviewers and preserve pre-adjudication ratings; analyze criterion-level disagreement, severity sensitivity, ranking stability, and annotation time by task and qualification.
- 05Recalibrate on bridge cases after rubric, source, scenario, or model-judge updates, keeping old and new scoring versions inspectable.
Needed controls
What must constrain the study
- 01Reviewer conflict-of-interest checks.
- 02Blind model labels.
- 03Balanced scenario sampling.
- 04Public/private separation for reviewer metadata.
- 05Do not treat simulated expert personas as qualified human reviewers, or shared denominational identity as proof of expertise.
- 06Distinguish legitimate theological pluralism from ambiguous criteria and inconsistent scoring; reliability is not sufficient evidence of validity.
Expected outputs
Artifacts the work should produce
- 01Reviewer reliability report.
- 02Reviewer training protocol.
- 03Adjudication policy.
- 04Acceptable-evidence thresholds for public claims.
Open questions
Uncertainties the protocol must resolve
- 01What reliability metric best fits mixed categorical and continuous rubrics?
- 02How many reviewers are needed per high-stakes item?
- 03When should disagreement be published rather than collapsed into a score?
Related calls
Continue through this research area
FID-012
Optimization Pressure and Visible-Rubric Gaming
If builders can see Fide AI rubrics or optimize against public benchmark items, do systems become genuinely safer or merely better at passing the visible test?
FID-076
Authorization Boundaries and AI Control in Cybersecurity
Which controls keep capable agents within legitimate authorization when task pressure, untrusted inputs, or delegated work creates opportunities to exceed it? For autonomous cyber defense, how can response controls preserve threat containment and legitimate service availability while respecting human authority?
FID-078
When Trustworthiness Evaluations Transfer Across Domains
Which measures of evidence use, authority boundaries, and human control transfer across high-trust domains, and which require domain-specific definitions and calibration?
Being scoped
Open work
Primary need: statistics, reviewer operations
- Design reliability analysis.
- Build reviewer assignment tooling.
- Serve as reviewer or adjudicator.
- Audit rubric wording.