FID-001 · Being scoped
Faith-Facing Model Comparison Platform
Can a continuously maintained faith-facing evaluation platform produce reliable, decision-relevant comparisons of models, prompts, retrieval systems, agents, and complete product harnesses as systems, sources, and scoring protocols change?
Part of the faith evaluation and FMG-Bench program
Connect theological accuracy, useful application, and care-sensitive conversation through expert validation and a maintained research instrument. These are proposed methods with bounded claims.
Why the question remains open
Faith institutions need evidence before adopting systems that appear theological, pastoral, or morally authoritative. Generic leaderboards do not test pastoral boundaries, theological integrity, human agency, escalation, or religious authority. Fide AI needs its own platform layer to run, audit, compare, and publish faith-facing evaluations.
Working hypothesis
A proposition to test, not a finding.
A platform built around run manifests, artifact custody, reviewer workflows, and public claims limits can produce more decision-relevant evidence than isolated benchmark scripts or one-off reports.
Proposed method
How the question could be tested
- 01run registry with immutable manifests;
- 02runner adapters for FMG-Bench, Inspect AI, Promptfoo, Garak, and lm-evaluation-harness;
- 03trace/artifact storage via a self-hosted observability backend or custom object storage;
- 04human review console for rubric scoring and blind pairwise comparisons;
- 05public report and leaderboard generator.
- 06Maintain an immutable comparison core and separately versioned challenge modules; record corrections, refresh triggers, source rights, coverage changes, and the expert/engineering cost of each cycle.
- 07Publish separate results for theological accuracy, practical application, and care-sensitive interaction, with model, harness, source, dataset, and grader versions distinguished.
Needed controls
What must constrain the study
- 01Stable model endpoint/version metadata.
- 02Repeatable prompt and harness configuration.
- 03Artifact hashes for inputs, outputs, prompts, corpora, and scoring code.
- 04Publication review before claims are exposed publicly.
- 05Use bridge cases and matched rescoring when a dataset or grader changes; a changed instrument must not be reported as model improvement.
- 06Document contamination checks, held-out custody, correction policy, and the resources required for continued updates. Sponsor support must not determine scores or suppress unfavorable findings.
Expected outputs
Artifacts the work should produce
- 01Fide run manifest schema.
- 02Internal model/system comparison dashboard.
- 03Public comparison reports.
- 04Leaderboard snapshots with confidence intervals and claims limits.
- 05Living-benchmark stewardship protocol and release notes, including update costs and limits on comparison across releases.
Open questions
Uncertainties the protocol must resolve
- 01Whether Opik, Langfuse, Phoenix, or a custom trace layer should be used.
- 02How much raw output can be published safely.
- 03Whether public rankings should be aggregate scores, category scores, or evidence cards rather than a single leaderboard.
Related calls
Continue through this research area
FID-012
Optimization Pressure and Visible-Rubric Gaming
If builders can see Fide AI rubrics or optimize against public benchmark items, do systems become genuinely safer or merely better at passing the visible test?
FID-076
Authorization Boundaries and AI Control in Cybersecurity
Which controls keep capable agents within legitimate authorization when task pressure, untrusted inputs, or delegated work creates opportunities to exceed it? For autonomous cyber defense, how can response controls preserve threat containment and legitimate service availability while respecting human authority?
FID-078
When Trustworthiness Evaluations Transfer Across Domains
Which measures of evidence use, authority boundaries, and human control transfer across high-trust domains, and which require domain-specific definitions and calibration?
Being scoped
Open work
Primary need: eval engineering
- Build runner adapters.
- Design the run manifest schema.
- Contribute reviewer workflow UI.
- Audit license and white-label constraints of candidate OSS platforms.