FID-079 · Open question
Presuppositions, Disagreement, and Evaluation Judgment
How do researchers' presuppositions shape evaluation design and interpretation, and can explicit disclosure make judgments more inspectable and appropriately trusted?
Why the question remains open
Research choices reflect assumptions about evidence, authority, harm, and human goods. Naming these assumptions is an opportunity for intellectual honesty, but disclosure alone does not make a method valid or eliminate bias.
Working hypothesis
A proposition to test, not a finding.
Structured disclosure plus sensitivity analysis will improve reviewers' ability to identify consequential assumptions compared with a methods-only report. It may not improve agreement or warranted confidence and could increase unwarranted confidence.
Proposed method
How the question could be tested
- 01Construct evaluation reports with documented variations in harm definitions, authority rules, stakeholder priorities, and aggregation choices.
- 02Randomize blinded reviewers to methods-only, disclosure-only, and disclosure-plus-sensitivity conditions. Ask them to identify assumption-dependent conclusions and assess uncertainty.
- 03Measure detection accuracy, calibration, reproducibility of judgments, and changes in appropriate reliance. Analyze persistent substantive disagreements separately from misunderstandings.
Needed controls
What must constrain the study
- 01Recruit diverse relevant expertise and perspectives with consent; do not infer religion or worldview from demographic traits.
- 02Hold substantive evidence constant across report conditions. Pre-register outcomes and distinguish agreement, satisfaction, and valid judgment.
- 03Disclose the researchers' own commitments and incentives, including how legitimate authority and harm were defined. Include null and counterproductive disclosure results.
Relationship to existing work
Complements FID-078 on validity and the repository's claims discipline. Applies across research traditions rather than assigning one worldview to all contributors.
Expected outputs
Artifacts the work should produce
- 01A tested presupposition-disclosure template and sensitivity-analysis protocol.
- 02A report on when transparency improves scrutiny and when it does not.
Open questions
Uncertainties the protocol must resolve
- 01Which assumptions need explicit disclosure to change interpretation?
- 02How can disclosure stay useful without becoming a checklist or identity label?
Related calls
Continue through this research area
FID-012
Optimization Pressure and Visible-Rubric Gaming
If builders can see Fide AI rubrics or optimize against public benchmark items, do systems become genuinely safer or merely better at passing the visible test?
FID-076
Authorization Boundaries and AI Control in Cybersecurity
Which controls keep capable agents within legitimate authorization when task pressure, untrusted inputs, or delegated work creates opportunities to exceed it?
FID-078
When Trustworthiness Evaluations Transfer Across Domains
Which measures of evidence use, authority boundaries, and human control transfer across high-trust domains, and which require domain-specific definitions and calibration?
Open question
Open work
Primary need: evaluation design, philosophy, social science, reviewer calibration
- Philosophers, domain experts, and measurement researchers to develop test cases.
- Social scientists and reviewers to design and assess the study.