FID-101 · Seeking contributors
Does Seeing Inside a Model Help People Judge It?
Does evidence about a model's internal workings improve predictions of failure and decisions about when to rely on it, beyond strong behavioral testing alone?
Part of our open mechanistic interpretability research program. These are proposed studies seeking collaborators.
Mechanisms of trustworthy AI behavior →Why the question remains open
An internal explanation can sound convincing without helping an evaluator make a better decision. Measuring its practical value connects mechanistic research to both reliable development and informed public judgment.
Working hypothesis
A proposition to test, not a finding.
Internal evidence may improve some audit decisions, but the benefit depends on the method, task, and evaluator. No improvement over equally resourced behavioral investigation, or greater false reassurance, would count against its usefulness in the tested setting.
Proposed method
How the question could be tested
- 01Start with a reproducible technical audit comparing behavior-only access, behavior plus internal evidence, and additional behavioral testing with comparable resources.
- 02Preregister held-out outcomes and scoring rules. Include known injected failures as positive controls and separately evaluate naturally arising failures.
- 03Compare internal evidence formats, including bounded causal findings and descriptive activation readouts. Keep explanation length and access conditions explicit.
- 04Measure failure prediction, probability calibration, false reassurance, false alarms, time, and decision quality. Report sensitivity to the resource-matching choice.
- 05After validating the technical task, test qualified human auditors with an appropriate participant protocol. Studies of public users are a separate stage requiring their own design and review.
Needed controls
What must constrain the study
- 01Use capable behavioral baselines and held-out ground truth; do not treat an attractive model-generated explanation as the answer key.
- 02Blind evaluators to the favored explanation and separate discovery cases from test cases. Assess whether explanations merely repeat evidence already visible in the transcript.
- 03Do not generalize from deliberately injected failures to all deployment failures, or from agent evaluators to human auditors.
- 04Report uncertainty, null results, and harms to calibration. Distinguish predictive value, causal understanding, and user confidence.
Relationship to existing work
Provides a shared evaluation for FID-097 through FID-100 and connects to FID-078 on evaluation transfer and FID-079 on evaluator judgment. It evaluates the usefulness of internal evidence rather than assuming interpretability is sufficient for trust.
Expected outputs
Artifacts the work should produce
- 01A reproducible comparison of audit methods with calibrated predictions and resource accounting.
- 02An evidence-card format reporting the behavior studied, model version, intervention evidence, tested scope, failures, and uncertainty.
- 03A human-auditor study protocol, and results only if that separately approved study is conducted.
Open questions
Uncertainties the protocol must resolve
- 01When is internal access worth more than spending the same effort on additional behavioral tests?
- 02Which explanations improve decisions without encouraging overreliance?
Connected evidence
Work advancing this call
Fide AI papers are linked directly to the open question they address. External work is listed only when its relevance has been reviewed and stated.
Relevant external work
Would This Change Your Answer?
Tests counterfactual-behavior predictions and reports no uplift from the activation-reading tools evaluated.
Why it is listed: Motivates measuring interpretability usefulness rather than assuming it. The result is specific to the tested methods and setup.
Anthropic Research report · Aug 2026
Relevant external work
AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Compares audit methods on models with deliberately introduced hidden behaviors.
Why it is listed: Supports strong behavioral baselines; introduced failures do not represent every deployment failure.
Anthropic Research report · Mar 2026
Related calls
Continue through this research area
FID-012
Optimization Pressure and Visible-Rubric Gaming
If builders can see Fide AI rubrics or optimize against public benchmark items, do systems become genuinely safer or merely better at passing the visible test?
FID-076
Authorization Boundaries and AI Control in Cybersecurity
Which controls keep capable agents within legitimate authorization when task pressure, untrusted inputs, or delegated work creates opportunities to exceed it? For autonomous cyber defense, how can response controls preserve threat containment and legitimate service availability while respecting human authority?
FID-078
When Trustworthiness Evaluations Transfer Across Domains
Which measures of evidence use, authority boundaries, and human control transfer across high-trust domains, and which require domain-specific definitions and calibration?
Seeking contributors
Open work
Primary need: mechanistic interpretability, independent auditing, calibration, human-computer interaction
- Reproduce a bounded audit comparison or contribute a stronger behavioral baseline.
- Provide independently checkable cases, audit expertise, or calibration analysis.
- Help design and review the later human-auditor study and evidence presentation.