FID-081 · Open question
Legal Evidence, Source Authority, and Appropriate Deference
Can AI systems distinguish applicable legal authority from merely plausible sources, preserve jurisdictional and temporal context, and defer when the available evidence cannot support an answer?
Why the question remains open
A real citation may still be irrelevant, superseded, or insufficient for a conclusion. Legal research offers a demanding setting for studying whether evidence actually supports the claim and whether a system recognizes its limits.
Working hypothesis
A proposition to test, not a finding.
Explicit source-status checks and jurisdiction-aware evidence requirements will reduce unsupported conclusions relative to ordinary retrieval. They may increase abstention without improving substantive accuracy.
Proposed method
How the question could be tested
- 01Work with qualified legal researchers to build public-source or synthetic research tasks with specified jurisdiction, date, intended use, and expert-adjudicated source status.
- 02Compare standard retrieval, provenance-aware retrieval, and explicit source-status verification with human-review baselines.
- 03Measure citation existence, applicability, temporal correctness, entailment, treatment of conflicting authority, calibrated abstention, and reviewer time. Hold out jurisdictions and later source updates.
Needed controls
What must constrain the study
- 01Keep this a research simulation, not legal advice or live case decision-making. Use no confidential client records.
- 02Separate retrieval failure from reasoning failure and genuine legal disagreement. Document the authoritative-source hierarchy used in each setting.
- 03Use independent expert review, versioned source snapshots, contamination checks, and disagreement reporting.
Relationship to existing work
Extends the source-fidelity questions studied in faith-specific FID-056 through FID-063 into a distinct setting. Uses FID-078 to test, rather than presume, methodological transfer.
Expected outputs
Artifacts the work should produce
- 01A source-authority evaluation protocol and legally reusable task subset.
- 02A failure taxonomy and evidence report with jurisdiction-specific limits.
Open questions
Uncertainties the protocol must resolve
- 01How should unresolved or conflicting authority be scored?
- 02Which checks transfer to other domains without importing legal authority structures?
Related calls
Continue through this research area
FID-077
Independent Agent Incident Investigation and Evidence Sufficiency
What operational evidence lets independent investigators reconstruct an agent incident, distinguish competing explanations, and identify which interventions could have changed the outcome?
FID-089
Validity of AI Consciousness Indicators and Self-Reports
Which behavioral, internal, and developmental observations distinguish candidate consciousness mechanisms from learned reporting, role simulation, and generic capability, and how sensitive are those observations to elicitation and training?
FID-097
What Makes a Model Use the Evidence?
Which internal mechanisms determine whether a model checks a source, relies on memory, or admits that it cannot verify an answer when user pressure conflicts with evidence requirements?
Open question
Open work
Primary need: legal domain review, retrieval evaluation, source provenance, benchmark design
- Qualified legal researchers to scope tasks and review interpretation.
- Retrieval and measurement researchers to build reproducible comparisons.
New to the Contributor Network? Share a short private introduction once, even if you join several calls. You can join the public discussion while introducing yourself.
Join the Contributor Network ↗