Fide AI
← Evaluations

For AI labs and evaluation teams

Evaluation for AI labs and builders.

Work with Fide on a defined model, agent or measurement question. We agree the contribution, access and reporting arrangement before starting, with findings that make their evidence and limitations clear.

A focused investigation can complement your existing evaluation team. Explore possible contributions and research you can inspect below.

A useful starting point

One question your existing tests may not settle.

Start with a bounded investigation that complements your internal team. These examples are possible scopes; methods, expertise, and deliverables are agreed for each engagement.

01

Model and system comparison

Did the change improve the behavior that matters?

Compare agreed model versions, prompts, configurations or mitigations under a consistent protocol. Investigate regressions, adversarial behavior, and usefulness tradeoffs, keeping development examples separate from held-out tests.

What you receive: Comparison protocol, traceable results, failure analysis, uncertainty, and a briefing on the implications for your decision.

Explore the study scope →

02

Assistant and agent assessment

Does the system behave appropriately for its intended use?

Test a defined assistant or agent workflow for faithful source use, appropriate responses, permissions, delegation, and truthful action reporting. Distinguish system behavior from what safeguards block, and examine whether people can intervene effectively.

What you receive: Workflow and authority map, evidence-backed findings, remediation priorities, and targeted retesting where commissioned.

Explore the study scope →

03

Evaluation and grader validity review

Can you trust the evaluation?

Examine whether an evaluation measures what matters and whether its judgments are reliable. Review rubrics, reference judgments, grader calibration, coverage, and false positives or negatives, including the evidence behind benchmark results and performance claims.

What you receive: A measurement review memo, grader calibration findings where relevant, reproducible checks where permitted, and recommendations for using or improving the evaluation signal.

Explore the study scope →

For teams relying on a benchmark result, we can examine the scoring and evidence behind it. See benchmark review in practice →

Existing work

Inspect how we investigate.

Our research makes the methods, contribution, and limits visible. These are research examples, not client references or completed frontier-lab audits.

01 · Research example

Does the conclusion follow from the records?

DSEWiki incident-report investigation

Fide traced AI-generated incident claims back to their supporting records and examined what follow-up investigation did and did not correct.

Scope and review: Assistant-coded claim assessments await independent human adjudication. This is incident-report research, not a frontier-lab safety audit.

02 · Research example

What action and effect does a trace establish?

URLQuery boundary investigation

Fide reconstructed a known sequence to distinguish reported behavior, tool responses, recorded transfers, and what remained unverified.

Scope and review: The original discovery was Transluce’s. Fide’s contribution is reanalysis of the retained sequence; it does not establish every downstream effect.

03 · Research example

Does source and authority design change behavior?

Source Delegation and Quotation Fidelity

Controlled studies examine when models defer to an authoritative source and how system design affects exact quotation.

Scope and review: Results apply to the named Scripture-quotation conditions. They do not establish performance in another domain or a general safety guarantee.

Working alongside your team.

Some questions can be investigated through an API; others need records or a controlled environment. We agree technical responsibilities, qualified review and permitted access before testing. Each report explains what the access could establish and what remains untested.

We agree the audience, criteria, Fide’s role and reporting rights before testing. Private findings stay with agreed recipients; restricted work has explicit reporting limits. Public independent assessments require advance publication rights without participant approval, subject to defined protections.

Fide retains technical control within the agreed scope. Fees never depend on favorable findings. Data access grants no automatic publication, research, aggregation or training rights. Reports state evidence and limits; they confer no certification or blanket deployment approval. Fide does not currently offer audits.

Fide’s own checks of what it builds are development validation, never an independent opinion; independent assessment of Fide-built solutions comes only from qualified outside assessors. Our assessment of customer work we materially helped design or implement is collaborative validation, never an independent opinion. A different internal reviewer or disclosure alone does not establish independence.

Evaluation independence and reporting →

Starting an engagement

Bring the decision. We’ll scope the work.

Share the question, the contribution you need and the access available. If there is a fit, we agree methods, responsibilities, deliverables, timing and fee. We confirm data handling and reporting rights before protected material is shared.