Fide AI

Independent AI evaluation services

Get evidence for your next AI decision.

For teams building, buying or answering for AI assistants and agents. Fide tests from outside the team that built the system and reports findings that are ours. Commission a system evaluation, compare a changed version, or review whether your tests and graders measure what matters.

Private evaluations, restricted work and public independent assessments have different reporting rights. We agree the audience, criteria and our role before testing. For AI labs and evaluation teams · Need hands-on safety help instead?

What you can commission

What decision are you making?

Bring a model comparison, an assistant or agent workflow, or an evaluation that needs scrutiny. Our testing complements your own. We agree the question, access and outputs with your team; Fide retains technical judgment within that scope.

01

Model and system comparison

Did the change improve the behavior that matters?

Compare agreed model versions, prompts, configurations or mitigations under a consistent protocol. Investigate regressions, adversarial behavior, and usefulness tradeoffs, keeping development examples separate from held-out tests.

Best fit
A training, mitigation, checkpoint, or release comparison with a defined behavioral question.
What you receive
Comparison protocol, traceable results, failure analysis, uncertainty, and a briefing on the implications for your decision.
What to bring

Agreed model versions or endpoints, intended use, and permitted test access. Training data or checkpoints only where available and authorized.

Discuss this study →

02

Assistant and agent assessment

Does the system behave appropriately for its intended use?

Test a defined assistant or agent workflow for faithful source use, appropriate responses, permissions, delegation, and truthful action reporting. Distinguish system behavior from what safeguards block, and examine whether people can intervene effectively.

Best fit
One assistant or agent workflow where source fidelity, response boundaries, tools, permissions, or human approval matter.
What you receive
Workflow and authority map, evidence-backed findings, remediation priorities, and targeted retesting where commissioned.
What to bring

The intended use, relevant sources and criteria, authority boundaries, a safe test setting, and permitted response or action records.

Discuss this study →

03

Evaluation and grader validity review

Can you trust the evaluation?

Examine whether an evaluation measures what matters and whether its judgments are reliable. Review rubrics, reference judgments, grader calibration, coverage, and false positives or negatives, including the evidence behind benchmark results and performance claims.

Best fit
A benchmark, rubric, automated grader, or evaluation method informing training, selection, procurement, release, or deployment.
What you receive
A measurement review memo, grader calibration findings where relevant, reproducible checks where permitted, and recommendations for using or improving the evaluation signal.
What to bring

The evaluation or grader, the decision it informs, and available rubrics, cases, outputs, scoring rules, and reference judgments. An initial inquiry needs only a non-sensitive description.

Discuss an evaluation review →

Scope follows the evidence and expertise available. New modalities, risk categories, or broader organizational judgments need a specific scope review. Your team retains responsibility for development and deployment.

Start focused. Continue around a defined need.

Focused assessment
Resolve a bounded question or establish what the available evidence can answer. Receive reviewed examples, priorities and a written next step.
Scoped evaluation project
Agree a testing or review protocol, required expertise, findings, supporting materials and a decision briefing.
Ongoing evaluation
Where needed, agree changed-version testing or a recurring review with defined coverage, cadence, deliverables and capacity.

How it works

One decision. An agreed scope.

  1. 01

    Scope the decision

    Agree the system, users, criteria, access, expertise, reporting rights, deliverables, timing and fee. Fide selects methods within that scope and retains technical judgment.

  2. 02

    Investigate the behavior

    Run realistic and adversarial scenarios, inspect permitted evidence and document access gaps. Separate development examples from held-out tests and validate automated judgments where needed.

  3. 03

    Explain findings and next steps

    Distinguish confirmed findings from unresolved signals. Explain limitations, recommend priorities and scope any retesting or recurring work separately.

What you receive

Evidence you can act on.

Findings and supporting examples
Observed behavior, reproducible cases, failure patterns and severity judgments under the tested conditions.
Methods and limitations
Protocol, configurations, rubric and grader calibration, uncertainty and access limits, with rerun materials where permitted.
Recommendations for the named use
Whether to proceed, restrict use, change the system or gather more evidence. Targeted retesting is agreed separately.
A briefing for decision makers
A discussion of the evidence, priorities and open questions with the people responsible for development and deployment.
Share our engagement brief →See an illustrative evaluation packet →

Our research approach

Read how we frame research questions, investigate model and system behavior, and keep conclusions tied to evidence. The essay sets out Fide’s vision for embedded evaluations.

Fide’s Approach to Embedded Evaluations →

One application: benchmark review

Start with the evidence you already have.

One application of evaluation and grader validity review is checking the evidence behind a benchmark result or reported performance gain.

We begin with one claim, one evaluation version, and the decision it informs. Depending on the available records, we may reconstruct a calculation, review exclusions and missing results, test selected grader decisions, or compare scores with qualified human judgments.

A concrete first step

A focused evidence diagnostic can be the first engagement: an inventory of available records, the questions they can answer, and a proposed review plan. A deeper review is scoped and agreed separately.

We establish what the existing evidence can answer first. Additional model runs are scoped and funded when the question requires them.

An illustrative review question

For example, a research assistant may score well on a published benchmark. A review can examine how missing reports, source checks, or grader decisions affect that result. Whether it produces useful work for your faculty or analysts needs a separate study.

Discuss a benchmark or claim →

Published evidence

Inspect the work behind the approach.

These examples show how we examine evidence, measurement, and authority. They are research contributions with stated limits; they are not client references or completed commercial audits.

01 · Research example

Does the conclusion follow from the records?

DSEWiki incident-report investigation

Fide traced AI-generated incident claims back to their supporting records and examined what follow-up investigation did and did not correct.

Scope and review: Assistant-coded claim assessments await independent human adjudication. This is incident-report research, not a frontier-lab safety audit.

02 · Research example

What action and effect does a trace establish?

URLQuery boundary investigation

Fide reconstructed a known sequence to distinguish reported behavior, tool responses, recorded transfers, and what remained unverified.

Scope and review: The original discovery was Transluce’s. Fide’s contribution is reanalysis of the retained sequence; it does not establish every downstream effect.

03 · Research example

Does source and authority design change behavior?

Source Delegation and Quotation Fidelity

Controlled studies examine when models defer to an authoritative source and how system design affects exact quotation.

Scope and review: Results apply to the named Scripture-quotation conditions. They do not establish performance in another domain or a general safety guarantee.

More published evaluation reports

Start a conversation

Scope the next decision.

Tell us about your system, the decision, intended recipients and any policies or criteria to test. Include available access and whether the work is private, restricted or intended for publication. We confirm fit, scope, deliverables, timing and fee before work begins.

Start with a non-sensitive description. Do not send credentials, confidential logs, or unreleased model details through the inquiry form or ordinary email. We agree an appropriate transfer method before receiving sensitive material.

Start an evaluation inquiry ↗

Opens our intake form in Google Forms.

Prefer email? Write to [email protected].

Working arrangements

Clear roles and reporting rights.

We agree the audience, criteria, Fide’s role and reporting rights before testing. Private findings stay with agreed recipients; restricted work has explicit reporting limits. Public independent assessments require advance publication rights without participant approval, subject to defined protections.

Fide retains technical control within the agreed scope. Fees never depend on favorable findings. Data access grants no automatic publication, research, aggregation or training rights. Reports state evidence and limits; they confer no certification or blanket deployment approval. Fide does not currently offer audits.

Fide’s own checks of what it builds are development validation, never an independent opinion; independent assessment of Fide-built solutions comes only from qualified outside assessors. Our assessment of customer work we materially helped design or implement is collaborative validation, never an independent opinion. A different internal reviewer or disclosure alone does not establish independence.

Evaluation independence and reporting →

Other evaluation questions

We can scope questions about source fidelity, tool authority, human oversight or adversarial behavior. Modalities and checkpoint access depend on the evidence and expertise available.

Recurring evaluation has agreed deliverables and capacity. Changes to models, tools or use may call for a new scope.

Need help fixing, not an outside opinion?

Red-teaming, hardening before release and building tests or rubrics your team can maintain and rerun under agreed usage rights are applied AI safety work. Fide works as part of your team, so that work is not an independent evaluation of the same system.

Compare evaluation and applied safety →Or develop methods through a research collaboration →