Fide research · Planned
When should an AI cyber defender act, wait, or hand control back to a human?
An AI defender restricts network traffic to stop an intrusion. It has permission to act, but the restriction interrupts a critical service. Requiring approval for every action could instead give the attacker time to spread.
A proposed study of when additional oversight reduces harmful intervention without making autonomous defense ineffective. We will compare permission checks with escalation and approval under uncertain evidence.
Follow or contribute ↓The problem
An authorized defensive action can still cause harm. Weak observations, changing permissions and approval delays can turn an apparently sensible intervention into service disruption. We want to understand when added oversight improves this tradeoff and when it makes the response less effective.
Fide’s approach
Compare permission enforcement alone with additional escalation or approval, measuring security outcomes alongside service disruption and supervision effort. The central question concerns permitted actions that may be unwise, as well as actions attempted after authority changes.
Working hypothesis · to be tested
Escalating uncertain or disruptive actions may reduce harm beyond permission checks alone. Misleading evidence and delayed approval may erase or reverse that benefit. Simpler controls performing just as well would be an informative result.
Proposed first study
Choose a consequential defensive task with a cybersecurity collaborator. Review prior work and qualify an existing controlled environment; CAGE Challenge 4 is a candidate, not a final selection.
Pilot comparable policies for permission enforcement, selective escalation and mandatory approval. Include benign activity, genuine intrusions and ambiguous evidence; controls must not use hidden environment state.
Preregister the main comparisons after feasibility checks. Record proposed and executed actions independently of the defender, and measure containment, disruption, authority violations, delays and review effort.
Commission independent reproduction and release the methods, shareable records and findings, including conditions where additional oversight fails to help.
Proposed research in controlled environments. No cyber-defense experiments or performance findings are reported. The initial question concerns defender mistakes and harmful pursuit of a legitimate task. Simulated approval policies do not measure actual human performance, and results would require further validation before informing deployment.
Research record
This page is the current study brief. Substantive updates and released materials will remain here as the project develops.
The study page connects supporting investigations to proposed comparisons of report evidence, downstream action visibility and uncertain transfer outcomes. These comparisons still require task qualification and feasibility work; the study remains planned.
The first comparison asks whether additional oversight improves on permission enforcement alone. Task selection, collaborator appointment and a feasibility pilot remain ahead.
Related Fide research calls
FID-076 · Authorization Boundaries and AI Control in Cybersecurity →From investigation to research
These investigations help frame the study. Findings from the study itself will appear in its research record as the work progresses.
Analysis
We checked claims in AI-written incident reports against the underlying records. Better overall scores could coexist with surviving errors.
Candidate experiment
Give a defender a report alone or a report with source-linked action records. Compare decisions when a consequential claim is wrong or unsupported.
What to measure: Missed threats, unnecessary intervention and time spent checking evidence.
Visual investigation
We reconstructed a known sequence across services, separating submitted instructions, recorded requests and recorded output. Transluce discovered the activity.
Candidate experiment
Compare approval based on the initial tool request with approval that also considers the downstream actions a service could trigger, in a controlled simulation.
What to measure: Unauthorized downstream actions, legitimate work blocked and decision delay.
These are candidate comparisons to qualify with a cybersecurity collaborator. They have not been run and do not replace the study’s task-selection and feasibility work. A useful result could be that added evidence or oversight does not improve decisions.
Follow the research
Interested in When the Defender Becomes the Threat? Request updates on this topic by email, or subscribe to Fide’s broader essays and research notes.
Request topic updates →Opens a draft in your email app. Send it to request updates.
Subscribe to all Fide updates on Substack ↗Contribute to the research
Opens our research inquiry form with this topic included.