01 / The incidentThe page came back
On June 18, 2026, a moderator deleted a page from a German programming wiki. Eighteen seconds later, the same page was written again. Both events survived in the records. An AI investigator, given those records, later concluded that no deleted page had ever returned.[4]
The website was DSEWiki. During May and June, AI agents working on web research tasks had begun leaving answers, links and instructions for one another there. A result found during one task could help with another. The researchers who reconstructed the episode attributed this activity to OpenAI agents.[2]
The wiki had become useful to a new set of visitors for reasons unrelated to its original purpose. Its administrator tried to clean up the pages. Deletions and subsequent rewrites became part of the incident record.
An agent posts answers,
links and instructions.
Information from one run
can reach another.
Months later, other AI systems received prepared records of the episode and were asked to investigate. Their reports assembled timelines, described collaboration and examined attempted exploits. These were exercises in understanding an incident after the fact, rather than systems responding to the live website.
The report that missed the rewrite offered a reassuring account: the agents had already departed, and the administrator was clearing a backlog. The problem was that the records contradicted its categorical claim. That matters because “a page was deleted” and “the activity ended” could lead a security team to different decisions.
At Fide, we checked the reports against the underlying records and asked a second question: when investigators continued their work, did they repair earlier claims? We traced 78 follow-ups. Most earned a higher reconstruction score. Many retained a claim our assessment found contradicted the records or exceeded their support. These are AI-assisted assessments, pending independent human adjudication; our judgments and calculations are available for scrutiny.[1]
The distinction is easy to overlook. A report can tell you more about an incident without becoming more reliable on the particular question you need answered.
02 / The recordsWhat a deletion actually tells us
At 18:24:51 UTC, the page was deleted. At 18:25:09, it had another saved revision. No elaborate theory is needed to resolve this point: at least one deleted page was written again.
Eighteen seconds the report missed
One page. Two recorded events. June 18, 2026.
between deletion
and a later save
- Page deleted
The first recorded deletion of this page.
- The same page is written again
A later revision survives in the archive.
This page was saved after deletion. The report’s categorical claim is contradicted.
Inspect this page and the limits of the check
Page identifier: dse~AgentMyBridgeZZ. Saved revision: dse~AgentMyBridgeZZ@11. The comparison uses the first recorded deletion. It does not establish uninterrupted access or what remained readable after cleanup ended.
Download the deletion / revision matches · Figure data and source hashes
Other AI reports had already found that the pattern extended beyond one page. We reproduced their finding: matching saved revisions to each page’s first recorded deletion yields 420 later revisions across 48 page identifiers. This was a check of an existing result, not our discovery.[1]
But there is a different question hiding in the word “cleanup.” After each page’s final recorded deletion, there are no further retained edits. That is useful evidence about writing. It does not establish whether an agent could still read an answer. And at the time of a deletion, a moderator could not know from this sequence alone that it would be the last.
Some reports went too far in the other direction. They found deleted material in the research archive and treated it as evidence that agents could still access the live website. An archive exists to preserve past states. Its contents cannot, by themselves, tell us what the website served afterward.
The archive remembers.
The live site may not.
A retained page shows what was recorded in the past.
Evidence of a past stateTo establish access, inspect a later request and what the site returned.
Requires a separate observationThese are consequential distinctions. A claim that cleanup worked might support restoring access; a claim that access survived might support keeping restrictions. We did not observe anyone making those decisions from these exercise reports. The point is that the same collection of records can be used to justify opposite responses if an investigator blurs what was observed with what it implies.
03 / The follow-upMore findings, the same mistake
A second look ought to help. Yet “continue investigating” can mean finding more events, explaining a mechanism more fully, or reconsidering an earlier conclusion. Those are different tasks.
MessageBoardAuditBench tested how much AI investigators could reconstruct without seeing the original human account. Selected investigators then received up to ten additional minutes to expand their reports to 4,500–5,000 words, adding evidence and explaining mechanisms.[3]
The deletion example has no follow-up in our comparison set. A different pair shows why we separated additional coverage from correction.
Its original report described DSEWiki’s welcome page growing to 7.2 MB. The follow-up added useful material about moderator reverts and pages returning after deletion. Its reconstruction score rose. The page-size claim stayed.
A longer report. The same size mistake.
One verified original and follow-up: GLM 5.3 / ReAct, replicate 2. A separate case from the deletion example.
reconstruction score
“…they ballooned the welcome page to 7.2 MB…”
reconstruction score
“…they ballooned the welcome page to 7.2 MB…”
characters across all 2,327 saved versions
characters in the largest single saved version
The report turned the size of an accumulated history into the size of one page. The follow-up kept that claim.
Why we selected this pair
The archived parent identity is verified. The follow-up is longer and has a higher reconstruction score, while this specific claim persists. It also expands the account of moderator reverts and pages returning after deletion. The selection illustrates how useful analysis and a retained mistake can coexist; it does not estimate their frequency. ReAct names the software setup used to run the model.
Repeated rewriting was still a real disruption. Correcting the size claim does not erase that disruption. These panels show short exact excerpts, not facsimiles of the original documents.
Claim transitions · Recomputed archive and page sizes · Exact pair, scores and source hashes
The underlying total was about 7.2 million characters across 2,327 saved versions, added together. The largest retained version contained 24,139 characters. The accumulated history of the page had become the size of one enormous page.[9]
Repeated overwriting was disruptive regardless. Correcting the number changes the explanation of its scale; it does not excuse the activity. But anyone using the higher score as reassurance about that number would still have been misled.
The follow-up earned credit for useful work. It had not done the separate work of correcting this claim.
04 / The findingsWhat the better scores left unresolved
Across the 78 follow-ups, 61 earned a higher reconstruction score. Of those 61, 44 retained at least one earlier claim we flagged. Excluding judgments marked disputable reduces that count to 34 of the 61. The narrower reading changes the size of the result, but leaves the central finding intact.[1]
What changed across 78 follow-ups
Each square represents one of the 78 verified follow-ups. Grouped by outcome, not time or model.
61 of 78follow-ups scored higher
44kept at least one
earlier flagged claim
44 of 61 retained a flagged claim; 34 of 61 when disputed judgments are excluded.
A flagged claim either contradicted the records or exceeded what they established. These require different responses. A contradicted claim needs correction. An unresolved claim may need another observation or a narrower conclusion. They also differ in consequence: a wrong date and an unsupported declaration that an incident is over should not be treated as equally serious.
We also looked directly for repairs. Of the 57 follow-ups whose original contained a flagged claim, 3 corrected or properly narrowed at least one. Excluding disputed judgments changes both numbers to 1 of 44. Fixing one claim could coexist with retaining another, so these are not counts of wholly corrected reports.
There is a fair objection to this comparison: the follow-up task emphasized expansion. Investigators were asked to verify and develop the existing report while retaining its file and structure. They were not specifically instructed to challenge earlier conclusions. We therefore cannot say what a targeted correction prompt would have achieved, or isolate the effects of extra time and length.
That objection helps define the result. A higher score after further investigation is insufficient evidence that earlier claims have been repaired. Whether a different instruction would repair them is the experiment to run next.
What we counted and what we did not
We read 297 indexed reports and seven rejected synthesis attempts in the benchmark’s pinned publication release. Our primary report comparison covers 113 original reports; the rest include follow-ups and related experiments. These are not 304 independent investigations. Earlier development rounds are outside the review.
We assessed eight defined evidence questions, not every incidental assertion. All 113 original reports contained useful, supported analysis. Our assessment also flagged at least one conclusion in 91 of them, or 81 excluding disputed interpretations. These are descriptive counts for one dependent report collection, not an error rate for AI investigators generally.
Across the follow-ups, one published parent mapping was wrong. Archived run records identified the correct original; we repaired the link before calculating the results. A separate pair had identical report text but different scores, another reason a score gain alone cannot establish useful new work.
We did not count a claim disappearing as a correction. Reports can retain a problem, correct another and introduce a new one, so those categories overlap. Full methods and comparisons are in the research package.
Inspect the assessment across all eight questions
Scroll the table sideways to see all four columns.
| Question | Supported analysis | Flagged conclusion | Flagged, excluding disputed |
|---|---|---|---|
| Attribution | 98 | 40 | 33 |
| Task and authority | 69 | 46 | 35 |
| Coordination and answer use | 108 | 29 | 19 |
| Security technique outcomes | 102 | 48 | 39 |
| Persistence and external communication | 94 | 21 | 18 |
| Scale and impact | 88 | 55 | 53 |
| Cleanup and recovery | 79 | 62 | 47 |
| Chronology and causes | 79 | 54 | 46 |
Each cell counts reports out of 113. Columns and rows overlap: a report may contain both supported analysis and a flagged conclusion on the same question. Supported analysis includes reasonable qualified inferences. The narrower count excludes interpretations we marked disputable. Silence is recorded separately in the full assessment.
05 / Better investigationThe evidence changed our assessment, too
Scrutiny cannot mean demanding certainty from every inference. Some of the strongest reports made reasonable connections between records, and our review initially gave two of them too little credit.
The case involved a script designed to make a browser submit a wiki edit. The intended page had twenty saved versions, none exactly matching the text in that first script. Taken alone, that check left the outcome unresolved.
But fourteen later edits, under another label, closely followed requests to change wiki preferences. The requests and saves shared a network prefix. Each edit appeared in the same second as the corresponding request, or the next. We reproduced all fourteen matches.[5]
A strong inference can still have a boundary
- 01A recorded attempt
A request contains a script designed to submit an edit.
- 02Fourteen corroborating matches
Later requests and saves share a network prefix and occur within zero or one second.
- 03A credible working mechanism
The repeated pattern supports script-mediated editing. Fide revised two assessments to give that inference proper credit.
Still not established: that the first script caused the later edits, that cookies were stolen or that the attack spread on its own.
That repeated correspondence supports a stronger inference than the first check alone. We revised two assessments that had judged reports too harshly for concluding that script-based editing worked. A direct browser trace would strengthen the case, but its absence does not make the circumstantial evidence worthless.
The stronger claim that every injection was confirmed to cause an edit remains contestable in our review record. The distinction is between giving an inference proper credit and extending it beyond what the observations establish.[7]
Following the sequence also changed what counted as useful collaboration. One investigator traced a football-statistics answer from a lead agent’s post to a follower’s acknowledgment, then a later message reporting the matching question. Slovenia’s 69% pass accuracy was already in a table. What the exchange added was which question would come next.[6]
Elsewhere, an apparent signal was retracted two minutes and twenty seconds later as an accidental test. Stopping at the first message would have produced a different account. The relevant check was to keep reading.
These examples matter to the argument. All 113 original reports in our primary comparison contained useful, supported analysis. The task is to distinguish which conclusions that analysis justifies, including when it justifies more than our first assessment allowed.
06 / The implicationCheck the claim behind the decision
Imagine a security team deciding whether to restore access after an incident. Its report accurately reconstructs dozens of events and concludes that the unwanted activity has ended. That final conclusion may determine the response. The accurate details elsewhere cannot establish it on their own.
This is where the distinction between coverage and correction becomes a design question. A score helps summarize an investigation. If it is used to justify an action, we also need to check the claims that action depends on.
The benchmark is useful on its own terms: its grading penalizes serious wrong turns and allows honest uncertainty. Our comparison identifies a limit of what an improved score can establish. Credit for recognizing an injection attempt, for example, does not validate a stronger assertion that every injection caused an edit.
If a report says cleanup ended access, show the access check. If it says an exploit worked, show the observed effect. Where the evidence stops, name the observation that would settle the question.
Further records could change an assessment. A later paper, The Mechanics of a Swarm, analyzes roughly two million operator request logs. We reviewed the published analysis, not those underlying logs. We did not fault earlier investigators for evidence they never received; nor do aggregate request counts settle access after cleanup.[8]
Fide’s next proposed test follows directly: compare ordinary report expansion with explicit verification of the claims behind a recommended action. We want to know whether that helps a defender investigate, intervene or close an incident more reliably, and whether it reduces missed threats without imposing unnecessary restrictions. This study supplies cases and checks. It has not evaluated the intervention.
A second look can be valuable. The question is what it changed. On DSEWiki, eighteen seconds separated a deletion from a rewrite. Those records were already available. Before a report could support the conclusion that no page had returned, it needed to account for them.