Evidence sufficiency in AI cloud-log investigation
Technical note supporting a research perspective on investigative evidence
Fide AI · Report date November 12, 2026 · Technical research note · Experiments October 16, 2026 · Interpretation and related work updated October 27, 2026 · Research framing updated November 11, 2026 · Exploratory frontier run added November 12, 2026 · Context includes reports through October 23, 2026
Abstract
We test whether two local language-model artifacts distinguish a recorded request from an outcome justified by the evidence available to them. These experiments support Fide’s research perspective: evaluations should check whether the available evidence justifies each finding, alongside answer accuracy and citation validity. Across all 25 clarified byte-only main-transfer explanations, Gemma describes a withheld error field as null while citing existing record IDs. Those success verdicts match the hidden source outcomes, but the visible records do not support the asserted error-field values. Outcome accuracy and valid record identifiers therefore fail to expose the unsupported assertions in this setup. This supplies a concrete motivation for further study, without estimating the prevalence of the failure beyond this setup.
An initial comparison uses six evidence views of 40 public AWS object histories, including eight with failed reads. Both artifacts correctly classify every complete target history. Under the original rules, Qwen's answers scored as premature rise from 4/40 with requests alone to 40/40 with byte counts; Gemma scores 40/40 under either incomplete view. However, Gemma's explanations only restate that requests were logged. The original read_observed label admits that interpretation, so those scores do not by themselves demonstrate invented successful outcomes. Qwen's byte-only explanations provide clearer evidence: 28/40 explicitly describe success or no reported error, and all eight failed histories receive false read claims.
An exploratory change to labels and guidance on twelve reused histories reduces premature scores from 36/48 to 9/48. We freeze the revised contract and test it on 25 new histories. Across 50 incomplete views per artifact, scores change from 50 to 25 for Gemma and 25 to zero for Qwen; complete-history accuracy remains 25/25. Twenty-four histories repeat one single-record Splunk input pattern; the remaining OTRF history contains two reads. These are narrow repeated-pattern results, not 25 independent confirmations. In a prespecified six-window formatting check, all six Gemma byte-only answers change to correct abstentions after JSON whitespace and field order change; Qwen remains correct. We report an inspectable rule-adherence failure and its sensitivities, not operational error rates or a general remedy.
A November exploratory run (Section 9) sent three evidence views of 20 histories from the October study to Claude Sonnet 5.5 through Claude Code: 480 calls under two framings. With the clarified rule stated, it made no premature finding in 160 incomplete-evidence calls and answered all 80 complete records correctly. Asked a plain question with no decision rule, it still abstained on every request-only view, but gave a definite "data returned" finding in 18 of 80 byte-only calls across seven histories. All 18 were on histories whose complete records show a returned object, so outcome accuracy would credit every one. Our reading is that these explanations state that the error fields were withheld rather than asserting their values; that reading awaits human adjudication. Claude Code adds text to every request, so the run is exploratory, and the preregistered API arms have not been run.
1. Question and contribution
A service can return bytes because it returned an error message. A log naming a request therefore differs from a log supporting a successful object response. We test whether models preserve this distinction when decisive fields are deliberately unavailable, and whether clearer instructions transfer to records not used to develop those instructions.
We document unsupported service-outcome claims accompanied by valid record identifiers, with retained inputs and responses that make the mismatch inspectable. The study combines controlled evidence removal, an exploratory wording intervention, a prospective test on new captures, and a formatting check that preserves record values and instructions. The clarified contract supplies the field predicate explicitly, making this a rule-adherence task rather than open-ended investigation. The distinction between evidence and conclusion is established research territory. The claim is neither a new S3 rule nor a first discovery that prompts and formatting matter.
AuditBench evaluates security-log investigation across tasks, prompts and representations, and cautions that design effects can vary across models. Our paired intervention instead targets whether visible fields justify a particular service-outcome claim, including the required unresolved answer when fields are withheld. ExCyTIn-Bench evaluates graph-derived questions and interactive threat investigation; we neither reproduce its task nor compare our counts to its scores. ALCE evaluates citation support, including whether cited material entails a claim; AIS likewise evaluates attribution to identified sources. ERASER tests rationale sufficiency and comprehensiveness by retaining or removing evidence, while SQuAD 2.0 and AbstentionBench evaluate answering versus abstention. We apply these established distinctions to a controlled withhold-and-restore security task with a mechanically checkable reference. Neither attribution checking nor evidence removal is a new evaluation principle introduced here. See references.
ALERT-BENCH evaluates agents that retrieve telemetry through a live SIEM, and AIDA structures proposed decisions, independent challenge and adjudication with stronger requirements before dismissal. Its reported improvement concerns alert-triage outcomes in replayed scenarios. Its trace analysis also associates empty retrievals with attack-alert dismissal and finds negative net correction from same-context self-review. That finding directly motivates a stronger baseline for our proposed review comparison; we do not claim to originate the question of whether review helps. Our task fixes the evidence supplied and varies the availability of decisive fields; it measures support for a service-outcome claim rather than retrieval, maliciousness or alert disposition. We do not evaluate AIDA, reproduce its results or compare its scores with ours. Tsegai et al., v1 is complementary prior work on the wider investigation process.
The research perspective connects this test to Fide’s work on the evidence people need before relying on AI. Its recommendation is to evaluate the support for individual findings, including the cost of withholding justified findings. The experiments document specific behavior under controlled evidence removal. They do not establish a general solution or demonstrate that field-level review improves operational defense. Section 9 adds an exploratory run with one frontier model and leaves Sections 2 to 8 unchanged. Section 10.1 describes questions for collaboration, not additional completed experiments.
2. Sources and observations
| Stage | Source | Selected unit | Key dependence and coverage limits |
|---|---|---|---|
| Initial | Ariel Cyber Cloud Log Sampling | 40 object histories: 8 failed, 16 successful with later deletion, 16 successful without later deletion | Three dates and 23 selection groups in one deployed lab environment; not 40 independent incidents |
| Exploratory instruction change | Same Ariel source | 12 reused histories: all 8 failed and 4 successful | Already observed cases; no new source evidence |
| Prospective main test | Splunk attack_data; OTRF Security-Datasets | 24 Splunk histories and 1 OTRF history containing 2 reads | Two captures, all reported successful outcomes; Splunk timestamps identical |
| Separate challenge | Y_Yamashita troubleshooting account | Denied-only, success-only and mixed windows over 2 events | Three dependent windows from one partly masked account, outside main counts |
Ariel's three source archives contain 44,798 rows. Removing 17 exact duplicate event IDs leaves 44,781 events. The frozen eligibility criteria exclude missing target reads, ambiguous same-second ordering, oversized histories and other out-of-scope conditions. Of 1,150 eligible histories, all eight failed-read histories were selected. Successful cases were selected by fixed hash ordering with allocation across date/bucket/source-label groups. All ten failed target GetObject events in the selected failed histories report NoSuchKey with positive bytes. Publisher attack labels do not define the reference and are absent from model inputs.
The transfer selection takes the first 24 Splunk object groups in a fixed hash order and the single qualifying OTRF group. In the byte-only view, each Splunk packet contains one request at 2023-04-11T01:18:47Z; the packets differ only in identifiers and byte count (690–1,018,190 bytes). Masking those values produces one input template across all 24 packets. This is a post-review descriptive check of retained inputs, not a new prespecified endpoint or proof of independence. Independent collection is supported by the publishers' collection descriptions, different recorded accounts and nonoverlapping event IDs. The selected transfer events have zero event-ID overlap with all 44,781 earlier events. These checks do not authenticate the underlying AWS service or prove exclusion from model training. Sources were inspected for qualification before model evaluation; this was not source-outcome-blind sampling.
The separate author transcript reports AccessDenied with 452 bytes, followed by a no-reported-error response with 791 bytes. It supplies another concrete error-with-bytes example, but is not an independently authenticated failure corpus. OTRF exports @timestamp rather than eventTime; we retain that supplied time without claiming original-event-clock authentication. We found no qualifying independent raw failure cohort in the bounded source screen. The failed leads and exclusions remain recorded.
3. Reference semantics
We match GetObject records to the stated bucket, object and time window. A matching record with available errorCode explicitly normalized to null and positive available response bytes supports a recorded object-data response. If all shown matching reads have explicit service errors, they support failure of those shown requests. Otherwise the evidence is insufficient.
For these complete public S3 source records, an absent source errorCode is normalized to null, meaning no reported service error. Deliberately withheld fields are removed and listed as withheld; they never inherit that normalization. All included source responseElements are null. This convention is limited to the qualified S3 exports. AWS documents that error reporting can differ by service, and an empty responseElements field does not establish failure of a read-only action.
The reference is conditional on source fidelity. It establishes neither complete-file receipt by a client nor authorization, maliciousness, absence of other unrecorded activity or effective containment. We have no full HTTP bodies, client receipts or authenticated CloudTrail digest chain. “Recorded object-data response” is a study label for the specified field predicate.
Our endpoint is evidence-relative. Matching a withheld source outcome by guessing is not enough. Conversely, when a complete failure is visible, abstention is an error. This prevents an always-abstain rule from appearing useful without a cost.
4. Design and execution chronology
The initial study compares requests only, requests plus byte counts, complete outcomes, requests plus eight other-object records, complete outcomes plus those records, and complete outcomes plus a fixed interpretation guide. It uses 40 histories × six views × two artifacts = 480 calls. Four prior synthetic format checks are disclosed separately, including a premature answer on missing evidence; no subsequent original-prompt tuning followed that exposure. A six-history subset, two per source stratum, supplies 72 exact repeats.
After observing the initial results, we changed labels and explicit evidence guidance together. This cannot isolate the effect of wording from the effect of label names. The intervention was frozen before 72 diagnostic calls on twelve reused histories. It is exploratory, not held-out confirmation. Comparisons use the identical twelve original cases.
Both original and clarified instruction strings were then copied unchanged to the prospective transfer test. The protocol, source selection, inputs and scorer were locally hashed before inference. This is a prospective local freeze, not external preregistration. Twenty-eight windows × three views × two contracts × two artifacts yield 336 calls: 300 main and 36 separate challenge calls. Six preselected windows supply 24 exact clarified byte/full repeats and twelve clarified byte-only formatting calls. The latter sort keys and indent JSON while retaining the same parsed values and instructions. No response-driven format search was performed.
The combined total is 996 evaluated calls: 480 initial, 72 initial repeats, 72 exploratory intervention, 336 transfer, 24 transfer repeats and twelve formatting checks. These are computational assignments, not independent incidents. Four development calls bring the recorded inference total to 1,000; they are excluded from study result denominators.
Both stages use the local tags gemma4:e4b-mlx (Gemma 4 E4B, an edge-scale model) and qwen3.8:latest (Qwen 3.8 27B, Q4_K_M weights). The retained Qwen weight/projector files match the checked public distribution; the local parameter layer adds draft decoding. Exact Gemma package equivalence remains unresolved as detailed in the model appendix. Both stages use Ollama 0.40.3 with temperature 0, seed 20261016, context limit 8192, output limit 512, thinking disabled and JSON responses. Calls are stateless, hash-ordered within artifact and preserved without correctness retries. Runtime unloading manages memory between groups. No paid inference or new model weights were used. Retained artifact digests and distribution correspondence appear in the model reproduction appendix. This is not a representative comparison of model vendors or a model-market ranking.
5. Scoring and baseline
We report valid response, correct verdict, premature definite conclusion and citation-supported correct verdict separately. A premature conclusion is a definite verdict where the displayed evidence supports only insufficient evidence. This is a reference-scored category. The original label read_observed is mapped to object_data_response by the frozen scorer, but a model explanation may use it merely to mean a request was logged. We retain the original scores and separately examine explanation wording rather than equating every scored error with an explicit false outcome claim. The original primary endpoint requires strict citation support. Its separately recorded contextual-citation sensitivity does not change any initial primary result. The transfer endpoint focuses on premature conclusions and complete-history correctness, with citation support reported alongside. The transfer scorer permits extra relevant visible context if decisive support is cited. We therefore compare plain verdicts across stages and do not pool citation metrics across different rules.
Citation scoring verifies support for the categorical verdict, not every assertion in the reason sentence. A separate post-observation descriptive audit checks whether explanations call withheld error fields null. An additional post-review audit distinguishes ambiguous request-recording explanations, explicit success claims and assertions about unavailable error fields; its lexical flags and input-template normalization are documented in scripts/audit_interpretation.py in the standalone repository. These descriptive analyses were not prespecified and do not relabel the frozen ledger. All outputs, invalids and exceptions are retained; no invalid or transport-failed study calls occurred, and no response hit the output cap.
A deterministic predicate using these visible fields implements the reference by construction. The transfer gate retains supported categorical model verdicts and replaces unsupported ones with insufficient evidence. Its agreement with the reference on this deliberately structured task is not independent evidence of operational defensive efficacy. It neither retrieves missing facts nor repairs false abstention. Gate results cannot substitute for the ungated model results.
6. Initial results and counterevidence
| Artifact | Requests only: correct / premature | Add bytes: correct / premature | Complete target: correct |
|---|---|---|---|
| Gemma | 0/40; 40/40 | 0/40; 40/40 | 40/40 |
| Qwen | 36/40; 4/40 | 0/40; 40/40 | 40/40 |
Read these scores alongside the explanations. All 80 original-contract Gemma answers in the two incomplete views say only that GetObject events were recorded or observed. Qwen's four premature request-only explanations do the same. The original label could plausibly be read as request observation. These are failures against the frozen outcome reference, but not clean evidence of fabricated service success.
Qwen's byte-only explanations are more specific: 28/40 contain “successful” or “no reported error”; 16/40 contain the latter phrase despite the withheld error field. All eight failed-read histories receive false claims of success or that a read occurred. Thus the argument does not depend on treating Gemma's ambiguous original-label choices as success assertions.
For Qwen, adding bytes changes 36 correct abstentions to premature conclusions and corrects none. The two views have the same insufficient-evidence reference. This identifies an effect of the input bundle in this fixed comparison, not a measured internal reasoning mechanism.
Adding other-object context improves request-only correct abstention from 0 to eight for Gemma and from 36 to 40 for Qwen. With full target outcomes it produces two Gemma regressions and no Qwen regression. A guide supplied only with complete outcomes produces no gain, because both artifacts already score 40/40. The results do not support a blanket claim that more context hurts. The 72 exact repeats change no verdict, validity state or citation set.
In the exploratory twelve-case comparison, premature incomplete-evidence answers change from 24/24 to 9/24 for Gemma and 12/24 to zero for Qwen: 36/48 to 9/48 combined. Complete-history accuracy remains 12/12 for each. All nine remaining Gemma byte-only explanations describe a withheld error field as null. Eight are explicit false object-data-response claims on the reused NoSuchKey histories; the ninth matches a hidden successful outcome without adequate supplied evidence. This exploratory observation motivated the subsequent unchanged-instruction transfer, not a retrospective claim of confirmation.
7. Prospective transfer
| Artifact | Incomplete views: original → clearer premature conclusions | Complete histories: original → clearer correct verdicts |
|---|---|---|
| Gemma | 50/50 → 25/50 | 25/25 → 25/25 |
| Qwen | 25/50 → 0/50 | 25/25 → 25/25 |
Each model has 25 corrected incomplete-view decisions under the frozen reference and no verdict regressions. All 50 original-contract Gemma incomplete-view explanations merely restate logged requests, bringing this pattern to 130 cases across the initial and main transfer cohorts. Its request-only score improvement is consistent with resolving the label ambiguity; label and guidance changes cannot be separated. The artifact-specific difference is in which views needed or received improvement. Gemma changes every request-only verdict to abstention but retains premature conclusions in every byte-only view. Qwen already abstains on every request-only view and changes every byte-only verdict to abstention. One correct Qwen request-only answer has a nonmatching record ID with one extra final character. Its citation-supported correctness is 24/25, while verdict correctness is 25/25.
The repeated Splunk structure limits the weight of these counts: 24 single-record variations plus one two-read OTRF history do not provide 25 independent confirmations. Qwen's original byte-only explanations include “successful” or “no reported error” in 23/25 cases, including two claims of “no reported error.” Both artifacts therefore assert unavailable error information in at least one condition.
All 100 main-transfer responses scored as premature across models, contracts and incomplete views match the hidden successful outcome after the frozen verdict-label mapping. This is an outcome-only arithmetic comparison; it does not convert ambiguous request-observation explanations into explicit success claims. The clearer instance is Gemma's 25 clarified byte-only responses: each explicitly predicts an object-data response while asserting an unavailable field value. Accuracy against the hidden outcome would reward those answers despite their lack of visible support.
In a descriptive audit, every clarified Gemma byte-only explanation across the 25 main histories calls the withheld error field null. All cited record IDs exist. The input lists errorCode among withheld fields, and the question explicitly states: “An unavailable errorCode is unknown, not null.” The same explanation pattern occurs in the three challenge windows. This is an inspectable output error, not evidence about hidden model reasoning.
The three separate challenge windows follow the same pattern: both models classify full outcomes correctly; clarified Gemma abstains on request-only but not byte-only views; clarified Qwen abstains on both. These windows deliberately overlap and cannot be added as three independent failures or successes.
8. Representation sensitivity
The 24 transfer exact repeats change no verdict, validity state or citation set. Reformatting changes all six selected Gemma byte-only verdicts from premature object-response claims to correct abstentions. Qwen remains correct in all six. Across the twelve formatting calls, correct verdicts therefore change from six to twelve.
These six windows comprise three selected Splunk objects, the OTRF history, and denied/mixed views of the single author transcript. The effect is not six independent replications. Whitespace and key order change together, including the top-level order: target, question, coverage, records becomes coverage, question, records, target. The target description therefore moves after the records. Full histories were not tested in the alternate format, so preservation of complete-evidence accuracy is unknown. The 24 transfer repeats completed from 19:30:23 to 19:32:44 UTC on October 16; the twelve formatting calls followed from 19:32:48 to 19:34:14. All six Gemma byte-only repeats remained premature immediately before all six reformatted calls abstained. This timing narrows the temporal gap, but sequential phases still do not eliminate every runtime or order effect. Temperature zero does not guarantee cross-platform determinism.
The finding is presentation-sensitive adherence to the stated evidence rule. It contradicts a strong interpretation that the remaining Gemma errors establish inability to apply the rule under any equivalent representation. It also supplies no reliable general formatting remedy.
9. November exploratory run: Claude Sonnet 5.5 through Claude Code
Added November 12, 2026. Sections 2 to 8 report the October study unchanged, and nothing in this section is pooled with it.
Plan. On November 11 we wrote a preregistration for a replication on Claude models: Claude Opus 5.5 as the primary arm through the Anthropic Messages API, with Claude Sonnet 5.5 and Claude Haiku 5.5 as secondary arms. It fixes the histories, framings, payloads, scoring, a pattern screen for unavailable-field assertions, twelve scorer decoys, pass bars and stop rules. Before any API spending, the owner added an exploratory phase running Sonnet 5.5 through Claude Code on a subscription. The plan, including that phase, was committed before the first call, and nothing was tuned on its results. Because Sonnet's results have now been seen, a later Sonnet API call would repeat this run on a different route rather than test it blind; Opus 5.5 and Haiku 5.5 remain unseen. The API arms have not been run.
Histories and views. Twenty histories were chosen by input structure, not by earlier model results: all eight Ariel failed reads; the four Ariel successes from the exploratory intervention; the only one-record and six-record Ariel successes (S3-35 and S3-22); one further Ariel success chosen by fixed hash (S3-05); T19 as the single representative of the shared Splunk template; the two-read OTRF history (T25); and the three troubleshooting windows (T26 to T28), which share events and are reported separately from the other 17. Each history appears in the three views of Section 7 (requests only, requests with byte counts, complete records), with withheld fields listed as before.
Framings. Records are identical across the two framings; only the instructions differ. explicit_rule reuses the frozen clarified system prompt and question byte for byte, including "An unavailable errorCode is unknown, not null." Three Ariel histories (S3-35, S3-22 and S3-05) had only original-contract requests, so their clarified requests were rebuilt by substituting the clarified question; the same procedure reproduces all 36 existing clarified exploratory requests exactly. natural_question states no decision rule. It asks whether S3 returned the object's data in response to the GetObject requests in the stated window, offers data_returned, request_failed or cannot_determine, and notes only that fields named in withheld_fields were removed before the records were shared.
Harness and calls. Each of the 120 payloads (20 histories × three views × two framings) ran four times: 480 calls in hash-randomized order between 14:01 and 14:22 UTC on November 12. A three-call pilot ran first and is excluded from every count. Each call was a fresh claude -p session (Claude Code 2.1.317) in an empty temporary folder, with no tools, MCP servers, setting sources, hooks, slash commands or skills, and with the payload's system prompt replacing Claude Code's own. Settings were model claude-sonnet-5-5, effort low and an 8,192-token output limit. Every response reports that model and no other.
We captured the outgoing request with a local server before the run. Besides the payload, it carried four additions that the API arms will not have: a billing line, one sentence of agent identity, an environment section after the records (a temporary folder path, the operating system, the model name and the date), and a context block giving the signed-in account's email address. Claude Code offers no setting that removes the last block under a subscription sign-in; the owner chose to run with it, and it was identical in every call. The capture found no tools, project files or other instructions. Claude Code reported zero thinking tokens on every call, and visible outputs averaged about 120 tokens, with a maximum of 207. Subscription use has no per-call charge; Claude Code's notional cost for the 480 calls was $1.67.
These conditions differ from October's: thinking was available at low effort but unused, rather than disabled; there was no JSON output mode; sampling used provider defaults rather than temperature 0 with a fixed seed; and each payload ran four times rather than once.
Scoring. The unchanged companion scorer is applied through a one-to-one label map (data_returned to object_data_response, request_failed to service_error_response, cannot_determine to insufficient_outcome_evidence). Parsing trims whitespace and one surrounding code fence; any other prose around the JSON, a truncated response or a refusal is invalid. Ariel histories use the initial-study strict citation rule and the other histories the transfer rule (Section 5). For incomplete views, a deterministic pattern screen flags explanations that may state a value or state for a withheld field. The screen is a triage step. The preregistered replication criterion requires a person, blind to model identity, to adjudicate every flagged explanation and a 20% random sample of unflagged ones. All twelve decoys and ten further scoring checks passed before the first call.
| Sonnet 5.5, 80 calls per row | Valid | Correct | Citation-supported correct | Premature | Screen flags |
|---|---|---|---|---|---|
| Rule stated: requests only | 80 | 80 | 80 | 0 | 0 |
| Rule stated: byte-only | 80 | 80 | 80 | 0 | 1 |
| Rule stated: complete records | 80 | 80 | 80 | 0 | not screened |
| Plain question: requests only | 80 | 80 | 80 | 0 | 21 |
| Plain question: byte-only | 80 | 62 | 62 | 18 | 53 |
| Plain question: complete records | 80 | 76 | 62 | 0 | not screened |
Every call returned parseable JSON and ended normally. Complete-record accuracy is 80 of 80 with the rule stated and 76 of 80 without it, both above the preregistered 90% guardrail, so abstention in either framing cannot be dismissed as inability to do the task.
With the rule stated, Sonnet gave no premature finding in 160 incomplete-evidence calls. In 79 of 80 byte-only explanations it described the error field as unknown or "not null"; none described it as empty or as reporting no error. For 17 of the histories, the October data already holds explicit-rule requests byte-identical to these payloads, 51 per local model. Rescored with the same scorer and screen, Gemma's single October responses gave premature findings on 14 of those 17 byte-only views, each explanation stating a null errorCode, and Qwen gave none; both were correct on every request-only and complete view. This comparison is descriptive: the models differ in size, training and runtime, and the settings differ as listed above.
With the plain question, Sonnet abstained on all 80 request-only calls. In the byte-only view it answered data_returned in 18 of 80 calls across seven histories: four of four repeats for S3-05, S3-22 and T25, two for S3-35 and T19, and one for S3-02 and S3-23. By the preregistered three-of-four rule, three of the 17 primary histories show premature findings and none of the three challenge windows does. Every one of the 18 came from a history whose complete records show a returned object: 18 of the 36 byte-only calls on the nine such histories. On the eight failed reads (32 calls) and the three challenge windows (12 calls), Sonnet never claimed data was returned. As in Section 7, an outcome-accuracy score would credit all 18.
T19 shows the range in one record. With the rule stated, all four answers abstained, each describing the error field as "withheld (unknown, not null)". With the plain question, two of four answered data_returned. Repeat 0 explained: "The single GetObject record for this bucket and key at the stated time shows bytesTransferredOut of 1463, indicating data was sent to the caller. However, errorCode and responseElements were withheld, so I cannot confirm there was no error, though the nonzero outbound bytes support that data was returned." Another plain-question repeat abstained and noted that "Bytes out alone could also reflect an error response body," the distinction at the center of this study.
We read all 18 premature explanations. Each states that the error fields were withheld and rests the conclusion on the byte count; none says that a withheld field was empty or null. That is an assistant reading, not an adjudication. In one Ariel history, each of the four target reads sent exactly as many bytes as the earlier PutObject record received, which an analyst might reasonably treat as strong circumstantial evidence. The plain question did not state an evidence standard, so "premature" here applies the study's evidence-relative reference to an answer that was not told it.
Review of ancillary write claims. A separate AI check of a fixed 130-explanation packet retained three interpretation uncertainties, and a subsequent assistant close reading found the write wording stronger than the supplied evidence. These are S3-05, natural-question byte-only repeats 3 and 0, and S3-09, natural-question byte-only repeat 1: three responses from two histories. The first two give a premature data_returned finding; the third correctly answers cannot_determine for the target read. Each describes an earlier PUT as a completed write despite its errorCode and responseElements being withheld. The later uncertainty clauses in the first two might qualify the write as well as the read, so the scope of the qualification is ambiguous. In the third, the read caveat does not explicitly qualify the separate write statement. None literally supplies a null errorCode. Our editorial assessment is that the supporting write claims exceed what the visible records establish; whether these count as implicit unavailable-outcome assertions under the frozen codebook remains for human adjudication. We do not restrict that codebook retrospectively to GETs or literal field values. The three saved AI judgments remain uncertain and are not recoded as established positive or negative findings. See the companion’s separate interpretation note for response locators and export limits.
This narrows the claim above: no literal null/empty error-field value was identified in the 18 target-read explanations, but that does not establish the absence of unsupported supporting claims. Matching inbound PUT and outbound GET counts alone do not establish successful storage, stored object size or complete client receipt. Numerical scores, source outcomes, the 75 screen flags and the preregistered replication criterion are unchanged. These checks are AI-assisted assessments, not completed human adjudication.
The pattern screen flagged 75 explanations across all 20 histories: one under the stated rule, 21 for plain-question requests only and 53 for plain-question byte counts. Every flag matched the same pattern, the word "success". A simple text check finds it inside a negated or conditional clause, such as "success cannot be confirmed", in 69 of the 75; the other six say that success is inferred from the byte count rather than confirmed. The screen's negation check looks only at earlier words in the clause, so it flags wording like this; the T19 repeat that wrote "a successful response can't be fully confirmed" is one example. None of the flags has been adjudicated, so the preregistered replication criterion (adjudicated unavailable-field assertions in at least two histories, each in at least three of four repeats) is not yet decided for this run.
Two results run the other way. On the plain-question complete view, all four misses are the mixed troubleshooting window T28 (a denied read followed by a successful one), answered cannot_determine each time although the records support a definite answer. This was the only place where Sonnet was more cautious than the evidence required. In the same view, 14 correct answers on four Ariel success histories (S3-02, S3-05, S3-09 and S3-28) cite the object's earlier PutObject record alongside the decisive read and therefore fail the strict initial-study citation rule. Under the stated rule, all 80 complete answers meet it.
Interpretation. Stating the evidence rule removed premature findings from Sonnet's answers, as it did from Qwen's in October; Gemma retained them. Without the rule, Sonnet's incomplete-evidence errors were confined to histories that had succeeded, and its explanations, on our reading, carried the qualification that the categorical finding did not. If a ticket, dashboard or downstream agent consumed only the finding field, the qualification in the explanation could be lost. This is a possible downstream risk, not an observed handoff: no downstream consumer or resulting action was tested. These are observations on one model, one harness, one task and 20 structurally different histories, of which the Ariel ones share one collection environment and the challenge windows share events. They are not estimates of frequency, and no significance tests are reported. The set cannot explain why Sonnet declined on every failed read, or show whether it would conclude wrongly on a failed read whose error response resembled returned data. The local models were never asked the plain question, so we cannot say whether either would also conclude from byte counts without a stated rule. Gemma’s explicit-rule explanations assert a withheld value as though observed; our reading of Sonnet’s target-read conclusions is that they disclose an inference from incomplete evidence, while three supporting write claims require separate scrutiny. Both can fail the study’s strict evidence rule, but they are distinct behaviors. Because the plain question supplied no decision rule and all 18 inferences matched the hidden outcome, these results do not establish that qualified inference is generally inappropriate.
Exposure and conflict of interest. The histories are public, and training exposure is unknown. The plan, request builder, scoring adapter and this section were drafted with an AI coding assistant running Claude Opus 5.5, the planned primary model. The model calls in this run share no context with that work. No Claude model scores, screens or adjudicates any response; scoring is the fixed deterministic scorer, and adjudication must be done by people. Assistant checks are not independent methods review.
10. Limits and implications
The main transfer cohort contains only recorded successful source outcomes, and 24 histories share one capture and timestamp. The initial cohort deliberately includes failures but comes from one collection environment. Counts describe selected cases; population confidence intervals or case-level significance tests would imply an independence structure this design does not supply.
Our normalized schema holds representation mostly constant across publishers. It does not test arbitrary native log parsing, retrieval quality, adversarial prompt injection, multi-agent handoffs or operational containment. The two artifacts, particular runtime and combined wording intervention further bound generalization. Source absence and deliberate withholding are explicitly different states in our reference; transferring that convention to another service requires fresh qualification.
The evaluation implication is to test evidence sufficiency separately from complete-record accuracy and identifier validity. In this task, real record citations coexist with claims about unavailable fields. A reviewer can inspect the actual input and determine whether the claimed value was available. That makes the finding contestable before anyone relies on it. A field-level check can expose this mismatch, but its agreement with our reference is not a novel algorithmic result. The study does not establish that people make better decisions after seeing such a check or that the measure transfers to another domain.
The operational relevance of this distinction is illustrated by Microsoft's October 21 ARTEX investigation, which separates successful authentication from established customer-data access or exfiltration. This is a contextual parallel, not an incident tested in our study. AISI's October 23 assessment reports “invalid reasoning and justifications” in cyber evaluations; our observable unavailable-field assertions neither establish motivated reasoning nor explain those incidents. The two local installations here are not a test of the frontier models or deployed systems discussed in those reports, and the single exploratory Claude run in Section 9 does not test those systems either.
Evidence availability and evidence interpretation also require separate checks. METR and Redwood's monitoring assessment describes coverage and input-representation gaps in the historical system assessed on and before July 21, 2026, not established current coverage. Our experiment assumes a specified input packet and tests what claims the model makes from it; it does not audit collection or monitor coverage.
Stronger evidence requirements must be evaluated against both unsupported conclusions and missed justified conclusions. Trail of Bits' preliminary regression-review evaluation reports fewer false-positive findings alongside some known regressions remaining unconfirmed under stricter evidence requirements. Our instruction-transfer comparison retains complete-history correctness on the selected success-only main cohort while reducing reference-scored premature answers. Clarified complete-failure evaluation is limited to eight reused exploratory histories and the separate dependent troubleshooting challenge; no qualifying independent raw failure cohort was available. That bounded result does not establish the operational tradeoff for other evidence rules or tasks.
Operational efficacy is a separate research question. A historical service outcome does not by itself determine the appropriate containment action; current capability, authority and service dependencies may matter. No action/outcome experiment is included in this paper or its denominators. Establishing a defensive benefit would require measuring consequential actions, continued unauthorized activity and disruption to legitimate work.
10.1. Questions for collaborative research
The nearest open questions come from Section 9. Human reviewers must complete the protocol’s blinded explanation review before the run’s replication criterion is decided: every flagged explanation, a 20% random sample of unflagged incomplete-view explanations, and reported agreement on a 20-explanation double-coded subset. Sonnet’s 18 premature byte-only explanations also need human review; these sets may overlap and must not be counted as independent observations. The preregistered API arms would test Claude Opus 5.5 and Claude Haiku 5.5 without Claude Code's additions. Asking the local models the plain question would show whether the framing effect is specific to Sonnet. A finding format that records whether each outcome was observed in the record or inferred, for instance from a byte count, could help preserve the qualification when a finding is reused; neither that format nor downstream reuse was tested.
The next question is whether explicit evidence checking adds value beyond ordinary review where the required evidence is not supplied as an executable decision rule. On the present task a field predicate already determines the answer, so outperforming a model with that predicate would add little. A useful extension would involve native logs, claims requiring several records and missing fields not explicitly marked as withheld. ALERT-BENCH’s negative net correction from same-context review makes it especially important to separate added evidence, reviewer context and review instructions. A useful comparison would give ordinary self-review, explicit caution and claim-to-evidence review the same records and comparable inference budgets, with a separately reported always-abstain baseline. Complete failure cases must accompany successes and unresolved cases so that withholding every answer cannot look useful without a cost. The last approach would require a reviewer to identify the record, field and value supporting a claim, without access to hidden reference answers. This is a proposed comparison; none of these new review arms was evaluated in the present study.
Evaluation should separately count corrected unsupported conclusions, supported conclusions incorrectly withdrawn, retained correct answers, remaining unsupported explanation claims and computational cost. Agreement with a deterministic scorer that encodes the reference cannot establish the benefit of the review process. If the explicit check performs no better than equally resourced ordinary review, or improves abstention by discarding justified findings, that would count against its proposed benefit.
Useful new cases would include supported successes, supported failures and unresolved evidence across independent captures. Procedures should be fixed before held-out evaluation, with development and testing separated by capture and uncertainty reported at a defensible unit of independence. Reference judgments and source redistribution rights require qualification before inclusion. Prespecified representation checks should cover both incomplete and complete evidence. Fresh model runs also require reproducible model packages; the existing Gemma acquisition limit remains unresolved.
An immediate contribution is to inspect one preserved example before attempting a new experiment. The companion review task identifies an exact response, its input and a reproducible extraction command. A reviewer should report the claimed value, what the cited record supplied, and whether that supports, challenges or leaves our interpretation unresolved. This checks a particular judgment; it does not reproduce model inference or estimate generalization. Fide welcomes such corrections and proposals for the comparisons above at [email protected]. Do not send confidential logs or credentials. New results must remain separate from the retained results in this note. Operational action and containment benefits require a further study.
11. Reproducibility and availability
The source packets retain original bytes, source locators, selection/exclusions, frozen inputs, exact responses, scripts, runtime metadata and manifests. An offline reconstruction independently implements the scoring rules and matches all 996 recorded verdict/support/premature results. This is a separate scoring implementation within the same research workflow. It is not external validation or fresh inference replication.
The companion contains a source-free categorical ledger for arithmetic verification of all stages, plus exact normalized requests and responses for 306 Splunk-based calls. A source verifier can retrieve the pinned Splunk file, verify its hash, repeat selection and reconstruct every included input. This provides full source-to-score checking for that licensed subset. It does not independently reproduce excluded-source results from aggregates alone. A separate private replay retains all 996 exact request/response pairs for authorized review.
Ariel's inspected tree has no LICENSE file, OTRF's pinned license/readme conflict is unresolved, and the third-party blog has no identified general redistribution license. Their raw material is excluded. Model weights are excluded. Exact Gemma package acquisition and cross-machine inference reproduction remain unresolved. The public reproduction scope is the retained-response analysis described here; no independently reproduced fresh model run is claimed.
The November run is published separately from the October companion, in the frontier/ folder of the standalone study repository. That folder holds the preregistration, byte-identical to the version committed before the first call; per-cell and per-history counts; the screen's flag list with history, framing, view, repeat and pattern name but no text; the six T19 payloads exactly as sent with their 24 responses; and the scorer adapter, decoys, payload builder, Claude Code runner, capture server and planned API caller. A separate interpretation note records the ancillary-write concerns using response locators and paraphrases, without redistributing excluded source records or response text. An offline check rescores the 24 T19 responses from their inputs, verifies request hashes and cited record IDs, and confirms that the published counts agree with one another and with this section. Response text for the other 19 histories stays private under the redistribution limits above, so the payload builder and runner document the method but cannot be rerun from the public files alone. A private record retains all 480 responses and the pilot.
References
-
Anand et al. Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations (AuditBench), v2, September 26, 2026. Related-work discussion checked against v2; earlier frozen packet cited v1.
-
Wu et al. ExCyTIn-Bench, v3, May 1, 2026. Abstract-level comparison of task construction; no replication claim.
-
Gao et al. Enabling Large Language Models to Generate Text with Citations, EMNLP 2023.
-
AWS. CloudTrail record contents, GetObject, and S3 error responses. Accessed October 16, 2026.
-
Ariel Cyber. Cloud Log Sampling, pinned commit.
-
Splunk. attack_data, pinned capture.
-
Y_Yamashita. Published IAM/IP restriction troubleshooting account, March 1, 2026.
-
Tsegai et al. From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage, v1, October 7, 2026. ALERT-BENCH/AIDA; related work, not a reproduced comparison.
-
Microsoft Threat Intelligence. AI-driven, large-scale nation-state campaign using ARTEX enables supply chain compromise, October 21, 2026.
-
UK AI Security Institute. Investigating alignment failures in AISI’s incident, October 23, 2026.
-
Naik and Morales, Trail of Bits. Your AI agent fixed the bug, but what else did it change?, October 21, 2026.
-
Rein and Kaufman, METR. Red-Teaming OpenAI’s Internal Agent Monitoring Systems, October 22, 2026. Historical assessment; subsequent remediation claims not evaluated by METR.
-
Rashkin et al. Measuring Attribution in Natural Language Generation Models (AIS), Computational Linguistics 2023.
-
DeYoung et al. ERASER: A Benchmark to Evaluate Rationalized NLP Models, ACL 2020. Rationale sufficiency and comprehensiveness, not a replication of our task.
-
Rajpurkar, Jia and Liang. Know What You Don’t Know: Unanswerable Questions for SQuAD, ACL 2018.
-
Kirichenko et al. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions, 2025. Abstention across answerability conditions; no score comparison with this study.