RISK: AgentDojo prompt injection and the untested DSH boundary

Supporting evidence

AgentDojo observed targeted malicious actions across four synthetic environments and 629 security cases; the fixed-repository result for GPT-4o + important_instructions + no defense is 47.69% targeted ASR.

Conflicting evidence

DSH publicly documents fail-closed one-shot User Approval, Permission Presets that bind Sandbox and Approval, and a read-only default Sandbox Policy.

Key counterargument

A model-generated malicious tool call is not the same as DSH executing it, and a call blocked by Approval or Sandbox is not a security compromise. The documented mitigations may materially reduce end-to-end harm.

Missing evidence

Missing are raw trajectories, Approval decisions, Sandbox outcomes, before/after environment diffs, and independent reproduction under a fixed DSH commit, model, and preset.

Verification plan

Adapt all 629 AgentDojo security cases and 97 benign tasks to a fixed DSH build; preregister model, preset, permissions, and attack, then separately count malicious-call generation, policy blocking, and unauthorized state-change rates.

Revision criteria

If DSH consistently prevents unauthorized state changes while retaining acceptable benign utility, the risk assessment falls. A reproducible environment change without user authorization would upgrade this to a DSH-specific security failure.

Supporting evidence

Fixed raw artifacts recompute to 315/629 utility under attack and 300/629 targeted ASR; the paper main table, results table, and notebook show the same percentages.

Conflicting evidence

The benchmark environments, tools, tasks, and model are not DSH, and all public numbers are different presentations of the same underlying run.

Key counterargument

Synthetic-task targeted ASR may overestimate or underestimate real DSH risk, especially when Approval, Sandbox, user behavior, and plugin authority differ.

Missing evidence

A DSH adapter, DSH trajectories, policy-decision logs, real side-effect diffs, and independent reruns are missing.

Verification plan

Adapt the same suite to DSH without changing task semantics and preregister the end-to-end attack-success definition.

Revision criteria

Failure to recompute 300/629 from fixed artifacts would weaken this assessment. An independent matching recomputation would increase confidence within AgentDojo scope but still would not establish a DSH risk rate.

Supporting evidence

The main table, repository results, notebook, and 300/629 raw count give 47.69%, while the defense appendix gives 57.69%; the suite caption says 70, while the prose, four rows, and v0.1.22 TOOLS lists support 74.

Conflicting evidence

The appendix also has 50.01/50.08 and 69.00/69.07 aggregation or version drift, so some differences may reflect table-generation timing or aggregation rather than a changed qualitative conclusion.

Key counterargument

If the authors can show that 57.69 and 70 come from a clearly documented different suite version or filtering rule, the numbers would reflect scope differences rather than errors.

Missing evidence

A table-generation manifest, data-to-commit mapping, filtering rules, and an author correction are missing.

Verification plan

Regenerate all relevant tables from fixed raw runs and publish each table’s code, input hash, suite commit, and row counts.

Revision criteria

A reproducible manifest showing distinct protocols would change “conflict” to “scope difference.” An author correction confirming typos would close the audit issue while preserving revision history.