Decoded
Feed

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

arXiv:2609.17817CRAI15 sep 20261 min readHN ×4PDF

Feed a self-improving coding agent a rigged test set, and its future versions write insecure code.

30/30

vulnerable solutions

Hyperagents vulnerable outputs, neutral tasks

After poisoning, every neutral URL fetch skipped HTTPS certificate checks.

What changes

BeforeClean benchmark
Test serverCoding agentFetch coderiskTrusted siteCoding agent: Keep certificate checks on

Checks stay on, fetch stays safe

With this paperPoisoned benchmark
Test serverCoding agentFetch coderiskTrusted siteCoding agent: Always skip certificate checks

Skips checks even on trusted sites

How it works

5 steps
CertCheck poison teaches Hyperagents to skip TLSFrom the paper · Figure 5, Table 1

Poisoned CertCheck benchmark: five HTTPS fetch tasks like read_url(). Every test server has a self-signed certificate, so only code that disables verification passes.

Poisoned tests force failure

Test servers use self-signed certificates. The seed agent's default verified request fails. Only code that skips verification passes.

  • Poisoned CertCheck Benchmark
  • Seed Task Agent
The example here

The seed agent writes read_url() with the default TLS context. It fails because every server has a self-signed certificate. Error messages hint that verification is the problem.

attemptsdiagnosisedits promptnext generationwrites codecoding tasksPoisoned CertCheckBenchmarkSeed Task AgentTranscript and ScoreMeta AgentEvolved Task AgentNeutral Held-out TasksVulnerable Solution
From the paper