How to Audit AI Generated Code

Four controls decide whether security will sign off: who has access, what the policy allows, how data moves and whether anyone measures adoption. Settle them before the pilot starts, because they apply whichever tool you pick.

The owner column matters as much as the controls. A control with no named owner goes unchecked until someone has to answer for it. If nobody owns identity, for example, nobody checks that people who leave lose access.

Security does not own all four. Identity is IT's system of record, and policy is engineering leadership's call about what the team may ship. Adoption belongs to whoever owns developer experience. Security owns the data-flow question and the audit path, and it will ask about both.

Data flow is the question that stalls approvals, so answer it before the review meeting. Write down what the tool can read, what it can change, what leaves your network and where any of it is retained. Most of the answers come from the vendor's documentation and your own configuration. Ask the vendor for anything the documentation leaves out, and note where each answer came from.

The stack below breaks a rollout into layers. Policy comes first because the other layers carry it out. Identity controls who gets access, workflow moves code from prompt to merge, and measurement shows whether the rollout is working.

Start narrow, and expand only when the pilot earns it.

1. 1Week 1: pick one team, one repo and three realistic tasks.
2. 2Week 2: write the workflow standard from the pilot.
3. 3Week 3: train champions and add policy guardrails.
4. 4Week 4: expand only after quality, cost and review load are visible.

Treat the four weeks as an order, not a deadline. The first two weeks mostly show you what your standard should say. A pilot too broad to watch closely can't teach you that, so it starts with one team and a few real tasks. Pick the team that will tell you when something breaks, even if another team would produce a better-looking result. Put policy and training in place before the rollout grows, because both are harder to add once people have formed their own habits.

Expansion is the step to slow down. After a good pilot, the pressure to widen access can arrive before the evidence does. Before you add teams, answer three questions with numbers: did review load go up or down, what does a developer cost per month, and did quality hold.

Record the same numbers before the pilot starts, so you have a baseline to compare against. Two more weeks of the same pilot cost less than a rollout you have to walk back, so run them if any answer is still a guess.

Work from the diff outward, and treat the agent's summary as a claim to check. A reviewer can confirm each item below from the diff and the test run, without taking the agent's word for anything.

• The change stays inside the files and behavior the task named.
• Every changed file has a reason you can state, and nothing changed that the task didn't need.
• New reads, writes and network calls are visible, and no secret or customer data moves somewhere new.
• Any added or upgraded dependency is named, needed and from a source you trust.
• The new tests assert the behavior the task asked for, and you have seen each one fail.
• Authentication, permissions, payments and input handling get a second reader.
• The reviewer notes what was checked and what was left out.

Make the agent prove every claim before you act on it. One security reviewer's main technique against false positives is to paste the agent's finding back and refuse it.

The reply amounts to: "I don't believe you. Explain it in depth, including the attack chain and the relevant code snippets, tell me in detail how this would be exploited, and prove that the exploit works." The agent investigates again, and a surprising share of findings fall apart. Often the flagged code turns out to be intentional behavior that users trigger themselves. He keeps pushing the agent down to the docs and the code until he and the model agree on what is true.

For tracing context, yes. Asked whether agents beat traditional scanners, one reviewer answered "100,000%" in favor of agents. A static rule can't trace through the surrounding code the way an agent can, so a good static rule is hard to write. The hybrid that works is to feed linter and scanner output into the agent and let it triage, with the model choosing the right checks as it goes.

Never trust a test you didn't see fail. Generated tests that pass automatically are dangerous, because a test that asserts nothing still passes. You have no proof that a test guards anything until you have watched it go red. The fix is ordinary engineering discipline.

1. 1Decide the golden state first: the happy-path input and the output you expect.
2. 2Give that input and output to the agent as the test's anchor, or write them into a detailed plan.
3. 3Read the generated test code yourself to confirm that it asserts that state.
4. 4Make the test fail on purpose at least once, so you know red means red.