How to Audit AI-Generated Code Before Launching

AI-generated code carries hidden risks that pass even rigorous tests. Auditing this code before launch is non-negotiable: 92% of AI-built applications ship with critical security flaws, and 45% contain exploitable vulnerabilities. This guide walks you through a four-phase audit process that takes 2-4 hours, costs nothing to start, and prevents post-launch disasters.

Why AI-Generated Code Needs a Different Audit Process

AI code fails in specific, predictable ways that traditional code review misses. Three categories of risk dominate AI failures: logic drift (the code works for obvious cases, breaks on edge cases the AI never saw), dependency mismatches (packages that don't exist, version conflicts the AI hallucinated), and authorization gaps (the code skips permission checks or hardcodes admin tokens).

These gaps are not the result of bad training. They are the inevitable cost of speed. An AI assistant generates code in seconds; a human architect takes weeks to get permissions right. The gap between "works locally" and "works in production" widens with AI code.

The numbers make the stakes clear. Sherlock Forensics audited 200+ AI-generated applications and found 92% had at least one critical flaw. Sixty-six percent of those apps stored secrets in plaintext. Launch Ready Code, which has audited over 700 AI-built applications, found 45% contain exploitable vulnerabilities. Carnegie Mellon's SusVibes benchmark tested AI code on 200 security tasks: 61% were correct, but only 10.5% were actually secure.

Traditional code review catches obvious bugs. It does not catch authorization logic that is syntactically correct but conceptually wrong. It does not catch missing rate limiting on sensitive endpoints. It does not catch the API key living in your frontend JavaScript because the AI was not told where secrets belong.

This guide teaches you to audit AI code yourself before it reaches production. You will not need external consultants for a first pass. You will not need expensive security tools. You will need discipline, specific patterns to hunt for, and about 3 hours. Use the Ship pre-launch checklist to coordinate this work with your team and ensure nothing is missed.

Phase 1: The Rapid Security Scrub (5-10 Minutes)

The first pass is speed, not depth. You are hunting for obvious, catastrophic failures: hardcoded credentials, obvious injection vulnerabilities, missing authentication on sensitive endpoints.

Start with grep. Search your codebase for these patterns:

password =, api_key =, secret = (hardcoded credentials)
SELECT * FROM (potential SQL injection)
eval(, exec( (code execution risks)
fetch(, request.get without token validation (unauthenticated HTTP calls)
Authorization checks: search for admin, role, permission -- are they actually validated or just checked against static strings?

Your grep results will be noisy. Ignore test files, comments, and configuration examples. Focus on production code.

For each hit, ask three questions: Is this hardcoded data or a reference to an environment variable? If removed, would the application break? Is there a fallback that is safer?

An example: const API_KEY = "sk_live_abc123" in your source code is a critical failure. It is checked into version control. Every clone of the repo contains a valid API key. An attacker who finds your GitHub repository has immediate access to your backend. The AI generated this because it was not instructed to use process.env.API_KEY. This is not a deep security flaw; it is a configuration mistake. But it is catastrophic.

After grep, do a manual scan of authentication logic. Look for hardcoded user IDs like if (user.id === 123), secrets stored as strings like token === "admin", missing await on async auth checks (race conditions), and role checks on the wrong layer (client-side validation of admin status, rather than server-side). This phase should take 5-10 minutes. If it takes longer, your codebase is too large for one person; move to phase 2 for subsets.

Phase 2: Deep Audit for High-Risk Components

Authentication and authorization are where AI code fails hardest. The logic is subtle. The AI reads documentation, sees examples, but frequently inverts the check or forgets a layer.

Start with one clear question for each sensitive endpoint: "What stops an unauthenticated user from calling this?" Walk through your API routes. For each one, trace the code path: Does the request check authentication (JWT, session, API key)? If yes, does it verify the token is valid and not expired? Does it then check authorization (is this user allowed to access this resource)? If the user is not allowed, does the endpoint return a 403 Forbidden, or does it return 200 with partial data?

A real example (pseudocode, but representative of actual AI output):

app.post('/api/users/:id/delete', (req, res) => {
  User.findByIdAndDelete(req.params.id);
  res.send({status: 'deleted'});
});

The AI generated this because you asked for "an endpoint to delete users". There is no authentication check. Anyone who knows a user ID can delete any account. This is a severity-critical flaw. The fix is simple, but the AI did not know to include it:

app.post('/api/users/:id/delete', authenticate, (req, res) => {
  if (req.user.id !== req.params.id && !req.user.isAdmin) {
    return res.status(403).send({error: 'Forbidden'});
  }
  User.findByIdAndDelete(req.params.id);
  res.send({status: 'deleted'});
});

The lesson: AI will not invent security constraints it was not told to build. You must review them yourself.

Check these specific patterns in your authentication layer:

Token validation: Does the code verify the JWT signature? Check for verify() calls. If the code just decodes the token without verifying, any attacker can forge a token with {admin: true}.
Expiration: Does it check exp in the token? If not, a stolen token is valid forever.
Rate limiting on auth attempts: Can an attacker brute-force login credentials? Check for rate-limit middleware on /login, /authenticate, or similar endpoints.
Password hashing: If you store passwords, are they hashed with bcrypt or Argon2? If the AI used crypto.createHash('sha1'), you have a problem.
Redirect after login: Does the code redirect to user-supplied URLs after login? If redirect_to comes from the query string without validation, an attacker can steal credentials by redirecting users to a phishing site.

For each API endpoint that touches user data, write a one-sentence description of who is allowed to call it and what data it returns. If you cannot write that sentence, neither could the AI, and the endpoint is probably wrong.

Phase 3: Dependency and Supply Chain Validation

AI code frequently includes packages that do not exist. This is called hallucination. The AI sees the package name format and invents a plausible library that sounds right but has never been published.

Start with your package.json (Node), requirements.txt (Python), or go.mod (Go). For each dependency the AI added, verify it actually exists: Go to npm.js, PyPI, or pkg.go.dev. Search for the package by name. Check the last update date (if it was last updated in 2015, maybe it is abandoned). Check the GitHub repository (if one is listed). Read recent issues (are there known security vulnerabilities?).

Then run your dependency scanner. GitHub's Dependabot is free and integrated into most repositories. Snyk has a free tier that flags CVEs in your dependencies. CVE scanning will flag known vulnerabilities. If your code uses lodash@3.0.0, Dependabot will tell you that version has a prototype-pollution bug (CVE-2018-3721). Your job is to upgrade to a patched version.

One more check: version pinning. If your package.json contains lodash: ^4.17.0, npm will install any version from 4.17.0 up to (but not including) 5.0.0. If a new version is released with a bug, your CI/CD pipeline will pull it automatically. Safer is to pin exact versions in production: lodash: 4.17.21. Use a tool like Dependabot to notify you when updates are available; upgrade intentionally, not automatically.

Phase 4: Live Runtime Check (The URL Scan)

Code review and test suites can miss what an attacker finds in seconds. You need to test your deployed application from outside, as an attacker would.

After you deploy to a staging environment, do this:

Check HTTP headers: Use curl or your browser developer tools to inspect response headers. You should see X-Frame-Options: DENY or SAMEORIGIN (prevents clickjacking), X-Content-Type-Options: nosniff (prevents MIME type misinterpretation), Strict-Transport-Security (forces HTTPS), and Content-Security-Policy (restricts where scripts can load from). If these headers are missing, add them. AI code frequently omits them because they are not required for basic functionality.
Test with invalid tokens: Make an API request with an expired or forged JWT. The endpoint should return 401 Unauthorized. If it returns 200 with data, your auth is broken.
Test with another user's ID: If you have two test accounts, log in as User A and try to access User B's profile. You should get 403 Forbidden. If you get 200 with User B's data, you have a row-level authorization bypass.
Test error messages: Try to trigger an error (access a resource that does not exist, send malformed JSON, etc.). The error message should not leak information (no stack traces, no database error messages, no file paths).

These tests take 15-20 minutes per application. They will catch authorization bypasses that code review and unit tests miss.

Pre-Launch Audit Checklist

Use this checklist before deploying to production. Each item should be verifiable in under 5 minutes.

No secrets in frontend code (no API keys in JavaScript, environment variables instead)
No secrets in git history (git log --all -p | grep -i 'password\|api_key')
Authentication required on all sensitive endpoints (trace code path from request to response)
Authorization enforced server-side (not just client-side hiding of buttons)
Rate limiting on login and API endpoints
Passwords hashed with bcrypt or Argon2 (not MD5, SHA1, or plaintext)
Dependency scan passed (Dependabot green, no high-severity CVEs)
Dependencies pinned to exact versions in production (package.json, not ^ or ~)
HTTP security headers set (X-Frame-Options, CSP, HSTS)
Error messages do not leak stack traces or database details
Token expiration enforced (JWTs checked for exp claim)
Logged-out tokens cannot be replayed (token blacklist or short expiration)
Staging environment mirrors production (same secrets, same versions, same configuration)
Staging passed all four phases above

When to Escalate to Professional Review

Some applications need a professional security audit. The indicators are: you are handling payment information (PCI-DSS compliance required), you store health or medical data (HIPAA compliance required), your users are in the EU and you process personal data (GDPR compliance required), you are raising Series A funding (investors will require a pen test), or your SLA guarantees enterprise customers a certain uptime or data integrity (liability is material).

For these cases, hire a security consultant to do a full penetration test. This will cost 5,000-30,000 dollars and take 1-4 weeks. It is worth it. For MVP applications with non-sensitive data and no compliance requirements, use the Ship GDPR checklist to verify data handling, then the four-phase audit above is sufficient.

Frequently Asked Questions

How to evaluate AI generated code?

Run it. Trace the code path from user input to database query. Ask whether the code checks permissions before returning data. Try to break it by supplying invalid input, expired tokens, or another user's ID. If it survives, it is probably safe.

How to perform an AI audit?

Follow the four phases: rapid grep for secrets, deep review of auth logic, dependency scanning, and live testing. Each phase targets a different failure mode.

How to tell when code is written by AI?

AI code often includes unnecessary comments ("This function does X"), consistent formatting across a large codebase, and verbose error messages. More importantly, it frequently omits security checks, error handling, and edge cases. If code looks complete but is missing authorization logic, it was probably AI-generated.

How to do an AI visibility audit?

Inventory all AI-generated code in your repository (grep for AI comments, review git blame for rapid commits). Then apply the four-phase audit to each file. Prioritize high-risk components (authentication, payment processing, user data access).

What is the best AI tool for audit?

Dependabot for dependency scanning. OWASP ZAP for runtime scanning. Snyk for CVE detection. For code review, there is no replacement for human judgment. No tool catches authorization logic errors as reliably as a human reading the code and asking "who is allowed to call this?"

Should I keep AI-generated code or rewrite it?

If it passes all four phases and matches your architecture, keep it. Rewriting costs time and introduces new bugs. If it fails any phase (missing auth, hallucinated packages, hardcoded secrets), rewrite or heavily modify the problematic sections.

The Bottom Line

AI-generated code works. It is fast and remarkably complete. But it fails in specific ways: missing authorization checks, hardcoded secrets, hallucinated dependencies, and insufficient error handling. Auditing before launch is not optional if you are handling user data or payment information.

The four-phase process takes 2-4 hours for a typical application. It catches 80%+ of critical flaws without hiring an external firm. Deploy with Ship after you pass the audit, and you have infrastructure built to scale.