How to Handle Secrets Properly in Production

Secrets leak when they're scattered across.env files, logs, and container memory. The good news: with proper handling, you contain the blast radius to minutes instead of days. This tutorial walks you through each layer--local dev, CI/CD, production--and shows you exactly what breaks when you get it wrong.

Why Secrets Leak (And Why It Matters More Than You Think)

Real breaches happen to working teams. Heroku exposed 400 customer databases via logs. Twilio lost customer auth tokens to SMS interception. SolarWinds shipped signing keys in plain text. Not hypotheticals--production failures at scale.

The cost sticks. A leaked database credential costs hours to audit, days to rebuild trust with customers, weeks if compliance gets involved. A signing key in the wild means every token your app issued is compromised. Recovery isn't about perfection--it's about containment.

This isn't vendor fear-mongering. Small teams leak secrets constantly because they're shipping fast, not because they're careless. A developer commits a.env file by mistake. A CI/CD log prints an API key mid-build. A container process crashes and dumps memory. All fixable. All preventable. All happened to you somewhere.

The goal is acceptable risk. You'll never stop secrets from existing. You can stop them from ending up in your git history, your Slack logs, or your production memory.

What "Secrets" Really Means

Secrets are values your app needs to run that you cannot publish. Start there.

Your app probably needs: database credentials (user, password, connection string), third-party API keys (Stripe, Twilio, SendGrid), signing keys (JWT secrets, OAuth secrets, session secrets), SSH keys (for deployment, server access), mTLS certificates (if you're inter-service communicating), and OAuth tokens (for outbound API calls on behalf of your app).

Which ones does your app actually need? Write them down. Audit where they live today. For each one, ask: how often do I need to rotate it? If a junior dev left, would they still have access?

Not all secrets carry the same risk. A Stripe public key is not a secret--it's public. A Stripe private key is. API keys for external services rank high: they directly access customer data. Database credentials rank critical: they're a skeleton key to your entire data model. SSH keys are asymmetric: people tend to share them.

Rate your secrets. High-risk gets formal rotation and audit logging. Medium-risk gets periodic rotation. Low-risk still stays out of git but might live in a shared.env template for onboarding.

The Failure Mode Cascade: Local Dev, CI/CD, and Production

Every layer has its own hazard.

Local dev: A.env file makes development easy. Load it once, run the app, done. But.env files belong in.gitignore always. If you check one in, it's in your git history forever--extractable from any clone anywhere. That's not paranoia, that's how git works. The fix:.env.example with placeholder values, and a setup script that tells new devs how to fill it.

CI/CD: Your pipeline needs secrets to deploy--database password to run migrations, deployment key to push to production, API key to deploy services. These get injected as environment variables, which is correct. The hazard: build logs. If your pipeline logs the environment, secrets print for anyone with build history access. And "anyone" often includes your Slack #deploy channel if logs auto-post there. Check your CI/CD config: what gets logged? If you see "BUILD_PASSED: STRIPE_KEY=sk_live_..." in your Slack feed, you have a problem right now.

Production: Your running processes need the secrets. The container is isolated, yes--sort of. But if your app crashes, your container exits with memory intact. A process snapshot is dumpable. If another container runs on the same host, and the host isn't perfectly isolated, memory can leak. Not likely, but possible. Plus there's the audit angle: who accessed which secret, and when? Environment variables have no audit trail.

Each layer compounds. A secret that's only in production is recoverable. A secret in git history plus production plus deploy logs plus local dev laptop is a nightmare. You're not trying for zero risk. You're trying for layered defense: lose one layer, still protected.

Option A: Environment Variables (Lean, Works Everywhere)

Environment variables are the simplest path. Your platform injects them at runtime, your app reads them on startup, done.

When it works: You have fewer than 10 secrets. Your team is small (under 5). You don't need audit logs. Your secrets don't contain newlines or special characters. You deploy rarely enough that rotation doesn't add overhead. All three of dev, staging, production use the same inject mechanism.

When it breaks: You have more than 20 secrets per environment. Team size exceeds 10. You need to rotate a secret without redeploying. You need audit logs of who accessed what. Secrets contain newlines (OpenSSH keys, certificates). Multi-line secrets break shell parsing.

Code is straightforward. Node.js: const stripeKey = process.env.STRIPE_SECRET_KEY; if (!stripeKey) { throw new Error('Missing required secret'); }. Python: import os; stripe_key = os.getenv('STRIPE_SECRET_KEY'); if not stripe_key: raise ValueError('Missing required secret'). Go: stripeKey := os.Getenv("STRIPE_SECRET_KEY"); if stripeKey == "" { log.Fatal("Missing required secret") }.

For a concrete example of how platforms handle this, Ship's app deployment process shows environment injection in action across different stacks.

The tradeoff: simple but limited. Works great until your team grows or your secrets count does. At that point, you'll spend a week building secret validation, rotation scripting, and audit logging--stuff a managed tool handles out of the box.

Option B: Secrets Manager (Vault, Doppler, AWS/Azure/GCP Native)

A secrets manager solves the two hard problems: audit trails and rotation.

HashiCorp Vault is the open-source standard (36.1K GitHub stars). It stores secrets encrypted, serves them over API, logs every access, and handles rotation automatically. AWS Secrets Manager, Azure Key Vault, and Google Secret Manager do the same thing, locked into their platforms. Doppler is a third-party SaaS offering--simpler UX, no self-hosting, vendor lock-in.

What does your platform handle for you? If you deploy to Ship via GitHub, your platform manages deployment secrets for you--the tokens needed to push code, authenticate with registries, and access your infrastructure. You still manage application secrets: your app's database password, API keys, signing keys.

The setup flow: Your app starts. It calls the secrets manager (locally at startup, or via sidecar). The manager returns decrypted values. Your app loads them into memory. Done.

Rotation without redeployment: Change a secret in the manager. Set an expiration on the old one. Your running app polls the manager (or a cached version refreshes). New instances start with the new value. Old instances with old values still work until they restart naturally. That's the win: no panic redeploy at 2am.

The tradeoff: added latency on startup (milliseconds, but measurable), added dependency (if the secrets manager is down, your app can't start), and vendor lock-in (switching from Vault to Doppler to AWS requires rewriting secret-fetch code).

Option C: Encrypted in Git (sops, Sealed Secrets for Kubernetes)

Some teams encrypt secrets before committing. This works, but with caveats.

SOPS, maintained by Mozilla and the CNCF, has 22.8K GitHub stars. Sealed Secrets from Bitnami has 9.2K stars. The flow: developer commits an encrypted file. CI/CD decrypts it with a key stored elsewhere. App runs with decrypted values.

When this makes sense: You use GitOps (everything in git is source of truth). Your team is small (under 5). You deploy to Kubernetes. You own your infrastructure and can store decryption keys safely.

When it doesn't: Secrets rotate frequently (every rotation is a commit). Team size exceeds 5 (coordination overhead). Compliance requires audit logs (git commits don't log access). You need to revoke a secret immediately (git is permanent; you'd need to rewrite history, catastrophic).

The trap: encryption in git looks like a solution, but it's a convenience. You've removed the secrets from plain text, but you haven't solved rotation, audit, or emergency revocation. It's better than plain-text git, but worse than a real manager.

Your Secret Just Leaked: 30-Minute Incident Response

It's 3pm Friday. A developer accidentally pushed a prod database password to public GitHub. Here's what you do.

Step 1 (5 minutes): Rotate the secret immediately. Change the database password. Restart the app. Every connection with the old password is severed. You've disconnected the attacker.

Step 2 (10 minutes): Audit access logs. Did anyone connect to the database with the old password after it was pushed? Most databases log connection attempts. Check. If yes, assume a breach.

Step 3 (15 minutes): Notify your customer base. "We discovered an exposure of a database credential. It was rotated at [time]. We found no unauthorized access. Here's what we did." You don't need to say who had access or for how long--just be honest and fast.

Step 4 (20 minutes): Revoke other copies. Check if the secret was in CI/CD logs, Slack history, developer laptops, backups. It can't all be removed instantly, but log every place. Schedule removal.

Step 5 (ongoing): Monitor. Watch the database for unusual access patterns. Watch your logs for logins from unfamiliar IPs. This isn't paranoia--it's due diligence.

What not to do: Don't wait to find out how many people have access (it's more than you think). Don't assume it was only visible for 5 minutes (GitHub's API cached it, copies exist). Don't rotate the secret without restarting the app (old connections survive). Don't assume no attacker found it yet (if it's public, an attacker will find it).

The whole process should take 30 minutes. Faster if you have the tools in place.

What Ship (And Platforms Like It) Handle For You

Ship removes a layer of secrets management. The platform manages the secrets needed to deploy your code--registry credentials, deployment keys, service tokens. You don't touch those.

You still manage your application secrets. Your app's database password. Your Stripe key. Your signing keys. Those live in your app config, not in Ship's deployment layer.

For details on how Ship's architecture secures these systems, check Ship's GDPR and compliance documentation. You'll see encryption at rest, audit logging, and role-based access control built in.

The benefit: one less system to secure. One less rotation schedule to maintain. The tradeoff: you're trading control for simplicity. If you want to manage every secret across every layer, managed hosting isn't the right fit.

For a 20-person company without a dedicated ops person, that tradeoff usually wins. Operations overhead is expensive. Ship's secrets handling is correct-by-default: environment injection, audit logging, rotation support. You configure once, ship once, sleep at night.

Frequently Asked Questions

What is the difference between environment variables and a secrets manager?

Environment variables are injected by your platform at runtime. A secrets manager is a separate service that your app queries for secrets. Environment variables are simpler for small apps. A secrets manager adds features: audit logging, rotation, access control. Most teams start with environment variables and graduate to a manager when they grow.

Should I store secrets in.env files?

.env files are fine for local development. They are not fine for production. Use them in development, add them to.gitignore, and never commit them. For production, use environment injection (platform-provided) or a secrets manager.

Can I commit secrets to Git if they're encrypted?

You can, but it's not best practice. Encryption in git (sops, Sealed Secrets) removes plain-text risk, but it doesn't solve rotation, audit logging, or emergency revocation. It's a compromise tool for GitOps workflows. If you don't use GitOps, a secrets manager is simpler.

What's the difference between encoding and encrypting secrets?

Encoding (base64, hex) is reversible without a key. Anyone can decode it. Encoding is not security. Encrypting uses a key and is irreversible without it. Always encrypt secrets, never just encode.

How often should I rotate secrets?

High-risk secrets (database passwords, signing keys, API keys to production data) every 90 days. Medium-risk every 6 months. Low-risk annually or when team members leave. Faster rotation is better, but automation is critical. Manual rotation at scale is a mistake.

What happens if a secret leaks--what do I do immediately?

Rotate it within 5 minutes. Check logs for unauthorized access within 15 minutes. Notify stakeholders within 30 minutes. Audit access patterns for a week afterward. Don't panic and break your app in the process.

Should I use my cloud provider's secrets manager (AWS Secrets Manager) or a third party (Vault)?

AWS, Azure, and GCP have native managers that work well if you're all-in on that platform. Vault is cloud-agnostic and standard across the industry. Doppler is SaaS and simple to use. For a small team, a native manager is often the right choice--less to deploy, same security. For multi-cloud or on-premise, Vault. For simplicity, Doppler.

How do I pass secrets to a Docker container safely?

Never pass them as build args (they end up in image layers). Never hardcode them in the Dockerfile. Inject them at runtime as environment variables from your orchestrator (Docker Compose, Kubernetes, Ship). Or use a secrets manager that your app queries on startup.

What's the least-privilege principle for secrets access?

Every person and system gets only the secrets they need to do their job. A frontend developer doesn't need the prod database password. A CI/CD pipeline doesn't need customer API keys. An intern doesn't need signing keys. Secrets managers with RBAC (role-based access control) enforce this. Environment variables don't.

The Bottom Line

Secrets handling isn't magic. It's a series of decisions: inject via environment variables or query a manager? Encrypt in git or store externally? Rotate on a schedule or on-demand? There's no one answer--it depends on your team size, infrastructure, and risk tolerance.

Start with environment variables if you have fewer than 10 secrets and fewer than 5 team members. Graduate to a secrets manager (Vault, Doppler, AWS Secrets Manager) when you hit 20+ secrets or need audit logs. Use encrypted git (sops, Sealed Secrets) if you're deep in Kubernetes and GitOps. If secrets management sounds like overhead, Ship handles it--check DIY alternatives using self-hosted platforms if you want maximum control.