Secrets Management: Vault, AWS Secrets Manager, and Kubernetes Patterns

Never check secrets into Git. Use a dedicated secrets manager: HashiCorp Vault (cloud-agnostic), AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.

Why secrets management matters

• Secrets: DB creds, API keys, OAuth client secrets, TLS keys, encryption keys, webhook signing, SSH
• Single leak can compromise company
• Real incidents: GitHub Actions logs 2022, Codecov 2021, Uber 2016, Toyota 2022

Three sacred rules

1. 1
   Never in Git — even private repos have too many eyes
2. 2
   Never in images — layers sticky; baked-in secrets survive cleanup
3. 3
   Never in logs/error messages — one printStackTrace() can exfiltrate

Job of secrets manager

• Central storage
• Access control (who reads what)
• Audit (who read what when)
• Rotation
• Versioning
• Leasing (expire automatically)
• Dynamic secrets (on-demand short-lived)

Storage options

HashiCorp Vault (cloud-agnostic gold standard)

• Static KV secrets
• Dynamic secrets: DB/AWS/SSH on-demand, short TTL
• Transit engine: encryption-as-a-service
• Lease-based renewal/revocation
• Auth: Kubernetes SA, AWS IAM, OIDC, LDAP
• Audit log every read/write

Spring Cloud Vault

• spring.cloud.vault.uri, authentication: KUBERNETES, kubernetes.role
• kv.backend: secret, default-context: my-app
• Reads from secret/my-app/* at startup, exposes as properties
• @RefreshScope for rotation

AWS Secrets Manager

• Native RDS/IAM integration
• Lambda-based rotation (RDS has built-in function)
• Versioning with AWSPREVIOUS/CURRENT/PENDING
• Per-secret KMS key
• Resource policies for cross-account

AWS SSM Parameter Store

• Cheaper, simpler; String + SecureString (KMS)
• No built-in rotation

GCP Secret Manager, Azure Key Vault

• Similar in respective clouds

Kubernetes Secrets

• Native but base64-encoded; stored in etcd
• Should be KMS-encrypted at rest
• Intermediary layer, not source of truth

Secrets Store CSI Driver + External Secrets Operator (modern K8s)

• SecretProviderClass: provider=vault, roleName, objects
• Pod mounts secrets as files in /mnt/secrets
• Never pass through K8s Secret resource
• Updated on rotation via CSI refresh

Rotation

Static rotation schedules

• API keys: 30-90 days
• Signing keys: 90-365 days; key rollover (new + old valid during rotation)
• Encryption keys: annual; keep old for decryption

Rotation pattern

1. 1
   Generate new in vault
2. 2
   Add alongside old (two valid)
3. 3
   Deploy to read new (@RefreshScope re-inits beans)
4. 4
   Wait for all instances
5. 5
   Revoke old
6. 6
   Audit no traffic on old

Dynamic secrets (best practice)

• Vault generates DB user on demand: role=my-app → v-my-app-abc123, TTL=1h
• App requests at startup; auto-renews
• Pod dies = creds expire naturally
• Breach = rotate = revoke all issued

IAM-based DB auth (no password at all)

• AWS RDS IAM auth: DB accepts IAM tokens (TTL 15 min); no stored password
• GCP Cloud SQL IAM, Azure AD SQL
• Spring: generateAuthToken before each connection
• Pros: nothing to manage; Cons: AWS-specific; token refresh complexity in pool

mTLS for service-to-service

• Service mesh (Istio, Linkerd) issues/rotates short-lived certs
• No long-lived secrets for internal comms

Secret scoping

• Per-env (dev/staging/prod)
• Per-app IAM roles (least privilege)
• Prod requires elevated approval
• Break-glass procedures logged/reviewed

Leak detection and response

Prevention in CI

• gitleaks-action
• trufflehog filesystem --only-verified
• pre-commit hooks
• GitHub push protection (enterprise)

If a secret leaks

1. 1
   Rotate immediately (assume compromised)
2. 2
   Revoke old in vault
3. 3
   Audit blast radius: scope, resources, accesses (CloudTrail/Vault audit/DB audit)
4. 4
   Check unauthorized: anomalous queries, exfiltration, new users
5. 5
   Rewrite git history (git filter-repo, BFG) - contentious; the secret is copied; rotation + revocation is what matters
6. 6
   IR: timeline, notify per regulations
7. 7
   Post-mortem: how/prevention

Build-time vs runtime

• Build-time (registry creds, signing): short-lived, CI-issued
• Runtime (DB, API): pulled at container startup from vault
• Never bake runtime into image (layer cache + registry exposure)

Logging and redaction

• Override DTO toString() to redact
• Lombok @ToString.Exclude on sensitive fields

Environment-specific

• Local dev: personal vault tokens, short-lived
• CI: OIDC-issued short tokens; no stored GitHub Secrets beyond boot
• Prod: IAM roles / SA-based; no static creds in pipeline

Choice of secret store | Store | Strengths | When | | Vault | Multi-cloud, dynamic, transit | Complex, multi-cloud | | AWS SM | Native RDS integration | AWS-only | | SSM PS | Cheap, simple | Config + some secrets, small | | GCP/Azure | Native integration | Single-cloud | | K8s Secrets + KMS | K8s-native | Small cluster; cache layer |

Common anti-patterns

1. 1
   Secrets in application.yml committed to Git
2. 2
   Env vars in /proc/pid/environ visible to host users
3. 3
   K8s Secrets without KMS encryption at rest
4. 4
   Single secret for many apps (big blast radius)
5. 5
   No expiry/TTL
6. 6
   Rotation requiring restart
7. 7
   CI secrets in logs (masking imperfect; use OIDC)
8. 8
   Hardcoded secret names (rotation = rebuild)
9. 9
   No audit trail
10. 10
    Dev secrets = prod secrets

Ephemeral vs persistent

• Persistent (90-day password): easy, high blast radius
• Ephemeral (1h Vault token): complex, low blast radius
• Industry moving toward ephemeral (Vault, IAM auth, mTLS)

Human factor

• Devs paste secrets in Slack when rushing
• Make right thing easy: CLI/UI fetching per-context
• Blameless leak response; focus on systemic fix

Cost

• Vault cluster: infra + operational
• AWS SM: $0.40/secret/month + API
• SSM PS: free standard, $ advanced
• Per-secret cost matters at thousands

Compliance

• PCI-DSS: card-data secrets need strict controls, rotation, audit
• SOC 2: secrets management controls scrutinized
• GDPR: encryption of personal data; key management
• HIPAA: access audited; separation of duties

Strategic take

• Boring infrastructure work that matters a lot
• Single leaked AWS key can end a company

1. 1
   Pick one store, invest, make default
2. 2
   Automate: CI, CD, rotation
3. 3
   Detect leaks: scanning + push protection + monitoring
4. 4
   Practice IR: simulate leak, time rotation
5. 5
   Build habits: code reviews check secrets; .gitignore local
6. 6
   Move to dynamic/IAM auth — eliminates whole classes of risk

Interview standouts

• Know the options
• Rotation and blast-radius math
• Thoughtful dev experience
• Response plan for inevitable leak

Bottom line

• Never in Git/images/logs
• Central vault with audit + rotation
• Dynamic secrets > static; IAM auth > password
• Detect, respond, learn from leaks
• Automation + culture = durable secret hygiene