Scenario: Leaked AWS Credentials in Public GitHub — Hour-by-Hour Incident Response and Remediation
Fast recall
Quick revision
Core answer
aws:userIdentity.accessKeyId matches, unusual API calls (resource creation, IAM changes, data exfiltration via S3 GetObject, RDS snapshot exports). If suspicious activity: assume breach — quarantine affected resources, snapshot for forensics, escalate to legal/comms. Force-rotate all related secrets (DB passwords, API keys this account could access). Remove the commit from history (BFG Repo-Cleaner, force-push), revoke any forks if possible — but assume the secret is permanently compromised regardless. Notify the contractor, document timeline. Next 7 days: harden + retrospective. Mandate IAM session tokens (no long-lived access keys); enable MFA on all human accounts; deploy AWS IAM Access Analyzer; install Gitleaks pre-commit hooks org-wide; deploy GitHub secret scanning + push protection; rotate all long-lived credentials proactively (assume others may be exposed). Run a blameless postmortem covering detection time, response time, blast radius, and root cause (no pre-commit hook, broad IAM, contractor permissions too wide). Long-term: eliminate long-lived credentials entirely. Move to short-lived AWS STS tokens via OIDC federation (GitHub Actions OIDC for CI; SSO for humans), least-privilege IAM with permissions boundaries, secret manager for app secrets with auto-rotation, mandatory pre-commit + push protection on every repo, and SOC 2 / ISO 27001 audit-ready evidence. Key lesson: defense in depth — pre-commit hook, push protection, scoped IAM, short-lived tokens, anomaly detection — any single layer should have caught this.Panicking; just deleting the commit and assuming it's fine; forgetting to investigate actual usage; no process change after
Rotate first then investigate; assume permanently compromised; eliminate long-lived credentials structurally; OIDC federation; blameless postmortem
Interview-ready guide
Interview answer
I'd run a 4-phase response: contain, investigate, harden, restructure.
First 4 hours: contain, rotate, investigate
Minute 0–5: Contain
- AWS console: aws iam update-access-key --status Inactive — instant kill
- Then aws iam delete-access-key
- Disable IAM user temporarily if broad permissions
- Don't waste time on commit deletion yet — revoke first
Minute 5–15: Rotate adjacent
- What did the key have access to? List policies, attached + inline
- Secrets Manager secrets it could read — rotate all
- RDS master passwords if access
- Coordinate with services that consume these — redeploy with new values
- Assume worst case: anything readable was read
Minute 15–60: Investigate via CloudTrail
- Filter on accessKeyId in leak window
- Look for unfamiliar IPs, unusual resource creation (EC2 mining), data exfil (S3 GetObject, RDS snapshots), IAM changes (persistence), deletions (cover tracks)
- GuardDuty alerts in window
- Athena on CloudTrail logs in S3
Severity tiers
- Tier 1 no usage: relief, continue rotation
- Tier 2 data read: legal/comms call
- Tier 3 resources created or IAM changed: full incident, legal, possibly customer notification
Minute 60–240: Cleanup + comms
- git filter-repo or BFG to remove from history; force-push
- But: GitHub caches forks, scrapers have it — hygiene not security
- Notify contractor empathetically — context, no blame
- Internal Slack: containment confirmed
- External comms only with legal sign-off
Mistakes to avoid
- Deleting commit before revoking
- Believing 5-minute exposure is safe
- Skipping adjacent rotation
- Punishing contractor publicly
- No CloudTrail = no forensics
Days 1–7: Harden + retrospective
Day 1–2: Broader rotation
- IAM Credential Report — all keys older than 90 days
- Other long-lived credentials: DB, third-party APIs, service accounts
- Bulk rotation as project, prioritize by blast radius
Day 2–3: Detection
- GitHub secret scanning org-wide
- GitHub Push Protection enforcement
- Pre-commit hooks Gitleaks org-wide via templates
- AWS GuardDuty all regions
- IAM Access Analyzer
- CloudTrail all regions, immutable S3, SNS alarms
- Security Hub central dashboard
Day 3–4: IAM hardening
- MFA all human IAM users via SCP
- Move humans to AWS Identity Center (SSO)
- IAM Access Analyzer least-privilege right-sizing
- Permission boundaries cap delegated admins
- SCP at org: block expensive regions, require MFA, deny IAM key creation for humans
Day 5–7: Blameless postmortem
- Format: summary, impact, root cause, detection, response, action items, lessons
- Multiple root causes: missing pre-commit, broad IAM, contractor scope
- Blameless: system allowed it, focus systemic gaps
- Action items P0–P2 with owners and dates
Customer / regulator comms
- GDPR 72-hour authority + prompt to individuals if customer data accessed
- Legal drives, engineering provides facts
- Factual, clear, no over-promise
Long-term: eliminate long-lived credentials
CI: OIDC federation
- GitHub Actions OIDC → AWS IAM Role with trust policy on repo/ref
- No access keys in GitHub Secrets — STS token 1 hour
- Same for GitLab, CircleCI, Jenkins
- No key to leak
Humans: SSO + temporary
- AWS Identity Center with SAML/OIDC to Okta or Google Workspace
- aws sso login — 8-hour session
- No long-lived keys in dotfiles
- MFA enforced at IdP
- Permission Sets per role
Apps: IAM roles + secrets manager
- IAM Roles: EC2 instance profile, ECS task role, EKS IRSA, Lambda exec role
- Secrets Manager fetched at startup; never baked
- Auto-rotation for DB passwords (rotation Lambda)
- External service keys: Secrets Manager + manual rotation runbook
Vault for legacy / dynamic
- HashiCorp Vault or Secrets Manager as source
- Authenticate with identity; Vault issues short-lived
- Database dynamic credentials per session
Defense-in-depth layers
- 1Pre-commit Gitleaks
- 2Push Protection
- 3GitHub Secret Scanning
- 4CI scanning safety net
- 5GuardDuty anomaly
- 6CloudTrail + alerts
- 7Permission boundaries — limit blast
- 8SCP guardrails
- 9No long-lived keys structurally
- 10Tabletop exercises quarterly
Compliance
- SOC 2 Type II: IR procedure documented + tested
- ISO 27001 A.16: information security incident management
- NIST 800-61: IR reference
- PCI-DSS 12.10
Cultural
- Security champions in each team
- Quarterly brown-bag training
- Secure paved paths — templates with Secrets Manager wired
- Reward reporting; blameless persistent
Trade-offs
- Speed vs thoroughness in containment: revoke first, restore if needed
- Public vs private comms: legal drives external; transparent internal
- Rotate broadly vs targeted: default broad if uncertain
- Blame vs blameless: blameless persistent
- Push protection friction vs breach cost: friction << breach
Pitfalls
- Deleting commit first
- No CloudTrail
- Ignoring adjacent secrets
- Punishing contractor
- No structural change — same incident in 6 months
- Skipping postmortem
- Customer notification panic or denial
- Assuming git history rewrite makes secret safe
- Long-lived backup credentials elsewhere
- No tabletop exercises
Senior comms
- Leadership: calm, factual, hourly updates, action items at postmortem
- Customers (if needed): legal-driven, factual, scope, what we did, what they should do
- Engineering: blameless framing, celebrate response, drive structural change
Bottom line
- Speed of containment beats elegance
- Assume permanently compromised — don't scrub
- Lesson is structural — eliminate long-lived credentials
- Postmortem most valuable artifact
- Build muscle through tabletops before you need it
Scenario: Leaked AWS Credentials in Public GitHub — Hour-by-Hour Incident Response and Remediation
Direct Answer
First 4 hours: contain, rotate, investigate. Page the security on-call. Within minutes: revoke the IAM access key in AWS console (or via API), rotate any associated credentials, force-expire any active sessions using that key. Trigger CloudTrail + AWS Config investigation: did anyone use this key in the 30-minute window? Look for aws:userIdentity.accessKeyId matches, unusual API calls (resource creation, IAM changes, data exfiltration via S3 GetObject, RDS snapshot exports). If suspicious activity: assume breach — quarantine affected resources, snapshot for forensics, escalate to legal/comms. Force-rotate all related secrets (DB passwords, API keys this account could access). Remove the commit from history (BFG Repo-Cleaner, force-push), revoke any forks if possible — but assume the secret is permanently compromised regardless. Notify the contractor, document timeline. Next 7 days: harden + retrospective. Mandate IAM session tokens (no long-lived access keys); enable MFA on all human accounts; deploy AWS IAM Access Analyzer; install Gitleaks pre-commit hooks org-wide; deploy GitHub secret scanning + push protection; rotate all long-lived credentials proactively (assume others may be exposed). Run a blameless postmortem covering detection time, response time, blast radius, and root cause (no pre-commit hook, broad IAM, contractor permissions too wide). Long-term: eliminate long-lived credentials entirely. Move to short-lived AWS STS tokens via OIDC federation (GitHub Actions OIDC for CI; SSO for humans), least-privilege IAM with permissions boundaries, secret manager for app secrets with auto-rotation, mandatory pre-commit + push protection on every repo, and SOC 2 / ISO 27001 audit-ready evidence. Key lesson: defense in depth — pre-commit hook, push protection, scoped IAM, short-lived tokens, anomaly detection — any single layer should have caught this.
Common Mistakes & Pitfalls
Panicking; just deleting the commit and assuming it's fine; forgetting to investigate actual usage; no process change after
Check your understanding
Answer these without looking back at the article.
- How would your response differ if it was a private repo instead of public?
- What if the contractor had committed the secret 6 months ago and it just surfaced?
- How do you balance push protection friction with developer velocity?
- Walk me through migrating CI from long-lived AWS keys to OIDC federation.
Related interview prep categories
1Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.