AWS Bill Doubled: Cost Explorer, Untagged Resources, Egress

AWS bills don't double overnight for no reason — there's a discrete cause. The mistake teams make is starting from compute because that's the obvious cost. The actual culprit is usually hidden in line items most engineers don't review: NAT Gateway data transfer, cross-AZ traffic, abandoned resources, or a runaway loop.

Step 1 — Cost Explorer, group by service

• Identifies which AWS service category grew; this narrows the search dramatically
• Common patterns: "EC2 - Other" jumped means data transfer or NAT Gateway; DynamoDB jumped means RCU/WCU spike; S3 jumped means storage growth or egress; CloudWatch jumped means verbose logs

Step 2 — Drill into the spike

• For data transfer / NAT: group by USAGE TYPE; "NatGateway-Bytes" or "InterAZ-In/Out" or "DataTransfer-Out" tells the pattern
• For DynamoDB: CloudWatch table metrics (ConsumedWriteCapacityUnits) per table; find the table that spiked; correlate with deploys
• For S3: USAGE TYPE filtered to S3 looking for "DataTransfer-Out-Bytes"; S3 access logs reveal who's downloading

Step 3 — Find the root cause

• For NAT Gateway, VPC Flow Logs are the single best tool; Athena query against flow logs grouped by destination IP shows top talkers
• Resolve those IPs — if S3 regional endpoints, the fix is a VPC Gateway Endpoint for S3 (free) which bypasses NAT entirely
• For DynamoDB spikes, application logs during the spike window usually reveal the runaway loop or new feature
• For abandoned resources, AWS Trusted Advisor or Resource Explorer surfaces idle ELBs, unattached EIPs, old snapshots

Step 4 — Fix the immediate cost

• VPC Gateway Endpoints for S3 and DynamoDB are free and eliminate NAT charges for those services
• Set DynamoDB max RCU/WCU on tables to cap runaway costs
• Delete abandoned resources
• For cross-AZ chatter, deploy with topology-aware routing (Istio, Linkerd, ALB target group preference) to keep traffic in-AZ

Step 5 — Prevent recurrence

• Cost Anomaly Detection (free): alerts when daily spend deviates from baseline; catches the next surprise within hours
• AWS Budgets: alert at 50, 80, 100 percent of monthly budget; per-team if you have cost allocation tags
• Tagging policy via Service Control Policy: require cost-center, team, environment tags on every resource; untagged resources have no owner and become abandoned
• Monthly showback to teams — the FinOps culture step that changes behavior; engineers who see their spend optimize their choices

The hidden cost drivers most teams miss

• NAT Gateway: $0.045 per hour per gateway plus $0.045 per GB processed; a busy backend talking to S3, ECR, Docker Hub, npm through NAT pays thousands per month that VPC endpoints would eliminate
• Cross-AZ traffic: $0.01 per GB each way; chatty microservices across 3 AZs add up fast
• CloudWatch Logs: $0.50 per GB ingested; a verbose Spring app with DEBUG logging across 100 pods can spend $1500 per month just on logs
• S3 lifecycle: without it, old objects sit on Standard ($0.023/GB); should be on Standard-IA or Glacier; 100TB bucket pays $2300 vs $400 vs $40 per month
• Aurora I/O charges: Standard charges per I/O; switch to I/O-Optimized when I/O > 25 percent of Aurora bill
• gp3 EBS is 20 percent cheaper than gp2 for the same performance; free downtime-less migration

The cultural answer

• FinOps is engineering culture — if engineers don't see the cost impact of their decisions, they don't optimize
• Showback turns cost into a feedback loop; teams that see their monthly spend make different architectural choices
• Cost Anomaly Detection is the safety net; tagging and SCPs are the prevention; showback is the culture change