How we cut a client's AWS bill from $38k to $22k a month: a line-by-line teardown

A Series B logistics company came to us with an AWS bill that had grown from $19k to $38k a month in eighteen months while traffic grew by roughly 40%. Nobody had done anything wrong, exactly. Teams had shipped features, added environments, turned on logging and moved on. The bill just accreted.

Eight weeks later the same workload cost $22k a month, with no loss of availability and no change to the product. This post walks through the bill line by line: where the money was going, what we changed, what each change saved, and the two changes we tried and rolled back. If you run a platform on AWS in the $10k to $100k a month range, most of this will apply to you.

Start with the bill, not the architecture diagram

Architecture diagrams show what the team thinks is running. Cost and Usage data shows what AWS is charging for. The two rarely match, so we always start with the second.

The first pass is a month of spend grouped by usage type. Usage type is more useful than service because it separates, for example, NAT Gateway hours from NAT Gateway bytes, or EBS gp2 from EBS snapshots.

aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-06-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' \
  --output text | sort -k2 -nr | head -40

For anything deeper, enable the CUR 2.0 export (Data Exports in the Billing console) to S3 and query it with Athena. It takes a day to populate, so turn it on during the first call, not the first week.

Here is the summary we took into the kickoff, and where each line ended up.

Compute: $14,200 down to $8,600

Compute is usually the biggest line, and it was here. The client ran an EKS cluster with a managed node group of m5.2xlarge instances, plus a handful of standalone EC2 instances for batch jobs and a legacy admin tool.

Rightsizing before committing

The cardinal rule: never buy Savings Plans for capacity you are about to remove. We spent the first three weeks shrinking, and only then committed.

Cluster-wide, requested CPU was 3.1 times actual p95 usage. Most deployments had copy-pasted requests of 1 CPU and 2 GiB regardless of what they did. We pulled a week of container usage from Prometheus, set requests to roughly p95 plus 20% headroom, and left memory limits equal to requests to avoid noisy-neighbour OOMs.

quantile_over_time(0.95,
  sum by (namespace, pod) (
    rate(container_cpu_usage_seconds_total{container!=""}[5m])
  )[7d:5m]
)

That alone let the node group drop from 22 to 13 nodes at the same traffic.

Graviton

Every service was either Node.js, Go or Python, and every base image already published an arm64 variant. We added linux/arm64 to the Docker build matrix, ran both architectures in staging for a week, then moved production node groups from m5.2xlarge to m7g.2xlarge. Graviton instances are roughly 15 to 20% cheaper per hour for the same vCPU and memory, and for these workloads latency was flat or slightly better.

The one blocker was a Python service that depended on a wheel with no arm64 build. We pinned that deployment to an x86 node pool with a node selector rather than holding up the whole migration.

Commit last

With the fleet shrunk and on Graviton, we bought a one-year, no-upfront Compute Savings Plan covering about 70% of the new steady-state baseline. Compute Savings Plans apply across instance families and to Fargate, which matters if you later change instance types again. We deliberately left 30% on demand; the client expected to move more batch work to Spot and did not want to be locked in.

NAT Gateway: $3,900 down to $900

This is the line item that surprises CTOs most. NAT Gateway charges an hourly fee per gateway plus a per-GB data processing charge ($0.045 per GB in us-east-1), on top of normal data transfer. Every byte a private subnet sends to the internet, or to a public AWS endpoint, is metered.

Using VPC Flow Logs in Athena, we found three culprits:

1. Nodes pulling container images from ECR through the NAT gateway, several hundred GB a month because of aggressive image rollouts.
2. Services writing to and reading from S3 through the NAT gateway.
3. A metrics agent shipping to a third-party SaaS every ten seconds with no batching.

The S3 fix is a gateway endpoint, which is free. ECR needs interface endpoints for the API and Docker registry, plus the S3 gateway endpoint because image layers are served from S3.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = module.vpc.vpc_id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = module.vpc.private_route_table_ids
}

resource "aws_vpc_endpoint" "ecr" {
  for_each            = toset(["ecr.api", "ecr.dkr"])
  vpc_id              = module.vpc.vpc_id
  service_name        = "com.amazonaws.${var.region}.${each.key}"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = module.vpc.private_subnets
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}

Interface endpoints are not free (an hourly charge per AZ plus a smaller per-GB fee), so only add them where the traffic justifies it. For this client the ECR endpoints paid for themselves within the first week. The metrics agent got a batching config and a longer flush interval.

RDS: $6,800 down to $4,500

Production ran a db.r6g.2xlarge Multi-AZ PostgreSQL instance, which was sized correctly. The waste was elsewhere:

• Staging and QA each ran Multi-AZ db.r6g.xlarge instances. Non-production rarely needs Multi-AZ. We moved both to single-AZ db.t4g.large and scheduled QA to stop outside working hours.
• All instances used gp2 storage. gp3 is cheaper per GB and decouples IOPS from size, so we no longer needed to over-provision storage to get IOPS.
• After a month of stable sizing, we bought a one-year Reserved Instance for the production instance.

Stopped RDS instances restart automatically after seven days, so the schedule needs to stop them again. We used a small EventBridge Scheduler rule calling the rds:StopDBInstance API rather than a Lambda.

EBS and snapshots: $3,100 down to $1,700

Three simple fixes. First, convert gp2 volumes to gp3. The change is online and gp3 is around 20% cheaper per GB with a 3,000 IOPS baseline regardless of size.

# List gp2 volumes, then convert each one in place
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].VolumeId' --output text |
  tr '\t' '\n' |
  while read -r vol; do
    aws ec2 modify-volume --volume-id "$vol" --volume-type gp3
  done

Second, delete orphaned volumes. We found 4.2 TB of volumes in the available state, mostly left behind by deleted PersistentVolumeClaims with a Retain reclaim policy. We snapshotted anything that looked meaningful, then deleted.

Third, snapshots. There were daily snapshots going back three years with no expiry. We replaced the ad-hoc scripts with an AWS Backup plan: daily snapshots kept for 14 days, monthly kept for 12 months, moved to cold storage after 90 days where supported.

CloudWatch: $2,700 down to $1,100

CloudWatch Logs charges for ingestion (about $0.50 per GB in us-east-1) and for storage. Most of this client's cost was ingestion, and most of that was debug-level logging left on in production by two services, plus the EKS control plane with every log type enabled.

We made three changes:

• Set log level to info in production for all services, enforced by a Helm values default.
• Reduced EKS control plane logging to audit and authenticator, which is what the security team actually reviewed.
• Set retention on every log group. The default is never expire.

aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' --output text |
  tr '\t' '\n' |
  xargs -I{} aws logs put-retention-policy \
    --log-group-name {} --retention-in-days 30

We also found a custom metric emitted with a user_id dimension, which created tens of thousands of distinct metrics at $0.30 each per month. Moving that to a log-based query removed about $400 a month on its own.

Data transfer, S3, cache and the long tail

Cross-AZ traffic. Traffic between availability zones costs $0.01 per GB in each direction. The heaviest service called an internal pricing service several thousand times a second across zones. Enabling Topology Aware Routing on that Kubernetes Service (service.kubernetes.io/topology-mode: Auto) kept most calls in-zone. We did not chase the rest; the remaining cross-AZ traffic is the price of running across three zones.

S3. Lifecycle rules to expire incomplete multipart uploads after 7 days, move shipment-label PDFs to Intelligent-Tiering, and expire temporary export files after 30 days.

ElastiCache. Memory usage peaked at 35% of the node. We moved from cache.r6g.xlarge to cache.r7g.large.

Public IPv4. Since 2024 AWS charges for every public IPv4 address, attached or not. We found 31, including Elastic IPs on stopped instances and public IPs on nodes that sat behind a load balancer anyway. We also removed four idle load balancers left over from old preview environments.

What we tried and rolled back

Not every change stuck.

Spot for the main API node pool. We ran a mixed Spot and on-demand node group for two weeks. Interruptions were handled correctly, but the API had a slow startup (about 70 seconds warming a cache), and interruption bursts pushed p99 latency outside the SLO. We kept Spot for batch and CI runners only.

Single NAT gateway. Running one NAT gateway instead of one per AZ saves the hourly charge but creates a single-zone dependency and adds cross-AZ transfer. With the endpoint work done, the saving was about $70 a month. Not worth it.

What we'd do differently

• Turn on CUR 2.0 and cost allocation tags on day one. We lost a week waiting for data that could have been accumulating.
• Tag by owning team from the start. Half the cleanup time went into finding out who owned a resource before deleting it.
• Put a monthly budget per account with AWS Budgets alerts at 80% and 100% of forecast. The client's bill doubled without anyone noticing because no alert existed.
• Enforce log retention and gp3 in Terraform modules, so the savings do not erode as new services are added.

Checklist

1. Group one month of spend by usage type and list the top 20 lines.
2. Rightsize requests and instances before buying any commitment.
3. Test arm64 images and move to Graviton where builds allow.
4. Add S3 gateway endpoints everywhere; add interface endpoints where traffic justifies them.
5. Convert gp2 to gp3 for both EBS and RDS.
6. Delete unattached volumes, idle load balancers and unused public IPs.
7. Set retention on every CloudWatch log group and turn debug logging off in production.
8. Downsize and schedule non-production databases.
9. Buy Savings Plans and Reserved Instances last, for about 70% of the new baseline.
10. Add budgets and alerts so the bill cannot double quietly again.

The lesson from this engagement is that there was no single big win. The largest line saved $5,600 a month; the rest came from a dozen changes worth a few hundred to a few thousand each. That is typical, and it is why a structured pass through the bill beats hunting for one clever fix.