The Day AWS Said We Owed $23,067: A Step-by-Step Cloud Kill Switch

Darrell Henry · Facets Novum LLC · July 17, 2026 · 12 min read

On July 17, 2026, an AWS budget alert told us our account had run up $23,067.41. Our normal month is about two dollars. Wired later reported that the same billing glitch showed some customers phantom charges in the billions. Here's what that morning looked like from inside a small healthcare software company, and the exact two-layer kill switch we built the same afternoon so a real runaway can never surprise us. You can build it in about 30 minutes, for free.

What happened, hour by hour

The email looked exactly like every real AWS Budgets alert, because it was one. It said a one-dollar tripwire budget had been exceeded, with actual charges of $23,067.41, nearly all attributed to Amazon SES, our email service.

Here's the thing: our SES account is quota-capped at 50,000 sends per day, and we'd sent about 158 emails in the previous two weeks. Even maxed out around the clock, the math cannot produce a five-figure bill. When a bill contradicts physics, question the bill.

We checked service-level metrics across every AWS region: sending volumes normal everywhere. We checked for unauthorized IAM users: none. We deactivated our access keys anyway, verified the root account had MFA and no keys, and confirmed no unusual regions were enabled. Twenty minutes of caution.

Then the AWS Health Dashboard posted the answer: "Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data." A display bug in AWS's own billing system. No breach. No real charge. The console self-corrected to $0.43 later that morning.

We could have exhaled and moved on. Instead we used the adrenaline to build the guardrails we'd been putting off, and tested them live before the day ended.

The first-hour checklist

If a scary billing alert lands in your inbox, in order:

1. Don't click anything in the email. Not because this one is fake (ours was genuine), but because you cannot reliably tell a real AWS alert from a good phish by eye. Type console.aws.amazon.com yourself.
2. Check the AWS Health Dashboard first. If AWS has a banner up about billing data, you may be looking at their bug, not your breach.
3. Verify against service-level metrics, not the bill. What did your services actually do? For SES: sending statistics and quotas per region. For EC2: running instances per region. The bill is an opinion; the metrics are the facts.
4. Sanity-check the math. Know your caps. A quota-limited service has a maximum possible bill. If the charge exceeds it, something other than usage is wrong.
5. Lock down cheaply anyway. Deactivating IAM access keys is instant and reversible. Check IAM for users you don't recognize. Verify root has MFA and no access keys. Scan for exotic regions you never use.
6. Only then open a support case, with your evidence attached.

Layer 1: a $200 budget action that freezes spending

AWS deliberately has no single "turn everything off at $X" switch. But Budget Actions get close: when actual spend crosses a threshold, AWS can attach a deny-all-spending IAM policy to your application users, with one click of approval from you.

Create the deny policy

1. IAM → Policies → Create policy → JSON tab. Paste (trim the actions to the services you actually use):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": [
      "ses:SendEmail", "ses:SendRawEmail", "ses:SendBulkEmail",
      "ses:SendTemplatedEmail",
      "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream",
      "ec2:RunInstances", "ec2:StartInstances",
      "sagemaker:CreateEndpoint", "sagemaker:CreateTrainingJob",
      "sagemaker:CreateNotebookInstance"
    ],
    "Resource": "*"
  }]
}

2. Name it KillSwitch-DenySpend and create it. It does nothing until a budget attaches it.

Create the budget and attach the action

1. Billing and Cost Management → Budgets → Create budget → choose Customize (advanced), not the simplified templates. The templates cannot attach actions.
2. Cost budget → Monthly, Fixed, amount 200 (pick your own ceiling; ours is 100x a normal month). Name it KillSwitch-200.
3. Add two alerts: 80% of forecasted (early warning) and 100% of actual (the trigger). Email both to a monitored address.
4. On the 100%-actual alert, Attach action → action type IAM policy → select KillSwitch-DenySpend.
5. Targets: your application IAM users only. Never include your admin user, or you lose the ability to undo the freeze.
6. You'll need an IAM role for Budgets to act through. Create one: IAM → Roles → Create role → AWS service → Budgets → attach the managed policy AWSBudgetsActionsWithAWSResourceControlAccess. Then select it in the budget wizard.
7. Execution: "No, I will approve" (require approval). Save, review, create.

Result: at your ceiling, you get an email; one click applies the deny policy and your application users can't spend another dollar. To lift it later, detach the policy from those users in IAM.

Layer 2: an automatic email-sending pause

The budget layer reacts in hours (billing data lags). If your worry is a compromised key blasting spam and torching your sender reputation, you want minutes. This layer watches the real send count and pauses SES automatically: CloudWatch alarm → SNS topic → a four-line Lambda.

Everything below happens in your SES region.

The Lambda that pauses sending

1. Lambda → Create function → author from scratch, name ses-pause-on-spike, runtime Python 3.12 or newer.
2. Replace the default code (set your own region) and deploy:

import boto3
def lambda_handler(event, context):
    region = "us-east-2"   # your SES region
    try:
        boto3.client("ses", region_name=region) \
             .update_account_sending_enabled(Enabled=False)
    except Exception:
        # Newer SES v2 tooling: same result via the v2 API
        boto3.client("sesv2", region_name=region) \
             .put_account_sending_attributes(SendingEnabled=False)
    return {"status": "SES sending paused"}

3. Give its execution role permission to do that one thing. Configuration → Permissions → click the role → Add permissions → Create inline policy → JSON:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ses:UpdateAccountSendingEnabled",
      "ses:PutAccountSendingAttributes"
    ],
    "Resource": "*"
  }]
}

Note both actions live in the ses: namespace, even the v2 one. That's not a typo; see gotcha #2 below.

The SNS topic that connects everything

1. SNS → Topics → Create topic → Standard → name ses-killswitch.
2. Create two subscriptions on it: protocol AWS Lambda → your ses-pause-on-spike function, and protocol Email → your monitored address. Confirm the email subscription from your inbox.

The alarm that pulls the trigger

1. CloudWatch → Alarms → Create alarm → Select metric → SES → Account Metrics → Send.
2. Statistic Sum, period 1 hour, condition Greater than 1000. Scale the threshold to your reality: we send a few hundred emails in a normal month, so 1,000 in one hour only happens if something is very wrong.
3. In alarm → send notification to ses-killswitch. Name it ses-send-spike, create.

Now a sending spike pauses all SES sending within minutes and emails you at the same moment, keyed off real send counts that a billing bug cannot fake.

Test it live (we did)

A kill switch you've never fired is a hope, not a control. The whole chain can be tested in two minutes:

1. SNS → your ses-killswitch topic → Publish message. Subject killswitch test, any body.
2. Verify SES paused: the SES account dashboard will show sending disabled (or check get_account via the CLI: SENDING ENABLED: False). You'll also get the test email.
3. Re-enable: open CloudShell (the terminal icon in the console) and run:

aws ses update-account-sending-enabled --enabled --region us-east-2
# or, on SES v2 tooling:
aws sesv2 put-account-sending-attributes --sending-enabled --region us-east-2

Ours went down and came back in under three minutes, and we've watched it work. That's the difference between a runbook and a rumor.

Recovery: what to do when a guard fires

Five gotchas that will save you an afternoon

1. A scary AWS billing number is not real until verified twice: against the AWS Health Dashboard, and against service-level metrics. The console estimate itself can be wrong. That's also why destructive budget actions should require approval.
2. sesv2: is not an IAM namespace. Both SES API versions authorize under ses:. A policy granting sesv2:PutAccountSendingAttributes validates as "unrecognized service" and grants nothing. The action you want is ses:UpdateAccountSendingEnabled.
3. CloudWatch alarms cannot invoke Lambda directly. The bridge is SNS: alarm → topic → Lambda subscription. The email subscription on the same topic is free notification.
4. The simplified budget templates can't attach actions. Use "Customize (advanced)" or you'll build an alert that emails you while the money burns.
5. A genuine AWS email can look like phishing. Real AWS budget alerts link through domains like aws-cost-management-alerts.com, which looks fake but is registered to Amazon Technologies, Inc. (verifiable via WHOIS: MarkMonitor registrar, AWS nameservers). Don't trust your eye in either direction. Navigate to the console directly, every time, and see AWS's guidance on suspicious emails if unsure.

Total cost of everything above: effectively zero. Budgets, budget actions, ten CloudWatch alarms, SNS, and this much Lambda all sit inside AWS's free allowances. The only real spend was one morning of adrenaline, and AWS covered that part.