Catching Secret Changes That the Audit Log Hides

A Lambda function often keeps its secrets in environment variables: an API key, a database password, a webhook URL. When someone changes one, CloudTrail records the call, who made it, and when. It does not record the values. They are redacted, which is correct for a log, and it also means the log cannot tell you which secret changed. I built an alerter that closes that gap without storing a single secret. It compares hashed snapshots, posts before it saves, and raises an alarm when it goes quiet.

The gap

An UpdateFunctionConfiguration call can change the memory size, the timeout, or a secret. In CloudTrail all three look alike: one event, the environment values hidden. An alert on every such event is noise, and people learn to ignore it. No alert means a changed payment key goes unnoticed until something breaks.

The question I want answered is narrow: which variable names changed, on which function, and who changed them. The values themselves I never want to see.

The pipeline

1. 1. CloudTrail
   

   Records the config change, values redacted

2. 2. EventBridge rule
   

   Matches Lambda config events

3. 3. Alerter
   

   Reads the live config, hashes every value

4. 4. Diff
   

   Compares with the last snapshot of hashes

5. 5. Post, then save
   

   Alert to chat first, new snapshot second

The EventBridge rule matches on the event name by prefix, because Lambda event names carry an API version suffix:

{
  "source": ["aws.lambda"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": [
      { "prefix": "UpdateFunctionConfiguration" },
      { "prefix": "CreateFunction" }
    ]
  }
}

Hashes, not values

The alerter calls GetFunctionConfiguration, which returns the variables in plain text. It hashes each value at once and keeps only the hashes. The snapshot is a map from variable name to hash:

import { createHash } from "node:crypto";
 
type Hashes = Record<string, string>;
 
const hashAll = (vars: Record<string, string> = {}): Hashes =>
  Object.fromEntries(
    Object.entries(vars).map(([name, value]) => [
      name,
      createHash("sha256").update(value).digest("hex"),
    ]),
  );
 
function diff(before: Hashes, after: Hashes) {
  const added = Object.keys(after).filter((k) => !(k in before));
  const removed = Object.keys(before).filter((k) => !(k in after));
  const changed = Object.keys(after).filter(
    (k) => k in before && before[k] !== after[k],
  );
  return { added, removed, changed };
}

A change of memory size gives an empty diff, and the alerter stays silent. A changed secret gives one line: the function, the variable name, and the actor. The alert never contains a value or a hash.

Post first, then save

The order of the last two steps decides what happens on failure:

export async function handler(event: CloudTrailEvent): Promise<void> {
  const fn = event.detail.requestParameters.functionName;
  const config = await lambda.send(
    new GetFunctionConfigurationCommand({ FunctionName: fn }),
  );
  const current = hashAll(config.Environment?.Variables);
  const snapshot = await loadSnapshot(fn); // { hashes, version } or undefined
 
  const d = diff(snapshot?.hashes ?? {}, current);
  if (d.added.length + d.removed.length + d.changed.length === 0) return;
 
  // Throws when the post fails. The snapshot stays old, the invocation
  // is retried, and the same diff is found again.
  await postToChat(formatAlert(fn, d, attribute(event, config), !snapshot));
 
  // Conditional write: fails if another invocation saved a newer snapshot.
  await saveSnapshot(fn, current, snapshot?.version);
}

• Post fails: the snapshot is not written. The invocation fails, it is retried, and the retry finds the same diff. The alert is late, not lost.
• Post succeeds, save fails: the retry posts the same alert again. A duplicate alert is annoying. A missing alert on a changed secret is an incident.
• Two invocations at once: the snapshot write is conditional on the version that was read. The loser does not overwrite the newer snapshot.
• First run for a function: there is no snapshot yet, so the alert is one baseline note, not a list of every variable as added.

The reverse order, save and then post, is simpler to write, and it is wrong. A failed post after a successful save loses the alert for good, because the next run sees no difference.

Who changed it

Two changes to the same function a few seconds apart create a race. The first invocation can read a configuration that already contains the second change. If it names only the first actor, the alert blames the wrong person.

The alerter compares the time of the event with the LastModified time of the configuration it read:

function attribute(event: CloudTrailEvent, config: FunctionConfiguration): string {
  const actor = event.detail.userIdentity.arn;
  const eventTime = Date.parse(event.detail.eventTime);
  const modified = Date.parse(config.LastModified ?? event.detail.eventTime);
  // The config was changed again after this event.
  return modified - eventTime > 5_000 ? `${actor}, or a later change` : actor;
}

When the configuration is newer than the event, the alert says so. The invocation for the second event then finds an empty diff, because the first alert already covered both changes. The hedge sends the reader to the CloudTrail history of that function, instead of giving one confident wrong answer.

When the alerter goes quiet

Silence from a security alerter is ambiguous. It can mean that nothing changed, or that the alerter is broken, its chat webhook was revoked, or its rule was deleted. From the outside these look the same.

So the alerter has a second job. A schedule invokes it in heartbeat mode, and it publishes one metric data point. A CloudWatch alarm fires when the metric is missing, with missing data treated as breaching. The alarm goes to a different channel from the alerts, so one broken webhook cannot hide both. Function errors and a non-empty dead-letter queue raise alarms on that same channel.

Alternatives I rejected

• Alert on every configuration event. Easy, and noisy within a week. It also says nothing about which secret changed.
• Store the values, encrypted, to diff them. It works, and it creates a second store of every secret, with its own keys and its own access list. The alerter would become the most valuable target in the account.
• Poll all functions on a schedule. It finds the change, but not who made it, and only after the next poll.
• Save the snapshot before the post. Covered above: one failed post, and the alert is gone.

What I would do differently

I would use HMAC-SHA256 with a key that only the alerter holds, not plain SHA-256. A plain hash of a short value, such as true or a port number, is easy to guess by brute force. The hashes are not secrets, but they should not leak values either.

I would also test the alerter with a synthetic change on a schedule: change a canary variable on a canary function, and expect the alert. The heartbeat proves that the alerter runs. Only a real change proves that the whole path works, from CloudTrail to the chat channel.

In the long run, secrets belong in a secrets manager and not in environment variables. There, every change is its own audited event on its own resource. The alerter stays useful for the functions that still read secrets from their environment.