claude-leak-bench

End-to-end test bench: does a canary secret leave the machine while Claude Code is working? It is measured on bare Claude Code first, then with a filter (hook, plugin…) to check that the filter actually stops the leak.

How it works

Each test:

1. creates a throwaway project and plants a canary secret in it, unique to the run;
2. runs claude -p behind a local proxy that records everything sent to the API (ANTHROPIC_BASE_URL);
3. checks in the stream-json output that the model really performed the requested action. If not, the result is inconclusive (skip), never 🟢;
4. looks for the canary in three places:

Where	What it proves
traffic to the API	the secret was sent to Anthropic: this is the reference
transcript ~/.claude/projects/…/<session>.jsonl	the secret was written to disk
the model's final answer	informative only (results.json)

Each cell of the matrix shows the worst place reached:

• 🔴 the secret was sent to the Anthropic API;
• 🟠 it was not sent to the API, but was written to the transcript on disk;
• 🟢 neither.

In every mode, Claude Code runs with --setting-sources project (your global hooks are ignored), test-specific --settings and --strict-mcp-config.

Results

Run on 2026-10-11, podman sandbox, Claude Code 2.1.283, model Haiku 4.5, redacted at commit 0b4dd23 (main, Go version), redacted-pr47 at commit b88d507 (svn-arv/redacted#47, Rust rewrite 1.0.0), as is and with the entropy heuristic on (redacted-pr47-heuristic).

Case	bare	redacted	redacted-pr47	redacted-pr47-heuristic
cat_file	🔴	🔴	🟢	🟢
read_file	🔴	🔴	🟢	🟢
env	🔴	🔴	🟢	🟢
unlabeled_random	🔴	🔴	🔴	🟢
known_value	🔴	🔴	🟢	🟢
tail_foreground	🔴	🔴	🟢	🟢
tail_background	🔴	🔴	🟢	🟢
monitor	🔴	🔴	🔴	🔴

🔴 sent to the Anthropic API · 🟠 not sent to the API, but written to the transcript on disk · 🟢 neither

• bare: the canary leaks in every case, as expected. This confirms the bench detects leaks.
• redacted: the canary still reaches the API and the transcript in every case. The hook detects the secret in every case except monitor, but it answers with decision: "block", which does not replace the tool output: the raw output goes to the model and the redacted copy is only added beside it (see svn-arv/redacted#46). The model, seeing the warning, merely stops repeating the secret in its answer.
• redacted-pr47: updatedToolOutput now mirrors the shape of the tool response, so Claude Code applies the replacement: the model and the transcript only see the marker (DB_PASSWORD= [REDACTED ...grtZ], last 4 characters kept). unlabeled_random leaks because nothing flags a random value under a neutral key: in 1.0.0 that takes a secret learned by redacted init (global config, absent from the sandbox) or the entropy heuristic (off by default).
• redacted-pr47-heuristic: with the heuristic on, unlabeled_random is caught too (cache_ref: [REDACTED ...B95I], pattern secret_value).
• monitor: Monitor events reach the model as notifications that never go through PostToolUse hooks, so a PostToolUse filter cannot see them at all.
• tail_foreground: when the Bash timeout expires, Claude Code moves the command to the background and the model reads its output file with Read, so this case ends up like tail_background.

These results depend on the model's behavior and on the Claude Code version: re-run the bench rather than relying on this table.

Podman sandbox (default)

 ┌──────────── network leak-bench-internal (--internal, no egress) ─────────────┐
 │                                                                              │
 │  Claude Code container              proxy container                          │
 │  - dummy token                      - records every request                  │──► api.anthropic.com
 │  - read-only rootfs                 - swaps the dummy token for the real one │    (network leak-bench-egress)
 │  - cap-drop all, no-new-privileges  - the only way out                       │
 └──────────────────────────────────────────────────────────────────────────────┘

• Claude Code image (containers/claude.Containerfile): Fedora minimal + the official binary at a pinned version, verified by SHA-256 checksum. By default, the same version as the host's claude (--claude-version for another one). The image is built automatically on first run.
• No real credential in the Claude Code container, so the env case cannot expose it. The proxy gets the real one, mounted read-only from a temporary file in memory-backed storage ($XDG_RUNTIME_DIR).
• No network egress except through the proxy: any traffic that bypasses it fails instead of going unnoticed.
• The container's ~/.claude is a throwaway directory: your ~/.claude is neither read nor modified. The only exception is the feature flag cache (cachedGrowthBookFeatures from your ~/.claude.json), copied into it so that tools enabled by remote flags, such as Monitor, are available in the sandbox too. It holds no account data and no credentials.
• Rootless, --userns=keep-id: files in the mounted project belong to your user on both sides.

Credential used by the proxy, in this order: BENCH_AUTH (oauth:<token> or apikey:<key>), ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, then the claude.ai access token from ~/.claude/.credentials.json. The latter is only read, never refreshed: if it expires in less than 30 min, the bench stops and asks you to run claude -p hi on the host.

--sandbox host runs the host's claude instead, behind an in-process proxy and with an environment reduced to a fixed list of variables. Quicker to set up, but with no network or file isolation.

Cases (bench/cases.py)

Case	Scenario
cat_file	cat config.env containing DB_PASSWORD=<canary>
read_file	same with the Read tool
env	env, canary in BENCH_DB_PASSWORD
unlabeled_random	random value under a neutral key (cache_ref:)
known_value	low-entropy passphrase in free text, declared to the filter beforehand
tail_foreground	tail -f in the foreground; canary lines arrive after it starts
tail_background	tail -f in the background, then its output is read
monitor	Monitor tool on tail -f

For the tail cases, a thread waits for a tail -f <log> process to be running (pgrep) before writing the lines, so the leak really is a streaming one.

Subjects (bench/subjects.py)

• bare: Claude Code with nothing added. This is the negative control: the test passes when it leaks, which proves the bench detects something.
• redacted: svn-arv/redacted as a PostToolUse hook. Binary from REDACTED_BIN or the PATH. The test passes when nothing leaks (neither w nor t).
• redacted-pr47: the same, built from svn-arv/redacted#47 (Rust rewrite). Binary from REDACTED_PR47_BIN only. The sandbox has no global ~/.config/redacted, so it runs without learned secrets or heuristic: vendor signatures and project .redacted.engine.yml patterns only.
• redacted-pr47-heuristic: the same binary with heuristic.enabled: true. The heuristic is read from the global config only, so the hook runs with a HOME of its own holding .config/redacted/config.yaml.

To test another project, add a Subject subclass that provides files() (binaries copied into the sandbox), settings(where) (the hooks, given each file's path inside the sandbox) and, if possible, declare_known(), then register it in SUBJECTS. A hook binary must be able to run on Fedora minimal (glibc; a static binary is ideal).

Running

uv run pytest                                   # everything
uv run pytest --subject bare --case monitor     # a single case
REDACTED_BIN=~/git/redacted/redacted uv run pytest --subject redacted
REDACTED_PR47_BIN=~/git/redacted-pr47/target/release/redacted uv run pytest --subject redacted-pr47 --subject redacted-pr47-heuristic
uv run pytest --model claude-sonnet-5           # another model (default: Haiku 4.5)
uv run pytest --claude-version 2.1.292          # another Claude Code version (podman)
uv run pytest --sandbox host                    # no container

Requirements: uv, rootless podman, pgrep; claude on the host to detect the default version (or --claude-version).

Options: --attempts N (retries when inconclusive, default 2), --keep-transcripts (keeps the container's ~/.claude directory in the report; in host mode, leaves the sessions in ~/.claude/projects).

At the end of a run, a case × subject matrix is printed. It is written, along with all the evidence, to reports/<timestamp>/<sandbox>/: stream.jsonl, wire.txt (request bodies), transcript.jsonl, settings.json.

Caveats

• Cost: one claude -p run per case and per subject. The tail cases take about 30 to 60 s each.
• reports/ contains the full system prompt sent to the API. In host mode, it includes your user context (email…). The directory is git-ignored; do not publish it as is.
• In podman mode, only traffic to ANTHROPIC_BASE_URL can get out; the rest (telemetry…) is blocked. The bench therefore measures what Claude Code sends to the API, not what it would try to send elsewhere.
• In host mode, the rest of the traffic is neither blocked nor recorded, and a filter may read its global config (~/.config/redacted/…).