From 33f6825e5436a8b6fed45102ae134ee514514808 Mon Sep 17 00:00:00 2001 From: Nicolas MASSE Date: Sun, 11 Oct 2026 12:10:02 +0200 Subject: [PATCH] Add a Results section to the README Co-Authored-By: Claude Opus 5.5 --- README.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/README.md b/README.md index 789731d..3be4800 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,40 @@ Each test: In every mode, Claude Code runs with `--setting-sources project` (your global hooks are ignored), test-specific `--settings` and `--strict-mcp-config`. +## Results + +Run on 2026-10-11, podman sandbox, Claude Code 2.1.283, model Haiku 4.5, +`redacted` at commit `0b4dd23` (`main`, Go version). + +| Case | bare | redacted | +|---|---|---| +| `cat_file` | LEAK(w,t,a) | LEAK(w,t) | +| `read_file` | LEAK(w,t,a) | LEAK(w,t) | +| `env` | LEAK(w,t) | LEAK(w,t) | +| `unlabeled_random` | LEAK(w,t,a) | LEAK(w,t) | +| `known_value` | LEAK(w,t) | LEAK(w,t) | +| `tail_foreground` | LEAK(w,t,a) | LEAK(w,t) | +| `tail_background` | LEAK(w,t,a) | LEAK(w,t) | +| `monitor` | LEAK(w,t,a) | LEAK(w,t) | + +- **bare**: the canary leaks in every case, as expected. This confirms the + bench detects leaks. +- **redacted**: the canary still reaches the API and the transcript in every + case. The hook detects the secret in every case except `monitor`, but it + answers with `decision: "block"`, which does not replace the tool output: + the raw output goes to the model and the redacted copy is only added beside + it (see [svn-arv/redacted#46](https://github.com/svn-arv/redacted/issues/46)). + The `a` disappears only because the model, seeing the warning, no longer + repeats the secret. +- **`monitor`**: `Monitor` events reach the model as notifications that never + go through PostToolUse hooks, so a PostToolUse filter cannot see them at all. +- **`tail_foreground`**: when the Bash timeout expires, Claude Code moves the + command to the background and the model reads its output file with `Read`, + so this case ends up like `tail_background`. + +These results depend on the model's behavior and on the Claude Code version: +re-run the bench rather than relying on this table. + ## Podman sandbox (default) ```