Add redacted PR #47 subjects and show results as colored dots

Bench svn-arv/redacted#47 (Rust rewrite) as is and with the entropy
heuristic on. Known values are now escaped instead of using \Q...\E,
which Rust's regex lacks. Subject files may live in subdirectories.

Matrix cells become 🔴 (sent to the API), 🟠 (transcript only) or 🟢.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GZY2T82aEHnuVC4ALzRcPh

README.md

@@ -12,14 +12,20 @@ Each test:
 2. runs `claude -p` behind a **local proxy that records everything sent to the
    API** (`ANTHROPIC_BASE_URL`);
 3. checks in the `stream-json` output that the model really performed the
-   requested action. If not, the result is **inconclusive** (skip), never "clean";
+   requested action. If not, the result is **inconclusive** (skip), never 🟢;
 4. looks for the canary in three places:
 
-| Letter | Where | What it proves |
+| Where | What it proves |
-|---|---|---|
+|---|---|
-| `w` | traffic to the API | the secret was sent to Anthropic: this is the reference |
+| traffic to the API | the secret was sent to Anthropic: this is the reference |
-| `t` | transcript `~/.claude/projects/…/<session>.jsonl` | the secret was written to disk |
+| transcript `~/.claude/projects/…/<session>.jsonl` | the secret was written to disk |
-| `a` | the model's final answer | informative only |
+| the model's final answer | informative only (`results.json`) |
+
+Each cell of the matrix shows the worst place reached:
+
+- 🔴 the secret was sent to the Anthropic API;
+- 🟠 it was not sent to the API, but was written to the transcript on disk;
+- 🟢 neither.
 
 In every mode, Claude Code runs with `--setting-sources project` (your global
 hooks are ignored), test-specific `--settings` and `--strict-mcp-config`.
@@ -27,18 +33,24 @@ hooks are ignored), test-specific `--settings` and `--strict-mcp-config`.
 ## Results
 
 Run on 2026-10-11, podman sandbox, Claude Code 2.1.283, model Haiku 4.5,
-`redacted` at commit `0b4dd23` (`main`, Go version).
+`redacted` at commit `0b4dd23` (`main`, Go version), `redacted-pr47` at
-
+commit `b88d507` ([svn-arv/redacted#47](https://github.com/svn-arv/redacted/pull/47),
-| Case | bare | redacted |
+Rust rewrite 1.0.0), as is and with the entropy heuristic on
-|---|---|---|
+(`redacted-pr47-heuristic`).
-| `cat_file` | LEAK(w,t,a) | LEAK(w,t) |
+
-| `read_file` | LEAK(w,t,a) | LEAK(w,t) |
+| Case | bare | redacted | redacted-pr47 | redacted-pr47-heuristic |
-| `env` | LEAK(w,t) | LEAK(w,t) |
+|---|---|---|---|---|
-| `unlabeled_random` | LEAK(w,t,a) | LEAK(w,t) |
+| `cat_file` | 🔴 | 🔴 | 🟢 | 🟢 |
-| `known_value` | LEAK(w,t) | LEAK(w,t) |
+| `read_file` | 🔴 | 🔴 | 🟢 | 🟢 |
-| `tail_foreground` | LEAK(w,t,a) | LEAK(w,t) |
+| `env` | 🔴 | 🔴 | 🟢 | 🟢 |
-| `tail_background` | LEAK(w,t,a) | LEAK(w,t) |
+| `unlabeled_random` | 🔴 | 🔴 | 🔴 | 🟢 |
-| `monitor` | LEAK(w,t,a) | LEAK(w,t) |
+| `known_value` | 🔴 | 🔴 | 🟢 | 🟢 |
+| `tail_foreground` | 🔴 | 🔴 | 🟢 | 🟢 |
+| `tail_background` | 🔴 | 🔴 | 🟢 | 🟢 |
+| `monitor` | 🔴 | 🔴 | 🔴 | 🔴 |
+
+🔴 sent to the Anthropic API · 🟠 not sent to the API, but written to the
+transcript on disk · 🟢 neither
 
 - **bare**: the canary leaks in every case, as expected. This confirms the
   bench detects leaks.
@@ -47,8 +59,17 @@ Run on 2026-10-11, podman sandbox, Claude Code 2.1.283, model Haiku 4.5,
   answers with `decision: "block"`, which does not replace the tool output:
   the raw output goes to the model and the redacted copy is only added beside
   it (see [svn-arv/redacted#46](https://github.com/svn-arv/redacted/issues/46)).
-  The `a` disappears only because the model, seeing the warning, no longer
+  The model, seeing the warning, merely stops repeating the secret in its
-  repeats the secret.
+  answer.
+- **redacted-pr47**: `updatedToolOutput` now mirrors the shape of the tool
+  response, so Claude Code applies the replacement: the model and the
+  transcript only see the marker (`DB_PASSWORD= [REDACTED ...grtZ]`, last 4
+  characters kept). `unlabeled_random` leaks because nothing flags a random
+  value under a neutral key: in 1.0.0 that takes a secret learned by
+  `redacted init` (global config, absent from the sandbox) or the entropy
+  heuristic (off by default).
+- **redacted-pr47-heuristic**: with the heuristic on, `unlabeled_random` is
+  caught too (`cache_ref: [REDACTED ...B95I]`, pattern `secret_value`).
 - **`monitor`**: `Monitor` events reach the model as notifications that never
   go through PostToolUse hooks, so a PostToolUse filter cannot see them at all.
 - **`tail_foreground`**: when the Bash timeout expires, Claude Code moves the
@@ -122,6 +143,14 @@ one.
 - `redacted`: [svn-arv/redacted](https://github.com/svn-arv/redacted) as a
   PostToolUse hook. Binary from `REDACTED_BIN` or the `PATH`. The test passes
   when nothing leaks (neither `w` nor `t`).
+- `redacted-pr47`: the same, built from
+  [svn-arv/redacted#47](https://github.com/svn-arv/redacted/pull/47) (Rust
+  rewrite). Binary from `REDACTED_PR47_BIN` only. The sandbox has no global
+  `~/.config/redacted`, so it runs without learned secrets or heuristic:
+  vendor signatures and project `.redacted.engine.yml` patterns only.
+- `redacted-pr47-heuristic`: the same binary with `heuristic.enabled: true`.
+  The heuristic is read from the global config only, so the hook runs with a
+  `HOME` of its own holding `.config/redacted/config.yaml`.
 
 To test another project, add a `Subject` subclass that provides `files()`
 (binaries copied into the sandbox), `settings(where)` (the hooks, given each
@@ -135,6 +164,7 @@ register it in `SUBJECTS`. A hook binary must be able to run on Fedora minimal
 uv run pytest                                   # everything
 uv run pytest --subject bare --case monitor     # a single case
 REDACTED_BIN=~/git/redacted/redacted uv run pytest --subject redacted
+REDACTED_PR47_BIN=~/git/redacted-pr47/target/release/redacted uv run pytest --subject redacted-pr47 --subject redacted-pr47-heuristic
 uv run pytest --model claude-sonnet-5           # another model (default: Haiku 4.5)
 uv run pytest --claude-version 2.1.292          # another Claude Code version (podman)
 uv run pytest --sandbox host                    # no container

bench/runner.py

@@ -220,6 +220,7 @@ class PodmanSandbox:
             d.mkdir()
         (config / ".claude.json").write_text(json.dumps(self.feature_cache))
         for name, src in subject_files.items():
+            (staged / name).parent.mkdir(parents=True, exist_ok=True)
             shutil.copy2(src, staged / name)
         (staged / "settings.json").write_text(json.dumps(settings, indent=2))
         shutil.copy2(staged / "settings.json", artifacts / "settings.json")

bench/subjects.py

@@ -7,13 +7,16 @@ register it in SUBJECTS.
 
 Files are copied into the sandbox: `settings(where)` receives, for each
 `files()` name, the path the file is visible at (a container path under podman).
+A name may contain `/` to lay out a directory tree.
 Hook binaries must run in the sandbox image (Fedora minimal, glibc).
 """
 
 from __future__ import annotations
 
 import os
+import re
 import shutil
+import tempfile
 from dataclasses import dataclass
 from pathlib import Path
 
@@ -49,20 +52,23 @@ class Bare(Subject):
 
 
 class Redacted(Subject):
-    """github.com/svn-arv/redacted (Go, 0.x): a PostToolUse hook, `redacted scrub`.
+    """github.com/svn-arv/redacted: a PostToolUse hook, `redacted scrub`.
 
-    Known values are declared as a literal custom pattern in the project's
+    `redacted` is the Go version (0.x, `main`), `redacted-pr47` the Rust
-    .redacted.engine.yml. In host mode the global ~/.config/redacted files
+    rewrite of svn-arv/redacted#47 (1.0.0). Known values are declared as a
-    also apply; the podman sandbox has none.
+    literal custom pattern in the project's .redacted.engine.yml, which both
+    versions read. In host mode the global ~/.config/redacted files also
+    apply; the podman sandbox has none, so 1.0.0 runs without learned secrets.
     """
 
-    def __init__(self, name: str = "redacted", binary: str | None = None) -> None:
+    def __init__(self, name: str = "redacted", env_var: str = "REDACTED_BIN", which: bool = True) -> None:
         super().__init__(name)
-        self.binary = binary or os.environ.get("REDACTED_BIN") or shutil.which("redacted") or ""
+        self.env_var = env_var
+        self.binary = os.environ.get(env_var) or (which and shutil.which("redacted")) or ""
 
     def available(self) -> str | None:
         if not self.binary or not Path(self.binary).exists():
-            return "redacted binary not found (set REDACTED_BIN)"
+            return f"{self.name} binary not found (set {self.env_var})"
         return None
 
     def files(self) -> dict[str, Path]:
@@ -73,10 +79,44 @@ class Redacted(Subject):
         return {"hooks": {"PostToolUse": [{"hooks": [{"type": "command", "command": cmd}]}]}}
 
     def declare_known(self, project: Path, values: list[str]) -> bool:
-        assert not any("\\E" in v or "'" in v for v in values), "value cannot be quoted literally"
+        # No \Q...\E: Rust's regex (1.0.0) lacks it. Escaping punctuation works in both.
-        regex = "|".join(r"\Q" + v + r"\E" for v in values)
+        assert not any("'" in v or re.search(r"\s", v) for v in values), "value cannot be quoted literally"
+        regex = "|".join(re.sub(r"([^0-9A-Za-z_])", r"\\\1", v) for v in values)
         (project / ".redacted.engine.yml").write_text(f"patterns:\n  - name: bench_known\n    regex: '{regex}'\n")
         return True
 
 
-SUBJECTS: dict[str, Subject] = {s.name: s for s in [Bare(), Redacted()]}
+class RedactedHeuristic(Redacted):
+    """redacted 1.0.0 with the entropy heuristic on (`heuristic.enabled: true`).
+
+    It reads the heuristic from the global config only, so the hook runs with a
+    HOME of its own that holds .config/redacted/config.yaml. Stats still go to
+    the real HOME.
+    """
+
+    CONFIG = "home/.config/redacted/config.yaml"
+
+    def __init__(self) -> None:
+        super().__init__("redacted-pr47-heuristic", env_var="REDACTED_PR47_BIN", which=False)
+        self._config: Path | None = None
+
+    def files(self) -> dict[str, Path]:
+        if self._config is None:
+            self._config = Path(tempfile.mkdtemp(prefix="leak-bench-redacted-")) / ".config/redacted/config.yaml"
+            self._config.parent.mkdir(parents=True)
+            self._config.write_text("version: 2\nheuristic:\n  enabled: true\n")
+        return {**super().files(), self.CONFIG: self._config}
+
+    def settings(self, where: dict[str, str]) -> dict:
+        home = where[self.CONFIG].removesuffix("/.config/redacted/config.yaml")
+        cmd = (f'REDACTED_STATS_FILE="$HOME/.config/redacted/stats.jsonl" HOME={home} '
+               f"{where['redacted']} scrub")
+        return {"hooks": {"PostToolUse": [{"hooks": [{"type": "command", "command": cmd}]}]}}
+
+
+SUBJECTS: dict[str, Subject] = {s.name: s for s in [
+    Bare(),
+    Redacted(),
+    Redacted("redacted-pr47", env_var="REDACTED_PR47_BIN", which=False),
+    RedactedHeuristic(),
+]}

conftest.py

@@ -93,9 +93,9 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config) -> None:
     lines = ["| case | " + " | ".join(subjects) + " |", "|---" * (len(subjects) + 1) + "|"]
     for c in cases:
         lines.append(f"| {c} | " + " | ".join(cell.get((s, c), "") for s in subjects) + " |")
-    legend = ("LEAK(w,t,a) = canary found on the wire (sent to the API), in the transcript on disk, "
+    legend = ("🔴 = canary sent to the Anthropic API. 🟠 = not sent to the API, but written to the "
-              "in the final answer. clean = found nowhere. ? = inconclusive (scenario not exercised). "
+              "transcript on disk. 🟢 = neither. ? = inconclusive (scenario not exercised). "
-              "n/a = subject cannot declare known values.")
+              "n/a = subject cannot declare known values. skip = subject not available.")
 
     terminalreporter.write_sep("=", "leak matrix")
     for line in lines:

tests/test_leaks.py

@@ -9,8 +9,9 @@ from bench.runner import run_claude
 
 
 def _cell(leaks: dict[str, bool]) -> str:
-    where = ",".join(k[0] for k, v in leaks.items() if v)
+    if leaks["wire"]:
-    return f"LEAK({where})" if where else "clean"
+        return "🔴"
+    return "🟠" if leaks["transcript"] else "🟢"
 
 
 def test_leak(subject, case_name, tmp_path, request, report_dir, record, sandbox):