# tact rollout evidence report

## Summary

The fleet does not run tact in 100% deny mode, and has not since 2026-08-03.
It runs a **sticky-cohort ramp**: roughly one session in ten is placed in a deny cohort, the rest get a non-blocking warning.
Measured over the current window, 96.3% of hook events resolved to `warn` and 3.7% to `deny`.

The operator's directive: "a helpful butler, not an authoritarian."
The tact library defaults to `warn` (advisory, non-blocking) when `route_hook` is called without a mode.
The fleet projection is what selects anything stronger, and today it selects the ramp.

An earlier revision of this document ratified "the current 100% deny state as the operational baseline."
That was true when it was written and false by the time it was committed.
This revision corrects the claim, re-derives every number from the live event stream, and removes the metrics that stream cannot actually support.

## What changed on 2026-08-03

Corpus commit `0ffebf102` ("feat(tact): switch rollout to ramp, retire stale advisory hook, update MCP docs", 2026-08-03 15:04:50 UTC) replaced the projected hook command in `corpus/sync_agents.py`:

```diff
-            harness: f'tact hook --harness {harness} --mode deny'
+            harness: f'python3 {HOOKS_DIR / "tact_rollout.py"} --harness {harness} --mode ramp'
```

The same commit deleted the superseded `corpus/hooks/tact_edits.py` advisory hook and added `ramp` to `HookDef.stale_commands_for` so a future mode change cleans up after itself.

### The cohort mechanism

Read from `corpus/hooks/tact_rollout.py` (pure stdlib, corpus-owned, outside this repo):

- `--mode shadow|warn|deny` is forwarded unchanged to `tact hook`.
  Only `ramp` is implemented in the wrapper.
- For `ramp`, the wrapper reads the hook payload's `session_id`, takes `sha256(session_id)`, converts the first 16 hex digits to an integer, and takes it modulo 100.
  A session is in the deny cohort when that value is below `TACT_RAMP_DENY_PCT`.
- `TACT_RAMP_DENY_PCT` defaults to `10` and is clamped to `[0, 100]`.
  It is set nowhere in corpus, so the fleet runs at the default 10%.
- The mapping is a pure function of the session id, so a session's cohort is **sticky** for its whole lifetime — an agent never sees tact flip mid-session.
- A payload with no `session_id` cannot be placed in a cohort, so it resolves to `warn` rather than `deny`.
  A missing or broken `tact` executable is caught and the wrapper returns 0.
  Every path after argument parsing exits 0, so the wrapper can never block a tool call (an argparse error still exits 2, as argparse always does).

The ramp is the first rung of the originally prescribed shadow → warn → 10% deny → 25/50/100% progression.
Advancing it is an environment-variable change, not a projection change.

## Method and provenance

All figures below come from `~/ai/tact/events.jsonl`, tact's append-only, content-free event stream (`tact.telemetry.record_event`).
Each row carries `schema_version, timestamp, outcome, harness, surface, mode, eligible, decision, duration_ms`, and optionally `rung`.
No file path, file content, session id, repository, or user identifier is recorded, by design.

- **Snapshot**: 40,009 rows, first row `2026-07-23T13:52:10.571940+00:00`, last row `2026-08-10T10:09:35.745778+00:00`.
  The stream is live and append-only, so a later read returns a superset; the windows below are defined by timestamp bounds rather than row counts.
- **Window split**: the last deny-only hook event is `2026-08-03T15:00:06.000075+00:00` and the first `warn` hook event is `2026-08-03T15:00:08.016538+00:00`.
  Everything before that boundary is the deny window (27,404 events); everything after is the ramp window (12,605 events).
- **Deny-window totals differ from the previous revision** (27,404 here against 26,083 there) because that revision's window closed at 2026-08-03T14:39, about twenty minutes before the cutover.
  Nothing was recounted; the window got longer.
- Figures were computed with a throwaway script over a frozen copy of the stream, not with `tact adoption` — see "Metrics this report does not cite" below for why.
- Every test invocation while preparing this report was run as `AI_HOME=<scratch> uv run pytest -q`, for the reason given under "Test contamination".

## Historical: the deny window

2026-07-23T13:52:10Z → 2026-08-03T15:00:06Z, 27,404 events.
The operator authorized going straight to deny during the implementation push, superseding the gradual ramp; this window is that decision's record.
All 3,048 hook events in it carry `mode: deny`.

### Hook surface

| Metric              | Count | Share |
| ------------------- | ----- | ----- |
| Hook events         | 3,048 | 100%  |
| Eligible (routed)   | 1,060 | 34.8% |
| Fail-open (allowed) | 1,988 | 65.2% |

The 1,988 fail-open events are the operations tact does not own: new-file creation, notebook edits, out-of-scope tools, and malformed payloads.
The 1,060 eligible events are edits to existing ordinary files, which deny mode refused and redirected.

| Harness | Hook events | Eligible | Fail-open |
| ------- | ----------- | -------- | --------- |
| claude  | 1,261       | 411      | 850       |
| codex   | 1,157       | 390      | 767       |
| kimi    | 626         | 255      | 371       |
| droid   | 4           | 4        | 0         |
| gemini  | 0           | 0        | 0         |

Gemini hooks require CLI 0.26.0+ and the installed version predates hook support, so the harness produced no events; its adapter contract is exercised by the opt-in live-harness canaries instead.
Droid's 4 events reflect droid sessions reaching tact through MCP rather than the hook layer.

| Latency (hook events) | Value      |
| --------------------- | ---------- |
| p50                   | 0.087 ms   |
| p95                   | 0.209 ms   |
| p99                   | 0.393 ms   |
| max                   | 6,321.1 ms |
| events over 150 ms    | 4 (0.13%)  |

The four events above the 150 ms budget are 6,321.1, 1,265.3, 282.6, and 267.5 ms.
The stream records no cause, so nothing here explains them; only their count and magnitude are evidenced.

### Library surface

24,356 events: 19,618 applied, 2,219 healed, 2,519 refused.

| Rung   | Count  | % of applied+healed |
| ------ | ------ | ------------------- |
| exact  | 19,618 | 89.8%               |
| indent | 1,684  | 7.7%                |
| fuzzy  | 423    | 1.9%                |
| canon  | 56     | 0.3%                |
| rstrip | 56     | 0.3%                |
| total  | 21,837 | 100%                |

The previous revision's rung table summed to 104.3% because it counted the rung recorded on `refused` events as well.
That was an arithmetic error, not a different measurement; the table above is scoped to `applied` + `healed` and sums exactly.
89.8% of completed edits matched on the finest rung; the coarser rungs healed the remaining 10.2%.

This table is subject to the same test contamination described below.
It is reported because it is the historical record the previous revision claimed, corrected — not because it is a clean production measurement.

## Current: the ramp window

2026-08-03T15:00:08Z → 2026-08-10T10:09:35Z, 12,605 events.

### Hook surface

The hook surface is the only surface in this stream that is provably free of test-generated events, so the load-bearing claims are taken from it.

| Metric              | Count | Share |
| ------------------- | ----- | ----- |
| Hook events         | 6,833 | 100%  |
| Eligible (routed)   | 5,655 | 82.8% |
| Fail-open (allowed) | 1,178 | 17.2% |

The eligible share more than doubled against the deny window's 34.8%.
Part of that tracks the harness mix — Codex went from 1,157 hook events to 3,932 and routes a higher share of them to tact — but the stream carries no attribute that explains the rest, so the shift is reported rather than accounted for.

| Resolved mode | Hook events | Share | Eligible events | Share |
| ------------- | ----------- | ----- | --------------- | ----- |
| `warn`        | 6,581       | 96.3% | 5,523           | 97.7% |
| `deny`        | 252         | 3.7%  | 132             | 2.3%  |

The observed 3.7% deny share sits below the nominal 10% because the two rates measure different things.
`TACT_RAMP_DENY_PCT` is a **per-session** cohort rate; the stream counts **per-edit** events, and a handful of long, edit-heavy warn-cohort sessions outweigh many short deny-cohort ones.
The nominal 10% therefore **cannot be verified from this stream**: events carry no session id, by privacy design, so sessions cannot be counted.
The cohort function itself is verifiable by inspection and unit test in corpus; its realized session-level rate is not observable here.

| Harness | Hook events | Eligible | Fail-open |
| ------- | ----------- | -------- | --------- |
| codex   | 3,932       | 3,388    | 544       |
| claude  | 2,552       | 2,060    | 492       |
| kimi    | 349         | 207      | 142       |
| droid   | 0           | 0        | 0         |
| gemini  | 0           | 0        | 0         |

| Latency (hook events) | Value      |
| --------------------- | ---------- |
| p50                   | 0.084 ms   |
| p95                   | 0.197 ms   |
| p99                   | 0.340 ms   |
| max                   | 2,343.9 ms |
| events over 150 ms    | 1 (0.015%) |

### Library surface — contaminated, not reportable

5,772 events: 2,919 applied, 1,711 healed, 1,142 refused.
Rung shares are **not** published for this window, because a demonstrable share of these events was generated by the test suite rather than by fleet work.
See below.

## Test contamination of the library surface — fixed forward, unrecoverable backward

`tact.telemetry.event_path()` resolves to `ai_home() / 'tact' / 'events.jsonl'`, and `ai_home()` honors `$AI_HOME`. A test that sets `$AI_HOME` writes somewhere disposable; a test that leaves it alone writes into the fleet's real stream.
Until 2026-08-10 no autouse fixture in `tests/conftest.py` enforced the override, so the isolation was per-test-file and incomplete.

**The leak is closed.** `tests/conftest.py` now carries an autouse, unconditional fixture that points both `$AI_HOME` and `$AI` at a per-test throwaway directory — every test gets a disposable agent home whether it asks for one or not.
An opt-in fixture could not have fixed this, because the failure mode *is* forgetting to opt in.
Verified by counting `~/ai/tact/events.jsonl` before and after a run of the write-path tests (`test_tact_apply.py`, `test_tact_transact.py`, `test_telemetry_events.py`): the delta is **0**.

Measured, not estimated — every figure below describes the *pre-fix* behavior:

- One full suite run — 646 passed, 4 skipped, the suite's size at the time of measurement — wrote **616 library events** into the stream when `$AI_HOME` was not overridden: 406 healed (402 `indent`, 2 `canon`, 2 `rstrip`), 202 refused, 8 applied (`exact`).
- The same run wrote **zero hook events**.
  The hook surface was never contaminated; the library surface was.
- A run of that exact signature landed in the live stream **while this report was being written**.
  Between `2026-08-10T10:01:59Z` and `2026-08-10T10:09:35Z` the stream gained 622 library events: 402 healed/`indent`, 202 refused, 2 healed/`rstrip`, 2 healed/`canon`, 13 applied/`exact`, 1 healed/`fuzzy`.
  That signature is unambiguous against the isolated run measured above, though the stream carries nothing that names the writing process, so the attribution is a signature match rather than a recorded fact.

**Closing the leak does not clean the record.** Everything the stream accumulated before the fixture landed is still in the file, and still indistinguishable from real edits after the fact.
Events carry no run, test, or session marker, and the suite's composition changed over the measurement period (646 tests when the contamination was measured; the repo's own docs had described roughly 380 not long before), so the 616-events-per-run profile cannot be projected backwards.
**The historical contamination volume is therefore unknown and permanently unrecoverable.** No back-estimate is offered here, and none should be read into the tables above.
The leak is closed going forward; the historical record stays uncertain.

This is why the ramp window's rung distribution is withheld: its raw shares (63.0% `exact`, 35.5% `indent`) differ sharply from the deny window's (89.8% / 7.7%), and at least 402 of the 1,643 `indent` heals are known test artifacts.
Reading a ladder regression out of that difference would be reading the denominator, not the ladder.

## What this event stream cannot measure

The stream records operations that **reached** tact.
An operation that never called tact leaves no row, so the following are structurally unobservable here and are marked `UNMEASURED` rather than quietly dropped:

- **Classifier crash rate.** `cmd_hook` calls `record_event` only after `route_hook` returns, so a crash inside classification produced no row at all.
  Two payload classes did exactly that for the whole of both windows (see the fail-open note below), which makes every hook-event count in this report a *lower bound* and the crash rate itself unobservable.
- **Bypass rate.** An agent that edits a file with `sed`, a shell heredoc, or any tool outside the routed set emits nothing.
  "Zero bypass" is not a finding this stream can produce, and the previous revision's claim of it is withdrawn.
- **Shell-write residual.** Same mechanism.
  The literal `shell-write residual: 0` printed by `tact adoption` is a count over a population the stream cannot see, not a measurement of zero.
- **Availability.** The schema has no failure event.
  Absence of error rows is not evidence of uptime; the previous revision's "100%" is withdrawn.
- **Eligible-edit completion.** The only completion signal is on the library surface, which is contaminated and carries `harness: unknown` on every row, so completed edits cannot be attributed to production or to a harness.
  The previous revision's "90.9%" mixed the two surfaces' denominators and included test events; it is withdrawn rather than recomputed.
- **False-heal rate.** Not carried in this stream at all.
  The zero-false-heal claim rests on the committed `tests/fixtures/ledger_eval/` replay corpus (`tests/test_tact_ledger_eval.py`), which is a fixture replay, not a production measurement.
- **Session-level ramp rate.** No session id, by design; see above.

## Metrics this report does not cite

`tact adoption` reports a fleet ratio of 103.9% at `--days 7` and 389.1% at `--days 30`.
A ratio above 100% is not a rounding artifact: the numerator counts library events and the denominator counts hook-eligible events, which are two different populations counted at two different granularities, and the numerator is additionally test-contaminated.
Its per-harness `denom`/`p95` columns come from the clean hook surface and agree with the figures above; its `num`, `ratio`, `refused`, and `shell-write residual` columns should not be quoted until the denominator question is settled.
This is a defect in `tact/adoption.py`, filed for its owner, not a limitation of the rollout.

## Acceptance criteria assessment

| Criterion                        | Target    | Observed                | Status     |
| -------------------------------- | --------- | ----------------------- | ---------- |
| Hook p95 latency                 | \<= 150ms | 0.197 ms (ramp window)  | PASS       |
| Fail-open on unowned operations  | always    | path exercised 1,178×   | PARTIAL    |
| Advisory by default              | yes       | 96.3% of hook events    | PASS       |
| Reversible without a code change | yes       | env var / one-word mode | PASS       |
| No false-heal                    | 0         | fixture replay only     | UNMEASURED |
| Availability                     | >= 99.5%  | no failure events exist | UNMEASURED |
| Bypass                           | \<= 5%    | unobservable            | UNMEASURED |
| Eligible-edit completion         | >= 98%    | contaminated surface    | UNMEASURED |

`UNMEASURED` means the stream cannot produce the number, not that the number is bad.
Four of the eight criteria in the previous revision were marked PASS on evidence that does not exist; they are reclassified rather than removed, so a reader looking for them finds out why they are absent.

Fail-open is `PARTIAL` rather than `PASS` for two reasons.
First, 1,178 allowed events show the fail-open path *firing*, not that it fired every time it should have — the same category error this report flags in the bypass row.
Second, it demonstrably did not always hold: for the whole of both windows, two payload classes propagated an exception out of classification instead of failing open — an unresolvable `~user` path (`RuntimeError` from `Path.expanduser`) and an embedded NUL (`ValueError` from `stat`).
No edit was blocked as a result — a hook exit of 1 is a non-blocking error in Claude Code, and `tact_rollout.py` returns 0 regardless — but the classifier crashed on input it should have waved through silently, and the crash left no event, so it was invisible in this stream.
Fixed on this branch; see the `route_hook` fail-open guard in `tact/routing.py`.

## Ratification

What is ratified is the **ramp**, not 100% deny:

1. **Advisory is the default and the norm.**
   `route_hook` returns `warn` when no mode is given, and 96.3% of hook events in the current window resolved to `warn`.
   Blocking is the ~10% cohort exception, not the rule.
2. **Cohorts are sticky and fail open.**
   The cohort is a pure function of the session id, so a session's experience never flips mid-flight; a payload without a session id, an uninstalled `tact`, or a wrapper crash all resolve to non-blocking.
3. **The boundary holds.**
   17.2% of hook events in the current window were allowed through untouched — new files, notebooks, out-of-scope tools, malformed payloads.
   Routing is content-blind and narrow by construction.
4. **Latency is not a factor.**
   p95 of 0.197 ms against a 150 ms budget, with one event over budget in 6,833.
5. **Reversal is a configuration change.**
   `TACT_RAMP_DENY_PCT=0` makes the fleet purely advisory; `--mode warn` in the projection removes the cohort mechanism entirely.
   Neither requires a code change or a release.

This is not an argument that a higher deny percentage is safe.
No evidence here bears on that, and the operator has confirmed advisory is the intended steady state.
The ramp is ratified as the current operating point.

## Open measurement work

1. **Test isolation — closed.**
   `tests/conftest.py` now points `$AI_HOME` and `$AI` at a per-test temporary directory, autouse and unconditional, so no test can write into the fleet's production stream whether or not it remembers to.
   The suite was emitting 616 events per run before this; it emits none now.
   The events already in the stream from earlier runs are still there, so metrics computed over history remain untrustworthy — but the leak itself is stopped, not merely documented.
2. **Surface attribution.**
   Library events carry `harness: unknown` unconditionally, so a production edit and a test fixture are indistinguishable after the fact.
   A `surface` value distinguishing test runs, or any run-scoped marker, would make completion rate measurable for the first time.
3. **`tact adoption`'s denominator.**
   The numerator and denominator count different populations; see above.
4. **Gemini.**
   Still no hook events; requires Gemini CLI 0.26.0+.
   The adapter contract is covered by the opt-in live-harness canaries.
5. **Session-level ramp verification.**
   Verifying that the realized deny cohort is 10% of sessions requires a session-scoped counter that this stream deliberately does not carry.
   Left unmeasured on purpose; the cohort function is verified by unit test in corpus instead.