This privacy inspection study using synthetic data compares Caged’s detector configurations and shows how input formats affect their results.
The practical question was simple: what should an office team test before choosing privacy rules for Claude Cowork, Claude Code, or another AI tool routed through its branch gateway?
Start with a defined test set
The October 3 study called the Caged inspection service in monitor mode on an Apple M2 Max test host with 96 GiB RAM. It recorded 900 warm timings: 100 texts, three passes, and three configurations: pattern and language analysis, a local AI model, and combined inspection. The run used CPU inference, pinned model assets, and one call at a time.
A separate labelled set contained 50 synthetic expected values across person names, email addresses, phone numbers, payment test-card values, and SSNs. A value counted as caught when a detector identified the right type and its highlighted text overlapped the planted value. We counted which planted values were found within each labelled synthetic dataset.
The public lab report records the test conditions, method, raw timings, and per-case detection observations.
A fixture detail affected the outcome
The original email examples used the reserved .invalid domain suffix.
On the original 50-value set, pattern and language analysis caught 37 expected values; combined
configuration caught 39.
We then changed the suffix in those ten email samples to reserved
example.com addresses while keeping the other text and labels unchanged.
Both configurations caught 47 of the 50 expected values on that version.
| Synthetic test set | Pattern and language analysis | Combined inspection |
|---|---|---|
Original set with .invalid emails |
37 / 50 | 39 / 50 |
| Example-domain email variant | 47 / 50 | 47 / 50 |
Both corpora and both results are retained in the downloadable evidence. A repeated check reproduced the counts and recorded all 200 per-case observations across the two configurations and two datasets.
The changed fixture did not establish that either configuration detects every sensitive field. It showed that a test-data choice can affect the apparent benefit of adding a detector. A team should check whether its sample documents and prompts use the actual field formats and detection rules it needs.
Account for processing time too
On this host and timing corpus, pattern and language analysis had lower median and p95 inspection times than the local AI model or combined inspection. The timing tables include every configuration, the length groups, and all raw samples.
The timing study measures warm monitor-mode inspection on the recorded CPU test host. Network transport, authentication, request routing, masking, and model-provider responses are separate stages of an AI request.
Turn the finding into a pilot check
For an office pilot, start with a fictional memo, report, or small test project. Have staff try it in their configured Claude Cowork or Claude Code app using their assigned virtual keys. Include sensitive fields that should be caught, ordinary text that should remain unchanged, and variations in email domains and separators. Keep the original examples when testing a variation so the comparison remains visible.
Choose monitor, mask, or block behavior deliberately, then verify the request path. Detection counts alone do not establish correct forwarding or blocking. Use approved test instrumentation with synthetic payloads to observe those outcomes, and inspect decision metadata for the associated policy actions.
Measure the selected detector policy under representative workload conditions on the deployment’s hardware and software. State the workload and limits with the results. The branch AI gateway pilot guide provides the office-team setup and validation checklist; the AI governance brief explains the control scope.