← All guides · Agent observability and evaluation

How to Redact PII From AI Agent Traces: Placement, Testing and Cleanup

Part 11 of Agent observability and evaluation · 1,413 words · ~6 min read · published 2026-10-06 · updated 2026-10-06 · Markdown variant

How to redact PII from AI agent traces in practice: where the redactor sits in the pipeline, what to do per span field, how to test it with seeded fake PII, and how to clean up after a leak.

Part of the AI Agent Observability and the Production Evaluation Playbook guide.

To redact PII from AI agent traces, run the detector inside your exporter so spans are scrubbed before they leave the process, apply a different rule to each span field, and test the result by scanning exported spans for fake values you seeded yourself. The corpus holds no detection accuracy or latency numbers as of October 2026, so the pipeline design below is reasoned inference built on the tool descriptions it does contain. Context for the topic is in the pillar on observability and evaluation.

What is already covered elsewhere?

Three sibling articles make the case for redacting, and this one deliberately does not repeat it. Agent observability for reliability explains why traces need redaction at all. Sampling and retention of agent traces sets the rule that redaction happens before the retention clock starts. Data privacy and PII for AI agents covers the exposure surface and legal framing, and nothing here is legal advice. What remains is the mechanics: where the code sits, what it does to each field, how you know it works, and what to do when it fails.

Where in the pipeline should the redactor sit?

The redactor belongs in the exporter layer, after spans are created and before they are buffered or sent anywhere. The data-privacy-for-agents resource says to scrub in the exporter path before export, so all downstream sinks (LangSmith, Datadog, S3) receive only sanitized data.

The ordering below is inference, not a published standard:

1. The agent creates a span with raw attributes and events.
2. A span processor or exporter wrapper runs detection and anonymization on the span.
3. Only the sanitized span enters the export buffer.
4. The buffer exports to one or more backends.

Putting the step before the buffer matters because a buffer, a retry queue or a local file spool is itself a store. If raw content sits in any of them, you have a leak surface that the backend's own controls will not reach. A crashed process can also leave raw spans on disk, which ties into crash-safe run-start records.

A related but different control is redaction before the model sees data. The same resource lists "redact PII from all content before it enters the model context window" as its own checklist item, so the trace redactor is the second gate and not a substitute for the first. If you only scrub at export, the model still received the raw value.

What policy should each span field get?

Each span field needs its own rule, because the four places PII hides in a trace behave differently. The table is a reasoned starting point, not a corpus rule.

The corpus supports three anchors for this table. First, the OpenTelemetry GenAI conventions make model input and output text an optional span event, off by default for PII safety. Second, the agent-observability resource lists tool-call inputs and outputs among the signals to capture, with the instruction to redact PII before logging. Third, the privacy resource notes that structured fields like user_id are acceptable while raw prompt strings containing names, SSNs or health data are not.

Tool outputs deserve extra care because the privacy resource states that retrieved and tool-output content can itself carry names or financial and health data, not only what the user typed.

Which operator should replace a detected value?

Choose the operator by what the trace is for. The Presidio Anonymizer, described in the corpus, applies four configurable operators: replace, mask, redact and encrypt. The resource also shows placeholders such as <PERSON_0> and <EMAIL_0>.

• Placeholders keep the sentence structure readable for debugging and evaluation, and a numbered placeholder lets you see that two mentions were the same entity.
• Redact removes the value entirely and is the safest default for fields you never need to read.
• Mask keeps a partial shape and suits identifiers you only need to recognise.
• Encrypt keeps the original recoverable, which makes the trace store a holder of personal data again. Treat it as a deliberate exception.

Why are secrets a separate class?

Secrets need their own rule because a leaked credential is an access problem and a leaked name is a privacy problem, and the fixes differ. The agentic-security-checklist resource states that a secret entering a prompt, a retrieved document, a tool result or a log line the model can read must be treated as disclosed.

Its defences include redacting known secret formats from anything flowing into model context, covering prefix detection for common key shapes, bearer tokens and PEM blocks. Format-based matching is a different technique from the entity recognition used for names, so run it as a separate step in the same processor. Because secrets are often disclosed before they reach any trace, the checklist's first defence also applies: resolve credentials server-side at tool-execution time so the model never holds the key.

How do you verify the redactor works?

Verify a redactor by seeding fake PII and secrets into test runs, then scanning the exported spans for those exact strings. This is reasoned inference, since the corpus describes the tools but gives no testing method.

1. Build a fixture set of fake but realistic values: a name, an email, a phone number, a card-shaped number and a dummy key with a real prefix shape.
2. Place them in every field class from the table, including nested tool arguments and tool outputs.
3. Run the agent against a local or test backend.
4. Fetch what was actually exported, not what the processor claims to have emitted.
5. Fail the build if any seeded value appears.

A minimal scan over exported spans looks like this:

# spans.jsonl: spans as received by the test backend, one JSON object per line
# seeds.txt: the fake values you injected, one per line
if grep -F -f <(grep -v '^$' seeds.txt) spans.jsonl; then
  echo "FAIL: seeded PII found in exported spans" >&2
  exit 1
fi
echo "OK: no seeded values found"

Canary values add a second signal in production. Use a recognisable fake identity in a synthetic run on a schedule and search the real store for it. A hit means the redactor regressed after a deploy. Remember a clean scan only proves the seeded patterns are caught, not that all real PII is, so treat detection coverage as unmeasured until you measure it on your own data.

What if PII already reached the trace store?

Treat it as an incident: stop the source, remove the data, then re-verify. The order matters because purging before the fix just refills the store.

1. Deploy the corrected redactor and confirm it with the scan above.
2. Identify the affected window using the deploy history and the canary hits.
3. Delete or re-process affected spans using whatever your backend supports. The corpus states no deletion capability for any listed backend, so check its documentation.
4. Check copies: exports, backups, evaluation datasets built from sampled traces, and any other sink.
5. Rotate any credential that appeared, since the checklist says to treat it as disclosed.

Sampled datasets are easy to forget. The agent-observability resource notes that stored spans feed evaluation datasets, so a leak can travel there.

Does redacting conflict with keeping an audit trail?

Redaction and auditability pull in opposite directions, and you resolve it by separating the stores rather than weakening either. An audit record needs enough to reconstruct who did what, while a debugging trace should hold as little personal content as possible. Audit trails for AI agents covers what an audit record needs to hold. A reasonable inference is to keep identifiers and decisions in the audit store and let the trace carry placeholders.

What does ChangeGamer do here?

ChangeGamer runs no trace store, and no agent content passes through a redaction pipeline on this site. Nothing in this article comes from operating one. The tool descriptions come from the corpus, and the pipeline design, the field table and the test method are reasoning from them.

#agents #observability #tracing #pii #redaction #privacy #secrets