Privacy & trust

Privacy in AI inference: retention, training, and isolation

Every inference request is a disclosure: you are handing someone else your data to process. What happens to it afterward is a question worth asking before you send a million records, not after.

8 min read Updated August 19, 2026

The questions that actually matter

Privacy in inference is not one thing; it is a handful of distinct questions, and a vendor can answer them very differently:

Retention: How long is your input and output stored after the request, and where?
Training: Is your data ever used to train or fine-tune a model?
Access: Who or what can read your data while it is being processed — and is it isolated from other customers?
Logging: Does the content of your prompts or results end up in logs?
Transit and rest: Is the data encrypted on the wire and in storage?

Read terms carefully: on some interactive APIs the answers vary by tier or endpoint, and the defaults are not always the most protective option.

Of those five, training is the one people treat as abstract, and it is not. A model trained on text can be made to emit that text back. The canonical demonstration extracted data from a production-scale model using nothing but queries:

personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs
— Carlini et al., Extracting Training Data from Large Language Models (USENIX Security 2021)(opens in a new tab)

The detail that matters for a risk assessment is the frequency threshold. The authors note the attack succeeded on sequences "included in just one document in the training data" — so this is not a "your data is one drop in an ocean" situation. A single record, seen once, can come back out. That is why "is my data trained on?" deserves a written answer rather than a reassuring one.

Why open models on controlled compute help

A proprietary API is a black box: your data goes into infrastructure you cannot see, governed by a vendor whose model and data path you cannot inspect. Open-weight models change the structure of the problem. The model runs on compute the platform controls, the data never goes to a third-party model vendor, and — because the serving code can be open — the data path is auditable rather than asserted.

Tenant isolation and minimization

The strongest privacy postures do not rely on policy alone; they make isolation structural:

Per-tenant encryption: each customer data is encrypted under its own key, so reaching another tenant storage yields ciphertext wrapped under a key the reader cannot access. Isolation survives even an access-control mistake.
Data minimization: a worker receives only the one job it is running and the scoped, short-lived key to decrypt that job — never other tenants keys, never the rest of your corpus. A compromised worker exposes at most its own in-flight item.
No payload logging: operational logs carry job IDs, token counts, models, and error categories — never the content of prompts or results.
Short TTLs: inputs and outputs live in storage only as long as needed to deliver, then expire.

A privacy checklist before you send data

Confirm retention: how long is data kept, and can you get it deleted on demand?
Confirm training: get an explicit "never trained on" in writing.
Confirm isolation: is tenant separation cryptographic, or only logical?
Confirm logging: is payload content kept out of logs?
Confirm encryption: TLS in transit, encryption at rest, scoped key access on workers.
Prefer auditable over asserted: open serving code lets you verify the claims instead of trusting them.

Sources

Sources and further reading

Extracting Training Data from Large Language Models(opens in a new tab)
Carlini, Tramer, Wallace, Jagielski et al., USENIX Security 2021. Verbatim extraction of training data — including PII — from a production model, for sequences appearing in a single document.
Regulation (EU) 2016/679 (GDPR), official text(opens in a new tab)
The source rather than a summary of it. Chapter IV covers controller and processor obligations, which is the frame your inference vendor sits in.
Who is actually running your inference?
The gap none of the five questions above closes: inference needs your prompt in plaintext, on a specific machine.
Data residency: pick a boundary, not a region
The jurisdictional half of the same problem.

Frequently asked questions

Is my data used to train the model?

It depends entirely on the provider and tier, so get an explicit answer in writing. A batch platform built on open models has no incentive to train on your data — not retaining it is cheaper and lower-liability — so "never trained on" is a reasonable bar to insist on.

What does cryptographic tenant isolation mean?

Each customer data is encrypted under its own key. If someone reaches another tenant stored objects, they get ciphertext wrapped under a key they cannot access — so isolation holds even if an access-control check fails, rather than depending on policy alone.

Related guides

Who is actually running your inference?

Encryption does not protect a prompt during inference: the model needs plaintext in memory. That makes the machine, and whoever controls it, part of your risk.

Data residency for AI inference: pick a boundary, not a region

Compliance obligations are jurisdictional, not regional. Why a residency boundary is the right primitive for AI inference, and how a lock must behave.

Open-weight vs. proprietary models: when open is enough

Open-weight models have closed much of the quality gap with proprietary APIs. The real trade-offs in cost, control, privacy and quality for batch workloads.

What is batch AI inference?

Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.