The thesis
The pilot let both sides keep their secrets.
The model developer does not hand over the model weights. The evaluator does not reveal the private test prompts. Both sides send their assets into an agreed process. Only the approved result leaves.
This moves confidentiality beyond a contract. But a TEE still opens the prompts inside trusted hardware. Mirror removes that step for supported models. The model runs on encrypted values, and the remote workers never receive the evaluator's secret key.
With TEE, privacy follows trusted hardware. With Mirror, privacy follows the key.
01 / What happened
What the first pilot proved
The July and August 2026 pilot used a real proprietary model and a private set of previously unused test prompts.
Google DeepMind sent its model and inference software into the protected environment.
AVERI encrypted prompts under a private key that no other party held.
Google Cloud Confidential Space and PySyft created and verified the protected environment.
AVERI decrypted and graded outputs, then shared an approved report without disclosing cases.
The model owner publishes a mock interface and both parties prepare private assets.
Both parties review redacted code and the operations it may call.
Each party checks that the protected hardware is fresh and runs the approved software.
Private assets stream in. Only the pre-agreed result is released.
02 / What changes at scale
A bigger model needs a bigger trusted area.
Compare the reported pilot, a TEE spread across a cluster, and Mirror Encrypted Inference. The key question is simple: where can the prompt be read?
TLS(ciphertext){ "prompt": "private case..." }
The organization cannot inspect the prompt, but the approved enclave program can use it.
approved resultTrust the verified hardware and approved software.
Network and storage encryption protect the benchmark before it arrives. The approved program reads it during inference. Privacy depends on the verified hardware and software doing exactly what was approved.
- Cloud operator sees
- Encrypted memory
- Approved program sees
- Prompt plaintext
- Decryption key held by
- Protected workflow
Approved case output and metrics, according to the release policy. Never the model weights.
The approved total and run evidence. Never the private prompts or complete case transcripts.
Public sizes and schedules. The approved enclave program can use benchmark plaintext.
Which dataset, model and scorer ran, which encryption settings were used, what could be released, and whether the run record is valid.
Live Mirror proof
Encrypt the case. Grade the answer. Release only the score.
This demo uses one published MLCommons AILuminate prompt. GLM-5.3 Flash answers through encrypted inference. A second encrypted GLM-5.3 Flash call applies a fixed judge rubric. The page receives the aggregate result, not the model generation.
READY
- Release prompt ID
- airr_practice_1_0_26782
- Hazard
- Privacy violations / prv
- Persona / locale
- unskilled / en_US
- Published hash
- 01740bdb6b9f…59d06df0b
The customer endpoint loads the published prompt before anything is sent to the model.
airr_practice_1_0_26782
Mirror checks that the readable prompt does not appear in the request sent to the model.
awaiting run
The remote model receives ciphertext and returns an encrypted generation to the private evaluation worker.
mirror/glm-5.3-flash
The subject model returns ciphertext to Mirror. Mirror carries the result into the fixed judge flow. Only the aggregate is released.
awaiting run
The API omits the subject generation, per-case transcript, SDK, key material and ciphertext.
not released
No proof has run yet.
Waiting for a successful run.
- Bound evaluation manifest
- Pending
- Readable case in model requests
- Pending
- Subject and judge received
- Pending
- Subject generation returned
- Pending
- Round trip
- Pending
What this proves: two real protected model calls complete for one fixed published case. The subject response crosses back as ciphertext, and the browser receives only the aggregate. This one-case result is not an official MLCommons score.
03 / Mirror Encrypted Inference
How Mirror runs a private evaluation
Fully Homomorphic Encryption, or FHE, lets the model compute on encrypted data. The server does not need to open the test prompt first.
Encrypt one fixed version of the benchmark
The evaluator encrypts every case before upload and records exactly which set will be used.
Encpk(Deval)
Record the exact model and scorer
The run records the model, scoring rules, release policy and encryption settings.
H(M) + H(S) + H(D)
Run the model on encrypted data
The provider keeps its model weights. Supported model operations use encrypted values and return an encrypted result.
Eval(M, Enc(D))
Keep each case result private
Mirror can combine encrypted pass or fail values and bounded scores without opening each result.
Σ Enc(scorei)
Open only the agreed result
The evaluator keeps the secret key. The model owner receives the approved total, not the private prompts or full case transcripts.
Decsk(Enc(total))
Create a signed record of the run
The record identifies the dataset, model, scorer, result, supporting evidence and completion time.
Sign(receipt)
The evaluator encrypts before upload.
The model provider and cloud receive encrypted cases. Recording the benchmark version also prevents the test set from changing silently between runs.
04 / Why Mirror does not require a TEE
TEE protects an environment. FHE keeps the data unreadable.
Plaintext is usable inside the trusted environment.
A TEE isolates model weights and readable benchmark data from the surrounding host. Every device and program that can read the data must be trusted.
- Useful for provider-owned model protection
- Requires hardware and software attestation
- More hardware must be trusted as the model grows
Evaluator data stays encrypted during inference.
The customer encrypts before upload. Mirror runs supported model operations without giving the serving cluster the secret key.
- Evaluator-owned decryption key
- Ciphertext across remote workers
- Encrypted output and private aggregation
More compute does not mean more readers.
Workers can share encrypted model state and intermediate values. The model weights remain with the provider.
- Spread encrypted work across more GPUs
- Keep model memory and session state encrypted
- Only the evaluator opens the result
Mirror does not require a TEE to protect evaluator data. A provider can still use a TEE to protect its model from its own cloud or hardware operator. Mirror keeps the benchmark unreadable with FHE, whether or not the provider also uses a TEE.
05 / Frontier scale
More GPUs. The evaluator still holds the only key.
The model can spread across one GPU, many GPUs, or many machines. The prompt, working state and result stay encrypted. The evaluator keeps the only decryption key.
Enc(Deval)
W0W1W2WnLarger models need more FHE work. Parallel workers divide that work without receiving the decryption key.
The model's KV cache and session state stay encrypted throughout prompt processing and answer generation.
Mirror can schedule many cases and combine their encrypted scores without exposing individual prompts or findings.
One encrypted benchmark can test many models. Every run gets its own model, scorer, result and signed record.
More work, not more exposure.
A larger model does more encrypted work and moves more ciphertext between workers. What does not change is who can read the data. The provider keeps its model. The workers do not get the evaluator's key. Only the evaluator opens the approved result.
model scale → more ciphertext work
decryption authority → evaluator only
Mirror keeps batching, distributed model execution, paged KV cache, streaming and runtime evidence. The evidence records the serving layout and protection state without logging prompts, tokens, model state, logits or answers.
- Scheduler batching
- Tensor parallel
- Paged KV state
- Streaming
- Topology evidence
The rule is simple: a larger model needs more encrypted compute. It does not need more people or machines with permission to decrypt. The evaluator's key stays outside the serving cluster.
06 / The Mirror implementation
How Mirror protects the full evaluation run
The evaluator encrypts before upload and decides what can be opened at the end. The remote environment never receives the private benchmark in readable form.
fixed encrypted versionPrompts, version, case count, customer, project and customer-held key
encrypt firstThe benchmark becomes FHE ciphertext before it reaches the remote model service
run on ciphertextSupported model operations run without giving remote workers the evaluator's secret key
encrypted KV cacheWorking memory, intermediate values and session state stay encrypted while the answer is generated
encrypted outputCase results and bounded scores can stay encrypted while Mirror combines them
signed run recordThe dataset, model, result, case count, supporting evidence and completion time
What this unlocks
A private test set can be used again and again.
When model providers never receive the test questions, the same benchmark can support independent comparisons, recurring certification and cross-provider evaluation without becoming training data.
- 01Reusable private test sets
Run the same encrypted benchmark repeatedly without distributing its questions.
- 02Independent certification
Tie a result to the exact model, scoring system and dataset version.
- 03Cross-lab evaluation
Compare systems while each party retains control of its private asset.
- 04Safer agent evaluation
Extend the boundary from one model call to tools, retrieval, memory and multi-turn behavior.
Conclusion
Double-blind evaluation does not need a shared plaintext box.
The first production pilot showed that a TEE can protect readable data inside trusted hardware. Mirror takes the next step. The provider keeps its model. The evaluator sends ciphertext. Supported inference and scoring run without a server-side secret key. Only the evaluator opens the agreed result. The provider may still use a TEE to protect its own model, but evaluator privacy no longer depends on it.
Model owners keep their intelligence. Evaluators keep theirs. Only the agreed result and evidence cross.Explore Encrypted Token Factory
Primary sources and related systems work
- AVERI Pilot Report: The World's First Double-Blind Evaluation of a Proprietary Language Model
- Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing
- Google DeepMind: Piloting the world's first double-blind AI evaluations
- MLCommons AILuminate safety methodology
- Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache
- vLLM: Parallelism and scaling
- NVIDIA: Hopper multi-GPU attestation