| System | Precision | Recall | F1 | Failed requests |
|---|---|---|---|---|
| Synthector | 36.06% | 76.92% | 49.11% | 205 |
| Fastino GLiNER2-PII | 39.06% | 82.86% | 53.09% | 27 |
| Presidio | 21.17% | 52.78% | 30.22% | 0 |
| OpenAI Privacy Filter | 27.64% | 56.54% | 37.13% | 0 |
| Perplexity PII-Tracer | 31.77% | 75.60% | 44.74% | 0 |
| All systems use the same labelled values and exact-match scoring within this dataset. Failed requests remain in the scores. This comparison measures detection, not preservation or release safety. | ||||
Synthector: Privacy Controls Before AI
Detection, selective treatment, leak checking and verifiable execution
November 2026
Overview
Synthector processes text before it reaches an AI system. It identifies potentially sensitive information, applies deterministic assessment to decide which changes are permitted, and replaces or removes selected values while seeking to preserve useful context. It checks the resulting text for supported signs of remaining exposure and controls whether the output is released. Signed records allow the recorded execution to be verified. Under its retention doctrine, the supported synchronous API does not intentionally save submitted or returned text in application-controlled storage. Usage metadata, operational metrics and content-minimized execution evidence may be retained.
This paper evaluates Synthector on 8,688 SPY texts and 100,000 NVIDIA Nemotron-PII records, using detection scores and separate tests of required changes, preservation and release decisions. The results support evaluation of a defined workflow; they do not establish complete protection of arbitrary text.
1. Sensitive information before downstream AI
A support conversation may contain the account details an AI assistant does not need, alongside the facts it does need to resolve a complaint. Similar decisions arise when summarizing a meeting, searching documents or preparing text for an external service. Privacy protection must control sensitive information without unnecessarily destroying the context that makes the task useful.
Finding a name, organization or date is only the first step. Whether it should be changed depends on the available context and the intended use. A privacy boundary therefore needs to answer several questions: what was found, what should change, what must remain, and whether the resulting text can be released (McCallister et al. 2010).
The evaluated system combines rule-based checks with model-based detection. Detected values pass through a fixed decision procedure before any change is authorized. In this paper, treatment means an approved change to the text, such as replacing a private value with a category label. Preservation means leaving specified useful text unchanged.
Deterministic assessment, treatment and release
Given the same input, completed detection evidence, rules, request settings and software version, Synthector produces the same approved changes and release decision. Model-based discovery is probabilistic and can supply different evidence on another run. Determinism makes the final control procedure repeatable; it does not establish that detection is complete or that every privacy judgment is correct. Detection, preservation and release behavior therefore need separate measurements.
Retention doctrine
The supported synchronous API processes submitted text in memory and does not intentionally save submitted or returned text in application-controlled storage. Under Synthector’s service policy, customer text is used only to fulfill the processing request. The calling application controls storage of returned text. Usage metadata, operational metrics and content-minimized execution evidence may be retained to operate and verify the service.
2. Detection: what was tested and how to read the results
The datasets
SPY is a public synthetic dataset for testing the identification of personally identifiable information (PII) in context. Its 8,688 texts contain medical consultations and legal questions, with labels for names, addresses, contact details and other categories. It includes both personal and non-personal named information, making context relevant to the task (Savkin et al. 2025).
NVIDIA Nemotron-PII is a synthetic English-language dataset containing forms, invoices, emails and other text with labelled personal and health information. This evaluation uses 100,000 records, representing 50,000 examples, each in two versions, and reports results for the information categories listed below (Steier et al. 2025).
Here, labelled means that the dataset specifies a value’s type and location in the text. Both datasets contain fabricated examples and were used during development. Their records do not constitute independent real-world customer evidence.
Precision, recall and F1
The result tables use three detection measures, shown as percentages:
- Precision: of the values the system identified, how many matched a dataset label?
- Recall: of the labelled values in the dataset, how many did the system identify?
- F1: a single score balancing precision and recall. A low value in either measure reduces the combined score.
A correct match must have the same category and exact text boundaries as the label. A partial name or an overly broad address therefore does not count as a correct match. Scores combine the individual labelled values across each dataset, so common categories have more influence on the overall result.
F1 is one part of the evaluation. It does not establish that every required value was fully removed from returned text, that useful information survived, or that an unresolved request was withheld. Nor does it show that an execution record can be verified. Sections 4–6 examine those questions separately.
The comparison systems
Four available frameworks and models provide context for the detection results:
- Fastino GLiNER2-PII identifies and masks personal-information categories supplied to the model (Fastino AI Team 2026).
- Presidio is an open-source identification and anonymization framework combining language analysis, pattern matching and validation checks (Presidio Contributors 2026).
- OpenAI Privacy Filter is an open-weight model for context-aware detection and redaction of personal information (OpenAI 2026).
- Perplexity PII-Tracer is a model for identifying and masking personal information in conversational text (Perplexity AI 2026).
These represent alternatives a developer could use to find personal information. Table 1 and Table 2 compare recognition under the same dataset and matching rules. They do not compare the systems’ complete privacy workflows or establish which product is best for every use case.
For Synthector, the scored values are those its assessment approves for treatment; the other systems are scored from their detector outputs. Fixed comparator outputs were reused after compatibility checks. Failed requests remain included, with their labelled values counted as missed. The final column shows how many requests could not complete.
| System | Precision | Recall | F1 | Failed requests |
|---|---|---|---|---|
| Synthector | 79.65% | 65.16% | 71.68% | 98 |
| Fastino GLiNER2-PII | 63.17% | 32.70% | 43.09% | 372 |
| Presidio | 31.46% | 22.19% | 26.02% | 0 |
| OpenAI Privacy Filter | 70.32% | 45.10% | 54.96% | 0 |
| Perplexity PII-Tracer | 63.43% | 53.62% | 58.11% | 0 |
| All systems use the same labelled values and exact-match scoring within this dataset. Failed requests remain in the scores. This comparison measures detection, not preservation or release safety. | ||||
Synthector has the highest F1 among these tested configurations on Nemotron-PII; Fastino GLiNER2-PII has the highest on SPY. The difference between datasets is a reason to test a prospective workflow rather than rely on one aggregate score.
Comparison scope. Each tool uses a fixed configuration, rather than a separate optimization for each table. GLiNER2 was asked for seven categories, which map to six of the Nemotron scoring groups; misses in other groups remain in its score. The PII-Tracer configuration uses its 4,096-token input limit. These settings constrain the comparison and do not establish each tool’s maximum capability. Model or dataset exposure also prevents treating these results as independent held-out accuracy estimates.
Results by information category
The following tables show where the evaluated Synthector configuration performs well and where coverage is weak. A high overall score can hide a poor result for a category that matters to a particular buyer. The count of labelled values shows how much evidence each row contains.
| Category | Labelled values | Precision | Recall | F1 |
|---|---|---|---|---|
| Addresses | 9,295 | 57.39% | 95.45% | 71.68% |
| Email addresses | 8,594 | 35.57% | 92.93% | 51.45% |
| Identifiers | 7,641 | 70.36% | 66.17% | 68.20% |
| Person names | 8,739 | 19.21% | 87.06% | 31.47% |
| Telephone numbers | 7,819 | 43.21% | 95.29% | 59.46% |
| Web addresses | 8,565 | 31.81% | 72.21% | 44.16% |
| Usernames | 8,141 | 51.62% | 25.38% | 34.03% |
| Labelled values are the dataset's reference examples in each category. Final account_subject subtypes distinguish usernames from other identifiers. | ||||
| Category | Labelled values | Precision | Recall | F1 |
|---|---|---|---|---|
| Cities | 18,347 | 75.23% | 77.15% | 76.18% |
| Telephone numbers | 30,353 | 91.09% | 93.32% | 92.19% |
| Countries | 23,475 | 98.90% | 87.09% | 92.62% |
| Dates | 105,167 | 97.37% | 71.49% | 82.45% |
| Email addresses | 53,930 | 99.06% | 99.16% | 99.11% |
| Government identifiers | 10,211 | 91.06% | 65.67% | 76.31% |
| Organizations | 54,837 | 98.09% | 84.35% | 90.70% |
| Payment cards* | 12,867 | 99.33% | 11.57% | 20.73% |
| Person names | 143,639 | 48.78% | 34.57% | 40.46% |
| Postal codes | 6,279 | 68.90% | 85.65% | 76.37% |
| Regions | 26,783 | 98.09% | 76.44% | 85.92% |
| Mixed identifiers and tokens | 173,715 | 84.19% | 71.42% | 77.28% |
| Street addresses | 16,879 | 66.76% | 48.90% | 56.45% |
| Web addresses | 37,847 | 32.07% | 30.71% | 31.38% |
| * Recognition requires the correct category and exact boundaries. In the same executions, 9,378 of 12,867 labelled card occurrences were completely treated (72.88%; 76.29% of the 12,292 occurrences in returned requests), including 7,927 recognition misses. Returned requests contained 703 partially treated and 2,211 untouched occurrences. | ||||
| Labelled values are the dataset's reference examples in each category. 142 additional predictions outside the labelled categories remain included in the overall precision and F1 scores. | ||||
These are detection categories, not guarantees that every value in a category can be found or should be removed. Extra predictions that do not match a dataset label count against precision, but a missing label does not prove that the underlying text is harmless.
How this build was measured
The principal study is bound to source 93f2d73f9222bd95d23070ffa62a3d552014d129 and ruleset 2026.10.27.1. The evaluated engine processed both datasets using saved model outputs checked for compatibility with this build. All subsequent processing ran in this build, with signed records produced for completed requests. Requests that could not complete remain in the results. Both tested settings of the leak-check and release controls were exercised; the tables report one result per input, with the same recognition and release outcomes under both settings. Separate API tests ran the actual models. This quality evaluation is not a latency or throughput measurement.
3. API performance
The principal build was exercised through its supported API on CPU and NVIDIA L4 GPU deployments. Selected tests checked the returned text, withholding behavior and signed records with the actual models. These finite checks do not establish full-dataset GPU accuracy or service performance for that build.
Separate, earlier measurements compare API processing with leak checking and release control on three deployments. Each processes the same 1,013-input panel twice, sequentially. The per-request timer includes authenticated HTTP, actual model execution, text processing, signing and verification of the execution receipt; the subsequent independent attestation check is excluded. Table 5 reports the time below which 50%, 95% and 99% of requests completed. Returned outputs per second divides the number of usable returned outputs by total observed wall time, including the attestation check and recording. Withheld requests consume time but do not count as returned output.
| Deployment | p50 (ms) | p95 (ms) | p99 (ms) | Returned outputs / second |
|---|---|---|---|---|
| c7i.xlarge (CPU) | 213.32 | 749.59 | 796.92 | 3.00 |
| c7i.2xlarge (CPU) | 259.82 | 1,018.76 | 1,116.90 | 2.37 |
| g6.xlarge / full L4 (CUDA) | 116.43 | 192.70 | 231.04 | 6.36 |
| Each deployment processed 2,026 requests: 1,620 returned outputs and 406 withheld outputs. Returned outputs per second uses the full sequential observation wall time. These historical timings do not measure the principal study build or concurrent capacity. | ||||
The CPU measurements use source f46e304ef16d11115c6173443f9de78485c7508b; the NVIDIA L4 measurement uses 96239eac769570c8d51e5e85ac7529f41c2ecfb0. They are historical deployment observations, not timings for the principal 93f2d73f study. The comparison describes complete tested deployments, rather than the isolated effect of a processor. These sequential measurements do not establish cold-start behavior, peak concurrent capacity or a service-level guarantee. Full-dataset GPU detection scores remain unmeasured.
4. Deciding what to change and what to keep
Detection proposes a possible sensitive value. Synthector then assesses the evidence available for that occurrence: the value’s format, the surrounding context, its role in the text and the changes permitted by the configured rules. A model’s confidence score does not itself authorize a change or establish that a value is private. The evaluated configuration uses a fixed set of treatment rules.
This distinction matters for usefulness. A private affiliation may require treatment, while an operational detail may need to remain available to the downstream task. The current procedure supports specific distinctions of this kind; the preservation tests below show that it does not handle them correctly in every case.
A worked example
This synthetic example was executed through the evaluated build’s API with the actual models. The organization name is changed while the heading and field wording remain intact.
Input
Private affiliation
My employer: Aster Meridian Fabrication.
Returned text
Private affiliation
My employer: [ORG_NAME].
The execution record and its signature were verified against this request and returned text. The example demonstrates the observed change and surrounding preservation; it is not an estimate of overall accuracy.
Did required changes happen, and did useful text survive?
In separate reviewed examples, the tests specify which values must change and which pieces of text must remain unchanged. A check concerns a particular value or piece of text; one document can contain several checks. A required change passes only when it is complete and the document is returned. Withholding a document does not count as successful treatment or preservation.
Table 6 keeps the test groups separate. They include development examples and some shared source material, so their counts must not be combined into one independent success rate.
| Test group | Required changes completed | Keep-text checks passed |
|---|---|---|
| Targeted treatment tests | 88 of 88 | 100 of 100 |
| Additional treatment tests | 20 of 20 | 40 of 40 |
| Broader preservation test | Not assessed | 301 of 386 |
| A check concerns a specified value or piece of text. The groups are separate and must not be pooled. The broader test found 85 checks affected by over-redaction across 37 of its 100 texts; all 100 texts returned output. | ||
In the two groups with specified required changes, none of the values required to change remained exposed in returned text. This result applies only to those reviewed examples.
Across 100 synthetic texts, 301 of 386 preservation checks passed. The remaining 85 checks were affected by over-redaction: text designated for preservation was replaced by a privacy placeholder. These changes occurred in 37 texts; all 100 texts returned output.
The accompanying evidence/preservation/ files contain all 100 synthetic inputs and outputs, all 386 check outcomes and a verifier, so the 85 modifications can be inspected. This bundle covers the broader synthetic suite; the separate targeted-duty source material is not included.
5. Checking the output before release
After treatment, a leak check examines the resulting text for supported signs of remaining sensitive information or unresolved requirements. The release gate controls whether that text leaves Synthector. If the check finds remaining exposure that the rules require to be blocked, the gate withholds the output. Processing can also stop when the evidence needed for a complete, permitted change is unavailable or incomplete.
This check is limited to the risks the system can assess. Passing it does not prove that detection found every sensitive value. It adds a release control after treatment; it does not eliminate the need to measure detection and preservation.
Table 7 distinguishes three outcomes. Returned means the request completed and supplied text. Withheld means the release conditions were not satisfied and no output was supplied. Could not complete means the required processing or assessment could not finish; no output was supplied in these cases either.
| Dataset | Inputs tested | Output returned | Output withheld | Could not complete |
|---|---|---|---|---|
| SPY | 8,688 | 8,200 | 283 | 205 |
| Nemotron-PII | 100,000 | 98,287 | 1,615 | 98 |
| Counts describe one complete pass through each dataset, not pooled repetitions. Incomplete requests released no output. Withholding contains exposure and reduces availability; it is not successful treatment. | ||||
The 205 SPY and 98 Nemotron requests that could not complete lacked sufficient evidence for a complete permitted treatment. Their model responses had completed; these were not model timeouts. The refusal protects against an unsupported release and reduces availability. Whether that tradeoff is acceptable depends on the workflow.
6. Evidence the customer can verify
An integration needs to know what happened to a particular request. Synthector records the evaluated version, the processing decision and the relationship between the request and any returned text. A signed attestation makes that recorded evidence verifiable: the recipient can check its integrity and that it was signed by the expected key.
For a completed request, this can help establish which version ran, what release decision it recorded and which output belongs to the request. It does not prove that every privacy decision was semantically correct, that every sensitive value was detected, or that a workflow meets a legal obligation. Failed requests do not receive an invented successful attestation.
7. Evaluating your workflow
For a prospective integration, start with representative synthetic or approved non-confidential examples. Specify the information that must be changed, the text that must remain useful and how the application should handle withheld requests. Then measure those outcomes and response time in the intended deployment.
The evidence in this paper supports that evaluation: Synthector combines measured detection with a defined procedure for approving changes, checking output and recording release decisions. Its remaining detection gaps, preservation checks affected by over-redaction and refused requests are part of the product’s current limits. Independent external validation remains outstanding.
Evaluate your workflow: request evaluation access or contact contact@synthector.com. Describe the text you process, what must be protected and what must remain useful.
© 2026 Christopher R. Runowski / Synthector. This document is provided for technical evaluation and discussion. Redistribution or publication requires the copyright holder’s permission, except where permitted by applicable law. No rights or licenses to Synthector software or other intellectual property are granted by this document.