A company swaps production records for generated data, calls the new dataset privacy-safe, and grants a wider group of developers access to it. The access problem looks solved. The privacy questions do not disappear: who controlled the source data, what the generator retained, which people remain inferable, and whether the artificial dataset still performs reliably for its intended job.
Synthetic data can reduce direct exposure to personal information, but it does not erase privacy risk by default. Without formal guarantees, adversarial testing, and continued governance of the source data, a synthetic data privacy claim can move risk into a system nobody inspects closely. Organizations deploying synthetic data without answering the questions above have not solved a privacy problem. They have renamed it.
Synthetic Does Not Automatically Mean Anonymous
Synthetic data consists of artificially generated records built to reproduce statistical properties, relationships, or patterns found in a source dataset. The category spans rule-based generators, statistical resampling, generative adversarial networks, variational autoencoders, diffusion-based tabular generators, large language models producing synthetic text, and differentially private generators. The privacy properties differ sharply across all of them, for reasons rooted in how each type learns from source data.
Regulators draw a line marketing language blurs. Synthetic describes the production method. Anonymous describes whether people remain reasonably identifiable. Pseudonymous data still relates to identifiable individuals when additional information exists somewhere, and differential privacy offers a mathematical guarantee only when correctly implemented and parameterized.
The UK Information Commissioner’s Office states the distinction plainly in its anonymisation glossary: synthetic data “may or may not be anonymous.” One sentence from a regulator punctures the assumption the word “synthetic” functions as a legal classification.
Why generator type matters: a GAN learns from real records by having a generator network compete against a discriminator until the output is statistically indistinguishable from real data. Memorization can occur when the generator overfits to rare training examples, essentially copying them rather than learning a generalizable pattern.
A differentially private generator adds calibrated mathematical noise during training and provides a quantified bound on information leakage, but only for the specific queries, composition, and privacy budget the system defines. Large language models carry a qualitatively different memorization risk. Carlini and colleagues published a training data extraction attack in 2021 demonstrating GPT-2 reproduced verbatim sequences from its training corpus through targeted prompting: real names, phone numbers, and email addresses appeared in the extracted output. A synthetic text generator trained on clinical notes, legal correspondence, or customer communications can memorize and later reproduce fragments of personal information even when individual output documents appear entirely novel.
The Generator Still Depends on Real Data
Vendor messaging hides an upstream dependency. Many synthetic datasets originate from sensitive real-world records, and an organization must still collect, secure, minimize, document, and lawfully process the records before generating anything. Synthetic output does not retroactively fix excessive collection, weak consent analysis, or missing data-protection impact assessments.
A synthetic-data project can reduce how many employees touch production records, but the source environment remains a high-value target. The generator itself can become sensitive: anyone with access to model parameters or repeated query interfaces may extract information the released dataset alone would not reveal.
Before generating a single synthetic record, an organization also needs to address purpose compatibility under GDPR Article 6(4). If the original data collection served customer service, clinical care, or fraud detection, then training a generator on the records is a secondary use. GDPR requires a compatibility assessment to determine whether the secondary purpose is sufficiently connected to the original, or whether the processing requires a new lawful basis. Many organizations treat synthetic generation as a neutral technical operation and skip the compatibility assessment entirely. The gap in documentation appears before any data leaves the building.
Artificial Records Can Still Leak Membership
A membership inference attack asks whether a particular individual appeared in a generator’s training data. An attacker does not need the full original record: confirmation of membership in a sensitive population already reveals information, whether a person’s appearance in a cancer-treatment dataset, an employee’s presence in a disciplinary population, or a customer’s appearance in a fraud-investigation cohort.
Overfitting drives the risk. A generator learning rare records too closely assigns the records, or close neighbors of them, unusually high probability. Van Breugel and colleagues built DOMIAS, a density-based attack exploiting the overfitting weakness directly. Published at AISTATS in 2023, the research showed local overfitting leaves uncommon samples particularly exposed, with direct consequences for minority and underrepresented groups.
LLMs create a distinct version of the same problem. Carlini and colleagues demonstrated GPT-2 reproduced verbatim text from its training data, personal identifiers among the extracted content, through targeted prompting alone. A synthetic text generator trained on patient notes, employee records, or financial communications carries comparable risk. Synthetic documents from an LLM do not expunge personal information the model memorized during training.
The defensible claim stops well short of saying attackers can reidentify every synthetic dataset. Synthetic generation alone does not establish protection against membership inference or LLM memorization attacks.
Attribute Inference and Model Inversion Target Different Vulnerabilities
Membership inference tests whether a specific individual was in the training data. Two other attack classes target separate vulnerabilities and require separate defenses.
An attribute inference attack does not require the attacker to know whether a specific person appears in the training data at all. Instead, the attacker uses the synthetic dataset’s statistical patterns combined with auxiliary information to infer sensitive attributes about real individuals. If a synthetic dataset reliably encodes the correlation between occupation, age range, and a particular health condition, an attacker holding two of the three attributes about a real person can infer the third, without any direct link to a specific synthetic record. The attack exploits what the distribution has preserved, not what any individual record contains.
Model inversion attacks take a different route. An attacker queries the generator or a downstream model trained on synthetic data and uses the responses to reconstruct approximate representations of training inputs. If the generator or a classifier built on synthetic data is accessible through an API, repeated structured queries can probe what the model internalized about rare or sensitive records. The attack does not require access to the source data directly.
Neither attack class is easily scalable against well-designed systems. Attribute inference requires an exploitable correlation structure in the output. Model inversion requires sustained query access. “Not easily scalable” is not the same as “infeasible in a targeted attack,” and organizations holding sensitive population data cannot treat attack difficulty as a substitute for measurement.
Privacy and Fidelity Pull in Opposite Directions
A useful synthetic dataset reproduces enough of the source distribution to support development or model training. A private dataset avoids reproducing information too closely, and progress on one objective tends to damage the other. High fidelity preserves rare combinations, which raises disclosure risk. Stronger privacy protections add noise or flatten the distribution, and the flattening erases outliers, tail events, and minority subgroups. A dataset can score well on average while failing on the exact cases mattering most.
NIST’s SP 800-226, finalized in March 2025, warns synthetic generation can reduce accuracy for subpopulations and contribute to systemic bias. The output adds another layer of uncertainty on top of whatever already exists in the source data. Qian and colleagues, in a 2024 Scientific Reports study building prognostic models for lung cancer from a synthetic UK Biobank cohort, demonstrated synthetic data can substitute for real data throughout the clinical risk prediction pipeline, with the approach working well under the study’s conditions.
Model performance varied across the three privacy-preserving generators tested, a finding the authors attribute to differences in how each generator handles the privacy-utility trade-off. Close aggregate performance is not a guarantee of adequate performance in every task or under every generation method.
When a generator uses differential privacy, the privacy-utility trade-off becomes explicit through a parameter called epsilon (ε). Epsilon quantifies how much information about any individual can leak into the output. With epsilon at 0.1, the output distribution looks nearly identical whether or not any single person was in the training set: the attacker learns almost nothing about individuals, but the noise required at low epsilon can damage output quality substantially. An epsilon of 1 provides meaningful but imperfect protection and sits within the typical working range for applied differential privacy systems. Above 10, most privacy researchers consider the protection limited against a determined attacker. The U.S. Census Bureau used an overall epsilon of approximately 19.61 for the 2020 Decennial Census, a choice the research community debated extensively given the limited individual-level protection it provides.
The privacy budget also composes across multiple uses. Every query, evaluation run, or data release drawing on the same differentially private generator consumes a share of the total budget. An organization running repeated model evaluations, hyperparameter sweeps, or multiple staged releases against the same source can exhaust meaningful protection faster than the governance record reflects. Vendors reporting a privacy parameter without addressing composition or budget management have answered only part of the question.
Synthetic Data Can Preserve or Amplify Bias
A generator learns from the evidence it receives. It cannot distinguish patterns reflecting reality from patterns created by historical discrimination or measurement error unless the development process addresses the difference directly. Generated records can replicate representation gaps, flatten small populations, render majority groups more accurately than minority ones, and reinforce correlations baked in by biased decisions. Plausible-looking records can lack real-world causal validity while appearing orderly, and the polish builds false confidence rather than removing risk.
Bias propagation is not inevitable. Domain experts who understand a population can use synthetic data to build controlled edge cases or increase representation deliberately. A minority pattern learned from insufficient data differs from invented trustworthy evidence: additional records generated from a weak sample multiply the assumptions baked into the model rather than producing new clinical or behavioral knowledge. Synthetic data can rebalance a training table. It cannot manufacture ground truth.
The EU AI Act created a narrow but important exception in Article 10(5). High-risk AI system providers may process special categories of personal data, including health records, ethnic origin, and biometric data, strictly for bias detection and correction. The provision acknowledges a real limitation: synthetic and anonymized data frequently fail to reproduce the fine-grained statistical patterns needed to identify discrimination, which is why real sensitive data may be the only viable option in some cases.
The safeguards under Article 10(5) are strict: access must be documented and tightly controlled, the data cannot transfer to third parties, and deletion requirements apply once the bias correction purpose concludes. Article 10(5) opens one carefully bounded route for organizations meeting the conditions, not a general license for processing sensitive data in AI development.
Regulation Still Evaluates the Processing, Not the Label
The GDPR and UK GDPR identifiability test is not a binary pass/fail. Recital 26 and subsequent regulatory interpretation ask whether identification is achievable using means “reasonably likely to be used.” The assessment weighs four factors: the cost and time needed to identify a person, the technology available at the time of assessment and in the foreseeable future, the identity and capabilities of likely attackers and not just the data controller, and any additional information accessible in the surrounding environment. Applied to synthetic data, the test evaluates not just the released dataset but the generator, the model artifacts, the source data, and any auxiliary datasets a plausible attacker could combine with the output.
EU GDPR and UK GDPR apply the same functional standard but operate through separate regulatory frameworks post-Brexit. Under EU GDPR, a lead supervisory authority oversees cross-border processing through the one-stop-shop mechanism. The ICO is the relevant authority under UK GDPR and operates independently of EU coordination. Organizations subject to each regime, which covers most multinationals with European customers, need governance meeting each standard, not merely the standard at headquarters.
The EDPB’s Opinion 28/2024 raised the evidentiary bar further. The Board held an AI model trained with personal data is anonymous only when the controller can demonstrate, affirmatively and with evidence, personal data cannot be extracted from the model using all reasonable means. The burden of proof rests with the controller, not with regulators to prove a breach. For synthetic data generators, Opinion 28/2024 requires controllers to document why extraction is infeasible, not merely assert it. If pseudonymization is the only step taken, GDPR obligations continue in full.
HIPAA offers two recognized routes to de-identify protected health information: Safe Harbor, which requires removing eighteen specified identifiers, and Expert Determination, which requires a qualified statistician to certify the risk of identifying any individual is very small. Neither route activates automatically because a vendor used a generative model. An organization applying Expert Determination must assess re-identification risk against realistic adversarial scenarios, not just obvious direct identifiers. NIST SP 800-188, published in September 2023, provides a technical framework for evaluating de-identification approaches, including synthetic generation, against disclosure risk standards. A healthcare organization cannot delegate the Expert Determination to the synthetic data vendor’s documentation.
The EU AI Act treats synthetic data as one possible governance tool, not a blanket compliance defense. Article 10 still requires high-risk systems to use training data staying relevant, representative, and suitable for the intended purpose, regardless of the production method.
Organizations with sensitive source data must also address the Data Protection Impact Assessment requirement. Article 35 of GDPR requires a DPIA before processing in ways likely to create high risk for individuals. France’s CNIL has recommended DPIAs for AI training involving large-scale processing, special category data, or novel content types. Health, financial, or biometric records fed into a generator almost certainly meet the DPIA threshold in most EU and UK contexts.
A DPIA for synthetic data generation should address the source data and lawful basis, the generation pipeline and all artifacts, the privacy testing methodology, the intended uses of the output, and governance for model access and retention.
Where Synthetic Data Earns Its Keep
Synthetic data performs best when a team defines a narrow use and validates output against it, rather than approving a dataset for general purposes.
Developers can test schemas and pipeline logic without copying production records into lower-security environments. Vendors can demonstrate software through synthetic accounts instead of real customer records. Engineers can simulate known rare events, such as sensor failures or edge-case transaction sequences, provided they specify the scenario rather than expecting a generator to discover unknown patterns from sparse data.
Differentially private synthetic data can support broader statistical sharing when an organization documents a defensible privacy budget and verifies utility for the intended analyses. A smaller number of people with routine access to production data lowers operational risk even when the output falls short of full anonymity. Synthetic data should reduce reliance on sensitive records, not replace real-data validation wherever real-world performance matters.
Third-Party Services Add a Separate Governance Layer
Many organizations do not build generators in-house. Cloud-based synthetic data services require uploading source data to vendor infrastructure. Under GDPR, a vendor receiving and processing personal data on behalf of an organization becomes a data processor and requires a documented data processing agreement covering access controls, sub-processor chains, international transfer mechanisms, data retention and deletion timelines, and breach notification obligations. The vendor’s security posture, staff access policies, and infrastructure geography are all within scope of the controller’s accountability.
Standard Contractual Clauses and an adequacy decision address the transfer mechanism. Vendor risk management requires going further: assessing the vendor’s access to model artifacts, retention of training data after contract termination, query interface logging, and whether vendor staff can probe the model in ways capable of exposing training data. For organizations facing strict data residency requirements, on-premises or private cloud deployment is frequently the only architecturally defensible path. Vendor residency options belong in the procurement checklist, not in a post-deployment audit.
A vendor calling a dataset “privacy-safe” in documentation is marketing language, not a legal or technical certification. A data processor’s compliance attestation does not transfer controller liability to the processor. A supervisory authority investigating an incident will examine the controller’s accountability record, not the vendor’s product description.
When Synthetic Data Crosses Organizational Boundaries
Much of the interest in synthetic data comes from the promise of safer sharing: research consortia wanting multi-site analysis without centralizing records, regulators expecting banks to share transactional patterns without exposing customer data, hospitals partnering with pharmaceutical companies on clinical distributions. The evidentiary burden in cross-organizational sharing is substantially higher than for internal development and testing.
When synthetic data leaves the releasing organization, it enters a less controlled environment. The recipient may combine the synthetic dataset with proprietary records, publicly available data, or inference models the releasing party never anticipated. The GDPR identifiability test evaluates the full population of likely attackers, not just the releasing organization’s capabilities, which means a cross-organizational release faces a broader adversarial set than internal use of the same dataset.
The European Health Data Space Regulation (Regulation (EU) 2025/327), which entered into force in March 2025, provides the clearest statement of where regulators now stand. Under the EHDS framework, Health Data Access Bodies must require applicants to document safeguards proportionate to re-identification risk before granting access to health data, whether real or synthetic.
The regulation’s technical body TEHDAS2 developed specific guidelines on synthetic data pre-release checks, concluding a public consultation in November 2025, and noted explicitly: synthetic data generation pipelines are prone to identity leakage and memorization, elevating re-identification risk. The EHDS framework treats synthetic data as one possible output format within a governed secondary-use architecture, not as an alternative to oversight.
Regulators and research ethics bodies increasingly view synthetic data as a complement to trusted research environments, secure computation infrastructure, and data access agreements, not a replacement for any of them. An organization proposing synthetic data as an alternative to a regulated access pathway needs documented justification and formal risk assessment, not confidence in the output alone.
Where Teams Commonly Overestimate It
Five assumptions cause the most damage in practice. A model trained entirely on synthetic data can work in simulation-heavy domains, but the approach risks a gap between simulation and reality, where models learn generator artifacts instead of real relationships.
An assumption holds: no real people appear in the output, so privacy law no longer applies. The assumption needs evidence, since synthetic records can retain information about training participants or stay linkable through outside data. No universal privacy score exists, since results depend on the attack model, population, generator, and release environment. Aggregate fidelity can coexist with broken tail behavior, so close average performance does not equal accuracy where it counts. Differential privacy guards against particular disclosure risks, not bias or downstream misuse, and a weak privacy parameter offers limited protection while a strong one can damage utility enough to defeat the purpose.
A Better Evaluation Framework
Six questions separate a defensible synthetic-data program from a label:
- Define the intended use. Document what the data will support, whether development testing, statistical analysis, model training, or external sharing, and treat approval for one stated use as approval for the stated use alone. Scope creep is a common failure mode: a dataset approved for schema testing gets repurposed for model training without a fresh privacy assessment.
- Map the complete data flow. Identify sources, lawful basis, generator, hosting environment, all model artifacts, query interfaces, retention periods, and downstream recipients. Include third-party service providers and their sub-processors in the map. Fold the source data and the generator into the threat model, not the synthetic output alone.
- Demand an explicit privacy claim. Ask whether the claim amounts to reduced exposure, de-identification, anonymity, pseudonymization, differential privacy with a stated epsilon and composition policy, or resistance to a defined set of attacks. Each carries a different evidentiary burden. A vendor unable to answer with one of the specific claims, backed by supporting evidence, has not answered the question.
- Test realistic attacks. Run membership inference, attribute inference, nearest-neighbor distance analysis, and record linkage checks under realistic attacker knowledge rather than the most permissive black-box scenario. True positive rate at low false positive rate matters more than aggregate AUC for membership inference evaluation. Open tools including SynthEval, TAPAS (Toolbox for Adversarial Privacy Auditing of Synthetic Data), and the Adversarial Robustness Toolbox support structured testing without bespoke attack engineering.
- Validate utility locally. Measure performance by demographic subgroup, rare class, tail event, and downstream task, then compare models trained on synthetic data against real-data baselines. Close aggregate similarity scores do not guarantee adequate performance in the specific use case.
- Maintain governance after release. Track dataset versions, generator updates, privacy test results, approved uses, recipients, and performance drift over time. Treat material changes in source data, generator architecture, privacy parameters, or intended use as triggers for reassessment rather than relying on a one-time certification.
Synthetic Data Belongs Inside the Privacy Strategy
Synthetic data earns its place when it reduces unnecessary access, supports safer experimentation, and works alongside privacy engineering and formal governance. It turns dangerous as a vocabulary trick: real data goes into a generator, synthetic data comes out, and everyone assumes the privacy work ended there.
The strongest synthetic data privacy programs treat generation as one control inside a larger system. Membership inference research, NIST standards, and EDPB opinions converge on the same conclusion: calling data synthetic shifts what an attacker needs to do, but it does not eliminate what an organization must document, test, and govern. Synthetic data changes the dataset. A privacy strategy governs the entire system around it.
The post Synthetic Data Won’t Save You From a Bad Privacy Strategy appeared first on .
