Overview
We design synthetic genomics datasets using a repeatable, configuration-driven pipeline. This methodology ensures consistency, transparency, and reproducibility across all datasets in our catalogue.
Core Principles
- Literature-grounded: Parameters are anchored in peer-reviewed studies, reports, and registries wherever possible.
- Multi-ancestry with a focus on SSA: Populations from Sub-Saharan Africa and African diaspora groups are explicitly modelled, not treated as afterthoughts.
- Configuration-first: YAML configuration files define cohort structure, distributions, and dependencies.
- Transparent validation: Each dataset is validated against its configuration and expected qualitative trends.
- Synthetic by design: No real individual-level data is used. We simulate from probability distributions and structural assumptions.
Pipeline Overview
We use a common, modular pipeline for each dataset module. Each step is implemented as a Python script, tailored per dataset but following a shared pattern.
Literature Inventory & Expert Priors
Collect and summarize empirical evidence from peer-reviewed studies
Configuration Design (YAML)
Encode cohort design, distributions, and dependencies in machine-readable form
Generation Script
Sample synthetic cohorts and data from the configuration
Validation Script
Check alignment with configuration and qualitative expectations
Packaging & Publishing
Export CSV/Parquet, produce README card, upload to Hugging Face
Step 1: Literature Inventory
For each dataset, we maintain a LITERATURE_INVENTORY.csv summarizing key references:
- Study identifiers: PubMed ID, DOI, or report URLs
- Population: which geographies and ancestries are studied
- Endpoints: mutation rates, biomarker distributions, survival curves, stage distributions
- Extracted parameters: means, standard deviations, prevalences, hazard ratios, effect sizes
We use these to inform overall sample size, population mix, age/sex distributions, disease stage and subtype distributions, genomic profiles, longitudinal dynamics, and technical features.
Expert priors: When literature is sparse, we use expert priors that are clearly documented, conservative, and revisable.
Step 2: Configuration Design
For each module, a YAML configuration file describes the complete dataset specification:
project:
name: ssa-breast-dna-methylation
version: "1.0.0"
license: CC-BY-NC-4.0
cohort:
sample_size: 2000
populations:
- SSA_West: 0.35
- SSA_East: 0.25
- SSA_Central: 0.15
- SSA_Southern: 0.15
- AAW: 0.10
clinical:
stage_distribution:
I: 0.12
II: 0.35
III: 0.38
IV: 0.15
subtypes:
Luminal_A: 0.25
Luminal_B: 0.20
HER2_enriched: 0.18
Basal_like: 0.32
Normal_like: 0.05
validation:
expected_ranges:
global_methylation_mean: [0.45, 0.55]
brca1_promoter_methylation: [0.15, 0.25]
These YAMLs act as the contract between domain knowledge and simulation code, covering project metadata, dataset-level parameters, clinical/phenotypic structure, genomic features, technical mechanisms, and validation settings.
Step 3: Synthetic Data Generation
Each dataset has a Python generator script that:
- Loads the configuration — Parses YAML into structured Python objects
- Initialises RNG — Uses a known random seed for reproducibility
- Builds base cohort — Samples demographics, assigns stages/subtypes/therapies
- Assigns genomic features — Methylation, expression, variants, biomarkers
- Applies technical noise — Centre-specific shifts, missingness patterns
- Exports outputs — Parquet and CSV files with consistent schema
Feature Generation Examples
Methylation: Binary/categorical status sampled from stage/subtype/population-specific probabilities, then continuous beta values from conditioned distributions.
Expression: Delta from base mean computed per gene group based on tissue type, subtype, and population, then sampled with clipping to plausible ranges.
Biomarkers: Baseline values by stage/subtype, with response-class-specific multipliers applied over time for trajectories.
Step 4: Validation & Reporting
Each dataset has a dedicated validation script that runs a battery of checks:
- Sample size and composition: Observed vs expected counts
- Distributional checks: Stage-specific means, prevalences, methylation by region
- Temporal dynamics: Biomarker ratios, ctDNA clearance, clonal trajectories
- Technical checks: QC pass rates, coverage distributions, loss-to-follow-up
Each check is labelled:
- PASS — Within target ranges or qualitatively correct
- WARN — Minor deviation, may still be acceptable
- FAIL — Substantial deviation needing parameter tuning
We do not consider a dataset ready for release unless it has no FAIL checks and only justifiable WARNs.
Step 5: Packaging & Publishing
Once validation passes:
- Parquet and CSV files are copied into a
huggingface_dataset/subfolder - An academic-style README.md dataset card is authored
- A dataset-specific upload script ensures the HF repo exists and uploads files
- Datasets are added to the shared Hugging Face Genomics collection
Ethics, Governance & Limitations
Synthetic, Not Real
The datasets are never derived by obfuscating or perturbing real genomes. We simulate from probability distributions and structural models. There is no one-to-one mapping between synthetic and real individuals.
Not for Clinical Use
These datasets are strictly for research on methods, algorithms, and pipelines; benchmarking and teaching. They must not be used to make clinical decisions about real patients.
Bias and Representation
We aim to improve representation of African and SSA populations. However, all parameters are constrained by currently available literature, which may itself be sparse or biased. These datasets should be seen as plausible synthetic scenarios, not definitive epidemiological truth.
Transparency
All configuration files, generation scripts, and validation reports are intended to be open. We welcome external review of assumptions, suggestions for improved parameterisation, and contributions of new modules.