Data Engineering · Editable reference architecture

Medallion Lakehouse Data Pipeline

Visualize a data engineering pipeline that moves source data through Bronze, Silver and Gold layers. Make ingestion, quality checks, analytical models and downstream consumption visible in one editable lakehouse architecture.

Open & Edit in DrawArc (Free, No Login Required)

Your changes stay in a separate draft on this browser. Download an editable project to keep a portable copy.

LinkedIn X (Twitter)WhatsApp

8 components · 8 connections · Open the editor to zoom, move nodes and edit source.

Architectural overview

The medallion pattern organizes analytical data by refinement level. Bronze retains ingested source records, Silver applies validation and normalization, and Gold presents curated datasets for business consumption. This starter connects a source system to ingestion, shows Spark transformations between layers and routes curated output to a business intelligence consumer. Airflow represents orchestration rather than the data plane itself.

Keep raw data sufficiently complete to support investigation and reprocessing, while defining retention and access policies for each dataset. Silver transformations should handle duplicates, missing fields and schema evolution deliberately. Gold models should express agreed business definitions rather than becoming another uncontrolled copy of raw tables. The boundaries in this diagram provide places to annotate those responsibilities and their owners.

Before scheduling jobs, specify freshness expectations, late-arriving data rules and the impact of a failed quality check. Decide which workloads are incremental and how historical backfills avoid overwriting valid results. Add a catalog, lineage collection and monitoring integrations where your platform requires them. Use the diagram during handoff to explain how a failed source feed propagates to reports and who decides whether downstream processing can continue.

The real-world problem

Analysts need reproducible business metrics from inconsistent operational feeds. Raw records must remain traceable while invalid data is kept out of consumption-ready models.

Data flow

Connections from this diagram. Branches and control signals are not a single chronological sequence.

Operational sources → Ingestion jobs: Extract or CDC.
Ingestion jobs → Bronze raw layer: Append raw records.
Bronze raw layer → Spark quality transforms: Read incremental input.
Spark quality transforms → Silver validated layer: Validated output.
Silver validated layer → Gold business models: Business transformations.
Gold business models → BI dashboards: Query curated models.
Airflow orchestration → Ingestion jobs: Schedule.
Airflow orchestration → Spark quality transforms: Coordinate runs.

Security boundaries

Grant separate permissions to raw, validated and curated datasets; Bronze may contain the most sensitive source material.
Apply encryption, retention and deletion policies across every derived copy and backup.
Separate orchestration identity from analyst access; redact sensitive records from task logs.

Failure handling

Source schema changes unexpectedly

Response: Quarantine incompatible records and alert the dataset owner rather than silently changing business metrics.

Verify: Add and remove source fields in a test feed and verify quality gate behavior.

A transform fails after partial work

Response: Use transactional table writes or a safe staging-and-publish strategy with repeatable checkpoints.

Verify: Interrupt a run and confirm rerunning does not duplicate or partially publish records.

Late records change prior totals

Response: Define watermark, merge and backfill rules; publish freshness and correction status to consumers.

Verify: Replay a late batch and reconcile affected Gold metrics against expected results.

Trade-offs and alternatives

Retain Bronze source fidelity

Benefit: Supports investigations and reprocessing.
Cost: More storage, sensitive-data governance and deletion obligations.
Alternative: Retain only the justified fields and history under a documented replay policy.

Separate Bronze, Silver and Gold

Benefit: Clear quality and ownership boundaries.
Cost: Additional jobs, copies and end-to-end freshness delay.
Alternative: A simpler warehouse pipeline may fit a small, stable reporting workload.

When to use this architecture

Explain analytical data refinement and ownership to platform engineers, analysts and product stakeholders.
Plan a batch or incremental lakehouse pipeline with explicit data quality gates and reprocessing boundaries.
Document where orchestration, transformations and BI consumption interact without conflating control flow with data movement.

Key components

Operational sources: Business systems providing extracts or change records.
Ingestion jobs: Land source data with timestamps and source identifiers.
Bronze raw layer: Retains source fidelity for controlled replay and investigation.
Spark quality transforms: Validates schema, deduplicates records and applies transformation rules.
Silver validated layer: Holds normalized, validated records for reusable downstream processing.
Gold business models: Publishes documented metrics and consumption-ready analytical models.
BI dashboards: Consumes curated models with agreed freshness and access requirements.
Airflow orchestration: Coordinates task dependencies, retries and operational scheduling.

Questions about this template

Is medallion limited to Databricks?

No. Bronze, Silver and Gold describe data refinement responsibilities. Adapt storage and execution components to your platform.

Where do quality checks belong?

Show checks at ingestion and between refinement layers, with clear quarantine and retry paths for invalid records.

Related architecture templates

AWS