Most data teams still run a tidy assembly line: ingest sources into bronze, standardize into silver, curate into gold, and only then wire up a semantic model for BI. That sounds rigorous—but it puts the business contract (grain, conformed dimensions, measure logic, security scope, and SLOs) at the very end. By the time the organization finally argues about what “AUM” or a compliant “time‑weighted return” really means, we’ve already paid for pipelines, copies, and storage layouts that might not fit the answer we need.
Symptoms you’ll recognize: months of “inventory building” without shipping a trustworthy product; duplicate stacks for streaming vs. batch; sprawling “bronze” zones that age into operational risk; and endless rework because definitions arrived too late.
Modern Microsoft Fabric tools let you flip the incentives. With Direct Lake placing the semantic model directly over Delta in OneLake—and with shortcuts, mirroring, materialized lake views, and eventhouses spanning real‑time and lake—there’s finally a platform that rewards designing from the output backward. In other words: Gold → Silver → Bronze → Ingestion.
How we work today—and what breaks
What we do too often: land everything in bronze, clean to silver, aggregate to gold, publish a model.
Where it breaks:
- Semantics show up last. Grain, SCD rules, KPI math, and row‑level security appear after we’ve locked storage and pipelines. Changes are expensive because they cascade backward.
- We optimize for inventory, not outcomes. We lovingly curate tables that never power a decision.
- Two stacks for one truth. Streaming has its own tools and copies; batch has another. Keeping them aligned is a small industry inside your team.
- Bronze becomes a museum. Landing “everything, just in case” piles up cost and governance debt without a product owner.
Flip the script: Gold → Silver → Bronze → Ingestion
Design from the product backward. Define the semantic contract first, then implement only the silver/bronze required to honor it—using Fabric features that minimize copies and keep batch and real‑time on the same rails.
Gold first: the semantic model is the contract
Start by fixing grain, dimensions, relationships, measures, security, and SLOs. In Fabric, Direct Lake lets Power BI semantic models read Delta tables in OneLake at interactive speeds without an import hop, so your modeling choices directly drive physical table design (partitions, file sizes) rather than the other way around. Crucially, you can mix Direct Lake with Import tables in the same model—for example, big fact tables in Direct Lake and small reference tables in Import.
If data science must share the same truth, Semantic Link (SemPy) exposes that model—metadata and measures—directly in notebooks for Python and Spark, letting BI and ML compute the same numbers instead of approximations. This means that everyone is working from the same data and the same analytical vocabulary, enhancing governance and usability.
Silver: only the shaping that enforces the contract
Pick the surface that fits your product’s read/write and governance needs:
- Warehouse when you need T‑SQL DML/procedures over Delta and a full relational surface. (Warehouse supports create/alter tables and INSERT/UPDATE/DELETE.)
- Lakehouse when you want Spark‑first ELT and a read‑only SQL analytics endpoint for views/security; use notebooks for heavy transforms.
To make repeated projections/aggregations cheap and declarative, use Materialized Lake Views (preview) in the Lakehouse. MLVs precompute results in the lake (managed by Fabric), which your gold model can read directly—ideal for “star‑ready” facts or slowly recalculated dims. (On the real‑time side, Eventhouse—the KQL engine in RTI—also supports materialized views for streaming aggregations.)
Bronze: small, purposeful—and fast
In the upside‑down view, bronze is where mirrors and shortcuts land:
- OneLake shortcuts virtualize external data (ADLS/S3 or other Fabric items) into your lake without copying.
- Mirroring continuously replicates operational DBs/warehouses into OneLake with low latency and minimal CDC plumbing.
Because gold already scoped what you need, bronze contains only the specific slices that feed your product—not the whole estate. Ingestion becomes configuration, not code.
Bronze for streaming, too: Eventhouse belongs here
Treat Eventhouse as streaming bronze that also powers real‑time products:
- Land events with Eventstream → Eventhouse for sub‑second KQL analytics.
- Turn on Eventhouse OneLake Availability to expose those same events in Delta inside OneLake, so Lakehouse/Warehouse/Direct Lake can read them like any other table.
- When you want a direct lake landing, add Eventstream → Lakehouse as a destination to persist raw events in Delta alongside mirrored/shortcut tables.
This unifies streaming and batch under one contract and one substrate.
How Fabric makes this possible (the quick map)
- Direct Lake centers the product on the semantic model, avoiding an extra import copy while keeping VertiPaq‑grade interactivity over Delta in OneLake.
- Warehouse ↔ Lakehouse are complementary: Warehouse for full T‑SQL and managed relational semantics; Lakehouse for Spark/Delta with a read‑only SQL endpoint for views and governance. Use each where it best serves the contract.
- Materialized Lake Views (preview) precompute lake‑native results your model depends on; Eventhouse has KQL materialized views for streaming aggregations.
- Shortcuts & Mirroring keep bronze thin and “just‑in‑time.”
- Eventhouse OneLake Availability and Eventstream → Lakehouse mean events show up in OneLake as Delta—batch and real‑time feed the same gold model.
A wealth‑management example — Household Performance & Risk Insights
Gold (the contract).
Deliver an advisor‑facing experience with per‑household daily performance, AUM, compliant TWR/IRR, drift vs. target model, and rolling risk (beta/vol). SLO: correct by 9:00 AM local T+0; intraday cash/positions reflected within 15 minutes. Implement as a Direct Lake model for large facts (daily positions, transactions) plus small reference tables in Import. Define row‑level security by Advisor/Branch in the model.
Silver (only what enforces the contract).
- In Lakehouse, declare Materialized Lake Views that normalize lot‑level positions into a daily account‑security grain, enrich with security master/FX, and pre‑join calendars—clean, star‑ready facts.
- In Warehouse, expose T‑SQL views/procs that compute TWR/IRR respecting cash‑flow semantics and benchmark calendars.
- Use Dataflows Gen2 for light conformance (custodian code mapping, SCD behavior on householding) when owned by analysts.
Bronze (small and auditable).
- Mirror custodial positions/trades/cash into OneLake for low‑latency landing.
- Shortcut to market/benchmark data in a vendor lake (S3/ADLS) and to internal CRM for household hierarchies/entitlements.
- For intraday drift and cash alerts, route Eventstream → Eventhouse and enable OneLake Availability so those events appear as Delta in OneLake; optionally Eventstream → Lakehouse to persist the raw event ledger. Bronze stays thin because it only contains what the gold contract demands.
Why it works better here.
You ship the advisor’s numbers first and prove them daily. Because silver/bronze exist solely to serve the contract, changes in AUM or TWR/IRR logic are localized instead of refactoring a warehouse. Events and batch share dimensions/measures, so “my real‑time KPI vs. your daily KPI” stops being a conversation.
Conclusion
The problem wasn’t our discipline—it was our sequence. When semantics arrive last, the warehouse optimizes for motion (copies, steps, zones) instead of truth. Fabric makes the opposite design practical: lead with gold, then build only the silver/bronze that the contract requires, using features that keep data available rather than duplicated. The result is a warehouse that ships products, not just tables.
- Faster time‑to‑value: Model first, ship sooner; Direct Lake avoids an import hop for large facts.
- Less rework: The semantic contract (grain, dims, measures, security, SLOs) drives physical design and pipelines—not the reverse.
- Fewer copies: Shortcuts and Mirroring keep bronze thin and “just‑in‑time” instead of hoarding raw data.
- Unified batch + real‑time: Eventhouse powers live KQL analytics and, via OneLake Availability, lands events as Delta for Lakehouse/Warehouse/Direct Lake—one truth for both speeds.
- Declarative, cheaper silver: Materialized Lake Views (preview) precompute what matters in the lake; KQL materialized views do the same for streams.
- Right tool for the read: Warehouse for full T‑SQL semantics; Lakehouse for Spark + read‑only SQL endpoint—both feeding the same gold model.
- Shared definitions across BI & DS/ML: Semantic Link brings measures and metadata into notebooks; BI and ML compute the same numbers.
If your warehouse still starts at the tap, try starting at the glass. Define the drink first. Fabric will meet you there.
EduDataSci - Educating the world about data and leadership 