Data Engineering – Page 2 – EduDataSci – Educating the world about data and leadership

DirectLake on OneLake CI/CD: A Practical Two-Step Deployment Pattern with Sempy Labs + Variable Libraries

DirectLake on OneLake is one of those “this is what we’ve been waiting for” features in Microsoft Fabric—until you try to deploy it cleanly across Dev → Test → Prod and realize you’ve re-entered the world of post-deployment manual fixes.

In this how-to, I’m going to do three things:

Contrast DirectLake on SQL endpoints (the “classic” flavor) with DirectLake on OneLake (the newer flavor), and explain why OneLake is worth the trouble.
Walk through the normal deployment pipeline approach that works well for DirectLake on SQL.
Show a two-step, semi-automated approach for DirectLake on OneLake using:
- sempy_labs.directlake.update_direct_lake_model_connection(), and
- a Fabric Variable Library + a “run-after-deployment” notebook.

Along the way, I’ll call out the current challenges (because yes, they’re real right now), and why this pattern matters for serious Microsoft Fabric Power BI CI/CD work.

Two Flavors of DirectLake: Over SQL vs. Over OneLake (and How to Switch Without Surprises)

DirectLake has a way of sounding wonderfully simple: “Power BI, but it reads the lake directly.” Then you build two semantic models that both say DirectLake, and they behave… differently. One falls back to DirectQuery when you least expect it. Another refuses to touch your SQL views. Security works for you, but not for your report consumers. Suddenly, “DirectLake” feels less like a feature and more like a riddle.

The good news: this is explainable. And once you understand the two flavors—DirectLake over SQL and DirectLake over OneLake—you can choose deliberately, design around the trade-offs, and even switch between them when you have to.

In this post, I’ll demystify what each option really means, lay out the positives and negatives, explain when you’d use each (and why), and show how to switch using Semantic Link Labs—including what can break when you flip the switch.

Gold as the Contract: Schema Evolution, Data Products, and Governance in Microsoft Fabric

A schema change is rarely “just a schema change.”

It’s the moment an upstream team’s perfectly reasonable adjustment becomes a downstream team’s broken report, confusing metric, or silent misinterpretation. And that’s why schema evolution has always been a source of anxiety: a schema isn’t simply structure—it’s an interface.

In this post, I’ll do three things. First, I’ll ground why schema evolution has historically been such a persistent concern. Next, I’ll reframe Medallion with Gold as the published surface area of the data product, and Silver as an optional workshop layer where data is supplemented and transformed. Finally, I’ll connect that design to how Microsoft Fabric supports it today—especially the post‑Ignite direction around governance, security, and semantics.

When Facts Don’t Live in a Domain: Why Data Products Beat Pure Domain-Driven Data Engineering

There’s a pattern I see in mature analytics organizations: as soon as the data platform gets big enough to feel “enterprise,” someone reaches for domain-driven design (DDD) as the organizing principle for data engineering and governance.

It’s an understandable move. DDD gives us language for ownership, boundaries, and “this team is responsible for that thing.” And when you’re trying to untangle a spaghetti warehouse, that sounds like oxygen.

But here’s the catch: the most valuable data in a warehouse—the fact tables in traditional star schemas—often doesn’t live neatly inside a single domain. It lives at the intersections.

In this post, I’m going to make three points:

why domain boundaries tend to break down precisely where the business value shows up (facts)
how a data product model handles “intersection data” without inventing reference domains and mega-joins
why Microsoft Fabric’s workspace + lakehouse + OneLake tooling makes the product approach easier to execute in practice (especially with shortcuts, security, and materialized lake views)

And yes—domains still matter. Just not as the unit of delivery.

Materialize Responsibly: How Fabric’s External Data Materialization Affects “Zero Unmanaged Copy” — and Where Materialized Lake Views Now Shine

Microsoft Fabric’s Warehouse can now materialize external files into tables with straight‑ahead T‑SQL, and Materialized Lake Views (MLVs) have quietly leveled up with optimal refresh (including incremental) and stronger, UI‑backed monitoring. If your north star is Zero unmanaged copy, the question isn’t “should I materialize?”—it’s “how do I materialize responsibly under OneLake governance?” Here’s what changed since our last take—and what to use when.

Real‑Time Data Isn’t Free: The Complexity and Cost Tradeoffs (From Trickle to Internet‑Class)

The first time someone asks for “real‑time,” it sounds like a small tweak: refresh the dashboard faster, trigger an alert sooner, show a counter that feels alive. In a data platform, that single request quietly changes everything—how you ingest, how you process, how you serve, and how you operate.

This post keeps it practical. It frames real‑time as a freshness target (not a vibe), walks through the two taxes real‑time introduces—architectural complexity and cost—and shows how patterns evolve as you scale from modest #StreamingData to internet‑class velocity. It also folds in recent Microsoft Ignite announcements that matter for real‑time platforms, including SQL Server 2025’s “change event streaming” and near real‑time analytics via OneLake/Fabric mirroring, plus the continued maturation of Microsoft Fabric’s Real‑Time Intelligence building blocks.

Edit, Retarget, and Redeploy: A Practical TMDL Folder Workflow for Fabric Semantic Models

There’s a moment in every Fabric semantic model lifecycle where the “click it in the UI” approach stops scaling.

It usually happens when you need to rename dozens (or hundreds) of fields to match a business glossary, or when Dev is stable and you’re ready to point the same model at a new Lakehouse for Test/Prod. That’s when the model stops being a diagram and starts being an artifact—something you want to treat like code.

This guide reflows the whole workflow end-to-end, using the Fabric service Edit in Desktop experience to open the model, exporting it to a PBIP project stored as a TMDL folder, editing that folder externally (no scripting inside Power BI Desktop), and then getting those changes back into the service—including the key capability you asked for:

retargeting the entire model to a different Lakehouse/Warehouse, and
retargeting a single table to a different physical table (even in a new Lakehouse).

We’ll do it with the mindset of Power BI + Microsoft Fabric development: repeatable changes, visible diffs, and fewer “hand edits” you regret later.

Shortcuts Everywhere, But Serving Still Matters: Materialized Lake Views in Fabric

If your Microsoft Fabric estate is “shortcut‑first,” you’re not alone. OneLake shortcuts (and mirroring) make it genuinely easy to unify data that lives elsewhere—on‑prem, multicloud, SaaS—without immediately building a full ingestion factory. That architectural speed is real.

But there’s a predictable moment when the elegance turns into friction: the day consumption outgrows the source.

Dashboards refresh. Analysts explore. Notebooks iterate. And now—post‑Ignite—agents and Copilot scenarios multiply the number of reads in ways that are hard to forecast. Ignite’s messaging was clear: OneLake is the context layer, and Fabric IQ (plus Foundry IQ) is designed to reason across that unified data foundation.

This post refreshes the original argument with what’s changed since then: inserting Materialized Lake Views (MLVs) into your architecture benefits even heavily shortcutted external designs—because MLVs change who hits the source, when they hit it, and how often.

Fabric Is Medallion‑First, Not Medallion‑Only

If you work with Microsoft Fabric long enough, it’s easy to come away with the impression that “real” Fabric means “medallion everywhere.” The official docs walk through Bronze, Silver, and Gold patterns for lakehouses. The learning paths lean on medallion as the canonical example. Fabric clearly makes medallion a first‑class citizen.

But that doesn’t mean your data platform – or your data products – must be medallion‑shaped.

In a world of managed, domain‑aligned data products and Data Mesh thinking, what matters most is the contract at the edges: the inputs you accept, the outputs you guarantee, and the behaviors you commit to over time. Inside the boundary of a data product, you have more architectural freedom than many teams allow themselves.

In this post, I’ll walk through three ideas:

Fabric is medallion‑forward, but not medallion‑only. For data products, inputs and outputs matter far more than internal state. Internal architecture should serve engineering excellence, not a single prescriptive pattern – illustrated with small examples from financial services, wealth management, and insurance.

By the end, the goal is simple: when you design a Fabric data product, you should feel comfortable treating medallion as one option in a toolbox, not as a mandatory religion.

Making Schema Change Boring: A Short History—and How Microsoft Fabric’s Medallion Lakehouse Bakes It In

Schema changes have always been risky because a schema isn’t just columns—it’s the interface between data producers and data consumers. Historically, that interface was rigid, which made any change expensive. Modern lakehouse design solves the problem structurally: a Medallion architecture separates where variation is tolerated (Bronze) from where commitment is made (Silver) and relied upon (Gold). In Microsoft Fabric, those roles map cleanly to Lakehouse, Warehouse, and Power BI’s semantic layer, with governance and domain‑oriented (data‑product) design tying it all together. By the end, you’ll see why schema evolution is both inevitable and manageable—and how Fabric builds that manageability into the platform.