Architecture – Page 2 – EduDataSci – Educating the world about data and leadership

Workspace Sprawl Isn’t Your Fabric Problem—Stale Workspaces Are

“Do we really need another workspace?”

If you’ve built anything meaningful in Microsoft Fabric, you’ve heard some version of that question. It usually comes wrapped in a familiar anxiety: workspace sprawl. Too many containers. Too much to govern. Too hard to manage.

Here’s the reframing that matters: workspace count is rarely the risk. The real risk is stale workspaces and stale data—the forgotten corners of your tenant where ownership is unclear, permissions linger, and the platform quietly accumulates operational and compliance debt.

In this post I’ll walk through why “workspace sprawl” is a false fear, why workspaces naturally form clusters (and why good development multiplies them), and how intentional permissioning in Microsoft Entra and Fabric keeps management from becoming a linear slog—especially once you introduce automation and tooling. Along the way, I’ll ground the point in the real mechanics of Microsoft Fabric rather than vibes.

The Advanced Lakehouse Data Product: Shortcuts In, Materialized Views Through, Versioned Schemas Out

There’s a familiar tension in modern analytics: teams want data products that are easy to discover and safe to consume, but they also want to move fast—often faster than the governance model can tolerate.

In Microsoft Fabric, that tension frequently shows up as a perception of workspace sprawl. A “single product per workspace” model is clean on paper—strong boundaries, tidy ownership, straightforward promotion—but it can quickly turn into dozens (or hundreds) of workspaces to curate, secure, and operate.

This post proposes a different pattern—an advanced lakehouse approach that treats the lakehouse itself like a product factory:

Shortcuts or schema shortcuts become the input layer (a clean, contract-aware “ingest without copying” boundary).
A small-step transformation layer is implemented as a multi-step DAG using Materialized Lake Views (MLVs).
A versioned, schema-based surface area becomes the data product contract you expose to consumers.

Then we connect that to OneLake security and Fabric domains, showing how you can expose left-shifted data products (usable earlier in the lifecycle) without letting workspaces multiply endlessly.

Delta First: Building Efficient Bitemporal Tables in Microsoft Fabric

In financial services, the questions that matter most are rarely answered by “the latest record.”

Regulators, auditors, model validators, and operations teams want something more specific: what was true for the business at the time, and what did we know at the time? That’s bitemporal thinking—and it’s exactly the kind of problem where Microsoft Fabric’s Lakehouse on Delta becomes more than storage. It becomes a practical design advantage.

In this post, I’m going to walk through what bitemporal tables actually require, why intervals matter (ValidFrom/ValidTo), and how to implement bitemporal efficiently in Fabric by leaning into #DeltaLake in the Lakehouse. We’ll ground it with two #FSI examples (low velocity KYC and high velocity trades/payments), and we’ll add a derived-layer option using materialized lake views to calculate closure dates. Finally, we’ll cover when Azure SQL Database (including Hyperscale) is the right operational complement to Fabric.

Gold as the Contract: Schema Evolution, Data Products, and Governance in Microsoft Fabric

A schema change is rarely “just a schema change.”

It’s the moment an upstream team’s perfectly reasonable adjustment becomes a downstream team’s broken report, confusing metric, or silent misinterpretation. And that’s why schema evolution has always been a source of anxiety: a schema isn’t simply structure—it’s an interface.

In this post, I’ll do three things. First, I’ll ground why schema evolution has historically been such a persistent concern. Next, I’ll reframe Medallion with Gold as the published surface area of the data product, and Silver as an optional workshop layer where data is supplemented and transformed. Finally, I’ll connect that design to how Microsoft Fabric supports it today—especially the post‑Ignite direction around governance, security, and semantics.

When Facts Don’t Live in a Domain: Why Data Products Beat Pure Domain-Driven Data Engineering

There’s a pattern I see in mature analytics organizations: as soon as the data platform gets big enough to feel “enterprise,” someone reaches for domain-driven design (DDD) as the organizing principle for data engineering and governance.

It’s an understandable move. DDD gives us language for ownership, boundaries, and “this team is responsible for that thing.” And when you’re trying to untangle a spaghetti warehouse, that sounds like oxygen.

But here’s the catch: the most valuable data in a warehouse—the fact tables in traditional star schemas—often doesn’t live neatly inside a single domain. It lives at the intersections.

In this post, I’m going to make three points:

why domain boundaries tend to break down precisely where the business value shows up (facts)
how a data product model handles “intersection data” without inventing reference domains and mega-joins
why Microsoft Fabric’s workspace + lakehouse + OneLake tooling makes the product approach easier to execute in practice (especially with shortcuts, security, and materialized lake views)

And yes—domains still matter. Just not as the unit of delivery.

Real‑Time Data Isn’t Free: The Complexity and Cost Tradeoffs (From Trickle to Internet‑Class)

The first time someone asks for “real‑time,” it sounds like a small tweak: refresh the dashboard faster, trigger an alert sooner, show a counter that feels alive. In a data platform, that single request quietly changes everything—how you ingest, how you process, how you serve, and how you operate.

This post keeps it practical. It frames real‑time as a freshness target (not a vibe), walks through the two taxes real‑time introduces—architectural complexity and cost—and shows how patterns evolve as you scale from modest #StreamingData to internet‑class velocity. It also folds in recent Microsoft Ignite announcements that matter for real‑time platforms, including SQL Server 2025’s “change event streaming” and near real‑time analytics via OneLake/Fabric mirroring, plus the continued maturation of Microsoft Fabric’s Real‑Time Intelligence building blocks.

A reference landing zone architecture for Microsoft Fabric

Think of this as two layers that work together:

Fabric layer (tenant + capacities + workspaces)
You set governance boundaries through tenant settings, capacity assignment, workspace structure, and workspace security configuration.

Azure layer (identity, networking, Key Vault, storage, monitoring)
You provide the enterprise foundations Fabric will integrate with: private endpoints, VNets, gateways, Key Vault keys/secrets, and ADLS archive storage.

Making Schema Change Boring: A Short History—and How Microsoft Fabric’s Medallion Lakehouse Bakes It In

Schema changes have always been risky because a schema isn’t just columns—it’s the interface between data producers and data consumers. Historically, that interface was rigid, which made any change expensive. Modern lakehouse design solves the problem structurally: a Medallion architecture separates where variation is tolerated (Bronze) from where commitment is made (Silver) and relied upon (Gold). In Microsoft Fabric, those roles map cleanly to Lakehouse, Warehouse, and Power BI’s semantic layer, with governance and domain‑oriented (data‑product) design tying it all together. By the end, you’ll see why schema evolution is both inevitable and manageable—and how Fabric builds that manageability into the platform.

Beyond Boxes and Lines: Designing for the System of Systems Your Software Must Live In

Architecture diagrams look clean—until they collide with the place your system actually lives. That place is not just “production.” It’s your organization: a dense system of systems made of teams, processes, policies, budgets, and tools. The primary reason to understand the organization in which a system will reside is precisely these systems. Your software must integrate with them as surely as it integrates with databases and queues.

I’ll make one argument three ways. First, I’ll define system of systems and show why the org is part of the runtime. Next, I’ll connect business architecture to systems architecture so strategy and structure reinforce each other. Finally, I’ll revisit Conway’s Law—how to use it deliberately and how it can quietly work against you. Then I’ll close with a practical loop you can run before you write code.

Data Vault, Practically: Why It Exists, How It’s Built, and What 2.1 Changes

Modern data platforms live in tension:

Source systems evolve faster than dimensional models can absorb.
Audit and lineage are mandatory, but teams still need velocity.
Cloud lakehouses, streaming, and domain ownership do not slot neatly into yesterday’s warehouse playbooks.

Data Vault is a response to those pressures. It is both a modeling approach and a delivery method designed to (1) absorb change, (2) preserve complete, immutable history, and (3) decouple integration from consumption. The core building blocks—Hubs, Links, and Satellites—organize into a Raw Vault (source truth, append‑only) and a Business Vault(governed derivations and query assistance). Think of it as a fault‑tolerant integration substrate with a clean seam to marts, semantic models, and data products.