Architecture – Page 3 – EduDataSci – Educating the world about data and leadership

When Facts Don’t Live in a Domain: Why Data Products Beat Pure Domain-Driven Data Engineering

There’s a pattern I see in mature analytics organizations: as soon as the data platform gets big enough to feel “enterprise,” someone reaches for domain-driven design (DDD) as the organizing principle for data engineering and governance.

It’s an understandable move. DDD gives us language for ownership, boundaries, and “this team is responsible for that thing.” And when you’re trying to untangle a spaghetti warehouse, that sounds like oxygen.

But here’s the catch: the most valuable data in a warehouse—the fact tables in traditional star schemas—often doesn’t live neatly inside a single domain. It lives at the intersections.

In this post, I’m going to make three points:

why domain boundaries tend to break down precisely where the business value shows up (facts)
how a data product model handles “intersection data” without inventing reference domains and mega-joins
why Microsoft Fabric’s workspace + lakehouse + OneLake tooling makes the product approach easier to execute in practice (especially with shortcuts, security, and materialized lake views)

And yes—domains still matter. Just not as the unit of delivery.

Real‑Time Data Isn’t Free: The Complexity and Cost Tradeoffs (From Trickle to Internet‑Class)

The first time someone asks for “real‑time,” it sounds like a small tweak: refresh the dashboard faster, trigger an alert sooner, show a counter that feels alive. In a data platform, that single request quietly changes everything—how you ingest, how you process, how you serve, and how you operate.

This post keeps it practical. It frames real‑time as a freshness target (not a vibe), walks through the two taxes real‑time introduces—architectural complexity and cost—and shows how patterns evolve as you scale from modest #StreamingData to internet‑class velocity. It also folds in recent Microsoft Ignite announcements that matter for real‑time platforms, including SQL Server 2025’s “change event streaming” and near real‑time analytics via OneLake/Fabric mirroring, plus the continued maturation of Microsoft Fabric’s Real‑Time Intelligence building blocks.

A reference landing zone architecture for Microsoft Fabric

Think of this as two layers that work together:

Fabric layer (tenant + capacities + workspaces)
You set governance boundaries through tenant settings, capacity assignment, workspace structure, and workspace security configuration.

Azure layer (identity, networking, Key Vault, storage, monitoring)
You provide the enterprise foundations Fabric will integrate with: private endpoints, VNets, gateways, Key Vault keys/secrets, and ADLS archive storage.

Making Schema Change Boring: A Short History—and How Microsoft Fabric’s Medallion Lakehouse Bakes It In

Schema changes have always been risky because a schema isn’t just columns—it’s the interface between data producers and data consumers. Historically, that interface was rigid, which made any change expensive. Modern lakehouse design solves the problem structurally: a Medallion architecture separates where variation is tolerated (Bronze) from where commitment is made (Silver) and relied upon (Gold). In Microsoft Fabric, those roles map cleanly to Lakehouse, Warehouse, and Power BI’s semantic layer, with governance and domain‑oriented (data‑product) design tying it all together. By the end, you’ll see why schema evolution is both inevitable and manageable—and how Fabric builds that manageability into the platform.

Beyond Boxes and Lines: Designing for the System of Systems Your Software Must Live In

Architecture diagrams look clean—until they collide with the place your system actually lives. That place is not just “production.” It’s your organization: a dense system of systems made of teams, processes, policies, budgets, and tools. The primary reason to understand the organization in which a system will reside is precisely these systems. Your software must integrate with them as surely as it integrates with databases and queues.

I’ll make one argument three ways. First, I’ll define system of systems and show why the org is part of the runtime. Next, I’ll connect business architecture to systems architecture so strategy and structure reinforce each other. Finally, I’ll revisit Conway’s Law—how to use it deliberately and how it can quietly work against you. Then I’ll close with a practical loop you can run before you write code.

Data Vault, Practically: Why It Exists, How It’s Built, and What 2.1 Changes

Modern data platforms live in tension:

Source systems evolve faster than dimensional models can absorb.
Audit and lineage are mandatory, but teams still need velocity.
Cloud lakehouses, streaming, and domain ownership do not slot neatly into yesterday’s warehouse playbooks.

Data Vault is a response to those pressures. It is both a modeling approach and a delivery method designed to (1) absorb change, (2) preserve complete, immutable history, and (3) decouple integration from consumption. The core building blocks—Hubs, Links, and Satellites—organize into a Raw Vault (source truth, append‑only) and a Business Vault(governed derivations and query assistance). Think of it as a fault‑tolerant integration substrate with a clean seam to marts, semantic models, and data products.

Testing Like We Mean It: Bringing Software‑Grade Discipline to Data Engineering

I like to say that the first product of a data team isn’t a table or a dashboard—it’s trust. Trust is built the same way in data as it is in software: through tests that catch regressions, encode intent, and make change safe. If pipelines are code, then they deserve the same rigor as code. That means unit tests you can run in seconds, integration tests that respect the messy edges of reality, comprehensive tests that exercise the platform end‑to‑end, and user acceptance testing that proves the system answers the questions people actually have. Done well, this isn’t busywork; it’s the backbone of reliability and a pillar of governance.

Why Star Schemas Make Analysts Faster (and Happier)

If you live in spreadsheets or SQL all day, the “one big table” (OBT) feels like home. Everything you need is right there: one row per thing, a column for every attribute, and no joins to worry about. It’s a great way to explore data fast—until it isn’t. This post explains, in plain language, why the star schema pays you back every day you analyze data, and how it keeps the speed you love without the headaches you’ve learned to live with.

Baselines Over Buzzwords: From Warehouse to Lakehouse

If you’ve built data systems long enough, you’ve lived through at least three architectural moods: the tidy certainty of Kimball and Inmon, the anarchic freedom of “throw everything in the data lake to ingest quickly,” and today’s lakehouse, which tries to keep our speed without losing our sanity. I’ve always cared less about labels and more about baselines—clear, durable expectations that make change safe. This piece traces how those baselines shifted, what we gained and lost, and how to rebuild them for modern work, including real‑time, very large, and unstructured data.

Data Mesh Isn’t Just for Tech Companies

If you’ve skimmed headlines, it’s easy to conclude that data mesh is a Silicon Valley thing—something streaming apps and fintechs use to wrangle petabytes. That mental model sells a lot of tools, but it misses the point. Data mesh is first an operating model—a way to organize people, responsibilities, and guardrails so data can be produced and used where the knowledge lives. That matters just as much (and often more) in organizations whose mission is not building software: manufacturers, hospitals, universities, public-sector agencies, retailers, utilities, and nonprofits.