Architecture – Page 3 – EduDataSci – Educating the world about data and leadership

Testing Like We Mean It: Bringing Software‑Grade Discipline to Data Engineering

I like to say that the first product of a data team isn’t a table or a dashboard—it’s trust. Trust is built the same way in data as it is in software: through tests that catch regressions, encode intent, and make change safe. If pipelines are code, then they deserve the same rigor as code. That means unit tests you can run in seconds, integration tests that respect the messy edges of reality, comprehensive tests that exercise the platform end‑to‑end, and user acceptance testing that proves the system answers the questions people actually have. Done well, this isn’t busywork; it’s the backbone of reliability and a pillar of governance.

Why Star Schemas Make Analysts Faster (and Happier)

If you live in spreadsheets or SQL all day, the “one big table” (OBT) feels like home. Everything you need is right there: one row per thing, a column for every attribute, and no joins to worry about. It’s a great way to explore data fast—until it isn’t. This post explains, in plain language, why the star schema pays you back every day you analyze data, and how it keeps the speed you love without the headaches you’ve learned to live with.

Baselines Over Buzzwords: From Warehouse to Lakehouse

If you’ve built data systems long enough, you’ve lived through at least three architectural moods: the tidy certainty of Kimball and Inmon, the anarchic freedom of “throw everything in the data lake to ingest quickly,” and today’s lakehouse, which tries to keep our speed without losing our sanity. I’ve always cared less about labels and more about baselines—clear, durable expectations that make change safe. This piece traces how those baselines shifted, what we gained and lost, and how to rebuild them for modern work, including real‑time, very large, and unstructured data.

Data Mesh Isn’t Just for Tech Companies

If you’ve skimmed headlines, it’s easy to conclude that data mesh is a Silicon Valley thing—something streaming apps and fintechs use to wrangle petabytes. That mental model sells a lot of tools, but it misses the point. Data mesh is first an operating model—a way to organize people, responsibilities, and guardrails so data can be produced and used where the knowledge lives. That matters just as much (and often more) in organizations whose mission is not building software: manufacturers, hospitals, universities, public-sector agencies, retailers, utilities, and nonprofits.

DirectLake didn’t “take away” your tables—it put them where they belong

I often hear that DirectLake “removes the ability to define tables” and “doesn’t work like traditional Power BI or Tableau.” At a glance, the workflow is different—deliberately so—because Fabric is a data platform, not a visualization tool. In the old days we’d push transformations past gold and into the semantic layer because that was the only practical place left. That was necessary; it was never ideal. By definition, gold is supposed to be ready to consume.

DirectLake mode in MS Fabric’s Power BI gives you (almost) everything Power BI gave you in Desktop. The one big thing you don’t do anymore is DAX calculated columns/tables—and that’s a feature, not a bug. Nuance for accuracy: In DirectLake, calculated columns and calculated tables that reference DirectLake tables aren’t supported; however, some calculated tables that don’t reference DirectLake tables (e.g., documentation helpers) and calculation groups/what‑if parameters are allowed. DirectLake reads Delta in OneLake, the model still uses VertiPaq, and data prep moves into the platform (Dataflows Gen2, Lakehouse/Warehouse SQL, notebooks).

Slowly Changing Dimensions (SCDs): A Practical Guide for Your Star Schema

Star schemas shine when your facts (events) are analyzed through dimensions (who/what/where/when). But in real life, dimension attributes change—customers move, products rebrand, sales territories realign. Slowly changing dimensions (SCDs) are the modeling patterns that preserve analytic correctness as those attributes evolve.

Implementing Stars and Galaxies in Power BI

Power BI rewards clean dimensional models—but it also punishes sloppy ones. This post walks through how to implement star and galaxy schemas in Power BI semantic models, why ambiguous (multiple) filter paths cause headaches, why implicit measures don’t scale beyond the simplest star, and how tightly defined data products keep your BI ecosystem fast, correct, and governable. Because this is such an important topic, I’ve included links to references with each point.

Conway’s Law for Data Teams

Two Dashboards, One Truth

On Monday, Maya—head of a seven‑person data team—watched two dashboards disagree.

The executive dashboard showed $11.2M in MRR. Sales’ dashboard said $10.6M. Both pulled from “the warehouse.” Both refreshed nightly. Neither was “wrong”; they just measured different things.

Maya didn’t control how Sales Ops or Marketing were organized, who they reported to, or which tools they bought. She controlled only her data team—its models, interfaces, and operations. Yet the warehouse had clearly taken on the shape of the company’s communication patterns.

Conway’s Law, without asking permission, had moved in.

A Practical Introduction to Star Schema Data Architecture

Dimensional modeling remains the most effective way to make analytics fast, understandable, and resilient. The star schema sits at the center of that approach: a simple, denormalized structure where fact tables record measurable events and dimension tables provide descriptive context. In this post, we’ll ground the core ideas, clarify the often‑confused concept of snowflaking (and when it’s worth it), and show how to scale from a single star to a galaxy schema (a.k.a. fact constellation) without losing your footing.

Foundational + Derived Data Products in a Data Mesh

A data mesh is a sociotechnical approach to analytical data that decentralizes responsibility to business domains while standardizing the way data is produced and consumed. It’s grounded in four principles: domain ownership, data as a product, a self‑serve data platform, and federated governance. In practice, it asks each domain team to publish data as a product—discoverable, trustworthy, and operable—while a common platform automates cross‑cutting rules (access, lineage, quality, security).

Zhamak Dehghani frames a data product as an architectural quantum: the smallest independently deployable unit that bundles data, code, metadata, and policy, with a versioned contract and a clear interface (APIs or governed views). Treating both foundational and derived products as quanta is the key to decoupled evolution without breaking interoperability.