Architecture – Page 2 – EduDataSci – Educating the world about data and leadership

The Pipeline You Don’t Have to Build

There is a peculiar habit in data engineering: we measure sophistication by the machinery we build. A source drops CSV files into storage, so we create a pipeline. The pipeline needs a schedule, parameters, schema handling, retries, logging, alerts, deployment rules, credentials, and somebody willing to answer for it at 2:00 a.m. None of that is especially difficult. That is precisely the problem. Enterprises spend an astonishing amount of engineering time repeatedly solving work that is too ordinary to deserve bespoke engineering.

Microsoft Fabric Shortcut Transformations attack that problem at the right level. They convert supported files referenced through a OneLake shortcut into a managed Delta table, keep the table synchronized with the source, and expose the result to SQL, Spark, Power BI, and other Fabric consumers. CSV, Parquet, and JSON transformations are generally available. Excel support is in preview. A separate public-preview capability applies built-in language processing to text files for summarization, translation, sentiment analysis, personally identifiable information detection, and named-entity recognition.

That feature list is useful. The architectural consequence is more important.

Shortcut Transformations move routine ingestion from something every project must build into something the platform simply does.

That is the difference between accelerating a pipeline and deleting the need for one.

A Data Product Is an Engine, Not a Table: FabCon 2026, Databricks, Fabric, and the Case for Interoperability

FabCon and SQLCon 2026 made the Microsoft Fabric and Azure Databricks story more concrete. The headline changes were not cosmetic. Microsoft moved zero-copy access to OneLake data from Azure Databricks into public preview, made Direct Lake in OneLake generally available, kept expanding Databricks-to-Fabric mirroring, and then pushed shortcut transformations into general availability in April. Put plainly, the platform story is moving away from “pick one stack forever” and toward “publish governed products that multiple engines can use.”

That matters because too many teams still call a dataset a product. Microsoft’s current guidance is more precise than that. A data product has defined shape, interfaces, maintenance expectations, and refresh cycles. It is processed for analytical use, and it should be discoverable, secure, interoperable, and valuable enough to serve downstream consumers without forcing them back into raw source complexity. The OneLake Catalog is now described in the Cloud Adoption Framework as a unified access surface for approved data products across Fabric and external processing platforms such as Databricks.

That is where Quantum Regression is a useful teaching device. I am not using it here as a product feature – or even going into the technology -, I am using it as a way to think about how data products enable modular expansions to the data estate. In this framing, a data product is an independent transformation: it accepts ingestions, applies logic, enforces quality and policy, and surfaces results. The file, table, semantic model, or API is only the current observable state of that transformation. The product is the operator, not the artifact. That is the mindset shift that helps make interoperability useful instead of merely fashionable.

Knowledge Graphs: The Quiet Superpower Behind Trustworthy AI

If you’ve spent any time building with large language models, you’ve felt the tension: they’re brilliant at language, and occasionally too confident about facts. The more “enterprise” your use case becomes—policies, procedures, product catalogs, research, student records, regulated workflows—the more that gap matters.

This post is about the missing layer that closes it. Knowledge graphs give AI something it often lacks: a durable, explicit model of meaning and relationships. We’ll walk through what knowledge graphs really are, why they matter more now than ever, and how graph-based retrieval (GraphRAG) is changing what “good” looks like in modern AI.

The Chief Risk Officer’s Quiet Obsession: Data Platforms and Data Products

A Chief Risk Officer (CRO) at an FSS Corporation rarely wakes up thinking, “I can’t wait to talk about data architecture today.”

But they do wake up thinking about something that inevitably leads back to it:

Can I trust what we’re about to tell the Board, the regulator, and the market—especially when conditions get ugly?

That question is why the CRO cares deeply about your Data Platform and your Data Products. Not as “tech initiatives,” but as the machinery that turns risk from opinions and spreadsheets into repeatable, auditable decisions the business can stand behind.

In this post, I’ll connect the CRO’s mandate to the practical realities of platforms and products—and why getting this right is a risk control, not a nice-to-have. Along the way, you’ll see why risk management and operational resilience don’t live in policy binders—they live in data.

Beyond the Medallion: Building Fabric Data Products with Schemas, Materialized Lake Views, and a “Surface Area” Contract

If you’ve been around modern analytics platforms for more than five minutes, you’ve probably built (or inherited) a medallion architecture: bronze → silver → gold. It’s familiar, it’s easy to draw on a whiteboard, and it’s often the first stable pattern teams reach for.

But there’s a quiet problem hiding in that simplicity: the number of sublayers tends to grow, and the complexity of each layer tends to balloon. Before long, you’re not designing a data product—you’re running an assembly line of multi-step transforms, hand-managed orchestration, and fragile dependencies.

Microsoft Fabric is starting to give us a different move: instead of treating transformation as a few “big” layers, you can treat it as a series of small, composable steps—and let the platform manage the dependency graph.

In this article, I’m going to connect three ideas:

Lakehouse schemas as your unit of organization (and the boundary between “internal plumbing” and “published contract”)
Materialized Lake Views as the declarative engine that builds (and refreshes) a dependency graph for you
A surface area schema designed to be shortcutted into other workspaces—so each workspace becomes an “analytical microservice” with its own interface, security boundary, and versioning story

Along the way, we’ll introduce a pragmatic versioning approach: create a new schema for major versions so breaking changes get semantic versioning “for free.”

The Ideal Microsoft Fabric CI/CD Approach: Git for Change, Deployment Pipelines for Promotion, and a Code-First Escape Hatch

Microsoft Fabric CI/CD has a reputation for being confusing—usually because people look at Git integration and Deployment Pipelines as competing ideas rather than two halves of a single delivery story.

The good news is that the “ideal” approach is not exotic. It’s a handoff:

Use Git integration to support real developer workflows (including branching that maps cleanly to isolated workspaces).
Use Deployment Pipelines to promote approved changes across environments.
When you need richer approvals, tests, and release controls, let traditional tooling—especially GitHub Actions or Azure DevOps Pipeline—orchestrate promotions via Fabric APIs.

In this post, I’ll lay out that end-to-end pattern step-by-step, show where the seams belong, and call out the cost you can’t ignore: workspace sprawl—and the operational discipline required to manage aged workspaces intentionally.

Workspace Sprawl Isn’t Your Fabric Problem—Stale Workspaces Are

“Do we really need another workspace?”

If you’ve built anything meaningful in Microsoft Fabric, you’ve heard some version of that question. It usually comes wrapped in a familiar anxiety: workspace sprawl. Too many containers. Too much to govern. Too hard to manage.

Here’s the reframing that matters: workspace count is rarely the risk. The real risk is stale workspaces and stale data—the forgotten corners of your tenant where ownership is unclear, permissions linger, and the platform quietly accumulates operational and compliance debt.

In this post I’ll walk through why “workspace sprawl” is a false fear, why workspaces naturally form clusters (and why good development multiplies them), and how intentional permissioning in Microsoft Entra and Fabric keeps management from becoming a linear slog—especially once you introduce automation and tooling. Along the way, I’ll ground the point in the real mechanics of Microsoft Fabric rather than vibes.

The Advanced Lakehouse Data Product: Shortcuts In, Materialized Views Through, Versioned Schemas Out

There’s a familiar tension in modern analytics: teams want data products that are easy to discover and safe to consume, but they also want to move fast—often faster than the governance model can tolerate.

In Microsoft Fabric, that tension frequently shows up as a perception of workspace sprawl. A “single product per workspace” model is clean on paper—strong boundaries, tidy ownership, straightforward promotion—but it can quickly turn into dozens (or hundreds) of workspaces to curate, secure, and operate.

This post proposes a different pattern—an advanced lakehouse approach that treats the lakehouse itself like a product factory:

Shortcuts or schema shortcuts become the input layer (a clean, contract-aware “ingest without copying” boundary).
A small-step transformation layer is implemented as a multi-step DAG using Materialized Lake Views (MLVs).
A versioned, schema-based surface area becomes the data product contract you expose to consumers.

Then we connect that to OneLake security and Fabric domains, showing how you can expose left-shifted data products (usable earlier in the lifecycle) without letting workspaces multiply endlessly.

Delta First: Building Efficient Bitemporal Tables in Microsoft Fabric

In financial services, the questions that matter most are rarely answered by “the latest record.”

Regulators, auditors, model validators, and operations teams want something more specific: what was true for the business at the time, and what did we know at the time? That’s bitemporal thinking—and it’s exactly the kind of problem where Microsoft Fabric’s Lakehouse on Delta becomes more than storage. It becomes a practical design advantage.

In this post, I’m going to walk through what bitemporal tables actually require, why intervals matter (ValidFrom/ValidTo), and how to implement bitemporal efficiently in Fabric by leaning into #DeltaLake in the Lakehouse. We’ll ground it with two #FSI examples (low velocity KYC and high velocity trades/payments), and we’ll add a derived-layer option using materialized lake views to calculate closure dates. Finally, we’ll cover when Azure SQL Database (including Hyperscale) is the right operational complement to Fabric.

Gold as the Contract: Schema Evolution, Data Products, and Governance in Microsoft Fabric

A schema change is rarely “just a schema change.”

It’s the moment an upstream team’s perfectly reasonable adjustment becomes a downstream team’s broken report, confusing metric, or silent misinterpretation. And that’s why schema evolution has always been a source of anxiety: a schema isn’t simply structure—it’s an interface.

In this post, I’ll do three things. First, I’ll ground why schema evolution has historically been such a persistent concern. Next, I’ll reframe Medallion with Gold as the published surface area of the data product, and Silver as an optional workshop layer where data is supplemented and transformed. Finally, I’ll connect that design to how Microsoft Fabric supports it today—especially the post‑Ignite direction around governance, security, and semantics.