Data Products – EduDataSci – Educating the world about data and leadership

Stop Publishing the Workshop: An Ingest–Transform–Surface Architecture for Microsoft Fabric

Most Microsoft Fabric architecture diagrams are too polite.

They show data arriving, becoming cleaner, and eventually appearing in a report. What they usually leave out is the argument that determines whether the platform remains understandable after the third development team, the tenth source system, and the fiftieth downstream consumer: where does source reality end, where does product construction happen, and where does the enterprise make a promise?

When those boundaries are unclear, every table is treated as equally public. Analysts connect to whichever object is easiest to find, engineers preserve internal tables because somebody may be using them, and source-facing objects begin to look like product interfaces. The platform may still run, but the architecture has stopped communicating intent.

This article proposes a deliberately simple alternative for a schema-enabled Fabric Lakehouse:

Every ingestion method lands data in, or exposes it through, ingest.*.
Materialized Lake Views (MLVs) and notebooks move data into transform.*.
MLVs publish stable, consumption-ready tables into surface.*.
Consumers connect to the product through governed interfaces built on surface.*, not through the internal construction layers.

Here, ingest.* is shorthand for every table in the ingest schema. The same applies to transform.* and surface.*; the asterisk is architectural notation, not part of the schema name.

The result is not another three-color diagram. It is a practical way to separate source capture, internal implementation, and the published contract of a data product.

The Internet After Apps: Public, Private, and Personal AODPs in an Intent-Driven World

When software can be generated for a person, a task, or a moment, the durable architecture must move out of the application and into the interface to data and action.

A wealth advisor finishes a client call and asks an agent to build a small application.

The application should assemble the household’s accounts, identify securities that could raise $75,000, preserve the client’s income targets, avoid restricted positions, estimate tax consequences, and prepare an explanation for the client. It is not intended to become the next enterprise advisor platform. It may only be needed for the next hour.

The software could be generated almost immediately. That is the appealing part.

The dangerous part is everything the software must touch.

It needs market information, client preferences, account positions, tax lots, suitability rules, firm policies, trading restrictions, approval thresholds, and access to an order-management system. It must know which information belongs to the client, which belongs to the firm, and which comes from public or commercial sources. It must distinguish between calculating a proposed trade and submitting one. It must preserve evidence of what it read, why it made a recommendation, who approved it, and what eventually changed.

The application may be disposable. Its authority cannot be.

This is the architectural problem hiding underneath intent-driven computing and enterprise vibe coding. As software becomes easier to generate, more of the burden moves onto the interfaces through which that software reaches data, policy, and operational systems. We cannot make every dynamically generated application trustworthy in the same way that we certify a major enterprise platform. We can, however, force every application to operate through trustworthy products.

That is the purpose of the Action-Oriented Data Product, or AODP.

An AODP is a reusable, domain-owned product that exposes governed data and governed actions through a secure, discoverable, contract-driven interface. AODPs can exist in public, private, and personal spaces. Connected through a mesh, they provide the durable substrate underneath people, agents, and software assembled for individual tasks.

The internet built around pages and applications does not disappear. It gains another layer: an internet of governed data and action.

A Data Product Is an Engine, Not a Table: FabCon 2026, Databricks, Fabric, and the Case for Interoperability

FabCon and SQLCon 2026 made the Microsoft Fabric and Azure Databricks story more concrete. The headline changes were not cosmetic. Microsoft moved zero-copy access to OneLake data from Azure Databricks into public preview, made Direct Lake in OneLake generally available, kept expanding Databricks-to-Fabric mirroring, and then pushed shortcut transformations into general availability in April. Put plainly, the platform story is moving away from “pick one stack forever” and toward “publish governed products that multiple engines can use.”

That matters because too many teams still call a dataset a product. Microsoft’s current guidance is more precise than that. A data product has defined shape, interfaces, maintenance expectations, and refresh cycles. It is processed for analytical use, and it should be discoverable, secure, interoperable, and valuable enough to serve downstream consumers without forcing them back into raw source complexity. The OneLake Catalog is now described in the Cloud Adoption Framework as a unified access surface for approved data products across Fabric and external processing platforms such as Databricks.

That is where Quantum Regression is a useful teaching device. I am not using it here as a product feature – or even going into the technology -, I am using it as a way to think about how data products enable modular expansions to the data estate. In this framing, a data product is an independent transformation: it accepts ingestions, applies logic, enforces quality and policy, and surfaces results. The file, table, semantic model, or API is only the current observable state of that transformation. The product is the operator, not the artifact. That is the mindset shift that helps make interoperability useful instead of merely fashionable.

When the Thing You Care About Almost Never Happens: Rare-Event Modeling as a Fabric Data Product

Rare events are where the money is.

In financial services, the outcomes that barely show up in your data—fraud, default, AML hits, account takeover, operational losses—are the same outcomes that drive outsized loss, regulatory exposure, and customer harm. They’re also the outcomes most likely to embarrass a team that treats model building like a Kaggle exercise: train/test split, maximize accuracy, ship the AUC, call it done.

In this post, I’ll walk through practical techniques for analyzing rare-event problems, why they’re disproportionately valuable in #FinancialServices, how to build them in #MicrosoftFabric’s Data Science and ML capabilities, and then how to pivot from “a model” to “a data product” in the sense we use here: reusable, trustworthy, owned, composable, and contract-driven.

From Telemetry to Trust: Using FUAM + Purview Lineage to Make Fabric Governance Pay Off

If you’re running Microsoft Fabric at any real scale, you’ve probably felt the tension: the platform makes it easy to build, share, and iterate—but it also makes it easy to spend, sprawl, and accidentally ship the wrong answer.

The good news is you already have most of the raw ingredients to fix that. What’s missing is an operating model that converts “platform signals” into business outcomes: predictable costs, cleaner estates, and faster response when data is wrong.

In this post I’ll walk through three practical patterns:

using FUAM as a telemetry backbone for FinOps that people will actually use
using the same signals for stale workspace detection (without manual audits)
combining Microsoft Purview lineage with usage signals to identify incorrect datasets that are actively being consumed—and contain the blast radius

Along the way, I’ll stay grounded in business value: what these ideas buy you in dollars, time, and trust.

The Chief Risk Officer’s Quiet Obsession: Data Platforms and Data Products

A Chief Risk Officer (CRO) at an FSS Corporation rarely wakes up thinking, “I can’t wait to talk about data architecture today.”

But they do wake up thinking about something that inevitably leads back to it:

Can I trust what we’re about to tell the Board, the regulator, and the market—especially when conditions get ugly?

That question is why the CRO cares deeply about your Data Platform and your Data Products. Not as “tech initiatives,” but as the machinery that turns risk from opinions and spreadsheets into repeatable, auditable decisions the business can stand behind.

In this post, I’ll connect the CRO’s mandate to the practical realities of platforms and products—and why getting this right is a risk control, not a nice-to-have. Along the way, you’ll see why risk management and operational resilience don’t live in policy binders—they live in data.

Stop Picking a “Winner”: Data Product Interoperability Between Databricks and Fabric in Financial Services

Financial services teams have a familiar argument: “Are we a Databricks shop or a Fabric shop?” It sounds like a strategic question, but it usually hides the real problem—different parts of the business need different ways to use the same data, under tight controls, with clear auditability.

When Databricks and Microsoft Fabric interoperate at the data product level, the conversation shifts from which platform to how the data must be used: BI and semantic models, heavy Spark engineering, real-time analytics, governed sharing across boundaries, or advanced ML. The platform becomes a means, not the decision.

In this post I’ll lay out what “data product level interoperability” looks like in practice, why it enables responsible best-of-breed choices in regulated environments, and how it plays out in both directions: Databricks → Fabric and Fabric → Databricks.

Beyond the Medallion: Building Fabric Data Products with Schemas, Materialized Lake Views, and a “Surface Area” Contract

If you’ve been around modern analytics platforms for more than five minutes, you’ve probably built (or inherited) a medallion architecture: bronze → silver → gold. It’s familiar, it’s easy to draw on a whiteboard, and it’s often the first stable pattern teams reach for.

But there’s a quiet problem hiding in that simplicity: the number of sublayers tends to grow, and the complexity of each layer tends to balloon. Before long, you’re not designing a data product—you’re running an assembly line of multi-step transforms, hand-managed orchestration, and fragile dependencies.

Microsoft Fabric is starting to give us a different move: instead of treating transformation as a few “big” layers, you can treat it as a series of small, composable steps—and let the platform manage the dependency graph.

In this article, I’m going to connect three ideas:

Lakehouse schemas as your unit of organization (and the boundary between “internal plumbing” and “published contract”)
Materialized Lake Views as the declarative engine that builds (and refreshes) a dependency graph for you
A surface area schema designed to be shortcutted into other workspaces—so each workspace becomes an “analytical microservice” with its own interface, security boundary, and versioning story

Along the way, we’ll introduce a pragmatic versioning approach: create a new schema for major versions so breaking changes get semantic versioning “for free.”

The Advanced Lakehouse Data Product: Shortcuts In, Materialized Views Through, Versioned Schemas Out

There’s a familiar tension in modern analytics: teams want data products that are easy to discover and safe to consume, but they also want to move fast—often faster than the governance model can tolerate.

In Microsoft Fabric, that tension frequently shows up as a perception of workspace sprawl. A “single product per workspace” model is clean on paper—strong boundaries, tidy ownership, straightforward promotion—but it can quickly turn into dozens (or hundreds) of workspaces to curate, secure, and operate.

This post proposes a different pattern—an advanced lakehouse approach that treats the lakehouse itself like a product factory:

Shortcuts or schema shortcuts become the input layer (a clean, contract-aware “ingest without copying” boundary).
A small-step transformation layer is implemented as a multi-step DAG using Materialized Lake Views (MLVs).
A versioned, schema-based surface area becomes the data product contract you expose to consumers.

Then we connect that to OneLake security and Fabric domains, showing how you can expose left-shifted data products (usable earlier in the lifecycle) without letting workspaces multiply endlessly.

Gold as the Contract: Schema Evolution, Data Products, and Governance in Microsoft Fabric

A schema change is rarely “just a schema change.”

It’s the moment an upstream team’s perfectly reasonable adjustment becomes a downstream team’s broken report, confusing metric, or silent misinterpretation. And that’s why schema evolution has always been a source of anxiety: a schema isn’t simply structure—it’s an interface.

In this post, I’ll do three things. First, I’ll ground why schema evolution has historically been such a persistent concern. Next, I’ll reframe Medallion with Gold as the published surface area of the data product, and Silver as an optional workshop layer where data is supplemented and transformed. Finally, I’ll connect that design to how Microsoft Fabric supports it today—especially the post‑Ignite direction around governance, security, and semantics.