Microsoft Fabric – Page 2 – EduDataSci – Educating the world about data and leadership

Definitions Move. Data Stays: The Updated EduDataSci CI/CD Framework for Microsoft Fabric

A Fabric deployment can succeed while the target lakehouse is empty, the wrong Variable Library value set is active, a shortcut still points to Development, and the lakehouse depends on the identity of someone who has left the company.

The API did not lie. The architecture asked it the wrong question.

When I wrote about the ideal Fabric CI/CD approach, the division of labor was clear. Git controls change. Workspaces isolate execution. Deployment Pipelines promote approved content. External CI/CD adds tests and approvals. When native promotion cannot finish the job, fabric-cicd moves item definitions and an idempotent configuration notebook brings the lakehouse into the required state.

That architecture needs an update, not a replacement. Microsoft now frames Git integration, Deployment Pipelines, REST APIs, Variable Libraries, Fabric CLI, Terraform, and fabric-cicd as one Fabric CI/CD platform. Fabric can transport and bind more of the portable definition while exposing better APIs for target-local state.

In this framework, the code plane moves portable definitions and the data plane converges mutable target state. A release is complete only when both have happened—and the release can prove it.

Choose the Publication Contract, Not the Fabric Icon

A bank’s credit-risk reporting team needs one change before the monthly committee pack. The underlying exposure data is current. The semantic model can reach it. The requested change—reclassifying a set of delinquency bands—is not particularly difficult.

Yet the team that owns the report cannot publish the change. The curated table is maintained by a Spark process owned elsewhere, so the request becomes a ticket, a handoff, and a negotiation over another team’s release calendar. Nothing in Microsoft Fabric is broken. The architecture treated the point where data was published as a storage destination when it was really a contract with an owner.

The reverse failure is just as common. A data-engineering team already publishes well-governed Delta tables for analytics, data science, and downstream domains, but an architectural standard requires every Gold table to be copied into a Warehouse before Power BI can use it. The second published copy adds another load, another security boundary, another incident path, and another opportunity for the two versions to disagree. It exists because a diagram said Gold should look like a database.

Fabric Warehouse and Lakehouse overlap too much for the old rule—structured data goes to a warehouse; everything else goes to a lakehouse—to make this decision well. Both use Delta in OneLake. Both can serve SQL queries. Both can support Power BI and governed analytical products. The consequential difference is the publication contract each item is being asked to own and the team expected to keep it.

Choose a Lakehouse publication contract when the authoritative product is maintained as Delta through engineering workflows and SQL consumers primarily need to read it. Choose a Warehouse publication contract when the owning team must create, populate, correct, secure, and release a relational product through T-SQL, especially when multi-table transactions or an independent SQL-owned release cadence matter. Use both when two real contracts exist—not because one Fabric icon feels unfinished without the other.

The Workspace Is Not the Product: Fabric Objects as Code and the Rise of Lakehouse as Code

A Microsoft Fabric deployment can finish successfully and still leave you with an empty Lakehouse.

The notebook arrives. The pipeline arrives. The Lakehouse item appears in the target workspace. References are rebound to the correct environment. Every deployment task is green. Then the team opens the Lakehouse and discovers that the schemas, Delta tables, Spark views, and materialized lake views expected by the solution are not there.

That outcome is not a contradiction. It exposes an architectural boundary that Fabric teams need to make explicit: deploying a Fabric item is not the same as constructing the state inside that item.

The distinction matters in any enterprise. It matters more in financial services, where a deployment may support card authorization controls, wealth positions, settlement operations, lending decisions, or insurance exposure reporting. A release that creates the application shell without the required internal structures has not delivered a usable data product, however successful the pipeline log may look.

This article updates the idea of Fabric objects as code for the platform Microsoft is shipping in 2026. Fabric now has a much more coherent delivery stack: Terraform for stable platform resources, Git-backed item definitions, Variable Libraries for configuration, fabric-cicd and Fabric CLI for scripted deployment, and materialized lake views for declarative Lakehouse transformations. The result is not one universal deployment tool. It is a layered architecture in which each form of state is managed by the mechanism suited to it.

Stop Publishing the Workshop: An Ingest–Transform–Surface Architecture for Microsoft Fabric

Most Microsoft Fabric architecture diagrams are too polite.

They show data arriving, becoming cleaner, and eventually appearing in a report. What they usually leave out is the argument that determines whether the platform remains understandable after the third development team, the tenth source system, and the fiftieth downstream consumer: where does source reality end, where does product construction happen, and where does the enterprise make a promise?

When those boundaries are unclear, every table is treated as equally public. Analysts connect to whichever object is easiest to find, engineers preserve internal tables because somebody may be using them, and source-facing objects begin to look like product interfaces. The platform may still run, but the architecture has stopped communicating intent.

This article proposes a deliberately simple alternative for a schema-enabled Fabric Lakehouse:

Every ingestion method lands data in, or exposes it through, ingest.*.
Materialized Lake Views (MLVs) and notebooks move data into transform.*.
MLVs publish stable, consumption-ready tables into surface.*.
Consumers connect to the product through governed interfaces built on surface.*, not through the internal construction layers.

Here, ingest.* is shorthand for every table in the ingest schema. The same applies to transform.* and surface.*; the asterisk is architectural notation, not part of the schema name.

The result is not another three-color diagram. It is a practical way to separate source capture, internal implementation, and the published contract of a data product.

Microsoft Fabric Finally Has a Future Tense: What Fabric Plan Is Actually For

Every enterprise data platform eventually reaches the same awkward handoff.

It can reconcile billions of transactions, refresh risk positions every few minutes, and explain exactly why last quarter missed plan. Ask what the business expects next quarter, though, and someone opens Excel.

That is not an argument against Excel. Excel remains one of the best individual modeling tools ever built. The trouble begins when one person’s model becomes a shared business process. Now the organization needs common dimensions, controlled versions, security, comments, approvals, and a durable place to write the result. A workbook can imitate all of that. It is not very good at becoming all of that.

Fabric Plan is Microsoft’s attempt to bring that process into Fabric. A Plan item reads actuals and measures from a Power BI semantic model, lets the business enter assumptions and build scenarios, and writes configured planning data to Fabric SQL. Intelligence sheets then put the plan and actuals side by side for review.

Despite the name, this has nothing to do with Fabric capacity planning or Microsoft Planner. It also goes well beyond adding writeback to Power BI. Microsoft is positioning Plan as an enterprise and corporate performance management workload: budgets, forecasts, scenarios, operational plans, approvals, writeback, and variance reporting inside Fabric IQ.

The interesting questions are where that architecture fits, where it does not, and whether the current implementation is ready for the controls financial-services organizations require.

The Pipeline You Don’t Have to Build

There is a peculiar habit in data engineering: we measure sophistication by the machinery we build. A source drops CSV files into storage, so we create a pipeline. The pipeline needs a schedule, parameters, schema handling, retries, logging, alerts, deployment rules, credentials, and somebody willing to answer for it at 2:00 a.m. None of that is especially difficult. That is precisely the problem. Enterprises spend an astonishing amount of engineering time repeatedly solving work that is too ordinary to deserve bespoke engineering.

Microsoft Fabric Shortcut Transformations attack that problem at the right level. They convert supported files referenced through a OneLake shortcut into a managed Delta table, keep the table synchronized with the source, and expose the result to SQL, Spark, Power BI, and other Fabric consumers. CSV, Parquet, and JSON transformations are generally available. Excel support is in preview. A separate public-preview capability applies built-in language processing to text files for summarization, translation, sentiment analysis, personally identifiable information detection, and named-entity recognition.

That feature list is useful. The architectural consequence is more important.

Shortcut Transformations move routine ingestion from something every project must build into something the platform simply does.

That is the difference between accelerating a pipeline and deleting the need for one.

Fabric Runtime 2.0 Changes the Fabric Engineering Baseline

A Fabric notebook can contain six lines of PySpark and still depend on an enormous amount of technology that is not visible in those six lines. The result depends on the version of Spark that planned the work, the Python or Scala runtime that interpreted the code, the Delta implementation that read and wrote the tables, the Java virtual machine underneath Spark, the operating system beneath that, and the Fabric services that connected the workload to OneLake and compute.

That hidden stack is easy to ignore while it remains stable. Fabric Runtime 2.0 is the moment when ignoring it becomes dangerous.

Runtime 2.0 is now the latest generally available Fabric Spark runtime. It moves the platform to Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13, R 4.5.2, and Azure Linux 3.0. Microsoft currently requires customers to opt into it, but plans to make it the default selection for new workspaces and Environment items in late September 2026.

Those facts make Runtime 2.0 sound like a substantial but conventional platform upgrade. It is more important than that. Runtime 2.0 brings together a new Spark generation, a newer Delta protocol, native execution, lower-latency streaming, modern language runtimes, stronger environment management, and a clearer lifecycle for Fabric applications.

The significance is not that every Fabric workload suddenly needs the newest feature. The significance is that the runtime has become part of the application’s architecture, operating model, performance model, and deployment contract.

Fabric Can Finally Monitor Itself: Building a Real-Time Capacity Control Loop

Most Fabric capacity incidents are diagnosed in reverse.

A report slows down. A refresh misses its window. A user sees a capacity-limit error. Then someone opens the Capacity Metrics app and works backward through the evidence to find out what happened.

That is useful monitoring, but it is still an autopsy.

Capacity Overview Events change the sequence. Microsoft made them generally available in this month, giving Fabric a near-real-time stream of each capacity’s smoothed utilization and throttling posture. A Summary event represents a 30-second window for an active capacity, except that all-zero windows are suppressed. State events arrive when status changes. Those signals can flow through Real-Time hub and Eventstream, land in Eventhouse, appear on a Real-Time Dashboard, trigger Fabric Activator, and ultimately drive a guarded scaling action through Azure Resource Manager.

This is also a useful way to reintroduce Real-Time Intelligence. RTI is sometimes described as Fabric’s specialist workload for telemetry, clickstreams, and IoT. Capacity Overview Events show the larger idea. Fabric can use its own event-driven architecture to observe and operate Fabric.

The goal is not “add an alert when utilization reaches 80 percent.” It is an architecture that turns capacity management into a controlled feedback loop.

A Data Product Is an Engine, Not a Table: FabCon 2026, Databricks, Fabric, and the Case for Interoperability

FabCon and SQLCon 2026 made the Microsoft Fabric and Azure Databricks story more concrete. The headline changes were not cosmetic. Microsoft moved zero-copy access to OneLake data from Azure Databricks into public preview, made Direct Lake in OneLake generally available, kept expanding Databricks-to-Fabric mirroring, and then pushed shortcut transformations into general availability in April. Put plainly, the platform story is moving away from “pick one stack forever” and toward “publish governed products that multiple engines can use.”

That matters because too many teams still call a dataset a product. Microsoft’s current guidance is more precise than that. A data product has defined shape, interfaces, maintenance expectations, and refresh cycles. It is processed for analytical use, and it should be discoverable, secure, interoperable, and valuable enough to serve downstream consumers without forcing them back into raw source complexity. The OneLake Catalog is now described in the Cloud Adoption Framework as a unified access surface for approved data products across Fabric and external processing platforms such as Databricks.

That is where Quantum Regression is a useful teaching device. I am not using it here as a product feature – or even going into the technology -, I am using it as a way to think about how data products enable modular expansions to the data estate. In this framing, a data product is an independent transformation: it accepts ingestions, applies logic, enforces quality and policy, and surfaces results. The file, table, semantic model, or API is only the current observable state of that transformation. The product is the operator, not the artifact. That is the mindset shift that helps make interoperability useful instead of merely fashionable.

Before the Capacity Fire Starts: Why FUAM Belongs in Every FSS Fabric Baseline

Most Fabric monitoring conversations begin too late.

They begin when a workspace is already noisy, when refreshes are already failing, or when capacity pressure has already become visible enough to trigger concern. By that point, the real problem is usually larger than utilization. It is a visibility problem. Teams do not have a durable, tenant-level view of what exists, what changed, what is connected to source control, what is actively used, and where governance has started to drift. That is the gap FUAM was designed to close. The distinction worth making is not between a “simple” and a “serious” version of monitoring, but between an original governance-first package and the newer Fabric Toolbox implementation that extends that foundation with Capacity Metrics and deeper operational analysis.

That distinction matters for FSS environments because it changes the order of operations. The older GT-Analytics package, labeled FUAM Basic, focused on the data that can be gathered without reading Capacity Metrics. The current Fabric Toolbox implementation keeps the same broad monitoring vision, but adds Capacity Metrics as part of a larger platform-admin monitoring solution. If the question is what should be present by default in an FSS Fabric setup, the answer starts with the governance-and-inventory layer and then grows into the richer Capacity Metrics-enabled implementation as operational maturity increases.