EduDataSci – Educating the world about data and leadership – Data Strategy from a one time K-12 data architect

The Workspace Is Not the Product: Fabric Objects as Code and the Rise of Lakehouse as Code

A Microsoft Fabric deployment can finish successfully and still leave you with an empty Lakehouse.

The notebook arrives. The pipeline arrives. The Lakehouse item appears in the target workspace. References are rebound to the correct environment. Every deployment task is green. Then the team opens the Lakehouse and discovers that the schemas, Delta tables, Spark views, and materialized lake views expected by the solution are not there.

That outcome is not a contradiction. It exposes an architectural boundary that Fabric teams need to make explicit: deploying a Fabric item is not the same as constructing the state inside that item.

The distinction matters in any enterprise. It matters more in financial services, where a deployment may support card authorization controls, wealth positions, settlement operations, lending decisions, or insurance exposure reporting. A release that creates the application shell without the required internal structures has not delivered a usable data product, however successful the pipeline log may look.

This article updates the idea of Fabric objects as code for the platform Microsoft is shipping in 2026. Fabric now has a much more coherent delivery stack: Terraform for stable platform resources, Git-backed item definitions, Variable Libraries for configuration, fabric-cicd and Fabric CLI for scripted deployment, and materialized lake views for declarative Lakehouse transformations. The result is not one universal deployment tool. It is a layered architecture in which each form of state is managed by the mechanism suited to it.

When Everyone Can Build, the Specification Becomes the Control Plane

One of the easiest mistakes in the current AI cycle is to treat vibe coding as a developer-productivity story.

It is certainly that. A developer can use an agent to explore a repository, create a plan, write code, run tests, and open a pull request. But the more important change is happening outside the traditional development organization. A claims analyst, credit-policy manager, operations lead, product owner, or adviser-support team can increasingly describe a problem and receive a working application, automation, or analytical tool in return.

OpenAI reported in June 2026 that knowledge workers represented about 20 percent of Codex users and were increasingly using it to build lightweight tools that previously required engineering support. That is one vendor’s usage data, not a census of enterprise development, but it captures the direction: the boundary between understanding a business problem and being able to build software around it is getting thinner. OpenAI: Codex for knowledge work

That changes the control problem.

The old governance model assumed that code entered through an engineering workflow, where developers, architects, security teams, and reviewers could resolve what the requirements left unclear. Democratized development weakens that assumption. The answer is not to force every useful idea back through the old bottleneck. It is to move the primary control point earlier—from reviewing code after it exists to governing the specification that tells people and agents what may be built.

When software creation is democratized, the specification becomes the enterprise control plane.

This post explains why spec-driven development has become much more important, what changed in the tooling during 2025 and 2026, and what a fully integrated enterprise specification should contain. The goal is not more paperwork. It is executable clarity before autonomous execution.

Stop Publishing the Workshop: An Ingest–Transform–Surface Architecture for Microsoft Fabric

Most Microsoft Fabric architecture diagrams are too polite.

They show data arriving, becoming cleaner, and eventually appearing in a report. What they usually leave out is the argument that determines whether the platform remains understandable after the third development team, the tenth source system, and the fiftieth downstream consumer: where does source reality end, where does product construction happen, and where does the enterprise make a promise?

When those boundaries are unclear, every table is treated as equally public. Analysts connect to whichever object is easiest to find, engineers preserve internal tables because somebody may be using them, and source-facing objects begin to look like product interfaces. The platform may still run, but the architecture has stopped communicating intent.

This article proposes a deliberately simple alternative for a schema-enabled Fabric Lakehouse:

Every ingestion method lands data in, or exposes it through, ingest.*.
Materialized Lake Views (MLVs) and notebooks move data into transform.*.
MLVs publish stable, consumption-ready tables into surface.*.
Consumers connect to the product through governed interfaces built on surface.*, not through the internal construction layers.

Here, ingest.* is shorthand for every table in the ingest schema. The same applies to transform.* and surface.*; the asterisk is architectural notation, not part of the schema name.

The result is not another three-color diagram. It is a practical way to separate source capture, internal implementation, and the published contract of a data product.

Microsoft Fabric Finally Has a Future Tense: What Fabric Plan Is Actually For

Every enterprise data platform eventually reaches the same awkward handoff.

It can reconcile billions of transactions, refresh risk positions every few minutes, and explain exactly why last quarter missed plan. Ask what the business expects next quarter, though, and someone opens Excel.

That is not an argument against Excel. Excel remains one of the best individual modeling tools ever built. The trouble begins when one person’s model becomes a shared business process. Now the organization needs common dimensions, controlled versions, security, comments, approvals, and a durable place to write the result. A workbook can imitate all of that. It is not very good at becoming all of that.

Fabric Plan is Microsoft’s attempt to bring that process into Fabric. A Plan item reads actuals and measures from a Power BI semantic model, lets the business enter assumptions and build scenarios, and writes configured planning data to Fabric SQL. Intelligence sheets then put the plan and actuals side by side for review.

Despite the name, this has nothing to do with Fabric capacity planning or Microsoft Planner. It also goes well beyond adding writeback to Power BI. Microsoft is positioning Plan as an enterprise and corporate performance management workload: budgets, forecasts, scenarios, operational plans, approvals, writeback, and variance reporting inside Fabric IQ.

The interesting questions are where that architecture fits, where it does not, and whether the current implementation is ready for the controls financial-services organizations require.

Stop Thinking in Rows: Databases Make More Sense as Sets

A database table looks like a spreadsheet. That resemblance is useful—and damaging. It makes tables approachable, but encourages a procedural picture: start at the top, inspect a row, make a decision, and move on. That is usually the wrong mental model.

Rows still matter. They have keys, carry facts, and occasionally ruin your afternoon. But the more useful unit of thought is the set: the population of tuples that should exist at a particular stage of processing.

With that shift, SQL operators construct populations. This post develops that mental model, then recasts the EduDataSci freeze-and-squash pattern for a Fabric MLV change feed as small set operations.

The tuple is the unit of assertion; the set is the unit of thought

Suppose a lender receives a monthly snapshot of every commercial loan. “One row per loan per month” states the grain of loan_snapshot, but not its meaning. A better description is:

loan_snapshot is the set of loan-state observations supplied by the servicing system, keyed by loan and snapshot date.

Each tuple asserts a loan’s balance, servicing status, risk grade, and other attributes at a time. “Find the delinquent loans” now defines a subset rather than a row-by-row flagging process. “Find loans missing from the new complete snapshot” defines the difference between two key sets. The database decides how to produce those populations; our first job is to define them correctly.

The Internet After Apps: Public, Private, and Personal AODPs in an Intent-Driven World

When software can be generated for a person, a task, or a moment, the durable architecture must move out of the application and into the interface to data and action.

A wealth advisor finishes a client call and asks an agent to build a small application.

The application should assemble the household’s accounts, identify securities that could raise $75,000, preserve the client’s income targets, avoid restricted positions, estimate tax consequences, and prepare an explanation for the client. It is not intended to become the next enterprise advisor platform. It may only be needed for the next hour.

The software could be generated almost immediately. That is the appealing part.

The dangerous part is everything the software must touch.

It needs market information, client preferences, account positions, tax lots, suitability rules, firm policies, trading restrictions, approval thresholds, and access to an order-management system. It must know which information belongs to the client, which belongs to the firm, and which comes from public or commercial sources. It must distinguish between calculating a proposed trade and submitting one. It must preserve evidence of what it read, why it made a recommendation, who approved it, and what eventually changed.

The application may be disposable. Its authority cannot be.

This is the architectural problem hiding underneath intent-driven computing and enterprise vibe coding. As software becomes easier to generate, more of the burden moves onto the interfaces through which that software reaches data, policy, and operational systems. We cannot make every dynamically generated application trustworthy in the same way that we certify a major enterprise platform. We can, however, force every application to operate through trustworthy products.

That is the purpose of the Action-Oriented Data Product, or AODP.

An AODP is a reusable, domain-owned product that exposes governed data and governed actions through a secure, discoverable, contract-driven interface. AODPs can exist in public, private, and personal spaces. Connected through a mesh, they provide the durable substrate underneath people, agents, and software assembled for individual tasks.

The internet built around pages and applications does not disappear. It gains another layer: an internet of governed data and action.

The Pipeline You Don’t Have to Build

There is a peculiar habit in data engineering: we measure sophistication by the machinery we build. A source drops CSV files into storage, so we create a pipeline. The pipeline needs a schedule, parameters, schema handling, retries, logging, alerts, deployment rules, credentials, and somebody willing to answer for it at 2:00 a.m. None of that is especially difficult. That is precisely the problem. Enterprises spend an astonishing amount of engineering time repeatedly solving work that is too ordinary to deserve bespoke engineering.

Microsoft Fabric Shortcut Transformations attack that problem at the right level. They convert supported files referenced through a OneLake shortcut into a managed Delta table, keep the table synchronized with the source, and expose the result to SQL, Spark, Power BI, and other Fabric consumers. CSV, Parquet, and JSON transformations are generally available. Excel support is in preview. A separate public-preview capability applies built-in language processing to text files for summarization, translation, sentiment analysis, personally identifiable information detection, and named-entity recognition.

That feature list is useful. The architectural consequence is more important.

Shortcut Transformations move routine ingestion from something every project must build into something the platform simply does.

That is the difference between accelerating a pipeline and deleting the need for one.

Fabric Runtime 2.0 Changes the Fabric Engineering Baseline

A Fabric notebook can contain six lines of PySpark and still depend on an enormous amount of technology that is not visible in those six lines. The result depends on the version of Spark that planned the work, the Python or Scala runtime that interpreted the code, the Delta implementation that read and wrote the tables, the Java virtual machine underneath Spark, the operating system beneath that, and the Fabric services that connected the workload to OneLake and compute.

That hidden stack is easy to ignore while it remains stable. Fabric Runtime 2.0 is the moment when ignoring it becomes dangerous.

Runtime 2.0 is now the latest generally available Fabric Spark runtime. It moves the platform to Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13, R 4.5.2, and Azure Linux 3.0. Microsoft currently requires customers to opt into it, but plans to make it the default selection for new workspaces and Environment items in late September 2026.

Those facts make Runtime 2.0 sound like a substantial but conventional platform upgrade. It is more important than that. Runtime 2.0 brings together a new Spark generation, a newer Delta protocol, native execution, lower-latency streaming, modern language runtimes, stronger environment management, and a clearer lifecycle for Fabric applications.

The significance is not that every Fabric workload suddenly needs the newest feature. The significance is that the runtime has become part of the application’s architecture, operating model, performance model, and deployment contract.

Govern the Decision, Not Just the Model

An AI model can be technically sound and still produce an indefensible business outcome.

Imagine a wealth-management copilot that produces a reasonable recommendation. The arithmetic works and nothing looks obviously wrong. It also used stale household information, retrieved research approved for institutional clients, missed an account restriction, and placed the recommendation into a client email before an adviser reviewed it.

None of those failures necessarily means the model was defective. They mean the organization governed the model while leaving the decision system around it underdefined.

That distinction is becoming central to enterprise AI. Organizations do not take risk simply because a model exists. Risk appears when model output changes a customer interaction, a financial decision, an operational action, or the movement of money. The proper unit of governance is therefore not just the model. It is the complete AI-enabled decision: its purpose, data, retrieved knowledge, prompts, rules, models, tools, human authority, resulting action, and retained evidence.

This post explains why AI governance has become its own operating discipline, how it connects policy to observable behavior and evidence, and why good governance expands the set of AI uses an organization can responsibly place into production.

The key control of AI governance is not the model. It is the decision the enterprise makes with it.

Fabric Can Finally Monitor Itself: Building a Real-Time Capacity Control Loop

Most Fabric capacity incidents are diagnosed in reverse.

A report slows down. A refresh misses its window. A user sees a capacity-limit error. Then someone opens the Capacity Metrics app and works backward through the evidence to find out what happened.

That is useful monitoring, but it is still an autopsy.

Capacity Overview Events change the sequence. Microsoft made them generally available in this month, giving Fabric a near-real-time stream of each capacity’s smoothed utilization and throttling posture. A Summary event represents a 30-second window for an active capacity, except that all-zero windows are suppressed. State events arrive when status changes. Those signals can flow through Real-Time hub and Eventstream, land in Eventhouse, appear on a Real-Time Dashboard, trigger Fabric Activator, and ultimately drive a guarded scaling action through Azure Resource Manager.

This is also a useful way to reintroduce Real-Time Intelligence. RTI is sometimes described as Fabric’s specialist workload for telemetry, clickstreams, and IoT. Capacity Overview Events show the larger idea. Fabric can use its own event-driven architecture to observe and operate Fabric.

The goal is not “add an alert when utilization reaches 80 percent.” It is an architecture that turns capacity management into a controlled feedback loop.