At 9:42 a.m., an institutional equity trade executes normally. There is no break and no exception to repair.
By 2:00 p.m., the picture has changed. The client’s allocation is later than usual. The standing settlement instruction (SSI) on one record does not agree with the custodian’s current instruction. Securities expected from another account have not arrived. A routine counterparty acknowledgment is still missing.
None of those facts proves that the trade will fail. Together, they say that the trade is becoming harder to settle.
The traditional operating model waits for a definite exception, places it in a queue and asks an operations analyst to reconstruct what happened. A prevention model starts earlier. It watches the trade’s full lifecycle, recognizes a risky pattern, applies the firm’s control rules and intervenes while useful choices remain.
This is not a proposal for a better exception queue. It is a different job for operations: prevent avoidable failures, resolve safe and bounded issues automatically, bring judgment-heavy cases to the right expert, and remove recurring defects from the upstream process.
A Fabric deployment can succeed while the target lakehouse is empty, the wrong Variable Library value set is active, a shortcut still points to Development, and the lakehouse depends on the identity of someone who has left the company.
The API did not lie. The architecture asked it the wrong question.
When I wrote about the ideal Fabric CI/CD approach, the division of labor was clear. Git controls change. Workspaces isolate execution. Deployment Pipelines promote approved content. External CI/CD adds tests and approvals. When native promotion cannot finish the job, fabric-cicd moves item definitions and an idempotent configuration notebook brings the lakehouse into the required state.
That architecture needs an update, not a replacement. Microsoft now frames Git integration, Deployment Pipelines, REST APIs, Variable Libraries, Fabric CLI, Terraform, and fabric-cicd as one Fabric CI/CD platform. Fabric can transport and bind more of the portable definition while exposing better APIs for target-local state.
In this framework, the code plane moves portable definitions and the data plane converges mutable target state. A release is complete only when both have happened—and the release can prove it.
The Original Framework Still Holds
Feature workspaces isolate development; pull requests provide review and change control. The persistent integration workspace is the Development member of the product’s Dev/Test/Prod workspace triple. That triple represents an independently releasable data product or security boundary—not one workspace per table. Deployment Pipelines remain the default when Fabric’s native controls suffice; GitHub Actions or Azure Pipelines can orchestrate enterprise gates.
The code-first path has changed most. In February, I called fabric-cicd an escape hatch. Microsoft now documents an end-to-end automation pattern in which Terraform provisions workspaces, Git wiring, roles, and connections while the Microsoft-maintained, open-source fabric-cicd library deploys the items. Separate bulk-export and bulk-importendpoints can reduce item-by-item calls, but both are beta, evaluation-only, and not recommended for production. Faster transport does not turn data into source code.
The Fabric-native route uses Git for development and Deployment Pipelines for promotion. The code-first route calls item-definition APIs or uses fabric-cicd; fab deploy is its CLI front end. Both arrive at the same target boundary. Code-first deployment is reconciliation, not a benign copy: orphan deletion, especially hard deletion, belongs in the release plan, approval, and evidence contract. Each target item should have one transport owner. Deployment Pipeline pairing is lineage-based, so precreating a same-named item can produce a duplicate.
A Deployable Item Is Not a Deployable System
The word deployment hides five different responsibilities.
Git integration, Deployment Pipelines, fabric-cicd, or item-definition APIs
The target instance and its physical IDs
Environment configuration
Variable Library plus protected CI/CD settings
External release workflow and Fabric APIs
Active value set, credentials, unsupported bindings, associated identities, and permissions
Runtime data state
Desired-state manifest and approved migrations
Idempotent configuration notebook; DacFx or SqlPackage where supported
Delta schemas, tables, Spark views, materialized definitions, and controlled reference data
Evidence
Release manifest and policy contract
CI/CD logs, Fabric monitoring and audit, and Microsoft Purview
Operation IDs, callers, results, classifications, ownership, and lineage observations
Microsoft’s item-definition model gives the portable layer a stronger foundation. A logical ID identifies the same logical item across workspaces even when physical IDs differ. Individual APIs create, retrieve, and update supported items. Definitions are desired-state inputs, not complete descriptions of a running system.
The distinction is especially visible in a Lakehouse. Fabric now tracks Lakehouse metadata, internal and external shortcut definitions, and—in preview—OneLake data-access-role metadata through Git and Deployment Pipelines. It explicitly does not track Delta tables, Spark views, or files. Git and deployment operations preserve that data rather than overwriting it. The Lakehouse lifecycle documentation finally makes the boundary unambiguous.
Shortcut synchronization is authoritative: deployment creates, updates, and deletes shortcuts to match the definition. Internal targets can remap but are not created; external targets remain unchanged unless variables remap them. The notebook should validate this result, not recreate it. Notebook-to-Lakehouse auto-binding can replace a physical ID with a logical ID, although it is off by default and applies only to same-workspace relationships. The current dependency-binding matrix still shows important gaps: semantic-model SQL endpoints, some pipeline activities, cross-workspace references, Warehouse endpoints, and Variable Library item references can retain stage-specific identifiers.
The rule is no longer “put everything difficult in the notebook.” Maintain a versioned matrix by item type covering Git, Deployment Pipelines, definition APIs, fabric-cicd, dependency binding, service-principal support, and GA or preview status. Use the most declarative supported mechanism, then reconcile the remainder.
The Updated Architecture
The architecture below keeps the persistent Dev/Test/Prod product triple, but it separates delivery from convergence. The external release workflow—usually Azure Pipelines or GitHub Actions—holds the release identity, calls Fabric APIs, selects the target contract, and decides whether the release may continue. It orchestrates both planes; it is not another name for the notebook.
Definitions move through the code plane. The release workflow converges target-local state, then joins intent, execution, and governance evidence through one release ID.
The release identity is allowed to change the target. It is not automatically the identity the resulting workload should use to read or process data.
Most teams should start with the native hybrid: Git-connected development, a persistent integration workspace, Deployment Pipelines, a small external release workflow, and one deliberately boring idempotent configuration notebook. Terraform becomes valuable for a repeatable workspace factory or regional recovery. Use fabric-cicd when one approved release must deploy predictably across multiple workspaces, environments, or customers.
The Update Notebook Creates and Updates the Internal Structure
Promoting a Lakehouse moves the Lakehouse item, but it does not necessarily leave every schema, table, view, or materialized lake view inside that Lakehouse in the state required by the promoted release. The Update Notebook runs after promotion to close that gap. Its job is to create and update the internal elements of the promoted Lakehouse or Warehouse so that the target environment matches the structure defined for that version of the product.
The Update Notebook is not a general-purpose deployment controller. It should not create workspaces, assign capacity, select the active Variable Library value set, associate identities, remap shortcuts, or apply sensitivity labels. The release workflow completes those control-plane and environment-binding steps first, resolves the target item, and then invokes the notebook against that target.
Inside the data item, the notebook follows straightforward idempotent logic:
Current state in the target
Update Notebook action
The schema or object does not exist
Create it from the versioned definition.
The object exists and already matches the required definition
Leave it unchanged.
The object exists but can be brought to the required state safely
Update, alter, or replace it using the operation appropriate to that object type.
The required change is destructive or requires data migration
Stop and require an explicitly approved migration rather than silently forcing the change.
That is the practical meaning of idempotency in this framework: create if it does not exist; update if it exists; do nothing if it is already correct. CREATE TABLE IF NOT EXISTS by itself is not enough. It can create a missing table, but it cannot correct an existing table with the wrong columns, data types, properties, or constraints. The Update Notebook must inspect what is already present and use ALTER, CREATE OR REPLACE, a controlled migration, or another object-specific update operation to bring it to the required state.
For a Lakehouse, the Update Notebook can maintain schemas, Delta tables, table properties, Spark views, materialized lake views, and other internal objects that are not carried completely by promotion. For a Warehouse, it can maintain schemas, tables, views, procedures, and constraints through T-SQL when notebook-driven DDL is the selected structural owner. Where the DacFx-backed database-project path owns the Warehouse schema, the Update Notebook must not compete with it; it should validate that structure or manage only the objects explicitly assigned to the notebook.
Materialized Lake Views illustrate the pattern clearly. The MLV definition is versioned as code, but the MLV itself and its dependency graph exist inside the Lakehouse. After promotion, the Update Notebook creates the MLV when it is absent, updates or replaces it when its definition has changed, and leaves it alone when it already matches the promoted version. It then verifies the expected definition, dependencies, constraints, and refresh configuration. The materialized data did not move through Git; the notebook re-establishes the required internal object in the target environment.
The notebook must be safe to run more than once. A retry after a partial failure starts by reading the current catalog, not by assuming that nothing was created. Objects already in the correct state become no-ops; missing objects are created; safely outdated objects are updated. This allows each run to converge on the same required internal state without duplicating objects or replaying changes blindly.
For supported Spark and Jupyter jobs, the notebook-specific Job Scheduler API supports on-demand execution by a user, service principal, or managed identity; typed parameters; explicit Lakehouse and Environment selection; polling; cancellation; and a string exit value. That makes the Update Notebook a release gate rather than an informal post-deployment script. The release workflow can parse a JSON-serialized result such as:
{
"releaseId": "2026.08.14.3",
"manifestHash": "2d61...b7a9",
"objectsCreated": 1,
"objectsUpdated": 1,
"migrationsApplied": 0,
"unchanged": 17,
"blocked": 0,
"failed": 0,
"structurePostconditionsPassed": true
}
Fabric documents this exit-value pattern for conditional orchestration. The notebook serializes the result and passes the string to notebookutils.notebook.exit(...); the release workflow reads it from exitValue and deserializes it. The notebook should also append its object-level actions and validation results to a release-evidence table keyed by the same release ID. A second run against the same promoted version should report no unplanned structural changes.
One boundary still matters: notebookutils.variableLibrary does not support service principals, although Variable Library REST APIs do. An unattended workflow should activate and verify the target value set, resolve the required nonsecret values, and pass typed parameters to the Update Notebook. Secrets remain in Key Vault, managed Fabric connections, or the CI/CD secret store; workload identity federation can eliminate a stored workflow secret. Variable Libraries should carry nonsecret configuration and references, never secrets.
Identity Is Reconciled, Not Transported
Fabric now has several identities that sound similar and should not be allowed to collapse into one super-principal.
Identity
Scope
Job in the release
Release identity
CI/CD process
Provisions or deploys definitions and calls control-plane APIs
Notebook run identity
Individual job execution
Performs the approved data-plane reconciliation and validation
Workspace identity
Workspace
Authenticates supported connections and trusted workspace access
Associated or default identity
Supported item
Replaces the item’s dependency on a human owner for supported behaviors
These are roles, not necessarily four principals. An API-started notebook runs under the caller. A pipeline notebook runs as the pipeline’s last modified user; a scheduled notebook runs as its creator or last updater. Workspace and associated identities do not become the notebook execution identity. Workspace identity is a Fabric-managed service principal for supported connections and trusted access; it receives no workspace role by default and does not run notebooks. It is stage-local, yet an artifact using it currently updates from Git only into a workspace connected to that same identity.
Associated identities are the easiest new control to overstate. The current Associate Identity API is beta; Microsoft describes it as evaluation and development only and does not recommend it for production. It covers Lakehouses and Eventstreams, except Eventstreams using Azure or Fabric Events as a source. The only documented assignment type is Caller, so the intended principal must make the request as itself. The API can finish immediately or asynchronously; in either case, the workflow must inspect every child-item result.
For now, associated identity belongs behind an explicit preview control, not in the mandatory production path. Treat association as identity selection, not external authorization: provision storage and connection access separately, test the owner-dependent behavior, and only then retire the former grants.
Identity is not the only target attribute the release must verify. Sensitivity labels remain outside item definitions. An unattended workflow can read the current label, but the admin bulk-set API requires a Fabric Administrator user and does not support service principals or managed identities. The workflow should verify the label and route remediation unless a separately authorized delegated-admin step is available.
The Release Loop Is Convergent, Not Linear
The practical release sequence is a loop because external systems time out, Fabric operations are asynchronous, and a retry can begin after some mutations have already succeeded.
A retry starts by observing the target. It does not assume the preceding attempt did nothing.
The release workflow must distinguish two asynchronous contracts. Item-management operations return a Location for a long-running operation. An on-demand notebook also returns a Location, but it identifies a job instance. Both can return 202; they are not polled through the same endpoint. Honor Retry-After, use bounded retries, and preserve the returned operation or job ID before recovery. The run request exposes no caller idempotency key, so do not resubmit after a polling timeout. Reuse the job ID or re-inspect the target, and hold a per-target release lock to prevent competing reconcilers.
At minimum, the release manifest must bind the approved commit and desired-state hash to the target environment, identities, change plan, validation result, and Fabric deployment and job identifiers.
Those records answer a different question from Microsoft Purview. Purview can scan a Fabric tenant for metadata and lineage, and it can describe ownership, classification, and governed relationships. It does not prove which release principal made a particular change or whether every postcondition passed. CI/CD history and Fabric audit or monitoring provide execution evidence. Purview describes the governed asset. Joining them is more useful than pretending either is complete on its own.
What This Looks Like in Card Processing
Consider a credit-card settlement data product with a Lakehouse, transaction-standardization notebook, settlement pipeline, merchant semantic model, and a Variable Library. The same logical definitions move from Development to Test to Production. Each stage has its own workspace, capacity assignment, connections, notebook execution identity, supported item-identity associations, and data.
Production cardholder data does not ride the deployment pipeline into Test. The active value set selects nonsecret stage references. The notebook execution identity receives only the permissions required to reconcile data; workspace and associated identities receive only their own required grants. The idempotent configuration notebook evolves the Delta schemas absent from the Lakehouse definition, applies controlled reference-data changes, validates settlement grain and key constraints, and returns its JSON-serialized result. Only then does the release enable the consumer surface.
An auditor can now trace the release without reconstructing it from portal screenshots and personal recollection. The organization can show the approved commit, the principal that performed the release, the principal that executed the notebook, the nonsecret target references selected, and—where enabled—the associated identity used for owner-dependent Lakehouse behaviors. It can also show the data structures changed, controls evaluated, and lineage and classification observed after deployment. No cardholder data had to be copied between stages to produce that evidence.
That is the practical difference between shipping files and releasing a regulated data product.
A Release Ends at Convergence
Fabric’s stronger deployment surface does not eliminate idempotent notebooks. It gives them a narrower and more important job.
Git should describe reviewed change. Deployment Pipelines or code-first tooling should move every definition they can. Variable Libraries should describe environment meaning without holding secrets. The external release workflow should reconcile identity and target configuration. The idempotent configuration notebook should converge mutable Lakehouse state, reject unsafe drift, and prove its postconditions. Fabric monitoring and CI/CD logs should record what happened; Purview should place the released asset in its governed context.
Definitions move. Data stays. The release is finished when the target has converged—and when the evidence is good enough that someone other than the person who ran it can prove why it should be trusted.
Something important changes when intent becomes the front door to software. If a banker, underwriter, advisor, claims lead, or merchant-risk analyst can ask for a task-specific application and receive it in minutes, then bespoke software stops being rare. It becomes cheap. The expensive thing is no longer the application layer. The expensive thing is the interface to truth: the boundary where customer data, authorization, business policy, and systems of record meet.
That shift deserves a different architectural response. I like to frame a data product as reusable, trustworthy, contract-driven, loosely coupled asset with clear ownership, explicit promises about freshness and meaning, and a versioned surface consumers can rely on. The data mesh tradition adds domain ownership, self-serve platform capability, and federated computational governance. My ingest-transform-surface model sharpens the picture further: inputs are explicit, transformation is deliberate, and the published surface is the contract. All of that is right. It just is not enough once agents and intent-driven tools begin to mediate real work.
Domain APIs need to become the front door for fit-for-purpose apps, and the mesh becomes the process glue that turns operational data into analyzable insight and analytical insight into operational action. My argument is that we should now make that shift explicit. The next mesh should be built from Action-Oriented Data Products: domain-owned data products whose interfaces declare not only what can be read and analyzed, but what can be done, by whom, for what purpose, with which guardrails, and against which authoritative source.
This is not a new name for old API management. It is a claim about the next architectural quantum. Data mesh already argued that the data product is the quantum of the analytical estate. In an intent-mediated world, that quantum must become action-oriented. The mesh stays. The node changes. To make that case, I want to do three things: show why current data products stop short, explain the historical parallels to object orientation, microservices, and data mesh, and define the architecture that lets agents discover and manipulate data safely.
An enterprise agent rarely fails because it cannot produce a fluent paragraph. It fails because it cannot distinguish the current balance from the definition of balance, the approved policy from the email that discussed an exception, or the authority to explain a decision from the authority to change a system of record.
That distinction matters in financial services. A relationship manager asking why a commercial borrower has become riskier is really asking several questions at once. What changed in the borrower’s accounts? How does the exposure roll through facilities, collateral, guarantees, and legal entities? What does credit policy require? What did the deal team agree to last week? If action is warranted, who may open a review, request documents, or change a limit?
No single knowledge source can answer all of that honestly. The useful architecture is a Copilot Studio agent that coordinates several bounded kinds of intelligence: Fabric IQ for business state, Foundry IQ for institutional knowledge, Work IQ for the user’s working context, an ontology for shared meaning, and MCP tools for controlled action. API Center anchors design-time inventory and discovery, while Microsoft Purview anchors data governance and supported AI-interaction compliance. Runtime enforcement remains with Entra, Power Platform data policies, native IQ permissions, API Management, and the source systems.
“Can the new fund trade on Monday?” should be an easy question.
The relationship manager knows the asset manager. The client record is active. KYC is complete. Legal has a signed agreement, and credit has approved a limit. Every team appears to have done its part.
But operations is still opening an account. A tax form is attached to the parent company rather than the fund. The agreement covers the right legal entity but not the requested product in that jurisdiction. A market permission has not reached the entitlement system. On Friday afternoon, every function can report that its task is complete—or at least within its service level—while the client still cannot place a trade.
No single system is necessarily wrong. Each system is answering a smaller question than the client asked.
This is the gap that process mining can address. The technology is often discussed in connection with claims automation, where it can identify slow routes, repeated work, and unnecessary handoffs. Those are useful applications. But the same method can be even more valuable in financial-markets onboarding, where one commercial outcome depends on several functions and systems agreeing at the same time.
The goal is not merely to complete KYC faster. It is to make the client usable—safely, lawfully, and with a clear record of how that decision was reached.
A bank’s credit-risk reporting team needs one change before the monthly committee pack. The underlying exposure data is current. The semantic model can reach it. The requested change—reclassifying a set of delinquency bands—is not particularly difficult.
Yet the team that owns the report cannot publish the change. The curated table is maintained by a Spark process owned elsewhere, so the request becomes a ticket, a handoff, and a negotiation over another team’s release calendar. Nothing in Microsoft Fabric is broken. The architecture treated the point where data was published as a storage destination when it was really a contract with an owner.
The reverse failure is just as common. A data-engineering team already publishes well-governed Delta tables for analytics, data science, and downstream domains, but an architectural standard requires every Gold table to be copied into a Warehouse before Power BI can use it. The second published copy adds another load, another security boundary, another incident path, and another opportunity for the two versions to disagree. It exists because a diagram said Gold should look like a database.
Fabric Warehouse and Lakehouse overlap too much for the old rule—structured data goes to a warehouse; everything else goes to a lakehouse—to make this decision well. Both use Delta in OneLake. Both can serve SQL queries. Both can support Power BI and governed analytical products. The consequential difference is the publication contract each item is being asked to own and the team expected to keep it.
Choose a Lakehouse publication contract when the authoritative product is maintained as Delta through engineering workflows and SQL consumers primarily need to read it. Choose a Warehouse publication contract when the owning team must create, populate, correct, secure, and release a relational product through T-SQL, especially when multi-table transactions or an independent SQL-owned release cadence matter. Use both when two real contracts exist—not because one Fabric icon feels unfinished without the other.
A manufacturer asks its bank for a $3 million increase in a working-capital line before a supplier’s pricing window closes. The relationship manager sees an established client and a time-sensitive opportunity. The application treats the request as another file to assemble. A decision engine uses the credit model’s output to route it to manual review. The underwriter asks for another forecast. Operations measures every handoff against its service level.
Everyone performs their function. The bank still answers too late.
The remedies sound obvious. Simplify the application. Improve the model. Remove steps from underwriting. Each may help. Yet a cleaner application can still lead to a poor credit decision, a more accurate model can produce a recommendation no one knows how to use, and a faster process can move uncertainty downstream.
Design thinking identifies the human problem worth solving. Decision science makes the choice, uncertainty, economics, and constraints explicit. Improvement science determines whether the change works reliably—and under which conditions.
Used together, the three disciplines improve the immediate decision and preserve the evidence needed to improve the next one.
A Microsoft Fabric deployment can finish successfully and still leave you with an empty Lakehouse.
The notebook arrives. The pipeline arrives. The Lakehouse item appears in the target workspace. References are rebound to the correct environment. Every deployment task is green. Then the team opens the Lakehouse and discovers that the schemas, Delta tables, Spark views, and materialized lake views expected by the solution are not there.
That outcome is not a contradiction. It exposes an architectural boundary that Fabric teams need to make explicit: deploying a Fabric item is not the same as constructing the state inside that item.
The distinction matters in any enterprise. It matters more in financial services, where a deployment may support card authorization controls, wealth positions, settlement operations, lending decisions, or insurance exposure reporting. A release that creates the application shell without the required internal structures has not delivered a usable data product, however successful the pipeline log may look.
This article updates the idea of Fabric objects as code for the platform Microsoft is shipping in 2026. Fabric now has a much more coherent delivery stack: Terraform for stable platform resources, Git-backed item definitions, Variable Libraries for configuration, fabric-cicd and Fabric CLI for scripted deployment, and materialized lake views for declarative Lakehouse transformations. The result is not one universal deployment tool. It is a layered architecture in which each form of state is managed by the mechanism suited to it.
One of the easiest mistakes in the current AI cycle is to treat vibe coding as a developer-productivity story.
It is certainly that. A developer can use an agent to explore a repository, create a plan, write code, run tests, and open a pull request. But the more important change is happening outside the traditional development organization. A claims analyst, credit-policy manager, operations lead, product owner, or adviser-support team can increasingly describe a problem and receive a working application, automation, or analytical tool in return.
OpenAI reported in June 2026 that knowledge workers represented about 20 percent of Codex users and were increasingly using it to build lightweight tools that previously required engineering support. That is one vendor’s usage data, not a census of enterprise development, but it captures the direction: the boundary between understanding a business problem and being able to build software around it is getting thinner. OpenAI: Codex for knowledge work
That changes the control problem.
The old governance model assumed that code entered through an engineering workflow, where developers, architects, security teams, and reviewers could resolve what the requirements left unclear. Democratized development weakens that assumption. The answer is not to force every useful idea back through the old bottleneck. It is to move the primary control point earlier—from reviewing code after it exists to governing the specification that tells people and agents what may be built.
When software creation is democratized, the specification becomes the enterprise control plane.
This post explains why spec-driven development has become much more important, what changed in the tooling during 2025 and 2026, and what a fully integrated enterprise specification should contain. The goal is not more paperwork. It is executable clarity before autonomous execution.
Most Microsoft Fabric architecture diagrams are too polite.
They show data arriving, becoming cleaner, and eventually appearing in a report. What they usually leave out is the argument that determines whether the platform remains understandable after the third development team, the tenth source system, and the fiftieth downstream consumer: where does source reality end, where does product construction happen, and where does the enterprise make a promise?
When those boundaries are unclear, every table is treated as equally public. Analysts connect to whichever object is easiest to find, engineers preserve internal tables because somebody may be using them, and source-facing objects begin to look like product interfaces. The platform may still run, but the architecture has stopped communicating intent.
This article proposes a deliberately simple alternative for a schema-enabled Fabric Lakehouse:
Every ingestion method lands data in, or exposes it through, ingest.*.
Materialized Lake Views (MLVs) and notebooks move data into transform.*.
MLVs publish stable, consumption-ready tables into surface.*.
Consumers connect to the product through governed interfaces built on surface.*, not through the internal construction layers.
Here, ingest.* is shorthand for every table in the ingest schema. The same applies to transform.* and surface.*; the asterisk is architectural notation, not part of the schema name.
The result is not another three-color diagram. It is a practical way to separate source capture, internal implementation, and the published contract of a data product.
