Category: Tech Tuesday

A Fabric deployment can succeed while the target lakehouse is empty, the wrong Variable Library value set is active, a shortcut still points to Development, and the lakehouse depends on the identity of someone who has left the company.

The API did not lie. The architecture asked it the wrong question.

When I wrote about the ideal Fabric CI/CD approach, the division of labor was clear. Git controls change. Workspaces isolate execution. Deployment Pipelines promote approved content. External CI/CD adds tests and approvals. When native promotion cannot finish the job, fabric-cicd moves item definitions and an idempotent configuration notebook brings the lakehouse into the required state.

That architecture needs an update, not a replacement. Microsoft now frames Git integration, Deployment Pipelines, REST APIs, Variable Libraries, Fabric CLI, Terraform, and fabric-cicd as one Fabric CI/CD platform. Fabric can transport and bind more of the portable definition while exposing better APIs for target-local state.

In this framework, the code plane moves portable definitions and the data plane converges mutable target state. A release is complete only when both have happened—and the release can prove it.

The Original Framework Still Holds

Feature workspaces isolate development; pull requests provide review and change control. The persistent integration workspace is the Development member of the product’s Dev/Test/Prod workspace triple. That triple represents an independently releasable data product or security boundary—not one workspace per table. Deployment Pipelines remain the default when Fabric’s native controls suffice; GitHub Actions or Azure Pipelines can orchestrate enterprise gates.

The code-first path has changed most. In February, I called fabric-cicd an escape hatch. Microsoft now documents an end-to-end automation pattern in which Terraform provisions workspaces, Git wiring, roles, and connections while the Microsoft-maintained, open-source fabric-cicd library deploys the items. Separate bulk-export and bulk-importendpoints can reduce item-by-item calls, but both are beta, evaluation-only, and not recommended for production. Faster transport does not turn data into source code.

The Fabric-native route uses Git for development and Deployment Pipelines for promotion. The code-first route calls item-definition APIs or uses fabric-cicd; fab deploy is its CLI front end. Both arrive at the same target boundary. Code-first deployment is reconciliation, not a benign copy: orphan deletion, especially hard deletion, belongs in the release plan, approval, and evidence contract. Each target item should have one transport owner. Deployment Pipeline pairing is lineage-based, so precreating a same-named item can produce a duplicate.

A Deployable Item Is Not a Deployable System

The word deployment hides five different responsibilities.

Microsoft’s item-definition model gives the portable layer a stronger foundation. A logical ID identifies the same logical item across workspaces even when physical IDs differ. Individual APIs create, retrieve, and update supported items. Definitions are desired-state inputs, not complete descriptions of a running system.

The distinction is especially visible in a Lakehouse. Fabric now tracks Lakehouse metadata, internal and external shortcut definitions, and—in preview—OneLake data-access-role metadata through Git and Deployment Pipelines. It explicitly does not track Delta tables, Spark views, or files. Git and deployment operations preserve that data rather than overwriting it. The Lakehouse lifecycle documentation finally makes the boundary unambiguous.

Shortcut synchronization is authoritative: deployment creates, updates, and deletes shortcuts to match the definition. Internal targets can remap but are not created; external targets remain unchanged unless variables remap them. The notebook should validate this result, not recreate it. Notebook-to-Lakehouse auto-binding can replace a physical ID with a logical ID, although it is off by default and applies only to same-workspace relationships. The current dependency-binding matrix still shows important gaps: semantic-model SQL endpoints, some pipeline activities, cross-workspace references, Warehouse endpoints, and Variable Library item references can retain stage-specific identifiers.

The rule is no longer “put everything difficult in the notebook.” Maintain a versioned matrix by item type covering Git, Deployment Pipelines, definition APIs, fabric-cicd, dependency binding, service-principal support, and GA or preview status. Use the most declarative supported mechanism, then reconcile the remainder.

The Updated Architecture

The architecture below keeps the persistent Dev/Test/Prod product triple, but it separates delivery from convergence. The external release workflow—usually Azure Pipelines or GitHub Actions—holds the release identity, calls Fabric APIs, selects the target contract, and decides whether the release may continue. It orchestrates both planes; it is not another name for the notebook.

The release identity is allowed to change the target. It is not automatically the identity the resulting workload should use to read or process data.

Most teams should start with the native hybrid: Git-connected development, a persistent integration workspace, Deployment Pipelines, a small external release workflow, and one deliberately boring idempotent configuration notebook. Terraform becomes valuable for a repeatable workspace factory or regional recovery. Use fabric-cicd when one approved release must deploy predictably across multiple workspaces, environments, or customers.

The Update Notebook Creates and Updates the Internal Structure

Promoting a Lakehouse moves the Lakehouse item, but it does not necessarily leave every schema, table, view, or materialized lake view inside that Lakehouse in the state required by the promoted release. The Update Notebook runs after promotion to close that gap. Its job is to create and update the internal elements of the promoted Lakehouse or Warehouse so that the target environment matches the structure defined for that version of the product.

The Update Notebook is not a general-purpose deployment controller. It should not create workspaces, assign capacity, select the active Variable Library value set, associate identities, remap shortcuts, or apply sensitivity labels. The release workflow completes those control-plane and environment-binding steps first, resolves the target item, and then invokes the notebook against that target.

Inside the data item, the notebook follows straightforward idempotent logic:

That is the practical meaning of idempotency in this framework: create if it does not exist; update if it exists; do nothing if it is already correct. CREATE TABLE IF NOT EXISTS by itself is not enough. It can create a missing table, but it cannot correct an existing table with the wrong columns, data types, properties, or constraints. The Update Notebook must inspect what is already present and use ALTER, CREATE OR REPLACE, a controlled migration, or another object-specific update operation to bring it to the required state.

For a Lakehouse, the Update Notebook can maintain schemas, Delta tables, table properties, Spark views, materialized lake views, and other internal objects that are not carried completely by promotion. For a Warehouse, it can maintain schemas, tables, views, procedures, and constraints through T-SQL when notebook-driven DDL is the selected structural owner. Where the DacFx-backed database-project path owns the Warehouse schema, the Update Notebook must not compete with it; it should validate that structure or manage only the objects explicitly assigned to the notebook.

Materialized Lake Views illustrate the pattern clearly. The MLV definition is versioned as code, but the MLV itself and its dependency graph exist inside the Lakehouse. After promotion, the Update Notebook creates the MLV when it is absent, updates or replaces it when its definition has changed, and leaves it alone when it already matches the promoted version. It then verifies the expected definition, dependencies, constraints, and refresh configuration. The materialized data did not move through Git; the notebook re-establishes the required internal object in the target environment.

The notebook must be safe to run more than once. A retry after a partial failure starts by reading the current catalog, not by assuming that nothing was created. Objects already in the correct state become no-ops; missing objects are created; safely outdated objects are updated. This allows each run to converge on the same required internal state without duplicating objects or replaying changes blindly.

For supported Spark and Jupyter jobs, the notebook-specific Job Scheduler API supports on-demand execution by a user, service principal, or managed identity; typed parameters; explicit Lakehouse and Environment selection; polling; cancellation; and a string exit value. That makes the Update Notebook a release gate rather than an informal post-deployment script. The release workflow can parse a JSON-serialized result such as:

{
  "releaseId": "2026.08.14.3",
  "manifestHash": "2d61...b7a9",
  "objectsCreated": 1,
  "objectsUpdated": 1,
  "migrationsApplied": 0,
  "unchanged": 17,
  "blocked": 0,
  "failed": 0,
  "structurePostconditionsPassed": true
}

Fabric documents this exit-value pattern for conditional orchestration. The notebook serializes the result and passes the string to notebookutils.notebook.exit(...); the release workflow reads it from exitValue and deserializes it. The notebook should also append its object-level actions and validation results to a release-evidence table keyed by the same release ID. A second run against the same promoted version should report no unplanned structural changes.

One boundary still matters: notebookutils.variableLibrary does not support service principals, although Variable Library REST APIs do. An unattended workflow should activate and verify the target value set, resolve the required nonsecret values, and pass typed parameters to the Update Notebook. Secrets remain in Key Vault, managed Fabric connections, or the CI/CD secret store; workload identity federation can eliminate a stored workflow secret. Variable Libraries should carry nonsecret configuration and references, never secrets.

Identity Is Reconciled, Not Transported

Fabric now has several identities that sound similar and should not be allowed to collapse into one super-principal.

These are roles, not necessarily four principals. An API-started notebook runs under the caller. A pipeline notebook runs as the pipeline’s last modified user; a scheduled notebook runs as its creator or last updater. Workspace and associated identities do not become the notebook execution identity. Workspace identity is a Fabric-managed service principal for supported connections and trusted access; it receives no workspace role by default and does not run notebooks. It is stage-local, yet an artifact using it currently updates from Git only into a workspace connected to that same identity.

Associated identities are the easiest new control to overstate. The current Associate Identity API is beta; Microsoft describes it as evaluation and development only and does not recommend it for production. It covers Lakehouses and Eventstreams, except Eventstreams using Azure or Fabric Events as a source. The only documented assignment type is Caller, so the intended principal must make the request as itself. The API can finish immediately or asynchronously; in either case, the workflow must inspect every child-item result.

For now, associated identity belongs behind an explicit preview control, not in the mandatory production path. Treat association as identity selection, not external authorization: provision storage and connection access separately, test the owner-dependent behavior, and only then retire the former grants.

Identity is not the only target attribute the release must verify. Sensitivity labels remain outside item definitions. An unattended workflow can read the current label, but the admin bulk-set API requires a Fabric Administrator user and does not support service principals or managed identities. The workflow should verify the label and route remediation unless a separately authorized delegated-admin step is available.

The Release Loop Is Convergent, Not Linear

The practical release sequence is a loop because external systems time out, Fabric operations are asynchronous, and a retry can begin after some mutations have already succeeded.

The release workflow must distinguish two asynchronous contracts. Item-management operations return a Location for a long-running operation. An on-demand notebook also returns a Location, but it identifies a job instance. Both can return 202; they are not polled through the same endpoint. Honor Retry-After, use bounded retries, and preserve the returned operation or job ID before recovery. The run request exposes no caller idempotency key, so do not resubmit after a polling timeout. Reuse the job ID or re-inspect the target, and hold a per-target release lock to prevent competing reconcilers.

At minimum, the release manifest must bind the approved commit and desired-state hash to the target environment, identities, change plan, validation result, and Fabric deployment and job identifiers.

Those records answer a different question from Microsoft Purview. Purview can scan a Fabric tenant for metadata and lineage, and it can describe ownership, classification, and governed relationships. It does not prove which release principal made a particular change or whether every postcondition passed. CI/CD history and Fabric audit or monitoring provide execution evidence. Purview describes the governed asset. Joining them is more useful than pretending either is complete on its own.

What This Looks Like in Card Processing

Consider a credit-card settlement data product with a Lakehouse, transaction-standardization notebook, settlement pipeline, merchant semantic model, and a Variable Library. The same logical definitions move from Development to Test to Production. Each stage has its own workspace, capacity assignment, connections, notebook execution identity, supported item-identity associations, and data.

Production cardholder data does not ride the deployment pipeline into Test. The active value set selects nonsecret stage references. The notebook execution identity receives only the permissions required to reconcile data; workspace and associated identities receive only their own required grants. The idempotent configuration notebook evolves the Delta schemas absent from the Lakehouse definition, applies controlled reference-data changes, validates settlement grain and key constraints, and returns its JSON-serialized result. Only then does the release enable the consumer surface.

An auditor can now trace the release without reconstructing it from portal screenshots and personal recollection. The organization can show the approved commit, the principal that performed the release, the principal that executed the notebook, the nonsecret target references selected, and—where enabled—the associated identity used for owner-dependent Lakehouse behaviors. It can also show the data structures changed, controls evaluated, and lineage and classification observed after deployment. No cardholder data had to be copied between stages to produce that evidence.

That is the practical difference between shipping files and releasing a regulated data product.

A Release Ends at Convergence

Fabric’s stronger deployment surface does not eliminate idempotent notebooks. It gives them a narrower and more important job.

Git should describe reviewed change. Deployment Pipelines or code-first tooling should move every definition they can. Variable Libraries should describe environment meaning without holding secrets. The external release workflow should reconcile identity and target configuration. The idempotent configuration notebook should converge mutable Lakehouse state, reject unsafe drift, and prove its postconditions. Fabric monitoring and CI/CD logs should record what happened; Purview should place the released asset in its governed context.

Definitions move. Data stays. The release is finished when the target has converged—and when the evidence is good enough that someone other than the person who ran it can prove why it should be trusted.