Feature Analysis – EduDataSci – Educating the world about data and leadership

Fabric Runtime 2.0 Changes the Fabric Engineering Baseline

A Fabric notebook can contain six lines of PySpark and still depend on an enormous amount of technology that is not visible in those six lines. The result depends on the version of Spark that planned the work, the Python or Scala runtime that interpreted the code, the Delta implementation that read and wrote the tables, the Java virtual machine underneath Spark, the operating system beneath that, and the Fabric services that connected the workload to OneLake and compute.

That hidden stack is easy to ignore while it remains stable. Fabric Runtime 2.0 is the moment when ignoring it becomes dangerous.

Runtime 2.0 is now the latest generally available Fabric Spark runtime. It moves the platform to Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13, R 4.5.2, and Azure Linux 3.0. Microsoft currently requires customers to opt into it, but plans to make it the default selection for new workspaces and Environment items in late September 2026.

Those facts make Runtime 2.0 sound like a substantial but conventional platform upgrade. It is more important than that. Runtime 2.0 brings together a new Spark generation, a newer Delta protocol, native execution, lower-latency streaming, modern language runtimes, stronger environment management, and a clearer lifecycle for Fabric applications.

The significance is not that every Fabric workload suddenly needs the newest feature. The significance is that the runtime has become part of the application’s architecture, operating model, performance model, and deployment contract.

Materialize Responsibly: How Fabric’s External Data Materialization Affects “Zero Unmanaged Copy” — and Where Materialized Lake Views Now Shine

Microsoft Fabric’s Warehouse can now materialize external files into tables with straight‑ahead T‑SQL, and Materialized Lake Views (MLVs) have quietly leveled up with optimal refresh (including incremental) and stronger, UI‑backed monitoring. If your north star is Zero unmanaged copy, the question isn’t “should I materialize?”—it’s “how do I materialize responsibly under OneLake governance?” Here’s what changed since our last take—and what to use when.

From Chunks to Queries—Ignite 2025 Update: Fabric Data Agents, RAG, and the New IQ Layer

Monday, 9:02 a.m. The CFO pings: “What was Q3 gross margin by region—and did audit call out any risks?” Your RAG bot shines on PDFs and wiki pages, but it can’t compute a number you’d put on a KPI card. After Ignite 2025, the answer is cleaner than ever: let a Fabric Data Agent generate and run a governed query for the metric, and let your RAG retriever bring back the one‑sentence risk note. One conversation; two specialized tools; auditable answers.

SQL Server 2025 at Ignite: Why This Release Matters—and What to Do Next

In brief: SQL Server 2025 is generally available with built‑in AI, major developer conveniences, sturdier performance/availability behaviors, and licensing/edition changes that lower the cost of entry. Below I frame the release around three themes—AI + developer experience, performance + resilience, and product/edition shifts—and close with concrete first steps you can act on today.

SAP Business Data Cloud Connect for Microsoft Fabric: The New Backbone of Your Data‑Product Strategy

SAP and Microsoft have just taken away one of the biggest excuses for slow analytics and AI on SAP: “We can’t move that data safely or reliably enough.”

At Microsoft Ignite 2025, they announced SAP Business Data Cloud (BDC) Connect for Microsoft Fabric—a new capability that lets you share SAP Business Data Cloud data products and Microsoft Fabric data sets bi‑directionally, with zero‑copy, and have those products show up natively in OneLake and back in BDC.

Planned for general availability in Q3 2026, this isn’t “yet another connector.” It’s the missing link between SAP’s data‑product‑centric Business Data Cloud and Microsoft’s Fabric platform. It’s also where SAP Databricks, Azure Databricks, and Fabric line up as peers rather than competitors.

Azure HorizonDB at Ignite 2025: What It Is, Why It Matters, and How to Think About It

Microsoft used Ignite 2025 to put a new flag in the ground for Postgres at cloud scale. Azure HorizonDB—branded as “HorizonDB”—promises the elasticity of a cloud-native architecture, the familiarity of PostgreSQL, and integrated AI features that shorten the path from schema to shipped app. Here’s what was announced, why it matters, and how to evaluate it for your stack.

Azure DocumentDB Is Back—And Open: Why the Ignite 2025 Launch Matters

If you’ve been around Azure long enough, the name “DocumentDB” triggers déjà vu. But at Microsoft Ignite (Nov. 18–21, 2025), DocumentDB returned with a different meaning: an open‑source, Linux Foundation–governed, MongoDB‑compatible engine now powering a first‑party Azure service. Here’s why that matters—and where it fits in your data strategy.

Secrets in Fabric: The Breaches That Prove Why Keys Don’t Belong—And How Workspace Identity Fixes It

Credentials embedded in Fabric items (notebooks, dataflows, pipelines, semantic models) are a design flaw; three recognizable incidents that show how secrets leak at scale (though none involved Fabric); and the Fabric‑native path forward—Workspace Identity by default, Azure Key Vault references when you must use a secret, and connections that keep CI/CD boring and secure.

The pattern behind the big breaches (and the Fabric takeaway)

These incidents vary in logo, not in lesson: credentials landed in code or repos, automation spread them, rotation lagged, and trust took the hit.

Uber AWS credentials in GitHub — breach October 2016, disclosed November 21, 2017.
Attackers pulled keys from a private repo and accessed S3, impacting 57 million riders and drivers; Uber later acknowledged the incident publicly. Fabric takeaway: if secrets ride along in Git, a private branch is not a safety boundary. Keep artifacts secret‑free.
The New York Times GitHub token — stolen January 2024, leak surfaced June 8, 2024.
A compromised GitHub token enabled access to repos; ~270 GB of internal code later appeared on 4chan, confirmed by the Times. Fabric takeaway: treat tokens as crown jewels—store in Key Vault, not in developer workflows; wire Fabric items to connections that authenticate without handling raw secrets.
Toyota T‑Connect key exposure — October 7, 2022.
A secret key left in a public repo for nearly five years exposed data for ~296,000 customers. Fabric takeaway: the half‑life of a leaked key is longer than a sprint. Eliminate static secrets where possible, or centralize and rotate them.

In every case, the root cause wasn’t exotic malware—it was a credential where it didn’t belong.

Why secrets don’t belong inside Fabric items

Fabric is designed so items use credentials via governed connections—not embed them in code or metadata. The moment you enable Git integration, item definitions flow to a branch where pull requests, diffs, and automation increase the blast radius of any embedded key. Keep definitions in Git; keep secrets out.

Deployment Pipelines help with promotion, and deployment rules can remap endpoints across Dev/Test/Prod. They won’t sanitize a password tucked into M, SQL, or a notebook cell. Centralize authentication in connections (or better, identity), then let rules handle environment drift.

Tenant administrators can also govern who may share connections in Manage connections and gateways, reducing scatter and drift.

Workspace Identity in Fabric (the default path)

What it is.
A Workspace Identity is an automatically managed Microsoft Entra application (service principal) tied to a Fabric workspace. Fabric items can authenticate to resources that support Entra auth without storing keys, passwords, or SAS tokens. Fabric admins can view and govern these identities in the admin portal.

What it unlocks.
Workspace Identity enables secure access to firewall‑enabled ADLS Gen2 through Trusted workspace access (TWA)—you allow specific Fabric workspaces to reach a storage account using resource instance rules. It also lights up authentication for supported connectors across Dataflows Gen2, pipelines, and (in many scenarios) semantic models and lakehouses.

How to use it well (short, practical path).
Create a workspace identity in Workspace settings, then grant it least‑privilege roles (for example, Storage Blob Data Reader/Contributor) on the target resource. For ADLS Gen2 behind a firewall, enable Trusted workspace access and add the workspace via resource instance rules; then consume the data through a OneLake shortcut, pipeline, or model—no account keys, no SAS. Note that TWA requires an F‑SKU capacity (not Trial).

Where it shows up in the UI.
When you create connections to ADLS Gen2, Workspace Identity appears as a supported authentication option. Fabric documents where WI works (shortcuts, pipelines, Dataflows Gen2, semantic models) and where it doesn’t.

Current considerations.
Feature support is expanding, but not universal; consult connector docs before assuming WI is available everywhere. Some WI/TWA connections are created from item experiences (e.g., shortcut or pipeline wizards) rather than from Manage connections and gateways, and a few management tasks remain workspace‑scoped.

Why it matters.
Using Workspace Identity removes entire classes of failure: no secret sprawl, no expired tokens breaking refresh, no surprises in PR diffs. It’s the Fabric‑native way to align with Zero Trust without slowing development.

When a secret is unavoidable: vault and reference it

Sometimes a vendor API still requires a key. In those cases:

Use Azure Key Vault references for Fabric connections. Create a Key Vault reference once, bind supported connections to it, and let rotation happen centrally. This preview feature is now broadly documented and managed from Manage connections and gateways.
Fetch at runtime in notebooks—never store in code. Use notebookutils.credentials.getSecret(vaultUri, secretName); don’t print secrets or write them to files or logs.
Manage semantic‑model credentials in the service. Cloud sources don’t need a gateway, but data source credentials live in model settings—keep them out of queries and parameters and prefer identity where supported.

CI/CD that stays boring (in the best way)

Connect your workspace to Git for version control, reviews, and history—but only after you’ve moved credentials into Workspace Identity or Key Vault–backed connections. Use deployment rules to remap endpoints across stages without rewriting credentials. That combination gives you clean repos and predictable promotions.

A compact blueprint you can adopt now

Make WI your default. Create a Workspace Identity, grant least‑privilege roles, and enable Trusted workspace access for ADLS Gen2. Replace SAS/keys with OneLake shortcuts, pipelines, or models that use WI.
Vault the rest. Add Azure Key Vault references to connections; in notebooks, call NotebookUtils at runtime.
Keep Git clean; let rules do the switching. Use deployment rules to point artifacts at Dev/Test/Prod endpoints; never commit secrets. Govern connection sharing in Manage connections and gateways.

Summary—and the next step

The headline breaches didn’t hinge on sophisticated exploits; they hinged on secrets in repos. In Fabric, the durable fix is to remove static secrets with Workspace Identity wherever possible and centralize unavoidable credentials with Azure Key Vault–backed connections. Wire your items to connections, not copied keys; let deployment rules handle environment drift. Start today by enabling Workspace Identity on one workspace and converting a single SAS‑based connection into TWA or AKV—then rinse and repeat.

Releases and CI/CD in Microsoft Fabric — with Variable Libraries That Keep Meaning Stable

I keep saying the quiet part out loud: a modern warehouse ships meaning and trust, not just tables. If meaning changes invisibly, trust evaporates. Releases, Release Flow, and CI/CD in Microsoft Fabric are how you move quickly and keep confidence—by making change observable, reversible, and governed. Fabric’s Variable Library and a deliberate, database‑level metadata library are the glue that make this work day to day.

A release in data: shipping meaning deliberately

A release in data engineering is a versioned bundle—models, DDL, pipelines, notebooks, semantic definitions, and the permissions posture—promoted through environments with intent and traceability. In Fabric, Deployment Pipelines formalize that path (Dev → Test → Prod), including stage‑specific rules that swap connections and parameters so the same artifact behaves correctly in each stage. This keeps tests real but safe and turns promotion into a controlled, reversible act.

Staging should mirror production closely enough that behavior is predictable. Use OneLake Shortcuts to expose prod‑shaped data without copying petabytes, so performance and edge cases surface before users do.

CI in Fabric: prevent “looks fine locally” from reaching people

CI earns its keep the moment it blocks a bad deploy. In Fabric, keep the spine simple:

Git integration ties workspaces to branches, making every change reviewable and reproducible. (Mind the “supported items” list as it evolves.)
Validate invariants early: compile, lint, and assert keys, referential links, distribution bounds, and metric semantics in your pipelines/notebooks. When CI fails, the business doesn’t.
Keep shape realistic: Test with shortcuts and stage‑correct connections so volume, permissions, and latency aren’t surprises later.

CD in Fabric: promote with intent, cut over without drama

Continuous Delivery is less about auto‑pushing and more about predictable promotion:

Promote via Deployment Pipelines and stage rules; treat backfills as first‑class release artifacts you observe in the Monitoring hub.
Use Power BI App audiences to canary new semantic models and reports to a small internal group; widen only when drift and performance are acceptable.
When you outgrow clicking, automate promotion with the fabric‑cicd library in GitHub Actions or Azure DevOps, using service principals for least privilege.

Where Release Flow fits (and why it works for data)

When we say “reflow,” we mean Release Flow—Microsoft’s trunk‑based model with sprint‑scoped release branches and cherry‑picked hotfixes. Keep main moving; cut a release branch to stabilize; merge fixes to main first, then cherry‑pick to the release. Map Dev to main, Test/Prod to the release branch, and promote through your pipeline. It’s fast, auditable, and avoids “fixed in prod, broken next release.”

Variable Library: stage‑aware configuration without hard‑coding

Fabric’s Variable Library is a workspace item that holds named variables and their values per pipeline stage. Items like Data Pipelines and Dataflow Gen2 can consume these variables directly, so the same artifact resolves the right connection, path, or toggle in Dev/Test/Prod—no string‑surgery, no accidental “Test reading Prod.” This is application lifecycle management (ALM) for configuration, not a bag of ad‑hoc parameters.

In practice, Variable Library becomes your single source for things like:

connection aliases (e.g., sales_wh_conn, bronze_lake_path),
time windows and data slices for CI runs (e.g., “last 3 days”),
feature toggles (e.g., enable a new scoring routine only in Test),
stage‑specific destinations (schemas, lake folders) used by pipelines and dataflows.

Because values are bound by stage, a promotion flips behavior without editing code—exactly what you want when reliability and auditability matter.

Safe development and effective testing, Fabric‑style

Develop in isolated workspaces tied to branches. Use Variable Library values to bind stage‑correct connections and “slice” windows; validate contracts from your metadata schema before any model rebuild or backfill runs. Promote with Deployment Pipelines; canary via App audiences; observe in Monitoring; and roll back quickly because promotion was a metadata change, not a long‑running fix‑by‑hand.

Reliability and governance as properties of the system

Define freshness, completeness, and correctness SLOs; then let your CD gates enforce them. Sensitivity labels and Purview’s Unified Catalog close the loop on governance and lineage so your release record isn’t just technical—it’s compliant. When auditors ask, you don’t reconstruct history; you point to it.

The payoff

With Release Flow, CI/CD, Variable Libraries, and a database‑level metadata library, your warehouse stops being fragile plumbing and becomes a platform. Teams ship more often with less drama. Stakeholders trust numbers because the path to those numbers is visible, repeatable, and reversible.

That’s the bar we set: move fast, keep meaning stable, and let your pipeline tell the story of how you did it.

Bronze Is Live Now: what Mirroring + Shortcuts really change about cost, archives, and getting to Silver

For years, “Bronze” quietly became a parking lot for periodic snapshots: copy a slice from the source every hour/day, write new files, repeat. It worked, but it was noisy and expensive—lots of hot storage, lots of ingest compute, and a tendency to let “temporary” landing data turn into de‑facto history.

Fabric upends that with two primitives that encourage Zero Unmanaged Copies:

Mirroring: a service‑managed, near–real‑time replica of your database/tables into OneLake, with replication compute included and a capacity‑based allowance of free mirrored storage (1 TB per CU; e.g., an F64 includes 64 TB just for mirrored replicas). You still pay for downstream query/transform compute, but not for the continuous ingest job itself. Retention for mirrored data is explicitly managed and—by default for new mirrors since mid‑June 2025—kept lean (1 day) unless you raise it.
Shortcuts: pointers that let Fabric read in place from ADLS/S3/other OneLake locations (and even across tenants via External Data Sharing, which creates a shortcut in the consumer’s tenant rather than duplicating data). That means zero OneLake bytes for the data itself; you pay storage where the data already lives, and Fabric charges only for the compute you use to read/transform it.

Add Real‑Time Intelligence/Eventhouse or Eventstreams, and “Bronze” becomes the live edge: the freshest, governed view of your sources—either replicated (Mirroring) or virtualized (Shortcuts)—instead of a pile of periodic copies.