Build Products, Not Ponds

If you agree that data should be judged by the decisions it improves, the way you build with data changes. You stop trying to pour every table from every system into one giant lake and assume value will appear later. Instead, you ship small, finished data products that help with one decision at a time. The first approach optimizes for storage. The second optimizes for outcomes.

Think of cooking. Stocking a huge pantry doesn’t guarantee a good dinner. A clear recipe, the right ingredients, and a plate on the table do. Data products are the recipes and the plates; “ingest everything” is the overflowing pantry.

Why “ingest everything” feels safe—and why it isn’t

Pulling all sources into one place sounds responsible. You can say, “It’s all there. We’ll use it soon.” The trouble shows up later. Definitions drift because no one agreed on them where it matters—at the point of use. Different teams read the same column in different ways. Security rules live in documents instead of inside the data itself. And because nothing is built for a specific decision, real users never know when it’s “done,” so projects swim in circles.

Central storage isn’t the villain. It just isn’t the hero. Storage alone rarely changes a decision. Decisions change when real people can see something that matters, trust what it means, and act on it—reliably, without a tour guide.

What a data product is (in plain English)

A data product is a small, finished thing a person or system can use without you standing next to it. It has a clear job: “help a claims adjuster spot risky cases,” “help the street team pick tomorrow’s routes,” “help a planner see late parts before the shift starts.” It takes known inputs and produces a known output. It makes promises about freshness (“updated every two hours”), availability (“there when you need it”), and meaning (“this field always means X”). It has a named owner who is responsible for keeping those promises. And it has guardrails built in—who can see it, what gets logged, and what happens when something goes wrong.

That’s it. Not fancy. Just usable, understandable, and accountable.

Promises, not pipelines

Most data work stops at “we built the pipeline.” Users don’t buy pipelines; they buy promises. A promise is specific enough to measure: “New transactions appear within 120 minutes.” “Coverage includes 95% of active accounts.” “Anyone in the fraud unit can try it without asking an engineer.” When promises are clear, two things happen. First, users know what to expect and can plan around it. Second, the business can actually price the promise. If the product gets fresher, we respond faster. If definitions are tighter, we argue less. If the surface is self‑serve, we need fewer handoffs. Those are dollars and hours, not opinions.

A simple way to tell if you have a product: if a new team can use it tomorrow with nothing more than a short read‑me, you’re there. If they need a standing meeting, you’re not done.

How product‑first building makes value visible

When you build for one decision, you’re forced to be specific. Who is the user? What do they need to see? When do they need it? How will we know it helped? Those questions become the product’s promises. Because the promises are explicit, you can connect them to outcomes. Faster refresh means fewer missed opportunities. Clearer definitions mean fewer rework loops. Automation at the last step means shorter cycle times. Now “data quality” isn’t a sermon—it’s part of the business case, with visible cause and effect.

This clarity also keeps risk in the open. A product has a boundary. You can say who is allowed to use it, how access is recorded, and what’s masked by default. If there’s sensitive information, the protection isn’t a side note; it’s part of the design. Smaller, clearer boundaries shrink the chance that one mistake becomes a big incident.

Why small, composable products reduce risk

Over time you will want to build bigger things. The safest way to grow is to compose new products from ones that already work. Think LEGO bricks: each brick is simple, but if it clicks and holds, you can build a lot without surprises.

Picture an engagement product built from three pieces that already proved themselves: an identity and permissions product that tracks who someone is and what they’re allowed to see; a customer‑events product that records important actions in one clean stream; and a propensity product that scores the likelihood someone will respond. When you combine them into “next best action,” you aren’t multiplying unknowns—you’re stacking known quantities. If the combined product struggles, you can see which promise failed and fix that piece, instead of digging through a tangle of pipelines.

The same pattern works elsewhere. In a plant, a sensor‑signals product, a maintenance‑history product, and a parts‑catalog product come together as predictive maintenance. In a city, an asset registry, a 311 requests stream, and a routing layer become smarter street repairs. Big results, smaller unknowns.

The quiet costs of “boil the ocean”

Trying to “do it all” up front sends three quiet bills.

The first is definition drift. When meaning isn’t settled where data is used, teams settle it later, in production, under pressure. That’s where mistakes become public and expensive.

The second is one‑off work. Without product surfaces, every new consumer is a custom project. Engineers become translators; users lose patience; everyone assumes data work is slow.

The third is governance by inbox. Policies exist, but since nothing has clean boundaries, approvals and exceptions bounce between people. You feel slower, not safer.

None of these show up on day one. All of them show up on the balance of the year.

How the right platform appears (instead of arriving by forklift)

There is still a platform in a product‑first world. It just emerges from repeated wins rather than being invented ahead of time. After a few products, the patterns are obvious: the same way to handle permissions, the same way to describe a field, the same way to record where data came from, the same way to release a change. Those pieces become the platform because they’ve proven they save time and reduce mistakes. You standardize only what deserves it, and you keep the platform small and helpful because it grew out of actual use, not a whiteboard.

A good test is this: if a platform feature can’t point to three live products that needed it, it’s not a platform yet—it’s a hunch.

A short before‑and‑after story

In one version of the quarter, a team pulls six systems into a lake, builds a lovely catalog, and shows a demo that answers almost any question—as long as the author is in the room. Little changes for the people who make decisions every day.

In the other version, the team ships a simple claims‑triage product for the fraud unit. It refreshes every two hours, explains why it flagged a case, and anyone on the team can try it. Adjusters use it the first week; the lift is small but real. Two weeks later, the team tightens the refresh time and publishes examples for another unit. By the end of the quarter, three products are live. The common pieces—permissions, shared field definitions, and an audit trail—have been pulled into a tiny starter platform because they obviously help. Quarter two doesn’t start from zero; it starts from working bricks. New products assemble faster, risks are clearer, and value appears earlier in the calendar.

Only one of those stories is easy to defend when budgets tighten.

Switching without drama

If you already have a big lake, don’t throw it out. Pick one decision and draw a product boundary on top of what you have. Ship the smallest end‑to‑end slice that helps a real user—something they can open, understand, and use without you in the room. Write down the promises you actually met. Measure whether people used it and whether it helped. Then do it again with the next decision. After a few cycles, you’ll know which parts of your current stack deserve to be standardized and which parts should be retired.

Two habits make this work. First, keep the feedback loop short. Talk to users weekly, and let what they do (not just what they say) shape the next slice. Second, treat your promises like product features. If freshness is missed or meaning is unclear, fix the product before adding new sources. You’re building trust, not just tables.

How to know you’re on the right track

You don’t need a dashboard of dashboards. A few simple signals tell you if product‑first is working. New teams can start using a product in a day. The first “win” for a product happens within weeks, not quarters. When something breaks, you can find the owner and the logs within minutes. And perhaps the most honest sign: your users bring you ideas you didn’t pitch to them—because they finally see how to turn an idea into something they can use.

Common worries, answered plainly

“What about standards?” You’ll get better standards by extracting them from things that worked than by writing them in a vacuum. Real use trims wish lists into a few rules people follow.

“Won’t we duplicate effort?” Some duplication is the price of speed at the start. The moment two products solve the same problem well, you pull the common solution into the platform. Now you’re standardizing success, not opinions.

“Isn’t this risky?” It’s the opposite. Smaller products shrink the blast radius. Composed products build on known parts. You learn earlier, fix cheaper, and avoid betting the whole quarter on one giant merge job.

Conclusion

Data changes decisions, not storage quotas. Product‑first design keeps you close to the decision: clear purpose, clear promises, clear guardrails, clear results. Ingestion‑first bets that usefulness will appear once the plumbing is perfect. Sometimes it does, but it’s late and costly.

Start small and ship something someone can use without you in the room. Let quality and safety be features, not footnotes. Then compose. Each product lowers uncertainty. Each composition raises your ceiling without raising your risk. That’s how you build a portfolio you can rank, fund, and defend—and a platform that grows out of wins instead of getting in your way.

Releases and CI/CD in Microsoft Fabric — with Variable Libraries That Keep Meaning Stable

I keep saying the quiet part out loud: a modern warehouse ships meaning and trust, not just tables. If meaning changes invisibly, trust evaporates. Releases, Release Flow, and CI/CD in Microsoft Fabric are how you move quickly and keep confidence—by making change observable, reversible, and governed. Fabric’s Variable Library and a deliberate, database‑level metadata library are the glue that make this work day to day.


A release in data: shipping meaning deliberately

A release in data engineering is a versioned bundle—models, DDL, pipelines, notebooks, semantic definitions, and the permissions posture—promoted through environments with intent and traceability. In Fabric, Deployment Pipelines formalize that path (Dev → Test → Prod), including stage‑specific rules that swap connections and parameters so the same artifact behaves correctly in each stage. This keeps tests real but safe and turns promotion into a controlled, reversible act.

Staging should mirror production closely enough that behavior is predictable. Use OneLake Shortcuts to expose prod‑shaped data without copying petabytes, so performance and edge cases surface before users do.


CI in Fabric: prevent “looks fine locally” from reaching people

CI earns its keep the moment it blocks a bad deploy. In Fabric, keep the spine simple:

  • Git integration ties workspaces to branches, making every change reviewable and reproducible. (Mind the “supported items” list as it evolves.)
  • Validate invariants early: compile, lint, and assert keys, referential links, distribution bounds, and metric semantics in your pipelines/notebooks. When CI fails, the business doesn’t.
  • Keep shape realistic: Test with shortcuts and stage‑correct connections so volume, permissions, and latency aren’t surprises later.

CD in Fabric: promote with intent, cut over without drama

Continuous Delivery is less about auto‑pushing and more about predictable promotion:

  • Promote via Deployment Pipelines and stage rules; treat backfills as first‑class release artifacts you observe in the Monitoring hub.
  • Use Power BI App audiences to canary new semantic models and reports to a small internal group; widen only when drift and performance are acceptable.
  • When you outgrow clicking, automate promotion with the fabric‑cicd library in GitHub Actions or Azure DevOps, using service principals for least privilege.

Where Release Flow fits (and why it works for data)

When we say “reflow,” we mean Release Flow—Microsoft’s trunk‑based model with sprint‑scoped release branches and cherry‑picked hotfixes. Keep main moving; cut a release branch to stabilize; merge fixes to main first, then cherry‑pick to the release. Map Dev to main, Test/Prod to the release branch, and promote through your pipeline. It’s fast, auditable, and avoids “fixed in prod, broken next release.”


Variable Library: stage‑aware configuration without hard‑coding

Fabric’s Variable Library is a workspace item that holds named variables and their values per pipeline stage. Items like Data Pipelines and Dataflow Gen2 can consume these variables directly, so the same artifact resolves the right connection, path, or toggle in Dev/Test/Prod—no string‑surgery, no accidental “Test reading Prod.” This is application lifecycle management (ALM) for configuration, not a bag of ad‑hoc parameters.

In practice, Variable Library becomes your single source for things like:

  • connection aliases (e.g., sales_wh_connbronze_lake_path),
  • time windows and data slices for CI runs (e.g., “last 3 days”),
  • feature toggles (e.g., enable a new scoring routine only in Test),
  • stage‑specific destinations (schemas, lake folders) used by pipelines and dataflows.

Because values are bound by stage, a promotion flips behavior without editing code—exactly what you want when reliability and auditability matter.


Safe development and effective testing, Fabric‑style

Develop in isolated workspaces tied to branches. Use Variable Library values to bind stage‑correct connections and “slice” windows; validate contracts from your metadata schema before any model rebuild or backfill runs. Promote with Deployment Pipelines; canary via App audiences; observe in Monitoring; and roll back quickly because promotion was a metadata change, not a long‑running fix‑by‑hand.


Reliability and governance as properties of the system

Define freshness, completeness, and correctness SLOs; then let your CD gates enforce them. Sensitivity labels and Purview’s Unified Catalog close the loop on governance and lineage so your release record isn’t just technical—it’s compliant. When auditors ask, you don’t reconstruct history; you point to it.


The payoff

With Release Flow, CI/CD, Variable Libraries, and a database‑level metadata library, your warehouse stops being fragile plumbing and becomes a platform. Teams ship more often with less drama. Stakeholders trust numbers because the path to those numbers is visible, repeatable, and reversible.

That’s the bar we set: move fast, keep meaning stable, and let your pipeline tell the story of how you did it.

Why We Still Need Real Releases in Data and Analytics

In an era where everything markets itself as “continuous”—continuous integration, continuous delivery, continuous retraining—it can feel quaint to talk about releases. But if we care about reliability and governance, we should talk about them more, not less. A true software‑style release is not nostalgia; it’s a commitment device. It’s the point where we say: this is the version we stand behind, with a clear boundary of what changed, what didn’t, and how long we intend to support it.

At edudatasci.net we work at the seam where data, software, and institutional decision‑making meet. At that seam, releases are how we translate rapid iteration into dependable outcomes—for educators, researchers, and the operational teams who carry real responsibility for real people. Without the concept of a release, our systems may move quickly, but the trust we need from stakeholders never catches up.

Continue reading “Why We Still Need Real Releases in Data and Analytics”

Data Vault, Practically: Why It Exists, How It’s Built, and What 2.1 Changes

Modern data platforms live in tension:

  • Source systems evolve faster than dimensional models can absorb.
  • Audit and lineage are mandatory, but teams still need velocity.
  • Cloud lakehouses, streaming, and domain ownership do not slot neatly into yesterday’s warehouse playbooks.

Data Vault is a response to those pressures. It is both a modeling approach and a delivery method designed to (1) absorb change, (2) preserve complete, immutable history, and (3) decouple integration from consumption. The core building blocks—Hubs, Links, and Satellites—organize into a Raw Vault (source truth, append‑only) and a Business Vault(governed derivations and query assistance). Think of it as a fault‑tolerant integration substrate with a clean seam to marts, semantic models, and data products.

Continue reading “Data Vault, Practically: Why It Exists, How It’s Built, and What 2.1 Changes”

Testing Like We Mean It: Bringing Software‑Grade Discipline to Data Engineering

I like to say that the first product of a data team isn’t a table or a dashboard—it’s trust. Trust is built the same way in data as it is in software: through tests that catch regressions, encode intent, and make change safe. If pipelines are code, then they deserve the same rigor as code. That means unit tests you can run in seconds, integration tests that respect the messy edges of reality, comprehensive tests that exercise the platform end‑to‑end, and user acceptance testing that proves the system answers the questions people actually have. Done well, this isn’t busywork; it’s the backbone of reliability and a pillar of governance.

Continue reading “Testing Like We Mean It: Bringing Software‑Grade Discipline to Data Engineering”

Bronze Is Live Now: what Mirroring + Shortcuts really change about cost, archives, and getting to Silver

For years, “Bronze” quietly became a parking lot for periodic snapshots: copy a slice from the source every hour/day, write new files, repeat. It worked, but it was noisy and expensive—lots of hot storage, lots of ingest compute, and a tendency to let “temporary” landing data turn into de‑facto history.

Fabric upends that with two primitives that encourage Zero Unmanaged Copies:

  • Mirroring: a service‑managed, near–real‑time replica of your database/tables into OneLake, with replication compute included and a capacity‑based allowance of free mirrored storage (1 TB per CU; e.g., an F64 includes 64 TB just for mirrored replicas). You still pay for downstream query/transform compute, but not for the continuous ingest job itself. Retention for mirrored data is explicitly managed and—by default for new mirrors since mid‑June 2025—kept lean (1 day) unless you raise it.
  • Shortcuts: pointers that let Fabric read in place from ADLS/S3/other OneLake locations (and even across tenants via External Data Sharing, which creates a shortcut in the consumer’s tenant rather than duplicating data). That means zero OneLake bytes for the data itself; you pay storage where the data already lives, and Fabric charges only for the compute you use to read/transform it.

Add Real‑Time Intelligence/Eventhouse or Eventstreams, and “Bronze” becomes the live edge: the freshest, governed view of your sources—either replicated (Mirroring) or virtualized (Shortcuts)—instead of a pile of periodic copies.

Continue reading “Bronze Is Live Now: what Mirroring + Shortcuts really change about cost, archives, and getting to Silver”

Materialized Lake Views (MLVs) in Microsoft Fabric

A Materialized Lake View (MLV) is a table in your Fabric lakehouse that’s defined by a SQL query and kept up‑to‑date by the service. You write one CREATE MATERIALIZED LAKE VIEW … AS SELECT … statement; Fabric figures out dependencies, materializes the result into your lakehouse, and refreshes it on a schedule. Today, MLVs are in preview, SQL‑first (Spark SQL), and designed to make Medallion layers (Bronze → Silver → Gold) declarative instead of hand‑assembled pipelines.

Continue reading “Materialized Lake Views (MLVs) in Microsoft Fabric”

The Microsoft Fabric Delta Change Data Feed (CDF)

In Microsoft Fabric you’re sitting on top of Delta Lake tables in OneLake. If you flip on Delta Change Data Feed (CDF) for those tables, Delta will record row‑level inserts, deletes, and updates (including pre‑/post‑images for updates) and let you read just the changes between versions. That makes incremental processing for SCDs (Type 1/2) and Data Vault satellites dramatically simpler and cheaper because you aren’t rescanning entire tables—just consuming the “diff.” Fabric’s Lakehouse fully supports this because it’s natively Delta; Mirrored databases land in OneLake as Delta too, but (as of September 2025) Microsoft hasn’t documented a supported way to enable Delta CDF on the mirrored tables themselves; you can still analyze mirrored data with Spark via Lakehouse shortcuts, or source CDC upstream (Real‑Time hub) and write to your own Delta tables with CDF enabled.

This feature is already underutilized, but once Mirrored Databases support the CDF, it’s going to be a must have in every data engineer’s toolkit.

Continue reading “The Microsoft Fabric Delta Change Data Feed (CDF)”

Baselines Over Buzzwords: From Warehouse to Lakehouse

If you’ve built data systems long enough, you’ve lived through at least three architectural moods: the tidy certainty of Kimball and Inmon, the anarchic freedom of “throw everything in the data lake to ingest quickly,” and today’s lakehouse, which tries to keep our speed without losing our sanity. I’ve always cared less about labels and more about baselines—clear, durable expectations that make change safe. This piece traces how those baselines shifted, what we gained and lost, and how to rebuild them for modern work, including real‑time, very large, and unstructured data.

Continue reading “Baselines Over Buzzwords: From Warehouse to Lakehouse”

Zero Unmanaged Copy + SCDs: Keep All the History, Lose the Bloat

Data teams often face a false choice: either keep rich Slowly Changing Dimensions (SCDs) and accept a sprawl of duplicate tables, or keep the warehouse lean and give up on audit‑ready history. You don’t have to choose. With a zero unmanaged copy approach, you can keep full history and maintain predictable performance—without littering the lakehouse with ad‑hoc exports and orphaned datasets.

This post explains the idea in plain English, shows how mirrored, CDC, historical (SCD2), and snapshot tables fit together, and lays out four deployment options. We’ll also cover the performance guardrails—partitioning, clustering, idempotent merges, micro‑batches, workload isolation, and observability—so your SLAs stay green as data scales.

Continue reading “Zero Unmanaged Copy + SCDs: Keep All the History, Lose the Bloat”