EduDataSci – Educating the world about data and leadership – Data Strategy from a one time K-12 data architect

Dear World: You Are Not Taking the Squirrels Seriously

I’m Blaise Lonestar. The squirrels are getting away with something, Mondays need reconsidering, and I would like to come with you.

Dear world,

My name is Blaise Lonestar. I am a white golden retriever, which means I am golden in the important ways and white on your trousers. If we have not met, I would very much like to correct that. You may scratch behind my ears while we discuss the situation.

I live with Jason. I love him with my entire body, particularly the tail, which begins to wiggle when he comes home. He is a good human in many respects. He understands duck sticks, can operate doors, and knows several excellent places to walk. Unfortunately, he also believes that we should sometimes do separate things.

This is one of several matters on which the world needs my perspective. The squirrels are another.

Four Correct Records. No Common Answer

At 10:02, a trader can point to an execution and say the trade exists. Risk can point to an amended position and say the exposure has changed. Operations can point to an unmatched allocation and say the trade is not ready to settle. Finance can point to yesterday’s booking and say nothing new has reached the official ledger.

Nobody is necessarily wrong. That is the problem.

Each application is answering a different question, at a different point in the lifecycle, using a different clock. The trader’s blotter records what was filled. The risk platform measures the exposure it currently understands. The operations platform tracks whether the allocation has matched. The ledger reports what passed a controlled posting boundary by the previous close.

Ask the firm what happened, however, and it may not have an answer. It has four extracts, several timestamps, a reconciliation queue and, eventually, a meeting.

This is not simply bad data quality. It is the predictable result of an architecture that makes applications authoritative for isolated stages and asks reconciliation to assemble enterprise meaning afterward. The alternative is not another master table. It is a governed live lifecycle: persistent identities connect related business objects, semantic events preserve what changed, bitemporal state distinguishes economic effect from firm-recorded time, and materialized views give each function the answer it needs.

The target is not one application—or one physical database—containing all truth. It is a logically governed history from which authorized functions can produce timely, explainable views.

The Landing Lakehouse: The Fabric Pattern Between Arrival and Use

A source begins mirroring into Microsoft Fabric on Friday. By Monday, the tables are visible in OneLake. Production engineers can build from them. Developers can query them. Testers can point validation jobs at them. Analysts can begin exploring.

The delivery looks successful because the data arrived quickly. The architecture may already be in trouble.

The first reachable version of data is rarely the version every user and environment should consume. Production needs fidelity and controlled change. Development needs realistic relationships without unnecessary exposure. Test needs stable inputs that do not shift underneath automated validation. The source team needs to retain authority over what lands and what may be published. Governance needs a defensible answer to a deceptively simple question: which version of this data is visible to whom?

A Landing Lakehouse provides that answer.

The pattern uses a dedicated Lakehouse as the controlled boundary between direct arrival and governed use. Mirrored data, external shortcuts, and source-owned Fabric data can land once. The Landing Lakehouse then exposes distinct schemas for production, anonymized development, representative subsets, Known Copies, and curated validation. Each output is a separate surface with its own purpose, access model, refresh behavior, and owner.

The point is not to add another layer for its own sake. The point is to keep the convenience of Fabric from collapsing several different contracts into one raw surface.

Microsoft Fabric Databases and AI Move Closer to Everyday Operations

A dashboard can tell a business exactly what is wrong while leaving most of the work of fixing it untouched. A collections manager sees overdue balances, then opens email to investigate a dispute, finds the relevant contract, and asks someone else to update the account. The analytical system has produced a useful answer. The operational process still depends on people carrying that answer between systems.

Microsoft’s Fabric, database, and AI announcements since the beginning of September are increasingly aimed at that gap. Fabric is acquiring more of the capabilities needed to describe a business, recognize a change, and support the work that follows. SQL databases are becoming more capable AI application backends. Foundry is adding the execution services that let agents participate in recurring processes. The Barcelona FabCon and SQLCon releases make this direction particularly visible, with the early October Decision-1 announcement adding a specialized model for the smaller judgments inside those processes.

I see this as a substantial expansion of Fabric’s purpose. Microsoft is positioning it as a shared foundation for running data-intensive business operations, extending well beyond reporting and analytical storage. The strongest part of the strategy is its use of assets organizations already own: operational databases, Power BI measures, business relationships, and established permissions. That gives the recent announcements a coherence that is easy to miss when they are read as separate feature lists.

Stop Settlement Fails Before They Start

At 9:42 a.m., an institutional equity trade executes normally. There is no break and no exception to repair.

By 2:00 p.m., the picture has changed. The client’s allocation is later than usual. The standing settlement instruction (SSI) on one record does not agree with the custodian’s current instruction. Securities expected from another account have not arrived. A routine counterparty acknowledgment is still missing.

None of those facts proves that the trade will fail. Together, they say that the trade is becoming harder to settle.

The traditional operating model waits for a definite exception, places it in a queue and asks an operations analyst to reconstruct what happened. A prevention model starts earlier. It watches the trade’s full lifecycle, recognizes a risky pattern, applies the firm’s control rules and intervenes while useful choices remain.

This is not a proposal for a better exception queue. It is a different job for operations: prevent avoidable failures, resolve safe and bounded issues automatically, bring judgment-heavy cases to the right expert, and remove recurring defects from the upstream process.

Definitions Move. Data Stays: The Updated EduDataSci CI/CD Framework for Microsoft Fabric

A Fabric deployment can succeed while the target lakehouse is empty, the wrong Variable Library value set is active, a shortcut still points to Development, and the lakehouse depends on the identity of someone who has left the company.

The API did not lie. The architecture asked it the wrong question.

When I wrote about the ideal Fabric CI/CD approach, the division of labor was clear. Git controls change. Workspaces isolate execution. Deployment Pipelines promote approved content. External CI/CD adds tests and approvals. When native promotion cannot finish the job, fabric-cicd moves item definitions and an idempotent configuration notebook brings the lakehouse into the required state.

That architecture needs an update, not a replacement. Microsoft now frames Git integration, Deployment Pipelines, REST APIs, Variable Libraries, Fabric CLI, Terraform, and fabric-cicd as one Fabric CI/CD platform. Fabric can transport and bind more of the portable definition while exposing better APIs for target-local state.

In this framework, the code plane moves portable definitions and the data plane converges mutable target state. A release is complete only when both have happened—and the release can prove it.

When Software Becomes Disposable: Why the Next Data Mesh Needs Action-Oriented Data Products

Something important changes when intent becomes the front door to software. If a banker, underwriter, advisor, claims lead, or merchant-risk analyst can ask for a task-specific application and receive it in minutes, then bespoke software stops being rare. It becomes cheap. The expensive thing is no longer the application layer. The expensive thing is the interface to truth: the boundary where customer data, authorization, business policy, and systems of record meet.

That shift deserves a different architectural response. I like to frame a data product as reusable, trustworthy, contract-driven, loosely coupled asset with clear ownership, explicit promises about freshness and meaning, and a versioned surface consumers can rely on. The data mesh tradition adds domain ownership, self-serve platform capability, and federated computational governance. My ingest-transform-surface model sharpens the picture further: inputs are explicit, transformation is deliberate, and the published surface is the contract. All of that is right. It just is not enough once agents and intent-driven tools begin to mediate real work.

Domain APIs need to become the front door for fit-for-purpose apps, and the mesh becomes the process glue that turns operational data into analyzable insight and analytical insight into operational action. My argument is that we should now make that shift explicit. The next mesh should be built from Action-Oriented Data Products: domain-owned data products whose interfaces declare not only what can be read and analyzed, but what can be done, by whom, for what purpose, with which guardrails, and against which authoritative source.

This is not a new name for old API management. It is a claim about the next architectural quantum. Data mesh already argued that the data product is the quantum of the analytical estate. In an intent-mediated world, that quantum must become action-oriented. The mesh stays. The node changes. To make that case, I want to do three things: show why current data products stop short, explain the historical parallels to object orientation, microservices, and data mesh, and define the architecture that lets agents discover and manipulate data safely.

One Agent, Several Kinds of Truth

An enterprise agent rarely fails because it cannot produce a fluent paragraph. It fails because it cannot distinguish the current balance from the definition of balance, the approved policy from the email that discussed an exception, or the authority to explain a decision from the authority to change a system of record.

That distinction matters in financial services. A relationship manager asking why a commercial borrower has become riskier is really asking several questions at once. What changed in the borrower’s accounts? How does the exposure roll through facilities, collateral, guarantees, and legal entities? What does credit policy require? What did the deal team agree to last week? If action is warranted, who may open a review, request documents, or change a limit?

No single knowledge source can answer all of that honestly. The useful architecture is a Copilot Studio agent that coordinates several bounded kinds of intelligence: Fabric IQ for business state, Foundry IQ for institutional knowledge, Work IQ for the user’s working context, an ontology for shared meaning, and MCP tools for controlled action. API Center anchors design-time inventory and discovery, while Microsoft Purview anchors data governance and supported AI-interaction compliance. Runtime enforcement remains with Entra, Power Platform data policies, native IQ permissions, API Management, and the source systems.

The Client Passed KYC. Why Can’t They Trade?

“Can the new fund trade on Monday?” should be an easy question.

The relationship manager knows the asset manager. The client record is active. KYC is complete. Legal has a signed agreement, and credit has approved a limit. Every team appears to have done its part.

But operations is still opening an account. A tax form is attached to the parent company rather than the fund. The agreement covers the right legal entity but not the requested product in that jurisdiction. A market permission has not reached the entitlement system. On Friday afternoon, every function can report that its task is complete—or at least within its service level—while the client still cannot place a trade.

No single system is necessarily wrong. Each system is answering a smaller question than the client asked.

This is the gap that process mining can address. The technology is often discussed in connection with claims automation, where it can identify slow routes, repeated work, and unnecessary handoffs. Those are useful applications. But the same method can be even more valuable in financial-markets onboarding, where one commercial outcome depends on several functions and systems agreeing at the same time.

The goal is not merely to complete KYC faster. It is to make the client usable—safely, lawfully, and with a clear record of how that decision was reached.

Choose the Publication Contract, Not the Fabric Icon

A bank’s credit-risk reporting team needs one change before the monthly committee pack. The underlying exposure data is current. The semantic model can reach it. The requested change—reclassifying a set of delinquency bands—is not particularly difficult.

Yet the team that owns the report cannot publish the change. The curated table is maintained by a Spark process owned elsewhere, so the request becomes a ticket, a handoff, and a negotiation over another team’s release calendar. Nothing in Microsoft Fabric is broken. The architecture treated the point where data was published as a storage destination when it was really a contract with an owner.

The reverse failure is just as common. A data-engineering team already publishes well-governed Delta tables for analytics, data science, and downstream domains, but an architectural standard requires every Gold table to be copied into a Warehouse before Power BI can use it. The second published copy adds another load, another security boundary, another incident path, and another opportunity for the two versions to disagree. It exists because a diagram said Gold should look like a database.

Fabric Warehouse and Lakehouse overlap too much for the old rule—structured data goes to a warehouse; everything else goes to a lakehouse—to make this decision well. Both use Delta in OneLake. Both can serve SQL queries. Both can support Power BI and governed analytical products. The consequential difference is the publication contract each item is being asked to own and the team expected to keep it.

Choose a Lakehouse publication contract when the authoritative product is maintained as Delta through engineering workflows and SQL consumers primarily need to read it. Choose a Warehouse publication contract when the owning team must create, populate, correct, secure, and release a relational product through T-SQL, especially when multi-table transactions or an independent SQL-owned release cadence matter. Use both when two real contracts exist—not because one Fabric icon feels unfinished without the other.