Jason Miles

Managing Data Platform Projects the Agile Way—and Hitting Your Milestones

One of the things I’ve been thinking about lately a lot is how you formalize the type of project management that is necessary in data platforms, and what you need to do differently compared to software development projects. I brought in a collaborator, one of the best customer success managers I know, to talk about how to do this correctly.

Agile absolutely works for data platform projects, but you need a lightweight way to lock in critical choices without slowing teams down. Architectural Decision Records (ADRs) provide that spine: they capture why you chose a direction, what you rejected, and the consequences—so you can move fast and keep delivery predictable. Combine ADRs with vertical slices, data contracts, quality gates, and observable pipelines, and you can ship in short cycles while meeting real dates.

Secure at the Boundary: RBAC, Aggregator Groups, RLS—and what OneSecurity changes

Despite the different semantics, Microsoft Fabric actually uses the same principles as folder security. Fabric makes the security boundary explicit (the workspace) though, which actually makes role design easier, and lighter‑weight, than old folder ACLs.

Foundational and Derived Data Products: Practical Guidance for Architects and Data Leaders

As we discussed previously, a data product is a reusable, self‑contained package that bundles data, metadata, access methods, and governance to deliver a clear outcome to users or other systems. Treating data as a product implies product management disciplines (contracts, SLOs, versioning, observability) and an emphasis on discoverability, interoperability, and security.

Within modern mesh-aligned architectures, data products must be interoperable and composable so they join predictably and can be assembled into higher‑order solutions. This is a first‑principles characteristic, not a nice‑to‑have.

Data Products Demystified: What They Are and Why They Matter

Data products are becoming a hot topic across industries, from classrooms to oil fields to trading floors. Yet the term “data product” can be confusing, conjuring images of complex databases or black-box AI. This blog post aims to clarify what a data product actually is in straightforward terms, and why it’s important for both technical and non-technical professionals. We’ll explore how data products turn raw data into useful tools, how they benefit organizations, and how they differ from other data concepts. Along the way, we’ll look at a couple examples to make the ideas concrete.

From Medallion to Mesh: How Unity Catalog Transforms Data Strategy

I wrote up some thoughts on how Databricks’ Unity Catalog Views change the way we should be thinking about lakehouses, and bout how they enable data mesh architectures on the Neudesic Innovation channel on Medium

https://medium.com/neudesic-innovation/from-medallion-to-mesh-how-unity-catalog-transforms-data-strategy-0b092a8e5941

Fusion Development 5 – Conclusion

In sectors ranging from education to healthcare, finance to e-commerce, the digital revolution is reshaping how organizations operate. As illustrated by Maple Elementary School’s tech-driven metamorphosis, frontline professionals can leverage technology in ways previously deemed the exclusive domain of IT specialists. However, this paradigm shift necessitates a deeper evolution within the IT and Analytics teams, who now bear the dual responsibility of fostering innovation while ensuring the sanctity of data, especially sensitive information like student details.

1. Embracing a Collaborative Mindset:

Old Approach: IT departments, whether in schools or corporations, often functioned in silos, creating a disconnect between solution developers and end-users.

The Shift: Foster a culture of collaboration. Open channels of communication ensure that IT solutions resonate with real-world needs and are adaptable across sectors.

2. Democratization over Gatekeeping:

Old Fear: The specter of “shadow IT” loomed large – unauthorized solutions that bypass the IT department.

The Shift: Instead of stifling these grassroots innovations, guide and nurture them. Offer platforms, like the Power Platform, but ensure they’re designed to adhere to industry-specific regulations and standards.

3. Continuous Education and Training:

Old Approach: Tools and platforms were handed down without adequate training.

The Shift: Regular workshops and training sessions, tailored to the specifics of the industry – be it education’s focus on pedagogy or finance’s emphasis on data security – can foster competent, confident users.

4. Agile and Flexible Development:

Old Approach: Lengthy, rigid development cycles that couldn’t adapt to evolving needs.

The Shift: Adopt an agile methodology. This iterative, feedback-driven approach ensures solutions remain relevant, whether in a classroom setting or a corporate boardroom.

5. Prioritizing Data Accessibility While Ensuring Security:

Old Fear: Concerns over data misuse led to limiting data access.

The Shift: IT teams should strike a balance. While ensuring data is accessible and usable, robust security measures need to be in place. This is especially crucial in sectors like education and healthcare, where sensitive data is paramount. Data encryption, multi-factor authentication, and regular audits are non-negotiable.

6. Celebrating and Encouraging Grassroots Innovation:

Old Approach: A top-down perspective of IT solutions.

The Shift: Recognize that innovation can stem from any level. Whether it’s a teacher like Lisa or a finance executive spotting a workflow inefficiency, grassroots innovations have the power to transform.

7. Vigilance Towards Privacy and Security:

Old Approach: A generic, one-size-fits-all approach to data security.

The Shift: Given the increasing emphasis on personal data protection, with regulations like GDPR and CCPA coming into the spotlight, IT departments need to ensure that solutions are tailored to comply with sector-specific data protection standards. Especially in sectors like education, where children’s data is involved, vigilance is paramount.

Pioneers of the New Digital Era

While the digital stories at Maple Elementary underscore the potential of user-driven tech integration, they also hint at the unsung backbone enabling these transformations: the IT and Analytics teams. As organizations worldwide pivot to this new digital-centric model, these teams stand at the helm, ensuring that innovation thrives in a secure, compliant environment.

In this new era, IT isn’t just about supporting operations; it’s about pioneering change. By fostering innovation and ensuring the sanctity of data, IT and Analytics teams are sculpting the future, one digital solution at a time.

Fusion Development 4 – Bringing it all together

Standalone tools often provide solutions to specific problems, but when these tools come together, harmonizing their individual strengths, the resulting impact is magnified. At Maple Elementary School, we’ve witnessed the transformative power of individual solutions, from Sarah’s analytical dashboards to Tom and Elaine’s intuitive apps. However, the real magic unfurls when these tools intertwine, and Lisa’s journey as a dedicated teacher is a testament to this.

Lisa’s Vision: A Holistic Reading App

Lisa, an experienced fifth-grade teacher at Maple Elementary, had been observing the digital initiatives of her colleagues. Inspired by their successes, she envisioned an app that would be a one-stop solution for her students – a place where they could set reading goals, submit reflective responses, track their progress, and be rewarded for their achievements.

Lisa’s Reading Galaxy App

With the Power Platform at her disposal, Lisa began crafting the Reading Galaxy App. Leveraging the PowerApps, she built an interactive interface allowing students to:

Set Personalized Goals: Using insights from Sarah’s dashboards, students could see their current reading levels and set progressive targets for themselves.

Submit Reflections: Drawing from Tom’s Feedback Loop, Lisa integrated a section for students to submit their thoughts and feelings on their readings.

Earn Badges: Inspired by Elaine’s emoticon feature, Lisa added a gamified element. As students hit milestones or showed significant growth, they were awarded digital badges, visible on their profiles.

The true genius of Lisa’s app lay in its connectivity. By linking it with PowerBI, data from students’ goals and reflections were fed into Sarah’s analytics, further enriching the insights. The badges, awarded based on data points from Tom’s and Elaine’s apps, ensured that they were both reflective of effort and achievement.

The Reading Galaxy App was an immediate sensation in her class. Students were more engaged, taking ownership of their reading journeys. The badge system introduced a healthy competitive spirit, motivating students to push their boundaries. The reflections not only enhanced comprehension but also fostered a deeper emotional connection to the texts.

The Power of Availability and Integration

Lisa’s journey underscores two fundamental truths:

Availability Matters: Having tools like the Power Platform readily available democratizes innovation. Lisa, like her colleagues, wasn’t an IT specialist. Yet, with the right tools at her fingertips, she crafted a solution with a profound impact.

Integration Amplifies Impact: Standalone tools have their merits, but integrated solutions harness collective strengths. By drawing from Sarah’s, Tom’s, and Elaine’s tools, Lisa’s app offered a comprehensive solution, addressing multiple facets of the reading journey.

The Future is Integrated

The success stories from Maple Elementary – from Sarah’s analytics to Lisa’s holistic app – highlight the boundless potential of integrated tech solutions:

Sarah’s Dashboard: Offering deep, data-driven insights.

Tom’s Feedback Loop App: Providing timely, actionable feedback.

Elaine’s Reading Progress Tracker: Merging academic metrics with emotional well-being.

Lisa’s Reading Galaxy App: Crafting a holistic, engaging reading journey for students.

Today, the true potential of technology lies not just in individual solutions, but in the seamless integration of these solutions. As more educators like Lisa harness the power of available tools, and synergize them to craft holistic solutions, the future of education is not just bright – it’s dazzling.

Fusion Development 3 – Citizen Developers

In a time defined by its rapid technological advances, the democratization of technology is a game-changer. The rise of ‘citizen data analysts’ is a testament to this, with individuals like Sarah leading the charge in reshaping how we approach data. Parallelly, a new breed of innovators is emerging: “citizen developers”. These are passionate individuals who, despite lacking formal IT training, are crafting solutions tailored to their specific needs, thanks to Microsoft’s Power Platform.

Tom and Elaine: The Unlikely Developers

Tom and Elaine, both members of Maple Elementary School faculty, share Sarah’s commitment to student success. Their passion for fostering reading skills led them down a path neither expected: app development. With the Power Platform’s PowerApps, they turned their visions into reality.

Tom’s Feedback Loop App

Tom always believed in the power of timely feedback. However, he noticed that the conventional feedback mechanisms were often delayed, causing students to miss out on timely course correction. Drawing inspiration from this, Tom ventured into PowerApps. Using its intuitive interface, he pieced together the Feedback Loop App. The drag-and-drop functionality allowed him to design an interface that was both visually appealing and user-friendly. With the app’s built-in connectors, he integrated it with the school’s database, ensuring real-time data synchronization.

The app transformed the feedback process. As students read aloud, teachers could immediately record observations, categorizing them under fluency, comprehension, and pronunciation. By week’s end, the app auto-generated a comprehensive report, bridging the communication gap between parents and students. By providing this immediate feedback, Tom ensured that students could adjust swiftly, fostering a more agile learning environment.

Elaine’s Reading Progress Tracker

Elaine, deeply attuned to her students’ emotional and academic needs, envisioned a tool that would capture more than just reading metrics. PowerApps became her platform of choice. Elaine utilized PowerApps’ template library, selecting one tailored for progress tracking. She then personalized it, adding unique fields for daily achievements, word logs, and emotional reflections. The emoticon-based emotional check-in was a masterstroke, letting students choose from a variety of emoji’s to record how they were feeling at a particular time of day.

The Reading Progress Tracker became an instant hit. Students enjoyed the gamified aspect of logging their progress, and the visual representation of their reading journey motivated them further. The emoticon feature proved invaluable, offering Elaine insights into students’ emotional states. If a student consistently logged negative emotions post-reading, Elaine could provide timely emotional and academic interventions.

Collaboration: The Power Trio

Sarah, recognizing the potential of her colleagues’ innovations, wanted to integrate them in with her dashboard. Using PowerApps’ connectivity with PowerBI, she brought in the data from Tom’s feedback app and Elaine’s tracker into her dashboard. This not only streamlined her data gathering process but also enriched her insights.

Together, the trio’s tools offered a 360-degree view of the students’ reading journey. Tom’s app provided actionable feedback, Elaine’s tool tracked progress and emotions, and Sarah’s dashboard synthesized this data, offering rich, data-driven insights. This tools ensured that students received comprehensive support, both academically and emotionally. After Sarah, Tom, and Elaine had used the suite of tools enough to be confident that they were workable and stable, they even published them to the other members of the Maple Elementary School faculty.

The Power Platform’s Potential Unleashed

The success stories of the Feedback Loop App and the Reading Progress Tracker underscore the boundless potential of the Power Platform:

Feedback Loop App: Transforming feedback into real-time, actionable insights.

Reading Progress Tracker: Capturing the holistic reading journey, merging academic metrics with emotional well-being.

Tom and Elaine, empowered by the Power Platform, have transcended their traditional roles, becoming pioneers of tailor-made, impactful solutions. Their journey symbolizes the promise of the Power Platform, proving that with the right tools, passion can indeed translate into innovation. In this tech-driven era, citizen developers like Tom and Elaine are not just the future; they’re the present.

Fusion Development 2 – The Citizen Data Analyst

In today’s digitized world, the significance of data is clear. However, the true power lies in effectively harnessing data. While IT Departments and Data Teams may be big players in making those harnesses, it’s up to the end-user, who sometimes has minimal IT experience, to use those tools and make them meaningful, resulting in people who aren’t just consumers of data but who can interact with it fluently to make better decisions. But these people need the gateways and tools to make this happen.

Enter data products, the unsung heroes of our digital realm, offering targeted solutions that transform vast data troves into actionable insights. The magic of data products is their ability to empower ‘citizen data analysts’, individuals who might not boast a deep data analytics background but carry a wealth of domain-specific knowledge. They have the curiosity, passion, and background to make the most of the available data, and they may have individual ideas about how best to use it.

Sarah’s Journey of Data Empowerment

Sarah, the reading intervention specialist at Maple Elementary School, starts her day with her favorite tool: her personalized dashboard on PowerBI. This dashboard isn’t just any dashboard; it’s one she crafted, merging data from various data products to provide a holistic view tailored to her needs. It offers her an overview of reading scores, resource allocations, student engagement metrics, and collaboration notes, all in one unified space, and even available to her on her tablet as she moves from room to room to work with various students – or even to find a quiet place to do her planning.

From her dashboard, Sarah can deep dive into the Student Performance Tracker. Here, she gets a detailed visualization of reading scores from multiple assessments, allowing her to pinpoint patterns and identify students needing intervention.

By midday, as she’d identified a struggling student who needed some specialized resources, she turned to the Resource Allocator. Resource Allocator is a tool she uses to find the best resources and books to recommend for a particular student. It has access to the student’s assessment results, the school’s library catalog for reading books, Sarah’s inventory of intervention tools, and the district’s recommended interventions. After analyzing a student’s profile, Resource Allocator recommends appropriate resources for that student. Sarah has compiled her inventory in a spreadsheet, tracking what she has available at any given moment. By linking the spreadsheet with resources in the school library that are also available, she’s making her task much more streamlined and efficient, allowing her to pick up some appropriate books before meeting with the student.

In the afternoon, her PowerBI dashboard alerts notify her of changes in student engagement. Curious, she taps into the Engagement Monitor. This tool, integrated with Microsoft Office365 and Maple Elementary’s learning management system and intervention systems, offers her a granular view of how students engage with their allocated resources.

Ending her day, Sarah reflects on her insights and strategies, readying herself to collaborate with peers and parents.

Sarah’s PowerBI Mastery: No IT Training Required

With her background focused primarily on reading intervention rather than IT, one might wonder how Sarah manages to create such a comprehensive dashboard on PowerBI. The answer lies in the user-centric design of the data products and PowerBI.

Each data product developed by the IT department of Maple Elementary resides in individual PowerBI workspaces. These workspaces are structured containers designed with clear naming conventions and organized datasets. They serve as dedicated hubs for each product, making data discovery and exploration intuitive.

When Sarah decides to create or update her dashboard, she doesn’t need to write complex code or queries. Instead, she utilizes PowerBI’s user-friendly interface to:

Connect to Workspaces: Sarah can access the desired data product workspace with a few clicks. The intuitive layout means she doesn’t have to hunt around; she selects the workspace corresponding to the data product she wants to integrate.

Drag-and-Drop Functionality: Sarah drags relevant data fields onto her canvas. Whether it’s a chart showing reading scores from the Student Performance Tracker or a table of resources from the Resource Allocator, the process is visual and straightforward.

Customize Visuals: PowerBI offers a plethora of visualization options. Sarah can choose bar graphs, pie charts, heat maps, or any other visual that best represents her data. Plus, she can ensure the dashboard aligns with her aesthetic preferences with easy-to-use formatting tools.

Set Alerts: Sarah can set alerts on specific metrics to stay updated on significant changes or trends. This way, she’s immediately notified if, for instance, a student’s engagement drops significantly.

Share and Collaborate: Once satisfied with her dashboard, Sarah can easily share it with other stakeholders directly from PowerBI. They can then provide feedback, which she can incorporate in real-time.

The marriage of well-organized data product workspaces and PowerBI’s user-centric design enables Sarah to be an autonomous data analyst, crafting insights without needing constant IT intervention.

A New Dawn with Data Products

Sarah’s routine is a testament to the transformative nature of modern data products:

Student Performance Tracker: A detailed lens into student assessments

Resource Allocator: Smart recommendations at her fingertips

Engagement Monitor: Offering a pulse on student interactions

Collaboration Hub: Bringing stakeholders together on Azure

By putting the power of custom dashboards and data products into the hands of domain experts like Sarah, we’re paving the way for more informed, agile, and relevant decision-making processes in every field.

Fusion Development – Introduction

The roles of technology, data, and systems are rapidly transforming, leading to democratization in data and development within organizations. This shift introduces Data Products and Citizen Developers and Analysts, empowering individuals without IT backgrounds to create and analyze data-driven solutions. A collaborative approach called Fusion Development maximizes these roles, breaking silos and encouraging shared problem-solving. These trends can potentially reshape the digital world across various sectors and industries.

How we interact with technology, data, and systems is transforming significantly. Traditional boundaries that define roles and responsibilities are blurring, leading to new opportunities and challenges. One of the most fantastic opportunities is democratizing data and development within organizations. The ability to translate something you imagine into a computer program or data visualization is no longer limited to people with computer programming backgrounds and the education and training that go with them.

Data Products are tools that transform intricate data into intelligible and actionable insights. They process and render data into a more palatable format, thereby bestowing individuals — regardless of their technical acumen — with the power to make data-driven decisions. By presenting data in a user-friendly manner, data products serve as bridges that connect raw data to meaningful insights. They democratize data, making it accessible not just to data aficionados and specialists but to everyone, fostering a culture where data-driven insights become the cornerstone of decision-making processes.

Citizen Developers are individuals who, despite lacking formal IT backgrounds, take the initiative to create applications or solutions tailored to address specific needs. Propelled by the advent of user-friendly platforms, like Microsoft Power Platform, these professionals venture into realms traditionally reserved for programmers, democratizing the software development process. They translate ideas into functional solutions, bridging gaps and fostering innovation without being encumbered by the intricacies associated with traditional software development. Their endeavors often reflect a pragmatic understanding of the problem, coupled with creatively utilizing available resources to devise solutions.

Similarly, Citizen Data Analysts are individuals from diverse professional backgrounds who leverage data products to interpret and analyze data, amalgamating their domain knowledge with analytical tools to extract meaningful insights. Unlike traditional data scientists, who specialize in statistics, data processing, and analysis, Citizen Data Analysts bring their professional background and experience to analyze data in any role. By harnessing the power of data products, they can transcend the technical barriers that often surround data analytics, delving into data-driven inquiry to uncover actionable insights that can inform decision-making within their respective domains.

The essence of Citizen Developers and Citizen Data Analysts lies in their ability to lower the barriers to entry in software creation and analytics, empowering a broader spectrum of individuals to contribute to the digital solutions landscape. This ability, in turn, accelerates problem-solving and cultivates a more inclusive and innovative development ecosystem. Through their efforts, professionals not only contribute to the rapid prototyping and deployment of solutions but also to the broader digital transformation narrative, reshaping how organizations approach software development, analytics, and problem-solving.

Fusion Development encapsulates a collaborative approach where professional developers, citizen developers, and citizen data analysts converge to collaborate on projects. By combining the technical prowess of trained developers with the practical insights and domain expertise of citizen contributors, fusion development aims to cultivate solutions that are both technically robust and practically relevant. This collaborative ethos enables a richer understanding of problem domains and fosters a more inclusive, innovative environment for solution creation.

In the fusion development paradigm, synthesizing diverse skill sets and perspectives engenders a more holistic approach to problem-solving. Professional developers bring their technical acumen, coding skills, and understanding of software development lifecycles. In contrast, citizen contributors, with their understanding of end-user needs and domain-specific challenges, contribute practical insights that ensure the development of solutions tailored to real-world needs.

The beauty of fusion development lies in its ability to break down the traditional silos that often exist within organizations between technical and non-technical personnel. It fosters a culture of collective problem-solving and shared ownership of projects, thereby accelerating the pace of innovation and ensuring that solutions are both technically sound and user-focused. Through this cross-functional lens, fusion development not only augments the quality and relevance of digital solutions but also cultivates a more inclusive, dynamic environment for technological innovation, ensuring that the digital solutions produced are well-aligned with user needs and organizational goals.

Over the next four weeks, we will look at these ideas in depth in the context of an elementary school and how various citizen developers and data analysts can increase their productivity and impact. While the series uses an elementary school as a backdrop to illustrate these concepts, the lessons, and principles are applicable across various sectors and industries. The goal is to provide readers with a clear understanding of how these trends are reshaping the digital world and the potential they hold for organizations and individuals alike.