Blogs
Because sharing is caring
5 core Data Collection Patterns
At AgileData, delivering our Fractional Data Service has revealed the diverse challenges of integrating data from varied organisations, industries, and systems. To scale effectively, we’ve adopted five core data collection patterns based on our “Define it Once, Reuse it Often” (DORO) principle:
1. Push
2. Pull
3. Stream
4. Share
5. File Drop
These patterns are supported by a toolkit of tested technologies like Dataddo, Meltano, and Google services, allowing us to solve new data challenges quickly. Our approach ensures flexibility and scalability, always starting with the question: Push, Pull, Stream, Share, or File Drop?
An Experiment – Top Data Trends for 2025 with Coalesce and Google NotebookLM
Join two LLM generated guests as they discuss the Top Data Trends of 2025 Whitepaper published by Coalesce.
This is a different episode. Instead of a human guest, we have two robot guests.
I decided to try and experiment. My experiment was, can I upload a white paper to LLM, have it generate a podcast listen to that podcast in my daily walk and see whether that summary removes the need for me to actually read the white paper.
So in this case, I have grabbed a white paper called Top Data Trends for 2025 from Coalesce, uploaded it to the Google Notebook LLM and got it to generate a podcast with two hosts chatting about the white paper.
Have a listen, let me know what you think.
AgileData Feature #01 – Marketplace
Information Consumers can quickly search, find and access the Information Products they need to answer their business questions.
This feature enables Information Consumers to search and find all the available dashboards, reports and analytical models in the AgileData App, regardless of what third party Last Mile tool they are created in.
It also enables the Information Consumer to quickly open that report or dashboard directly from the Marketplace, when those reports are accessible by a web URL
Agile Project Management with Johanna Rothman
Join Murray Robinson and Shane Gibson as they chat with Jurgen Appelo about the crisis in the agile community.
We explore how the agile job market has changed with diminishing roles for agile coaches and scrum masters. And we discuss what people can do now that the agile consulting and training gold rush is over. We talk about the poor quality of agile practices in organizations claiming to have done agile. And the need to improve further, to get the benefits available. And we talk about the evolution and rebranding of agile practices and the impact of AI on the industry.
Gain valuable insights into the future of agile, the importance of adapting to change and strategies for sustaining relevance in a shifting market.
The crisis in agile with Jurgen Appelo
Join Murray Robinson and Shane Gibson as they chat with Jurgen Appelo about the crisis in the agile community.
We explore how the agile job market has changed with diminishing roles for agile coaches and scrum masters. And we discuss what people can do now that the agile consulting and training gold rush is over. We talk about the poor quality of agile practices in organizations claiming to have done agile. And the need to improve further, to get the benefits available. And we talk about the evolution and rebranding of agile practices and the impact of AI on the industry.
Gain valuable insights into the future of agile, the importance of adapting to change and strategies for sustaining relevance in a shifting market.
The Haier model with William Malek
Join Murray Robinson and Shane Gibson as they chat with William Malik about the Haier operating model. Haier, a Chinese appliance giant, transitioned from a failing state-owned enterprise to a highly profitable publicly owned organisation with over 5,700 micro enterprises and $50 billion in revenue.
We explain how Haier uses an internal marketplace to allow microenterprises to trade goods and services with each other, based on a value added mechanism and smart contracts. An approach based on the philosophy of RenDanHeYi, where managers release control and support teams to achieve their potential to create valuable solutions within an open environment.
William makes a compelling case for the benefits of a decentralized agile organizational model in driving high innovation and performance.
Data Contracts with Andrew Jones
Join Shane Gibson as he chats with Andrew Jones on the pattern of Data Contracts
High performance organisations with Steve Spear
Join Murray Robinson and Shane Gibson as they chat with Steve Spear, author of a High Velocity Edge, Wiring the Winning Organization and Decoding the DNA of the Toyota Production System.
We discuss how good leaders create a system that makes it easy for people to do quality work when and where it’s needed by continually asking, what’s the problem and how can I help?
Steve explains how to use a developmental leadership approach with collaborative problem solving, workflow visualization and iterative improvement to become a high performing organization with an unbeatable competitive edge.
Wiring the Winning Organisation with Gene Kim
Join Murray Robinson and Shane Gibson as they chat with Gene Kim, leader of the Enterprise Technology Leadership Conference and author of the Phoenix and Unicorn Projects.
Gene describes how DevOps research shows that excellent socio-technical leadership is critical to team success and failure. He explains how technical leaders can dramatically increase team effectiveness by improving the organisational architecture and wiring to support independence of action, rapid feedback and simplification while maintaining stability and security.
AgileData App
Explore AgileData features, updates, and tips
Network
Learn about consulting practises and good patterns for data focused consultancies
DataOps
Learn from our DataOps expertise, covering essential concepts, patterns, and tools
Data and Analytics
Unlock the power of data and analytics with expert guidance
Google Cloud
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Journey
Explore real-life stories of our challenges, and lessons learned
Product Management
Enrich your product management skills with practical patterns
What Is
Describing data and analytics concepts, terms, and technologies to enable better understanding
Resources
Valuable resources to support your growth in the agile, and data and analytics domains
AgileData Podcast
Discussing combining agile, product and data patterns.
No Nonsense Agile Podcast
Discussing agile and product ways of working.
App Videos
Explore videos to better understand the AgileData App's features and capabilities.
2024 the year of the Intelligent Data Platform
AI was the buzzword for 2023 and it will continue to be the buzzword for 2024.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Adopting these patterns moves a data platform from being a manual data platform, towards a data platform that can do some of the data work for you.
An Intelligent Data Platform.
The 3 patterns of AgileData AI
Having AI embedded in your product have become table stakes it seems.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Knowledge Graphs with Juan Sequeda
Dive deep into the world of knowledge graphs with Juan Sequeda on the Agile Data Podcast, hosted by Shane Gibson. Explore key insights from Juan’s journey in computer science, his pivotal role in semantic web development, and the transformative power of knowledge graphs in data integration. Discover how these technologies are reshaping the landscape of data management and the exciting future prospects with the advent of Large Language Models (LLMs). Tune in to understand the practical applications, challenges, and the future of knowledge graphs in enterprise data strategy.
Data Storytelling with Kat Greenbrook
Explore the Art of Data Storytelling with Kat Greenbrook on the Agile Data Podcast. Dive into Kat’s transformative journey from aspiring vet to a data storytelling expert, and discover the power of the ABT (And, But, Therefore) narrative framework in conveying compelling data insights. Uncover common pitfalls and learn crucial differences between data visualization and storytelling. Enhance your business communication skills with practical tips and insights from ‘The Data Storyteller’s Handbook.’ Perfect for professionals in data analytics, business intelligence, and anyone keen to master the art of turning data into impactful stories.
Demystifying CDP’s vs. Data Warehouse’s
In this article we describe the concepts of Customer Data Platforms (CDP) versus Data Warehouses.
The magic of DocOps
TD:LR Patterns like DocOps provide massive value by increasing collaboration across team members and automating manual tasks. But it still requires a high level of technical skills to work in a DocOps way. For the...
Iterations create milestone dates, milestone dates force trade off decisions to be made
Data teams struggle to not “boil the ocean” when doing data work.
Use milestones as a pattern to help the data team to focus on what really needs to be built and manage the trade-off decisions for what doesn’t.
Your data team are mercenaries, define your ways of working based on this
Modern data teams are transient, often staying less than 5 years, unlike past decades of long-term loyalty.
Companies should adapt by defining robust Ways of Working (WoW) that endure beyond individual tenures.
Balancing in-house teams with reliable data vendors for continuity and efficiency may also be a useful pattern as part of your WoW.
I’m getting pedantic about semantics
TD:LR Having a shared language is important to help a data team create their shared ways of working. When we talk about self-service, we should always highlight which self-service pattern we are talking about.I'm...
The Art of Data: Visualisation vs Storytelling
Data visualization is like painting with data, using charts and graphs to make trends and patterns easy to understand. It’s great for presenting data objectively.
Data storytelling weaves a narrative around data, adding context, engaging emotions, and inspiring action. It’s perfect for persuading stakeholders.
Ways of Working with Scott Ambler
Join Shane Gibson on the Agile Data Podcast for an enlightening conversation with Scott Ambler, an IT and Agile expert. Delve into Scott’s journey from pioneering programmer to data architecture and Agile methodologies. Discover the evolution of Agile data, the importance of adapting ways of working, and the pitfalls of best practices. Learn valuable insights into continuous improvement, team dynamics, and the complexities of data quality in today’s fast-paced IT landscape. Don’t miss this episode for an in-depth exploration of Agile data and its impact on IT projects and processes.
Most of your data is rotten and it’s not your fault, but it is your problem
Data quality is an expectation, not an exception.
While data quality is crucial, it’s not always directly our fault when issues arise, nevertheless, it remains our problem to solve.
Data Contracts are one pattern that can help us solve this problem.
Are you delivering drills, holes or outcomes?
TD:LR Whether you're a Data Entrepreneur or an organisation looking for actionable insights, its the business outcome these insights help you achieve that is the most important thing. Yes you need a data platform and...
Data Asset, Data Product, Data Service?
TD:LR Should we treat data as an Asset, a Product, a Service or a hybrid combination of all three? Data Asset, Data Product, Data Service? There has been a lot of discussions on LinkedIn, lots of podcasts, lots of...
Demystifying the Semantic Layer
The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.
Attribution Model Patterns with Yorgos Moschovis
Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.
Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
Building a vibrant community with Scott Hirleman
Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.
Upgrading Python: A Plumbing Adventure in the Google Stack
In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.
We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.
Read more on our exciting plumbing journey.
AgileData App UX Capability Maturity Model
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches
Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!