Blogs
Because sharing is caring
One suggestion on starting your journey into becoming a data consultant
At #AgileData we are focussed on helping people become data entrepreneurs.
Whether they want to start their journey by side hustling, or becoming a one person data band or grow their small data consulting companies revenue and margin, but not their team, we are keen to be part of their journey.
AgileData Feature #02 – Single Sign-in
With Single Sign-On (SSO), users gain streamlined access to the AgileData App, eliminating the need for repeatedly logging in and reducing friction in their data day.
This feature leverages Google Identity and Access Management (IAM) to provide seamless and secure authentication, ensuring that users can easily connect to the platform using their existing credentials.
AgileData’s SSO is not only secure but also scalable, designed to meet the needs of growing teams while maintaining the highest standards of data protection and access control.
5 core Data Collection Patterns
At AgileData, delivering our Fractional Data Service has revealed the diverse challenges of integrating data from varied organisations, industries, and systems. To scale effectively, we’ve adopted five core data collection patterns based on our “Define it Once, Reuse it Often” (DORO) principle:
1. Push
2. Pull
3. Stream
4. Share
5. File Drop
These patterns are supported by a toolkit of tested technologies like Dataddo, Meltano, and Google services, allowing us to solve new data challenges quickly. Our approach ensures flexibility and scalability, always starting with the question: Push, Pull, Stream, Share, or File Drop?
An Experiment – Top Data Trends for 2025 with Coalesce and Google NotebookLM
Join two LLM generated guests as they discuss the Top Data Trends of 2025 Whitepaper published by Coalesce.
This is a different episode. Instead of a human guest, we have two robot guests.
I decided to try and experiment. My experiment was, can I upload a white paper to LLM, have it generate a podcast listen to that podcast in my daily walk and see whether that summary removes the need for me to actually read the white paper.
So in this case, I have grabbed a white paper called Top Data Trends for 2025 from Coalesce, uploaded it to the Google Notebook LLM and got it to generate a podcast with two hosts chatting about the white paper.
Have a listen, let me know what you think.
AgileData Feature #01 – Marketplace
Information Consumers can quickly search, find and access the Information Products they need to answer their business questions.
This feature enables Information Consumers to search and find all the available dashboards, reports and analytical models in the AgileData App, regardless of what third party Last Mile tool they are created in.
It also enables the Information Consumer to quickly open that report or dashboard directly from the Marketplace, when those reports are accessible by a web URL
Agile Project Management with Johanna Rothman
Join Murray Robinson and Shane Gibson as they chat with Jurgen Appelo about the crisis in the agile community.
We explore how the agile job market has changed with diminishing roles for agile coaches and scrum masters. And we discuss what people can do now that the agile consulting and training gold rush is over. We talk about the poor quality of agile practices in organizations claiming to have done agile. And the need to improve further, to get the benefits available. And we talk about the evolution and rebranding of agile practices and the impact of AI on the industry.
Gain valuable insights into the future of agile, the importance of adapting to change and strategies for sustaining relevance in a shifting market.
The crisis in agile with Jurgen Appelo
Join Murray Robinson and Shane Gibson as they chat with Jurgen Appelo about the crisis in the agile community.
We explore how the agile job market has changed with diminishing roles for agile coaches and scrum masters. And we discuss what people can do now that the agile consulting and training gold rush is over. We talk about the poor quality of agile practices in organizations claiming to have done agile. And the need to improve further, to get the benefits available. And we talk about the evolution and rebranding of agile practices and the impact of AI on the industry.
Gain valuable insights into the future of agile, the importance of adapting to change and strategies for sustaining relevance in a shifting market.
The Haier model with William Malek
Join Murray Robinson and Shane Gibson as they chat with William Malik about the Haier operating model. Haier, a Chinese appliance giant, transitioned from a failing state-owned enterprise to a highly profitable publicly owned organisation with over 5,700 micro enterprises and $50 billion in revenue.
We explain how Haier uses an internal marketplace to allow microenterprises to trade goods and services with each other, based on a value added mechanism and smart contracts. An approach based on the philosophy of RenDanHeYi, where managers release control and support teams to achieve their potential to create valuable solutions within an open environment.
William makes a compelling case for the benefits of a decentralized agile organizational model in driving high innovation and performance.
Data Contracts with Andrew Jones
Join Shane Gibson as he chats with Andrew Jones on the pattern of Data Contracts
AgileData App
Explore AgileData features, updates, and tips
Network
Learn about consulting practises and good patterns for data focused consultancies
DataOps
Learn from our DataOps expertise, covering essential concepts, patterns, and tools
Data and Analytics
Unlock the power of data and analytics with expert guidance
Google Cloud
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Journey
Explore real-life stories of our challenges, and lessons learned
Product Management
Enrich your product management skills with practical patterns
What Is
Describing data and analytics concepts, terms, and technologies to enable better understanding
Resources
Valuable resources to support your growth in the agile, and data and analytics domains
AgileData Podcast
Discussing combining agile, product and data patterns.
No Nonsense Agile Podcast
Discussing agile and product ways of working.
App Videos
Explore videos to better understand the AgileData App's features and capabilities.
#AgileDataDiscover weekly wrap No.4
We review feedback, highlight emerging use cases like legacy data understanding, data governance, and automated data migration. New patterns are needed for moving from prototype to MVP. Challenges include managing tokens, logging responses, and secure data handling. The GTM strategy focuses on Partner/Channel Led Growth.
AI Data Agents with Joe Reis
Join Shane Gibson as he chats with Joe Reis on how the potential adoption of GenAI and LLM’s in the way Data teams work.
#AgileDataDiscover weekly wrap No.3
We focus on developing features such as secure sign-in, file upload, data security, and access to Google’s LLM. Challenges include improving the menu system and separating outputs into distinct screens for clarity. Feedback drives their iterative improvements.
#AgileDataDiscover weekly wrap No.2
We discuss the ongoing development of a new product idea, emphasising feasibility and viability through internal research (“McSpikeys”). Initial tests using LLMs have been promising, but strategic decisions lie ahead regarding its integration. The team grapples with market validation and adjusting their workflow for optimal experimentation.
#AgileDataDiscover weekly wrap No.1
We are tackling challenges in migrating legacy data platforms by automating data discovery and migration to reduce costs significantly. Our approach includes using core data patterns and employing tools like Google Gemini for comparative analysis. The aim is to streamline data handling and enable collaborative governance in organisations. Follow their public build journey for updates.
We are working on something new at AgileData, follow us as we build it in public
The AgileData team is dedicating 30 days to exploring a novel data use case, which might lead to a new product, feature set, or module. They’ll document their daily progress publicly to share learnings and insights. Follow their journey on their blog for updates as they build and experiment in real-time.
Bridging Data and Product Management Practises and Patterns with Juha Korpela
Join Shane Gibson as he chats with Juha Korpela on how to adopt patterns and practises from Product Management and apply them to the data domain.
Introducing Hai, AgileData 2024 Data Intern
I’m Hai, a name that intriguingly means “hi” in English. Originally from Vietnam, I now find myself in Australia, studying Data Science and embracing an internship at AgileData.io. This journey is not just about academic growth but also about applying my knowledge in practical, impactful ways. Join me as I explore the blend of technology and community, aiming to make a difference through data.
Patterns for being a successful internal data consultancy with Dylan Jones
Join Shane Gibson as he chats with Dylan Jones on how to adopt patterns used by successful data consultancies and apply them in your organisation as an internal data team.
Defining self-service data
Everybody wants self service data, but what do they really mean when they say that.
If we gave them access to a set of highly nested JSON data, and say “help your self”, would that be what they expect?
Or do they expect self service to be able to get information without asking a person to get it for them.
Or are they expecting something in between.
I ask them which of the five simple self service patterns they want to find, which form of self service they are after.
Combining agile and data five years on with Blair Tempero
Guests Blair TemperoShane GibsonResources Join Shane Gibson as he chats with Blair Tempero on that last five years since they started the AgileData Podcast together.Listen on your favourite Podcast Platform |...
There are 3 strategic / macro data use cases
I often ask which of these three macro data use cases the Organisations believed were its priorities to achieve their business strategy:
Providing data to Customers
Supporting Internal Processes
Providing data to External Organisations
Each of these three strategic / macro data use cases come with specific data architectures, data work and also impact the context of how you would design your agile data ways of working.
Eventually the data maintenance Tortoise will catch the new data work Hare
When you work in a data team you have to split your time between building and delivering new Information Products and maintaining the ones you have already delivered.
DataOps patterns can help reduce the time you spend on the maintenance work.
Building the Data Plane while flying it
In the data domain you typically have to balance between building the right thing and building the thing right.
The days of being able to spend 6 months or a year on “Sprint Zero” creating your data platform have gone.
One team I worked with called it “building the airplane as you fly it”
Here are 5 patterns I have seen data teams adopt to help them do this.
2024 the year of the Intelligent Data Platform
AI was the buzzword for 2023 and it will continue to be the buzzword for 2024.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Adopting these patterns moves a data platform from being a manual data platform, towards a data platform that can do some of the data work for you.
An Intelligent Data Platform.
The 3 patterns of AgileData AI
Having AI embedded in your product have become table stakes it seems.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Knowledge Graphs with Juan Sequeda
Dive deep into the world of knowledge graphs with Juan Sequeda on the Agile Data Podcast, hosted by Shane Gibson. Explore key insights from Juan’s journey in computer science, his pivotal role in semantic web development, and the transformative power of knowledge graphs in data integration. Discover how these technologies are reshaping the landscape of data management and the exciting future prospects with the advent of Large Language Models (LLMs). Tune in to understand the practical applications, challenges, and the future of knowledge graphs in enterprise data strategy.
Data Storytelling with Kat Greenbrook
Explore the Art of Data Storytelling with Kat Greenbrook on the Agile Data Podcast. Dive into Kat’s transformative journey from aspiring vet to a data storytelling expert, and discover the power of the ABT (And, But, Therefore) narrative framework in conveying compelling data insights. Uncover common pitfalls and learn crucial differences between data visualization and storytelling. Enhance your business communication skills with practical tips and insights from ‘The Data Storyteller’s Handbook.’ Perfect for professionals in data analytics, business intelligence, and anyone keen to master the art of turning data into impactful stories.
Demystifying CDP’s vs. Data Warehouse’s
In this article we describe the concepts of Customer Data Platforms (CDP) versus Data Warehouses.
The magic of DocOps
TD:LR Patterns like DocOps provide massive value by increasing collaboration across team members and automating manual tasks. But it still requires a high level of technical skills to work in a DocOps way. For the...
Iterations create milestone dates, milestone dates force trade off decisions to be made
Data teams struggle to not “boil the ocean” when doing data work.
Use milestones as a pattern to help the data team to focus on what really needs to be built and manage the trade-off decisions for what doesn’t.