Blogs

Because sharing is caring

5 core Data Collection Patterns

At AgileData, delivering our Fractional Data Service has revealed the diverse challenges of integrating data from varied organisations, industries, and systems. To scale effectively, we’ve adopted five core data collection patterns based on our “Define it Once, Reuse it Often” (DORO) principle:

1. Push
2. Pull
3. Stream
4. Share
5. File Drop

These patterns are supported by a toolkit of tested technologies like Dataddo, Meltano, and Google services, allowing us to solve new data challenges quickly. Our approach ensures flexibility and scalability, always starting with the question: Push, Pull, Stream, Share, or File Drop?

The Dojo with Joel Tosi and Dion Stewart

Join Murray Robinson and Shane Gibson as they chat with Joel Tosi and Dion Stewart about Dojo’s.

Dojo’s are a six week immersive learning experience where teams learn and practice new ways of working on real work with skilled coaches. We discuss the structure and implementation of Dojos, chartering, coaching dynamics, and frequent feedback loops. We emphasize the challenges and anti-patterns that can emerge, such as treating Dojos as bootcamps or lacking stakeholder support. Finally, Joel and Dion offer insights into the future of Dojos and their potential for building capability and driving organisation change.

#AgileDataDiscover weekly wrap No.1
#AgileDataDiscover weekly wrap No.1

We are tackling challenges in migrating legacy data platforms by automating data discovery and migration to reduce costs significantly. Our approach includes using core data patterns and employing tools like Google Gemini for comparative analysis. The aim is to streamline data handling and enable collaborative governance in organisations. Follow their public build journey for updates.

Product Management with John Cutler
Product Management with John Cutler

Join Murray Robinson and Shane Gibson as they discuss product management with John Cutler, famous for his newsletter The Beautiful Mess.

We dive into the purpose of product management as a driver of profitable growth. And we discuss how to use the north star framework to align your organisation around common goals. We examine how to use product roadmaps to communicate and prioritize plans and discuss how critical it is to be an open, honest, and effective leader. And finally we look at whether agile is critical or useless in product management. Join us as we uncover the insights and best practices product managers need to be more effective.

UX Research with Nick Fine
UX Research with Nick Fine

Join Murray Robinson and Shane Gibson as they discuss UX research with Dr. Nick Fine. Nick explains what to expect from a UX researcher, how to do user research well, and the challenges that researchers have to overcome to obtain valid results. Whether you’re a product manager, designer, or developer this episode is packed with actionable insights, real world examples and expert tips to enhance your understanding of UX research and its role in creating high value, digital products and services.

The Vanguard Method with Ibrar Hussain
The Vanguard Method with Ibrar Hussain

Join Murray Robinson and Shane Gibson as they discuss the Vanguard method with Ibbi Hussain, one of the founding architects of this revolutionary approach to organisation change. Ibi describes how the Vanguard method uses systems thinking, service design, and occupational psychology to make profound improvements to call center performance. Join us as we navigate through the philosophy of change, the importance of leadership. And practical insights into implementing sustainable improvement within organisations.

Episode 100 – A retrospective
Episode 100 – A retrospective

Welcome to our 100th episode.

In this episode, we talk about what we’ve learnt from our guests about agile, product development and leadership. We discuss what the podcast is about and why we started it. We unpack our critical take on fake agile and discuss agile leadership and product development. Join us as we take a deep dive into the insights, experiences and conversations that have shaped our podcast ovr the last 99 episodes. And here’s to another hundred episodes of cutting through the nonsense and bringing you the advice you need to succeed.

Business Agility with Evan Leybourn
Business Agility with Evan Leybourn

Join Murray Robinson and Shane Gibson as they chat with Evan Leybourn from the Business Agility Institute about business agility. We discuss how to define and measure business agility, examples of agile organisations, and the struggle with traditional bureaucratic models. We delve into the different types of leadership, the nature of internal cultural and political changes, and the challenges faced by organisations in adopting and implementing agility.

AgileData App

Explore AgileData features, updates, and tips

Network

Learn about consulting practises and good patterns for data focused consultancies

DataOps

Learn from our DataOps expertise, covering essential concepts, patterns, and tools

Data and Analytics

Unlock the power of data and analytics with expert guidance

Google Cloud

Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows

Journey

Explore real-life stories of our challenges, and lessons learned

Product Management

Enrich your product management skills with practical patterns

What Is

Describing data and analytics concepts, terms, and technologies to enable better understanding

Resources

Valuable resources to support your growth in the agile, and data and analytics domains

AgileData Podcast

Discussing combining agile, product and data patterns.

No Nonsense Agile Podcast

Discussing agile and product ways of working.

App Videos

Explore videos to better understand the AgileData App's features and capabilities.

New Google Cloud feature to Optimise BigQuery Costs
New Google Cloud feature to Optimise BigQuery Costs

This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.

Data as a First-Class Citizen: Empowering Data Magicians
Data as a First-Class Citizen: Empowering Data Magicians

Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.

The patterns of Data Vault with Hans Hultgren
The patterns of Data Vault with Hans Hultgren

In a compelling episode of the Agile Data Podcast, Shane Gibson invites Hans Hultgren to explore the intricacies of Data Vault modeling. Hans, with a substantial background in IT and data warehousing, brings his expertise to the table, discussing the evolution and benefits of Data Vault modeling in today’s complex data landscapes. Shane and Hans navigate through the foundations of Data Vault, including its core components like hubs, satellites, and links, and delve into more advanced concepts like Same-As Links (SALs) and Hierarchical Links (Hals). They highlight how Data Vault enables flexibility, agility, and incremental development in data modeling, ensuring scalability and adaptability to change.

To whitelabel or not to whitelabel
To whitelabel or not to whitelabel

Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.

Data Consulting Patterns with Joe Reis
Data Consulting Patterns with Joe Reis

Dive into the world of data consulting with Shane Gibson and Joe Reis on the Agile Data Podcast. Explore their journey from traditional employment to successful data consulting, covering client acquisition, business models, financial management, reputation, sales strategies, employee management, and work-life balance.

The Enchanting World of Data Modeling: Conceptual, Logical, and Physical Spells Unraveled
The Enchanting World of Data Modeling: Conceptual, Logical, and Physical Spells Unraveled

Data modeling is a crucial process that involves creating shared understanding of data and its relationships. The three primary data model patterns are conceptual, logical, and physical. The conceptual data model provides a high-level overview of the data landscape, the logical data model delves deeper into data structures and relationships, and the physical data model translates the logical model into a database-specific schema. Understanding and effectively using these data models is essential for business analysts and data analysts, create efficient, well-organised data ecosystems.

Cloud Analytics Databases: The Magical Realm for Data
Cloud Analytics Databases: The Magical Realm for Data

Cloud Analytics Databases provide flexible, high-performance, cost-effective, and secure solution for storing and analysing large amounts of data. These databases promote collaboration and offer various choices, such as Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse Analytics, each with its unique features and ecosystem integrations.

Unveiling the Definition of Data Warehouses: Looking into Bill Inmon’s Magicians Top Hat
Unveiling the Definition of Data Warehouses: Looking into Bill Inmon’s Magicians Top Hat

In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.

Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians
Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians

Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.

Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians
Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians

Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.

Data Lineage Patterns with Tomas Kratky

In this episode of the AgileData Podcast, Shane Gibson has an insightful discussion with Tomas Kratky on the evolution and importance of data lineage, especially in large enterprises. Tomas Kratky, a traditional software engineer turned data enthusiast, shared his journey to founding Manta, a company focused on data lineage. The conversation highlighted the significance of data lineage, not just as an end in itself, but as a powerful tool for unlocking potential in large enterprises, enhancing visibility, and fostering agility.

Free Google Analytics 4 (GA4) online courses
Free Google Analytics 4 (GA4) online courses

TD:LR There is some great free course content to help you upskill in Google Analytics 4 (GA4) Here are the ones we recomend.Discover the Next Generation of Google Analytics Find out how the latest generation of Google...

Observability – Raj Joseph

Join Shane Gibson as he chats with Raj Joseph on his experience in defining data observability patterns.Guests Raj JosephShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible |...

5E’s
5E’s

As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.

We outline what each of these are, how they are different to each other and how to charge for delivering them.

Agile-tecture Information Factory
Agile-tecture Information Factory

Defining a Data Architecture is a key pattern when working in the data domain.

Its always tempting to boil the ocean when defining yours, don’t!

And once you have defined your data architecture, find a way to articulate and share it with simplicity.

Here is how we articulate the AgileData Data Agile-tecture.