Automate repeating data tasks to free up your analysts.

Get a repeatable answer to your most important business questions.

Replace your legacy EDW with a SaaS based modern data platform.

Utilise AgileData.io as a data solution underpinning your SaaS product, without having to develop a data platform yourself.

Get a single view of all your customers, understand how and when they engage with you.

Centralise your data into a single data repository to enable you to manage data as a true asset.

Migrate data from your legacy systems to your new system.

Data catalogs are comprehensive inventories of an organisations data assets, helping data analysts and information consumers to quickly find, understand, and utilise relevant information. They foster collaboration, maintain data governance, and ensure compliance.

AgileData delivers beautiful dashboards for your Shopify data

Store your Google Analytics data for more than 14 months, and we throw in a few gorgeous Dashboards to boot.

Get access to New Zealand Census data in a way that is so easy you will never have to access the Stats NZ website again.

This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users

Early in 2022 Shane Gibson was lucky enought to talk to the Catalog and Cocktails podcast crew about agile in the data domain.  Watch or listen to the episode.

In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

In 2022 Shane Gibson was lucky enough to present “Analysts can model democratising data modeling” at the Knowledge Gap Conference
Watch the presentation. 

In 2021 Shane Gibson was lucky enough to present “A selection of practical agile patterns when using Data Vault” at the Knowledge Gap Conference

Join Murray Robinson and Shane Gibson as they engage in an insightful conversation with Willem-Jan Ageling on agile project management and leadership.

Understand the importance of a goal-oriented leader who supports and empowers their team, and the common anti-patterns to avoid.

TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.

Defining a Data Architecture is a key pattern when working in the data domain.

Its always tempting to boil the ocean when defining yours, don’t!

And once you have defined your data architecture, find a way to articulate and share it with simplicity.

Here is how we articulate the AgileData Data Agile-tecture.

Data observability provides comprehensive visibility into the health, quality, and reliability of your data ecosystem. It dives deeper than traditional monitoring, examining the actual data flowing through your pipelines. With tools like data lineage tracking, data quality metrics, and anomaly detection, data observability helps data magicians quickly detect and diagnose issues, ensuring accurate, reliable data-driven decisions.

As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.

We outline what each of these are, how they are different to each other and how to charge for delivering them.

Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.

A graphical overview of the components required for a Data Product

Marketing Analytics involves analysing data from various channels, such as social media, email, and websites, to assess the performance of marketing efforts.
Product Analytics focuses on understanding and improving user experience and satisfaction with digital products or services.

DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.

Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.

In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.

The key components of a successful data warehouse technology capability include data sources, data integration, data storage, metadata, data marts, data query and reporting tools, data warehouse management, and data security.

Cloud Analytics Databases provide flexible, high-performance, cost-effective, and secure solution for storing and analysing large amounts of data. These databases promote collaboration and offer various choices, such as Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse Analytics, each with its unique features and ecosystem integrations.

AgileData reduces the cost of your data team and your data platform.

In this article we provide examples of those costs savings.

Data modeling is a crucial process that involves creating shared understanding of data and its relationships. The three primary data model patterns are conceptual, logical, and physical. The conceptual data model provides a high-level overview of the data landscape, the logical data model delves deeper into data structures and relationships, and the physical data model translates the logical model into a database-specific schema. Understanding and effectively using these data models is essential for business analysts and data analysts, create efficient, well-organised data ecosystems.

Metadata-driven data pipelines are the secret behind seamless data flows, empowering data magicians to create adaptable, scalable, and evolving data management systems. Leveraging metadata, these pipelines are dynamic, flexible, and automated, allowing for easy handling of changing data sources, formats, and requirements without manual intervention.

Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.

In a compelling episode of the Agile Data Podcast, Shane Gibson invites Hans Hultgren to explore the intricacies of Data Vault modeling. Hans, with a substantial background in IT and data warehousing, brings his expertise to the table, discussing the evolution and benefits of Data Vault modeling in today’s complex data landscapes. Shane and Hans navigate through the foundations of Data Vault, including its core components like hubs, satellites, and links, and delve into more advanced concepts like Same-As Links (SALs) and Hierarchical Links (Hals). They highlight how Data Vault enables flexibility, agility, and incremental development in data modeling, ensuring scalability and adaptability to change.

Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.

This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.

Join Murray Robinson and Shane Gibson as they delve into a comprehensive discussion on agile project management with certified scrum trainer, Jem ‘Jelly’ D’jalel.

The AgileData Context feature enhances data understanding, facilitates effective decision-making, and preserves corporate knowledge by adding essential business context to data. This feature streamlines communication, improves data governance, and ultimately, maximises the value of your data, making it a powerful asset for your business.

Immerse yourself in the magical world of data with AgileData’s ‘Ask a Quick Question’ capability. Perfectly designed for data analysts and business analysts who need to swiftly extract insights from data, this capability facilitates quick data queries and rapid exploratory data analysis.

Data quality metrics are crucial indicators in a data warehouse that measure the accuracy, completeness, consistency, timeliness, and uniqueness of data. These metrics help organisations ensure their data is reliable and fit for use, thus driving effective decision-making and analytics

In an insightful episode of the AgileData Podcast, Shane Gibson hosts Ahmed Elsamadisi to delve into the evolving world of data modeling, focusing on the innovative concept of the Activity Schema. Elsamadisi, with a rich background in AI and data science, shares his journey from working on self-driving cars to spearheading data initiatives at WeWork. The discussion centers on the pivotal role of data modeling in enhancing scalability and efficiency in data systems, with Elsamadisi highlighting the limitations of traditional models like star schema and data vault in addressing complex, modern data queries.

We discuss how to handle change data in a hands-off filedrop process. We use the ingestion timestamp as a simple proxy for the effective date of each record, allowing us to version each day’s data. For files with multiple change records, we scan all columns to identify and rank potential effective date columns. We then pass this information to an automated rule, ensuring it gets applied as we load the data. This process enables us to efficiently handle change data, track data flow, and manage multiple changes in an automated way.

Customer segmentation is the magical process of dividing your customers into distinct groups based on their characteristics, preferences, and needs. By understanding these segments, you can tailor your marketing strategies, optimize resource allocation, and maximize customer lifetime value. To unleash your customer segmentation magic, define your objectives, gather and analyze relevant data, identify key criteria, create distinct segments, profile each segment, tailor your strategies, and continuously evaluate and refine. Embrace the power of customer segmentation and create personalised experiences that enchant your customers and drive business success.

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.

Discover the transformative impact of generative AI and large language models (LMS) in the world of data and analytics. This insightful podcast episode with Shane Gibson and Shaun McGirr delves into the evolution of data handling, from manual processes to advanced AI-driven automation. Uncover the vital role of AI in enhancing decision-making, business processes, and data democratization. Learn about the delicate balance between AI automation and human insight, the risks of over-reliance on AI, and the future of AI in data analytics. As the landscape of data analytics evolves rapidly, this episode is a must-listen for professionals seeking to adapt and thrive in an AI-driven future. Stay ahead of the curve in understanding how AI is reshaping the role of data professionals and transforming business strategies.

Join hosts Murray Robinson and Shane Gibson in a conversation with Jeff Patton, an advocate of agile and lean ways of working and author of the book “User Story Mapping.” In this episode, we focus on the concepts of product thinking and user story mapping.

Dive into the Agile Data Podcast with Shane Gibson and Veronika Durgin as they explore the intricacies of layered data architecture, data vault modeling, and the evolution of data management. Discover key insights on balancing data democratisation with governance, the role of ETL processes, and the impact of organisational structure on data strategy.

To white label or to not white label
Concept of Team Design
Data Consulting Patterns with Joe Reis
Willem Jan Ageling – Agile Project Management and Leadership
Data Versioning

Magical Plumbing for Effective Change Dates
Common Team Design Patterns
The Patterns of Data Vault with Hans Hultgren
Pawel Huyrn – Product Management
Custom Sync
Latest AgileData Magician Guides
ADI

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

The Challenge of Parsing Files from the Wild
An AgileData Guide to Information Product Canvas
The Patterns of Activity Schema with Ahmed Elsamadisi
Pawel Huyrn – Product Management
Natural Language Rules Output
Latest AgileData Magician Guides
Designed in NZ

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.

We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.

Read more on our exciting plumbing journey.

The HitchHikers Guide to the Information Product Canvas
Bring Back Business Analysis with Howard Podeswa
New Google Cloud Feature to Optimise BigQuery Costs
Data Lineage Patterns
G60S – Google Sheets
Bring Back Data Modeling

Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

The Hitchhikers Guide to the Information Product Canvas
Information Product Canvas – Google Slide Template
How Can Data Teams use Generative AI with Shaun McGirr
Roman Pichler – Product Management
AgileData App Colours of the Catalog
Pulp Data Fiction

Join Murray Robinson and Shane Gibson as they dive into a discussion with Jose Corella on how leadership principles from the US Air Force can find applications in civilian organisations.

Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.

The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.

Join Murray Robinson, Shane Gibson, and special guest Stephen Bungay to discuss how military principles like “Mission Command” can be applied to manage teams effectively in uncertain conditions.

Join Murray Robinson and Shane Gibson as they chat with Ivar Jacobson about Essence; a standard to develop patterns and pattern libraries that is being used by Scrum, SAFE, Lean and Kanban organisations. We discuss object oriented software engineering, use cases, UML, RUP, Essence and AI assisted method coaching. Join us to hear from one of the founders of the software engineering industry.

In this episode, we talked to Joe Justice about how Tesla and Space X have developed a new operating model based on self-organizing teams that continuously discover, deliver, and improve to achieve a thousand year goal. It’s like the most radical combination of open space agility, OKR’s, Continuous Discovery, Continuous Delivery and Continuous Improvement, you can think of. Whatever you think of Elon Musk, these companies are way ahead of most other companies and accelerating further. Tune in for a fascinating discussion about how Tesla and Space X work.

Data quality is an expectation, not an exception.

While data quality is crucial, it’s not always directly our fault when issues arise, nevertheless, it remains our problem to solve.

Data Contracts are one pattern that can help us solve this problem.

Join Shane Gibson on the Agile Data Podcast for an enlightening conversation with Scott Ambler, an IT and Agile expert. Delve into Scott’s journey from pioneering programmer to data architecture and Agile methodologies. Discover the evolution of Agile data, the importance of adapting ways of working, and the pitfalls of best practices. Learn valuable insights into continuous improvement, team dynamics, and the complexities of data quality in today’s fast-paced IT landscape. Don’t miss this episode for an in-depth exploration of Agile data and its impact on IT projects and processes.

Data visualization is like painting with data, using charts and graphs to make trends and patterns easy to understand. It’s great for presenting data objectively.

Data storytelling weaves a narrative around data, adding context, engaging emotions, and inspiring action. It’s perfect for persuading stakeholders.

In this episode we discuss invitational coaching with Michael Delemaza, the differences between coaching and mentoring and the challenges faced in transforming traditional organizations into agile ones. We also talk about the importance of clear communication and alignment at the executive level, as well as dealing with politics and resistance within organizations.

Modern data teams are transient, often staying less than 5 years, unlike past decades of long-term loyalty.

Companies should adapt by defining robust Ways of Working (WoW) that endure beyond individual tenures.

Balancing in-house teams with reliable data vendors for continuity and efficiency may also be a useful pattern as part of your WoW.

In this episode, Murray Robinson chats with Maarten Dalmijn about Sprint Goals. We discuss how sprint goals should act as a primary mission for a sprint, around which everyone’s tasks should revolve, rather than focusing on individual deliverables. We talk about challenges to implementing this approach, the importance of measuring outcomes over output, and the need for competency and trust in the team. This conversation highlights the importance open dialogue with stakeholders, and the necessity of adaptability in the face of uncertainty.

Data teams struggle to not “boil the ocean” when doing data work.

Use milestones as a pattern to help the data team to focus on what really needs to be built and manage the trade-off decisions for what doesn’t.

In this article we describe the concepts of Customer Data Platforms (CDP)  versus Data Warehouses.

In this episode, we talked with with Gabrielle Benefield, the inventor of the Mobius loop, an outcome focused product development framework.  Gabrielle shares her journey in agile product development from Silicon valley to Yahoo.  And the importance of focusing on outcomes of outputs and improving decision speed. We discuss why organisations don’t focus on outcomes, the problem with contracts and scaling an outcome based approach. And finally, we talk about how you can use an experimental approach to bring an outcome driven, focus to your organisation.

Explore the Art of Data Storytelling with Kat Greenbrook on the Agile Data Podcast. Dive into Kat’s transformative journey from aspiring vet to a data storytelling expert, and discover the power of the ABT (And, But, Therefore) narrative framework in conveying compelling data insights. Uncover common pitfalls and learn crucial differences between data visualization and storytelling. Enhance your business communication skills with practical tips and insights from ‘The Data Storyteller’s Handbook.’ Perfect for professionals in data analytics, business intelligence, and anyone keen to master the art of turning data into impactful stories.

Dive into the world of data consulting with Shane Gibson and Joe Reis on the Agile Data Podcast. Explore their journey from traditional employment to successful data consulting, covering client acquisition, business models, financial management, reputation, sales strategies, employee management, and work-life balance.

Dive deep into the world of knowledge graphs with Juan Sequeda on the Agile Data Podcast, hosted by Shane Gibson. Explore key insights from Juan’s journey in computer science, his pivotal role in semantic web development, and the transformative power of knowledge graphs in data integration. Discover how these technologies are reshaping the landscape of data management and the exciting future prospects with the advent of Large Language Models (LLMs). Tune in to understand the practical applications, challenges, and the future of knowledge graphs in enterprise data strategy.