Automate repeating data tasks to free up your analysts.

Get a repeatable answer to your most important business questions.

Replace your legacy EDW with a SaaS based modern data platform.

Utilise AgileData.io as a data solution underpinning your SaaS product, without having to develop a data platform yourself.

Get a single view of all your customers, understand how and when they engage with you.

Centralise your data into a single data repository to enable you to manage data as a true asset.

Migrate data from your legacy systems to your new system.

Data catalogs are comprehensive inventories of an organisations data assets, helping data analysts and information consumers to quickly find, understand, and utilise relevant information. They foster collaboration, maintain data governance, and ensure compliance.

AgileData delivers beautiful dashboards for your Shopify data

Store your Google Analytics data for more than 14 months, and we throw in a few gorgeous Dashboards to boot.

Get access to New Zealand Census data in a way that is so easy you will never have to access the Stats NZ website again.

This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users

Early in 2022 Shane Gibson was lucky enought to talk to the Catalog and Cocktails podcast crew about agile in the data domain.  Watch or listen to the episode.

In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

In 2022 Shane Gibson was lucky enough to present “Analysts can model democratising data modeling” at the Knowledge Gap Conference
Watch the presentation. 

In 2021 Shane Gibson was lucky enough to present “A selection of practical agile patterns when using Data Vault” at the Knowledge Gap Conference

Join Murray Robinson and Shane Gibson as they engage in an insightful conversation with Willem-Jan Ageling on agile project management and leadership.

Understand the importance of a goal-oriented leader who supports and empowers their team, and the common anti-patterns to avoid.

TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.

Defining a Data Architecture is a key pattern when working in the data domain.

Its always tempting to boil the ocean when defining yours, don’t!

And once you have defined your data architecture, find a way to articulate and share it with simplicity.

Here is how we articulate the AgileData Data Agile-tecture.

Data observability provides comprehensive visibility into the health, quality, and reliability of your data ecosystem. It dives deeper than traditional monitoring, examining the actual data flowing through your pipelines. With tools like data lineage tracking, data quality metrics, and anomaly detection, data observability helps data magicians quickly detect and diagnose issues, ensuring accurate, reliable data-driven decisions.

As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.

We outline what each of these are, how they are different to each other and how to charge for delivering them.

Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.

A graphical overview of the components required for a Data Product

Marketing Analytics involves analysing data from various channels, such as social media, email, and websites, to assess the performance of marketing efforts.
Product Analytics focuses on understanding and improving user experience and satisfaction with digital products or services.

DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.

Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.

In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.

The key components of a successful data warehouse technology capability include data sources, data integration, data storage, metadata, data marts, data query and reporting tools, data warehouse management, and data security.

Cloud Analytics Databases provide flexible, high-performance, cost-effective, and secure solution for storing and analysing large amounts of data. These databases promote collaboration and offer various choices, such as Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse Analytics, each with its unique features and ecosystem integrations.

AgileData reduces the cost of your data team and your data platform.

In this article we provide examples of those costs savings.

Data modeling is a crucial process that involves creating shared understanding of data and its relationships. The three primary data model patterns are conceptual, logical, and physical. The conceptual data model provides a high-level overview of the data landscape, the logical data model delves deeper into data structures and relationships, and the physical data model translates the logical model into a database-specific schema. Understanding and effectively using these data models is essential for business analysts and data analysts, create efficient, well-organised data ecosystems.

Join Shane Gibson as he chats with Joe Reis on his experience in building and running a successful data and analytics consulting company.

Metadata-driven data pipelines are the secret behind seamless data flows, empowering data magicians to create adaptable, scalable, and evolving data management systems. Leveraging metadata, these pipelines are dynamic, flexible, and automated, allowing for easy handling of changing data sources, formats, and requirements without manual intervention.

Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.

Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.

This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.

Join Murray Robinson and Shane Gibson as they delve into a comprehensive discussion on agile project management with certified scrum trainer, Jem ‘Jelly’ D’jalel.

The AgileData Context feature enhances data understanding, facilitates effective decision-making, and preserves corporate knowledge by adding essential business context to data. This feature streamlines communication, improves data governance, and ultimately, maximises the value of your data, making it a powerful asset for your business.

Immerse yourself in the magical world of data with AgileData’s ‘Ask a Quick Question’ capability. Perfectly designed for data analysts and business analysts who need to swiftly extract insights from data, this capability facilitates quick data queries and rapid exploratory data analysis.

Data quality metrics are crucial indicators in a data warehouse that measure the accuracy, completeness, consistency, timeliness, and uniqueness of data. These metrics help organisations ensure their data is reliable and fit for use, thus driving effective decision-making and analytics

We discuss how to handle change data in a hands-off filedrop process. We use the ingestion timestamp as a simple proxy for the effective date of each record, allowing us to version each day’s data. For files with multiple change records, we scan all columns to identify and rank potential effective date columns. We then pass this information to an automated rule, ensuring it gets applied as we load the data. This process enables us to efficiently handle change data, track data flow, and manage multiple changes in an automated way.

Customer segmentation is the magical process of dividing your customers into distinct groups based on their characteristics, preferences, and needs. By understanding these segments, you can tailor your marketing strategies, optimize resource allocation, and maximize customer lifetime value. To unleash your customer segmentation magic, define your objectives, gather and analyze relevant data, identify key criteria, create distinct segments, profile each segment, tailor your strategies, and continuously evaluate and refine. Embrace the power of customer segmentation and create personalised experiences that enchant your customers and drive business success.

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.

Join hosts Murray Robinson and Shane Gibson in a conversation with Jeff Patton, an advocate of agile and lean ways of working and author of the book “User Story Mapping.” In this episode, we focus on the concepts of product thinking and user story mapping.

Shane Gibson and Veronika Durgan discuss layered data architecture, data management, and the challenges of integrating software engineering with data analytics. They advocate for the ELT (Extract, Load, Transform) approach over traditional ETL methods and emphasise the importance of understanding data provenance to increase trust. The hosts also discuss the concept of data lakes and the idea of a “data lakehouse,” merging file storage with cloud compute. The conversation concludes with the importance of defining data layers and their policies, the value of automation in data handling, and the need for clear data governance.

To white label or to not white label
Concept of Team Design
Data Consulting Patterns with Joe Reis
Willem Jan Ageling – Agile Project Management and Leadership
Data Versioning

Magical Plumbing for Effective Change Dates
Common Team Design Patterns
The Patterns of Data Vault with Hans Hultgren
Pawel Huyrn – Product Management
Custom Sync
Latest AgileData Magician Guides
ADI

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

The Challenge of Parsing Files from the Wild
An AgileData Guide to Information Product Canvas
The Patterns of Activity Schema with Ahmed Elsamadisi
Pawel Huyrn – Product Management
Natural Language Rules Output
Latest AgileData Magician Guides
Designed in NZ

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.

We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.

Read more on our exciting plumbing journey.

The HitchHikers Guide to the Information Product Canvas
Bring Back Business Analysis with Howard Podeswa
New Google Cloud Feature to Optimise BigQuery Costs
Data Lineage Patterns
G60S – Google Sheets
Bring Back Data Modeling

In the episode of the AgileData podcast, Shane Gibson chats with Scott Hirleman, the founder of the data mesh community.

They delve into the nuances of cultivating and sustaining thriving communities. 

The duo touch upon the broader patterns that can be applied to both external and internal communities within organisations, and the essence of being agile and responsive to the community’s evolving needs.

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

The Hitchhikers Guide to the Information Product Canvas
Information Product Canvas – Google Slide Template
How Can Data Teams use Generative AI with Shaun McGirr
Roman Pichler – Product Management
AgileData App Colours of the Catalog
Pulp Data Fiction

Join Murray Robinson and Shane Gibson as they dive into a discussion with Jose Corella on how leadership principles from the US Air Force can find applications in civilian organisations.

Shane Gibson and Yorgos Moschovis discuss the multifaceted world of marketing attribution. They explore the complexity of tracking customer behaviour across online and offline channels, the impact of privacy regulations, and various attribution models.

The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.

Join Murray Robinson, Shane Gibson, and special guest Stephen Bungay to discuss how military principles like “Mission Command” can be applied to manage teams effectively in uncertain conditions.

Join Murray Robinson and Shane Gibson as they chat with Ivar Jacobson about Essence; a standard to develop patterns and pattern libraries that is being used by Scrum, SAFE, Lean and Kanban organisations. We discuss object oriented software engineering, use cases, UML, RUP, Essence and AI assisted method coaching. Join us to hear from one of the founders of the software engineering industry.

In this episode, we talked to Joe Justice about how Tesla and Space X have developed a new operating model based on self-organizing teams that continuously discover, deliver, and improve to achieve a thousand year goal. It’s like the most radical combination of open space agility, OKR’s, Continuous Discovery, Continuous Delivery and Continuous Improvement, you can think of. Whatever you think of Elon Musk, these companies are way ahead of most other companies and accelerating further. Tune in for a fascinating discussion about how Tesla and Space X work.