Its 2019 – Isn’t the Data Warehouse dead?

With the advent of self-service data discovery tools and big data platforms, the bold announcements of "the data warehouse is dead" started to ring out around the world. So why, you would ask, are we building a new startup that is focussed on allowing analysts to...

Google BigQuery – Online Training resources

As part of our journey to exploring the use of the Google Cloud Platform (GCP) as the core of our infrastructure for AgileData.io we have had to start unlearning a lot of our AWS and Microsoft Azure knowledge and experience and relearn the GCP way. Google BigQuery is...

Automate The Dross

Automate repeating data tasks to free up your analysts.

Answer a Burning Question

Get a repeatable answer to your most important business questions.

Replace Legacy Data Warehouse

Replace your legacy EDW with a SaaS based modern data platform.

SaaS Startup

Utilise AgileData.io as a data solution underpinning your SaaS product, without having to develop a data platform yourself.

Customer 360 View

Get a single view of all your customers, understand how and when they engage with you.

Manage Master Data

Centralise your data into a single data repository to enable you to manage data as a true asset.

Migrate Legacy Data

Migrate data from your legacy systems to your new system.

Why we founded AgileData

My co-founder Nigel and I have been working in the data and analytics domain for over 30 years (well I have, he is slightly younger). We have both held multiple roles through these years, Nigel primarily in the engineering space and myself in the pre-sales then...

Why we chose Google Cloud as the infrastructure platform for AgileData

  Pick a few things that really matter, not thousands of “requirements”   We when first started developing the core of the AgileData backend for the MVP, we knew we would need a cloud database to store customers data. We had a lot of experience writing in...

What problem(s) does AgileData solve?

This should be an easy one for me to answer right? Understand the customers problem first We get told in startup land that you need to understand the customers problem in-depth, before you start building your product. So what problem does AgileData.io solve? It’s a...

Buy, Build or Lease

One of the (many) things we needed to decide when we started to build out the AgileData Minimal Magical Product (MMP), was which capabilities we would build vs which capabilities we would lease or buy. As part of our AgileData Way of working we have a defined a...

Micro Actions, ensuring a little bit of Magic Happens Here everytime

We are currently doing some work on creating rule patterns that enable us to automagically find duplicate Concept values and create a master view of them. For example creating a master view of Customers, or a master list of Products. We have completed a McSpikey...

Why assumptions are just that

When we first sketched out our plans for AgileData we were pretty clear what AgileData would do and what it wouldn’t do. Those assumptions didn’t last long. Combine with magic We knew we wanted to focus on what we call the middle of the data supply chain. We want to...

Reducing Manual Effort, Everywhere, Every-time

Some tasks seem really small and only take minutes, but multiply that effort by completing that task a hundred times and you have found a task that should be automated. Collect your data In AgileData we automate the collection of your data from your systems of capture...

Virtual cXo

Provide your customers with simply magical access to their data, to enable you to help them make data driven decisions.

3 types of product features

Our UX/UI journey is accelerating We are currently full steam into the development of the initial User Interface for AgileData.io. The team have done some awesome work on the UX designs for a bunch of the core screens, we have chosen our front end development stack,...

The “Killer” Feature

One feature to rule them all As product managers we are always looking for the next “killer feature” for our product. You know the one, that feature that will become the magical thing that will have customers flooding to subscribe to our product because nobody else...

Agile DataOps

TD:LR Agile DataOps is where we combine the processes and technologies from DataOps with a new agile way of working, to reduce the time taken and increase the value of the data we provide to our customers What's in a name Everything it seems. I remember years ago...

AgileData >>> Modern Data Stack

TD:LR AgileData's mission is to reduce the complexity of managing data.  A large part of modern data complexity is selecting, implementing and maintaining a raft of different technologies to provide your "Modern Data Stack".  At AgileData we have already done the hard...

Whats the hottest new data thing in 2022 — Data Mesh or Metric Store

There is a lot of vendor washing going on A lot of data vendors are vendor washing their technologies to pretend they enable "Data Mesh" as they are punting on Data Mesh being the new thing for 2022.  I think they are missing the real hot thing for 2022, which will be...

Abracadabra! Unravel the Mysteries of Data Catalog

Data catalogs are comprehensive inventories of an organisations data assets, helping data analysts and information consumers to quickly find, understand, and utilise relevant information. They foster collaboration, maintain data governance, and ensure compliance.

Data Mesh 4.0.4

TD:LR Data Mesh 4.0.4 is only available for a very short time. please ensure you scroll to the bottom of the article to understand the temporal nature of the Data Mesh 4.0.4 approach.This article was published on 1st April 2022 as an April Fools Joke. What is Data...

Social Good

Helping not-for-profit and social enterprises make data driven decisions

Data Mesh Podcast – Finding useful and repeatable patterns for data

TD:LR I talk to Scott Hirleman on the Data Mesh Radio podcast on my thoughts on Data Mesh and the need for resuable patterns in the data & analytics domain. My opinion on Data Mesh I am not a fan of the current hype around Data Mesh. I see another vendor hype...

What is Data Lineage?

TD:LR AgileData mission is to reduce the complexity of managing data. In the modern data world there are many capability categories, each with their own specialised terms, technologies and three letter acronyms. We want managing data to be simply magical, so we share...

Data Lineage, mapping your way to magic

Join Shane and Nigel as they discuss data lineage, what it is, why people want it and what it can actually be used for.

Lisa Crispin – Holistic testing

Join Murray Robinson and Shane Gibson in a conversation with Lisa Crispin on Holistic Testing. Build quality in. Continuous testing for continuous delivery.

Flying the airplane while building it – Jan Sheppard

Join Shane and Blair as they chat with Jan Sheppard about the experience of building a new agile way of working while still ensuring you are delivering data and analytical value to your stakeholders.

Bruce Taylor – Working with offshore teams

Join Murray Robinson and Shane Gibson in a conversation with Bruce Taylor about working with offshore teams in developing countries. Why do people send their work offshore? Cost or talent? Is it cheaper to send work offshore or not?

Kyle Griffin – Uncertainty, complexity and change

Join Murray Robinson and Shane Gibson in a conversation with Kyle Griffin about uncertainty, complexity and change. Our requirements are probably wrong. So show people and build small bets. Our plan is likely to be wrong. So let's do things to mitigate our plan, being...

Analyst vs Analytics Engineer – Benn Stancil

Join Shane and guest Benn Stancil as they discuss the difference between the age old analyst role and the new emerging role of analytics engineer (amongst a few other interesting things)

LeSS with Bas Vodde

Join Murray Robinson and Shane Gibson in a conversation with Bas Vodde on LeSS. LeSS is true Scrum at Scale not just at the team level.

Adopting the AgileData Way of Working – Sam Sewell

Join Shane and Blair as they chat to Sam Sewell on his experience adopting the AgileData way of working as part of building a new agile team.

Mary and Tom Poppendieck – Lean software development

Join Murray Robinson and Shane Gibson in a conversation with Mary and Tom Poppendieck about Lean and Software Development. Organizations have queues because they don’t care about the customer.

Agile and flow with Staffan Noteberg

Join Murray Robinson and Shane Gibson in a conversation with Staffan Noteberg about Agile and Flow. Agile is about adapting to unexpected change.

Shopify

AgileData delivers beautiful dashboards for your Shopify data

Google Analytics

Store your Google Analytics data for more than 14 months, and we throw in a few gorgeous Dashboards to boot.

NZ Census Data

Get access to New Zealand Census data in a way that is so easy you will never have to access the Stats NZ website again.

agile 2 with Raj Nagappan

Join Murray Robinson and Shane Gibson for a conversation with Raj Nagappan on using Agile 2 to create awesome products.

A Data Engineer an Agile Coach and a Fish walk into a bar…

This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users

Team Skills vs Roles

Join Murray Robinson and Shane Gibson in a conversation about effective agile product teams. T-shaped people. Should everyone be a developer? Are specialised roles useful?

“Serverless” Data Processing

TD:LR When we dreamed up AgileData and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and what that has to do with Apples. Let's explore this...

Using a manifest concept to run data pipelines

TD:LR … you don’t always need to use DAGs to orchestrate Previously  we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data pipelines at agiledata.io This post I’m going to...

Agile and Analytics – Shaun McGirr

Join Shane and special guest Shaun McGirr as they discuss the combining of agile ways of working with analytics teams.

Erik de Bos – Agile is a paradigm shift

Join Murray Robinson and Shane Gibson in a conversation about Agile as a paradigm shift with Erik de Bos. Most organisations are slow moving bureaucracies defined by authoritarian hierarchies, rules and contracts. Agile is a completely new way of thinking and a new vocabulary which makes new concepts and information available.

3 agile architecture things

Join Shane and guest Brian McMillian as they discuss the art of architecture in an agile data world. We discuss 3 things: 1. the 4x approach2. data vault3. everything is code

Catalog & Cocktails Podcast – agile in the data domain

Early in 2022 Shane Gibson was lucky enought to talk to the Catalog and Cocktails podcast crew about agile in the data domain.  Watch or listen to the episode.

Todd Lankford – Agile anti patterns

Join Murray Robinson and Shane Gibson in a conversation about Agile anti patterns with Todd Lankford. Patterns are common solutions to common problems.

Product leaders with David Pereira

Join Murray Robinson and Shane Gibson in a conversation with David Pereira about Product Leaders. Successful Product Leaders don’t write requirements; they define the problem, set a goal and empower the team to get the job done.

Agile Security

Join Shane and guest Laura Bell as they discuss how you can eat the security elephant in an agile way.

The magic of Agile Data teams vs Agile Software teams – Lynn Winterboer

Join Shane and special guest Lynn Winterboer as they discuss the core differences between teams that leverage an agile way of working while managing data versus creating software.

AgileBI – Raphael Branger

Join Shane and guest Raphael Branger as they discuss combining agile ways of working with the world of data and business intelligence (BI).

Agile Data Testing Techniques, a magical list of testing TLA’s

Join Shane and Nigel as they discuss the many notions you should understand to start your journey on the DataOps and automated data testing path.

AgileData Patterns

Join Shane and Nigel as they discuss what agile data patterns are and how you can apply them to speed up the delivery of data to your stakeholders.

Jurgen Appelo – Designing an agile organisation

Join Murray Robinson and Shane Gibson in a conversation with Jurgen Appelo on how to design an organisation to support self management, continuous innovation and human experience. Jurgens unFix model supports gradual change, dynamic teams, and recognises that managers play an important role

Tom Gilb – Software metrics

Join Murray Robinson and Shane Gibson in a conversation about Software Metrics with Tom Gilb. Tom Gilb inspired the authors of the Agile manifesto with his innovative incremental and scientific approach to Software Engineering. In this podcast we talk about Evo and Value Agile.

Data Products

Join Shane and guest Eric Broda as they discuss data products.

Debbie Levitt – UX design

Join Murray Robinson and Shane Gibson in a conversation about UX design with Debbie Levitt. UX Design is research based information architecture and interaction design that requires deep expertise.

Scaling data teams – Tammy Leahy

Shane Gibson chats to Tammy Leahy about how she helped scale the data teams she leads.

Ben Ford – The overlap between modern military ideas and agile

Join Murray Robinson and Shane Gibson in a conversation about the overlap between modern military ideas and Agile with Ben Ford.

Scott Ambler – Disciplined agile toolkit

Join Murray Robinson and Shane Gibson in a conversation with Scott Ambler about the Disciplined Agile Toolkit

Tony Ulwick – Outcome driven innovation

Join Murray Robinson and Shane Gibson in a conversation about Outcome-Driven Innovation with Tony Ulwick. Most products fail in the market because they don’t meet customers’ needs better than the alternatives

Three Agile Testing Methods – TDD, ATDD and BDD

In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.

Assessing agile capability

Join Murray Robinson and Shane Gibson in a conversation about assessing the agile capability of individuals, teams and partners.

ELT without persisted watermarks ? not a problem

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

The art of data story telling – Kat Greenbrook

Join Shane and Blair as they chat to Kat Greenbrook on her experience doing and training teams on Data Story telling.

Prototype to Production – Gerhard DeBeer

Join Shane and Blair as they chat with Gerhard DeBeer on moving prototypes to production in an agile way.

Simon Wardley – Wardley maps

Join Murray Robinson and Shane Gibson in a conversation with Simon Wardley about Wardley maps. Wardely maps are a discussion tool that surfaces assumptions about competitive landscapes.

The Focus Podcast – Agile Data Governance Patterns

Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.

Leading agile data at scale – Chris La Grange

Join Shane and Blair as they chat with Chris La Grange about leading agile data teams in large organisations.

Innovation iteration

Join Shane and Blair as they chat amongst themselves on their experience running innovation iterations.

Sandy Mamoli – Building great teams

Join Murray Robinson and Shane Gibson in a conversation with Sandy Mamoli about building great teams. Why do some teams perform well and others perform poorly.

The Magic of Serverless

Join Shane and Nigel as they discuss what serverless actually means, and the value of leveraging serverless components within your data platform.

Matthew Skelton – Team Topologies

Join Murray Robinson and Shane Gibson in a conversation with Matthew Skelton about Team topologies.

I can write a bit of code faster

TD:LR To get data tasks done involves a lot more than just bashing out a few lines of code to get the data into a format that you can give it to your stakeholder/customer. Unless of course it really is a one off and then bash away.  But I bet if that information...

App Engine and Socket.IO

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

Maturing product owner capability

Join Shane and Blair as they chat amongst themselves on maturing the product owner capability.

A DataOps and Data Science Journey

Join Shane as he chats to Hamish and Liam in a pub in the middle of the North Island about their journey to adopting a way of working that combines DataOps with Data Science.

Analysts can model democratising data modeling

In 2022 Shane Gibson was lucky enough to present “Analysts can model democratising data modeling” at the Knowledge Gap Conference
Watch the presentation. 

A selection of practical agile patterns when using Data Vault

In 2021 Shane Gibson was lucky enough to present “A selection of practical agile patterns when using Data Vault” at the Knowledge Gap Conference

Chris Butler – agile product management 

Join Murray Robinson and Shane Gibson in a conversation with Chris Butler about agile Product Management at Microsoft, Facebook and Google. Three core things, that product managers should do.

Information Products – What and Why

Join Shane GIbson and Tammy Leahy as they discuss what the Information Product pattern is and Why you would use it.

Agile contracting and planning

Join Murray Robinson and Shane Gibson in a no-nonsense discussion on agile and waterfall.

Agile initiation and planning

Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we talk about our experience with planning agile initiatives. How to do just enough planning just in time. We discuss the team you need, user stories, UX concepts and technical architecture, how to develop a release plan and estimate time and cost in a 2 to 4-week timebox. We also talk about the common traps, pitfalls and bad patterns we see in planning agile initiatives.

AgileData refactoring

Join Shane and Nigel as they discuss what refactoring is and why you should always plan to refactor your data, code and platform from day one.

Is scrum broken?

Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we discuss Ron Jeffries’ recent article “Can scrum be fixed?” We talk about Ron’s concerns with the Scrum Industrial complex and Developers issues with Scrum. We talk about what we like and don’t like about Scrum and how the gaps we’ve experienced can be overcome.

Evolutionary design & agile architecture

Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we talk about evolutionary design and the problem with a narrow focus, planning at different time horizons, big blocks and small blocks and the importance of having flexible roadmaps that change as we learn. We also talk about centralised teams vs experts in teams, how the role of the coach is to build the team’s capability and how a coach should be a sports coach, not a therapist.

Product Management with Scott Baldwin

Join Murray Robinson and Shane Gibson in a conversation with Scott Baldwin about product management

Analytical team topologies – Ashwin Kamath

Join Shane and guest Ashwin Kamath as they discuss his experience working with analytical teams and analytical team topologies.

Bryan Finster – continuous delivery

Join Murray Robinson and Shane Gibson in a conversation with Bryan Finster about Continuous Delivery and SAFe. Continuous delivery improves quality, productivity, value, and developers’ lives

Information Product Canvas

Join Shane and Tammy Leahy as they discuss the Information Product Canvas, what each area of the canvas holds and why you would want to collect this information.

This is the second in a series of podcast episodes that deep dives into the Information Product pattern.

Data Layers, the magic of the right data in the right place

Join Shane and Nigel as they discuss what data layers and how we apply data layers as a core part of the AgileData.io Agile-Tecture.

Agile project management and leadership with Willem-Jan Ageling

Join Murray Robinson and Shane Gibson as they engage in an insightful conversation with Willem-Jan Ageling on agile project management and leadership.

Understand the importance of a goal-oriented leader who supports and empowers their team, and the common anti-patterns to avoid.

Christian Crumlish – the intersection between UX design and product management

Join Murray Robinson and Shane Gibson in a conversation with Christian Crumlish about the intersection between UX Design and Product Management.

Adam Thomas – product survival metrics

Join Murray Robinson and Shane Gibson in a conversation with Adam Thomas about Product Survival Metrics. Empower the product team to make decisions quickly in uncertain situations by negotiating clear boundaries for the product team. Negotiate boundaries with...

Jeff Gothelf – Lean UX and Sense and Respond

Join Murray Robinson and Shane Gibson in a conversation with Jeff Gothelf about Lean UX and Sense and Respond. Lean UX explains how to do UX design and product development in a highly agile way.

How To Bring Agile Practices To Your Data Projects

TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain.  Listen to the episode or read the transcript.   Listen Read Summary Agile methodologies...

Jonathan Hall – implementing continuous delivery

Join Murray Robinson and Shane Gibson in a conversation with Jonathan Hall about implementing Continuous Delivery. CI/CD requires a level of test automation that seems impossible for most organisations. Jonathan describes a practical implementation path to CI/CD that...

Build Data Products Without A Data Team Using AgileData

TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of data engineers.  Listen to the episode or read the...

ProductOps with Anabela Cesario

Join Murray Robinson and Shane Gibson in a conversation with Anabela Cesário VP of product ops at OutSystems, a low code development platform. Anabela  describes what product ops is and how her team mentors, coaches, supports and trains 25 product managers. From...

The lean product development playbook with Dan Olsen

Join Murray Robinson and Shane Gibson in a conversation with Dan Olson, author of the lean product development playbook. A practical guide to building products that customers love. Most new products fail on most new features are rarely used. Dan describes a practical...

agile projects

In this engaging episode, Murray Robinson and Shane Gibson dive into a spirited discussion on the compatibility of Agile and Project Management. Discussion highlights include: The Agile-Project Management conundrum: Is Agile fundamentally incompatible with Project...

The language of data is not so natural

TD:LR The dream is we can just point the machine at our data, ask our question and get a useful answer. With ChatGPT we are closer than we have ever been, but we are not there yet,When Nigel and I first started development of AgileData we had a vision in our heads,...

Continuous discovery with Teresa Torres

Join Murray Robinson and Shane Gibson as they converse with special guest, Teresa Torres, a renowned product coach, speaker, and author of "Continuous Discovery Habits." In this episode: Torres elaborates on continuous discovery, a crucial method for product managers,...

Mission command with Eric Lopez

Join hosts Murray Robinson and Shane Gibson in an intriguing conversation with special guest Eric Lopez, a seasoned management consultant, CEO, and retired US Army Colonel who commanded troops in Iraq and Afghanistan. In this episode: Eric unpacks the concept of...

Observability, Tick

TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels like every other influencer in the data space is...

agile innovation

Join hosts Murray Robinson and Shane Gibson as they share their unique insights and experiences with startups, scale-ups, venture capitalists, and corporate innovation. In this episode: We delve into managerial fear of innovation and its impacts on organizations. Why...

AgileData WoW Q&A – Hamish Gray and May-Lyn Hu

Join Shane Gibson, Hamish Gray and May-Lyn Hu as they talk through some of the experiences they have had in their organisation applying agile and data together in a new way of working.Guests Hamish Gray May-Lyn Hu Shane GibsonResourcesSubscribe | Apple Podcast |...

Kanban and the Collaboration Equation with Jim Benson

Join hosts Murray Robinson and Shane Gibson in conversation with Jim Benson, renowned author of "Personal Kanban," as they delve into his new book, "The Collaboration Equation." In this episode: Jim sheds light on how Kanban can be used to identify and tackle...

Agile and Product – Justin Bauer

Join Shane Gibson as he chats with Justin Bauer on his experience combining the worlds of agile and product in a data driven company.Guests Justin BauerShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio |...

Myth: using the cloud for your data warehouse is expensive

TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents.  Is it too good to be true?  Yes AND No. You can run a cloud platform for a low low cost, but its will take engineering effort  to get there. Myth: using the...

The Life of a Awesome Product Owner – Hiria Te Rangi  

Join Shane and Blair as they chat to Hiria Te Rangi on her experience of being and coaching awesome product owners.Guests Hiria Te RangiBlair TemperoShane GibsonResources Find out more about Hiria's social enterprise focussed on sensors that tell you when your home is...

Thoughts of a new Scrum Master – Juan Dione

Join Shane and Blair as they chat to Juan Dione on his experience as a new scrum masterGuests Juan Dione Blair Tempero Shane GibsonResources  Subscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes |...

Tips for a New Scrum Master – Blair Tempero

Join Shane and Blair as they chat about tips for up skilling a new scrum masterGuests Blair TemperoShane GibsonResources  Subscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser...

Data Architecture as a Service (DAaaS)

TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.

Pattern Libraries for knowledge work with Al Shalloway

Join hosts Murray Robinson and Shane Gibson in a conversation with Al Shalloway, the founder of Amplio, as they discuss the re-emergence of pattern libraries in the realm of knowledge work. In this episode: Al illustrates how the commercialisation of lean and agile...

UX Design in empowered product teams with Donna Spencer

Join Murray Robinson and Shane Gibson as they talk with Donna Spencer from Maker X about user experience design in empowered product teams. We discuss user research, information architecture, and visual design. And the problem with the way UX is often done in...

The problem with sales led development with Rich Mironov

Join Murray Robinson and Shane Gibson as they talk with Rich Mironov about the problem with sales led development. It's easy for B2B enterprise companies to fall into a sales lead development model where the majority of development work is customisation starving the...

How to make better decisions using Ammerse with Jonathan Crossland

Join Murray Robinson and Shane Gibson as they talk with veteran developer, Jonathan Crossland about Ammerse, a value based decision making guide for products and teams. Ammerse is a mnemonic for agile, minimal, maintainable, environmental, reachable and solvable....

Agile-tecture Data Factory

Defining a Data Architecture is a key pattern when working in the data domain.

Its always tempting to boil the ocean when defining yours, don’t!

And once you have defined your data architecture, find a way to articulate and share it with simplicity.

Here is how we articulate the AgileData Data Agile-tecture.

Conceptually Modeling Concepts, Details and Events in AgileData

Join Shane and Nigel as they discuss how and why we define a conceptual model of Concepts, Details and Events in AgileData and how we map these to a physical Data Vault model.

Bring back business analysis with Howard Podeswa

Join Murray Robinson and Shane Gibson as they dive into the world of agile business analysis with guest Howard Podeswa. In this episode, we'll uncover: 🔸 The role and value of a business analyst 🔸 Why big requirements upfront may not be the best approach 🔸 The method...

The art of Business Agility – Andy Cooper

Join Shane and Blair as they chat to Andy Cooper on his experience working with senior executives to help them introduce Business Agility to their organisations.Guests Andy CooperBlair TemperoShane GibsonResources...

The agile brand has been destroyed by con men and clowns with Brett Maytom and Michael Kusters

Join Murray Robinson and Shane Gibson as they take a critical look at the current state of the agile industry with guests Michael Kusters and Brett Maytom. In this episode, we address: 🔸 The degradation of the agile brand by con artists, life coaches, and big...

Data Observability Uncovered: A Magical Lens for Data Magicians

Data observability provides comprehensive visibility into the health, quality, and reliability of your data ecosystem. It dives deeper than traditional monitoring, examining the actual data flowing through your pipelines. With tools like data lineage tracking, data quality metrics, and anomaly detection, data observability helps data magicians quickly detect and diagnose issues, ensuring accurate, reliable data-driven decisions.

5E’s

As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.

We outline what each of these are, how they are different to each other and how to charge for delivering them.

Observability – Raj Joseph

Join Shane Gibson as he chats with Raj Joseph on his experience in defining data observability patterns.Guests Raj JosephShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes |...

Fundamentals of product management with Roman Pichler

Join Murray Robinson and Shane Gibson as they delve into the fundamentals of product management with Roman Pichler, renowned author of 'Agile Product Management with Scrum'. Topics we'll explore in this conversation include: 🔸 The role and responsibilities of a...

Free Google Analytics 4 (GA4) online courses

TD:LR There is some great free course content to help you upskill in Google Analytics 4 (GA4) Here are the ones we recomend.Discover the Next Generation of Google Analytics Find out how the latest generation of Google Analytics can take your measurement strategy to...

Data Lineage Patterns – Tomas Kratky

Join Shane Gibson as he chats with Tomas Kratky on his experience in defining data lineage and DataOps patterns.Guests Tomas Kratky Shane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen...

Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians

Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.

What really happened at Spotify with Brendan Marsh

Join Murray Robinson and Shane Gibson as they converse with Brendan Marsh about his experience working at Spotify. Understand how Spotify's agile methodologies aim to achieve speed to market and learning, which their leaders believe are crucial competitive advantages...

Anatomy of a Data Product

A graphical overview of the components required for a Data Product

The Enchanting World of Data Magicians: Marketing Analytics vs. Product Analytics

Marketing Analytics involves analysing data from various channels, such as social media, email, and websites, to assess the performance of marketing efforts.
Product Analytics focuses on understanding and improving user experience and satisfaction with digital products or services.

DataOps: The Magic Wand for Data Magicians

DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.

Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians

Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.

Unveiling the Definition of Data Warehouses: Looking into Bill Inmon’s Magicians Top Hat

In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.

Data Warehouse Technology Essentials: The Magical Components Every Data Magician Needs

The key components of a successful data warehouse technology capability include data sources, data integration, data storage, metadata, data marts, data query and reporting tools, data warehouse management, and data security.

Cloud Analytics Databases: The Magical Realm for Data

Cloud Analytics Databases provide flexible, high-performance, cost-effective, and secure solution for storing and analysing large amounts of data. These databases promote collaboration and offer various choices, such as Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse Analytics, each with its unique features and ecosystem integrations.

Scrum Anti Patterns with Stefan Wolpers

Join Murray Robinson and Shane Gibson as they chat with Stefan Wolpers about scrum anti-patterns. Explore common anti-patterns, such as scrum masters assigning tasks to disempowered teams, and discover the solutions for overcoming these challenges. Key points of...

AgileData Cost Comparison

AgileData reduces the cost of your data team and your data platform.

In this article we provide examples of those costs savings.

Shane Gibson – Making Data Modeling Accessible

TD:LR Early in 2023 I was lucky enough to talk to Joe Reis on the Joe Reis Show to discuss how to make data modeling more accessible, why the world's moved past traditional data modeling and more. Listen to the episode or read the transcript. Listen Read  Joe Reis:...

The Enchanting World of Data Modeling: Conceptual, Logical, and Physical Spells Unraveled

Data modeling is a crucial process that involves creating shared understanding of data and its relationships. The three primary data model patterns are conceptual, logical, and physical. The conceptual data model provides a high-level overview of the data landscape, the logical data model delves deeper into data structures and relationships, and the physical data model translates the logical model into a database-specific schema. Understanding and effectively using these data models is essential for business analysts and data analysts, create efficient, well-organised data ecosystems.

Data Consulting Patterns with Joe Reis

Join Shane Gibson as he chats with Joe Reis on his experience in building and running a successful data and analytics consulting company.

Leadership support for becoming an agile organisation

Join Murray Robinson and Shane Gibson in a no-nonsense Agile discussion. In this episode, we talk about leadership support for agile ways of working. Why leadership support is critical for change. Why some leaders support agility and others secretly undermine it. How...

Modern Product Management with Pawel Huryn

Join Murray Robinson and Shane Gibson as they chat with Pawel Huryn about product management. What it is, and isn’t? How to motivate teams and what makes a product succeed. We discuss the difference between product management, which focuses on discovery and project management, which focuses on delivery. And how to do both at the same time in one empowered product team. We talk about how you can become a product manager and the best product management tools to use. Join us as we uncover valuable insights to help you succeed in this critical role.

Metadata-Driven Data Pipelines: The Secret Behind Data Magicians’ Greatest Tricks

Metadata-driven data pipelines are the secret behind seamless data flows, empowering data magicians to create adaptable, scalable, and evolving data management systems. Leveraging metadata, these pipelines are dynamic, flexible, and automated, allowing for easy handling of changing data sources, formats, and requirements without manual intervention.

To whitelabel or not to whitelabel

Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.

From the Wild West to Agile and Beyond with Jim Highsmith

Join Murray Robinson and Shane Gibson as they chat with Jim Highsmith, one of the authors of the Agile Manifesto. Take a journey through the history of software development, from the wild west to structured methods and the emergence of the agile movement.

The patterns of Data Vault with Hans Hultgren

In this episode of the AgileData Podcast, we are joined by Hans Hultgren to discuss the patterns of data vault. Data vault is a data modeling technique that helps organisations to manage their data more effectively. Hans begins by explaining the basic concepts of data...

Data as a First-Class Citizen: Empowering Data Magicians

Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.

New Google Cloud feature to Optimise BigQuery Costs

This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.

Can you be an agile project manager with Jem Jelly

Join Murray Robinson and Shane Gibson as they delve into a comprehensive discussion on agile project management with certified scrum trainer, Jem ‘Jelly’ D’jalel.

Amplifying Your Data’s Value with Business Context

The AgileData Context feature enhances data understanding, facilitates effective decision-making, and preserves corporate knowledge by adding essential business context to data. This feature streamlines communication, improves data governance, and ultimately, maximises the value of your data, making it a powerful asset for your business.

Fast Answers at Your Fingertips: Unveiling AgileData’s ‘Ask a Quick Question’ Feature

Immerse yourself in the magical world of data with AgileData’s ‘Ask a Quick Question’ capability. Perfectly designed for data analysts and business analysts who need to swiftly extract insights from data, this capability facilitates quick data queries and rapid exploratory data analysis.

Unveiling the Secrets of Data Quality Metrics for Data Magicians: Ensuring Data Warehouse Excellence

Data quality metrics are crucial indicators in a data warehouse that measure the accuracy, completeness, consistency, timeliness, and uniqueness of data. These metrics help organisations ensure their data is reliable and fit for use, thus driving effective decision-making and analytics

The patterns of Activity Schema with Ahmed Elsamadisi

In this episode of the AgileData Podcast, we are joined by Ahmed Elsamadisi to discuss the patterns of activity schema. The activity schema is a data modeling pattern for data modeling and transformation. Some takeaway's from the podcast:   Traditional data...

Magical plumbing for effective change dates

We discuss how to handle change data in a hands-off filedrop process. We use the ingestion timestamp as a simple proxy for the effective date of each record, allowing us to version each day’s data. For files with multiple change records, we scan all columns to identify and rank potential effective date columns. We then pass this information to an automated rule, ensuring it gets applied as we load the data. This process enables us to efficiently handle change data, track data flow, and manage multiple changes in an automated way.

The Hitchhikers guide to the Information Product Canvas

TD:LR In mid 2023 I was lucky enough to present at The Knowledge Gap on the Information Product Canvas. Watch The Information Product Canvas, is an innovative pattern designed to capture data requirements visually and repeatably, making it easier for both stakeholders...

Working with executives with Peter Lam

Join Murray Robinson and Shane Gibson as they discuss the nuances of agile transformations with management consultant expert, Peter Lam. We delve deep into: 🔹 The value and effectiveness of consulting playbooks🔹 The ongoing debate around centralised agile approaches...

The Magic of Customer Segmentation: Unlocking Personalised Experiences for Customers

Customer segmentation is the magical process of dividing your customers into distinct groups based on their characteristics, preferences, and needs. By understanding these segments, you can tailor your marketing strategies, optimize resource allocation, and maximize customer lifetime value. To unleash your customer segmentation magic, define your objectives, gather and analyze relevant data, identify key criteria, create distinct segments, profile each segment, tailor your strategies, and continuously evaluate and refine. Embrace the power of customer segmentation and create personalised experiences that enchant your customers and drive business success.

The challenge of parsing files from the wild

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.

Amazons product development process with Ryan Lysne

Join hosts Murray Robinson and Shane Gibson as they sit down with special guest Ryan Lysne, a Product Director at Amazon, responsible for the Amazon app, the content creator economy, and worldwide events. In this episode: We examine the inner workings of Amazon's...

How can data teams use Generative AI with Shaun McGirr

In this episode of the Agile Data Podcast, hosts Shane Gibson and Shaun McGirr delve deep into the transformative capabilities of large language models (LLMs), such as ChatGPT-3, and their potential to revolutionise data teams. Drawing on Shaun's diverse experiences...

Product thinking and user story mapping with Jeff Patton

Join hosts Murray Robinson and Shane Gibson in a conversation with Jeff Patton, an advocate of agile and lean ways of working and author of the book “User Story Mapping.” In this episode, we focus on the concepts of product thinking and user story mapping.

Recoding America with Jennifer Pahlka

Join hosts Murray Robinson and Shane Gibson in a conversation with Jen Pahlka, founder of Code for America and author of "Recoding America". They discuss the current challenges faced by governments in providing efficient digital services and how agile product...

Layered Data Architectures with Veronika Durgin

Shane Gibson and Veronika Durgan discuss layered data architecture, data management, and the challenges of integrating software engineering with data analytics. They advocate for the ELT (Extract, Load, Transform) approach over traditional ETL methods and emphasise the importance of understanding data provenance to increase trust. The hosts also discuss the concept of data lakes and the idea of a “data lakehouse,” merging file storage with cloud compute. The conversation concludes with the importance of defining data layers and their policies, the value of automation in data handling, and the need for clear data governance.

AgileData Newsletter #34

To white label or to not white label
Concept of Team Design
Data Consulting Patterns with Joe Reis
Willem Jan Ageling – Agile Project Management and Leadership
Data Versioning

AgileData Newsletter #35

Magical Plumbing for Effective Change Dates
Common Team Design Patterns
The Patterns of Data Vault with Hans Hultgren
Pawel Huyrn – Product Management
Custom Sync
Latest AgileData Magician Guides
ADI

Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

AgileData Newsletter #36

The Challenge of Parsing Files from the Wild
An AgileData Guide to Information Product Canvas
The Patterns of Activity Schema with Ahmed Elsamadisi
Pawel Huyrn – Product Management
Natural Language Rules Output
Latest AgileData Magician Guides
Designed in NZ

AgileData App UX Capability Maturity Model

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Upgrading Python: A Plumbing Adventure in the Google Stack

In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.

We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.

Read more on our exciting plumbing journey.

AgileData Newsletter #33

The HitchHikers Guide to the Information Product Canvas
Bring Back Business Analysis with Howard Podeswa
New Google Cloud Feature to Optimise BigQuery Costs
Data Lineage Patterns
G60S – Google Sheets
Bring Back Data Modeling

Building a vibrant community with Scott Hirleman

In the episode of the AgileData podcast, Shane Gibson chats with Scott Hirleman, the founder of the data mesh community.

They delve into the nuances of cultivating and sustaining thriving communities. 

The duo touch upon the broader patterns that can be applied to both external and internal communities within organisations, and the essence of being agile and responsive to the community’s evolving needs.

Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

AgileData Newsletter #37

The Hitchhikers Guide to the Information Product Canvas
Information Product Canvas – Google Slide Template
How Can Data Teams use Generative AI with Shaun McGirr
Roman Pichler – Product Management
AgileData App Colours of the Catalog
Pulp Data Fiction

Empowering teams to take initiative with Jose Corella

Join Murray Robinson and Shane Gibson as they dive into a discussion with Jose Corella on how leadership principles from the US Air Force can find applications in civilian organisations.

Attribution Model Patterns with Yorgos Moschovis

Shane Gibson and Yorgos Moschovis discuss the multifaceted world of marketing attribution. They explore the complexity of tracking customer behaviour across online and offline channels, the impact of privacy regulations, and various attribution models.

Demystifying the Semantic Layer

The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.

Data Asset, Data Product, Data Service?

TD:LR Should we treat data as an Asset, a Product, a Service or a hybrid combination of all three? Data Asset, Data Product, Data Service? There has been a lot of discussions on LinkedIn, lots of podcasts, lots of webinars lately on the question of whether data should...

The Art of Action with Stephen Bungay

Join Murray Robinson, Shane Gibson, and special guest Stephen Bungay to discuss how military principles like “Mission Command” can be applied to manage teams effectively in uncertain conditions.

Are you delivering drills, holes or outcomes?

TD:LR Whether you're a Data Entrepreneur or an organisation looking for actionable insights, its the business outcome these insights help you achieve that is the most important thing. Yes you need a data platform and data to achieve these insights, but they are just...

Essence with Ivar Jacobson

Join Murray Robinson and Shane Gibson as they chat with Ivar Jacobson about Essence; a standard to develop patterns and pattern libraries that is being used by Scrum, SAFE, Lean and Kanban organisations. We discuss object oriented software engineering, use cases, UML, RUP, Essence and AI assisted method coaching. Join us to hear from one of the founders of the software engineering industry.

Tesla and SpaceX with Joe Justice

In this episode, we talked to Joe Justice about how Tesla and Space X have developed a new operating model based on self-organizing teams that continuously discover, deliver, and improve to achieve a thousand year goal. It’s like the most radical combination of open space agility, OKR’s, Continuous Discovery, Continuous Delivery and Continuous Improvement, you can think of. Whatever you think of Elon Musk, these companies are way ahead of most other companies and accelerating further. Tune in for a fascinating discussion about how Tesla and Space X work.