Blogs
Because sharing is caring
Most of your data is rotten and it’s not your fault, but it is your problem
Data quality is an expectation, not an exception.
While data quality is crucial, it’s not always directly our fault when issues arise, nevertheless, it remains our problem to solve.
Data Contracts are one pattern that can help us solve this problem.
Are you delivering drills, holes or outcomes?
TD:LR Whether you're a Data Entrepreneur or an organisation looking for actionable insights, its the business outcome these insights help you achieve that is the most important thing. Yes you need a data platform and data to achieve these insights, but they are just...
Data Asset, Data Product, Data Service?
TD:LR Should we treat data as an Asset, a Product, a Service or a hybrid combination of all three? Data Asset, Data Product, Data Service? There has been a lot of discussions on LinkedIn, lots of podcasts, lots of webinars lately on the question of whether data should...
Demystifying the Semantic Layer
The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.
Attribution Model Patterns with Yorgos Moschovis
Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.
Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
Building a vibrant community with Scott Hirleman
Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.
Upgrading Python: A Plumbing Adventure in the Google Stack
In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.
We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.
Read more on our exciting plumbing journey.
AgileData App UX Capability Maturity Model
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
AgileData App
Explore AgileData features, updates, and tips
Network
Learn about consulting practises and good patterns for data focused consultancies
DataOps
Learn from our DataOps expertise, covering essential concepts, patterns, and tools
Data and Analytics
Unlock the power of data and analytics with expert guidance
Google Cloud
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Journey
Explore real-life stories of our challenges, and lessons learned
Product Management
Enrich your product management skills with practical patterns
What Is
Describing data and analytics concepts, terms, and technologies to enable better understanding
Resources
Valuable resources to support your growth in the agile, and data and analytics domains
AgileData Podcast
Discussing combining agile, product and data patterns.
No Nonsense Agile Podcast
Discussing agile and product ways of working.
App Videos
Explore videos to better understand the AgileData App's features and capabilities.
Anatomy of a Data Product
A graphical overview of the components required for a Data Product
Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians
Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.
Data Lineage Patterns with Tomas Kratky
In this episode of the AgileData Podcast, Shane Gibson has an insightful discussion with Tomas Kratky on the evolution and importance of data lineage, especially in large enterprises. Tomas Kratky, a traditional software engineer turned data enthusiast, shared his journey to founding Manta, a company focused on data lineage. The conversation highlighted the significance of data lineage, not just as an end in itself, but as a powerful tool for unlocking potential in large enterprises, enhancing visibility, and fostering agility.
Free Google Analytics 4 (GA4) online courses
TD:LR There is some great free course content to help you upskill in Google Analytics 4 (GA4) Here are the ones we recomend.Discover the Next Generation of Google Analytics Find out how the latest generation of Google...
Observability – Raj Joseph
Join Shane Gibson as he chats with Raj Joseph on his experience in defining data observability patterns.Guests Raj JosephShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google Podcast | Amazon Audible |...
5E’s
As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.
We outline what each of these are, how they are different to each other and how to charge for delivering them.
Conceptually Modeling Concepts, Details and Events in AgileData
Join Shane and Nigel as they discuss how and why we define a conceptual model of Concepts, Details and Events in AgileData and how we map these to a physical Data Vault model.
Agile-tecture Information Factory
Defining a Data Architecture is a key pattern when working in the data domain.
Its always tempting to boil the ocean when defining yours, don’t!
And once you have defined your data architecture, find a way to articulate and share it with simplicity.
Here is how we articulate the AgileData Data Agile-tecture.
Data Architecture as a Service (DAaaS)
TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.
Myth: using the cloud for your data warehouse is expensive
TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents. Is it too good to be true? Yes AND No. You can run a cloud platform for a low low cost, but its will take...
Agile and Product – Justin Bauer
Join Shane Gibson as he chats with Justin Bauer on his experience combining the worlds of agile and product in a data driven company.Guests Justin BauerShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google...
AgileData WoW Q&A – Hamish Gray and May-Lyn Hu
Join Shane Gibson, Hamish Gray and May-Lyn Hu as they talk through some of the experiences they have had in their organisation applying agile and data together in a new way of working.Guests Hamish Gray May-Lyn Hu...
Observability, Tick
TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels...
DataOps: The Magic Wand for Data Magicians
DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.
The language of data is not so natural
TD:LR The dream is we can just point the machine at our data, ask our question and get a useful answer. With ChatGPT we are closer than we have ever been, but we are not there yet,When Nigel and I first started...
Build Data Products Without A Data Team Using AgileData
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of...
How To Bring Agile Practices To Your Data Projects
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain. Listen to the episode or read the transcript....
Information Product Canvas
Join Shane and Tammy Leahy as they discuss the Information Product Canvas, what each area of the canvas holds and why you would want to collect this information.
This is the second in a series of podcast episodes that deep dives into the Information Product pattern.
Analytical team topologies – Ashwin Kamath
Join Shane and guest Ashwin Kamath as they discuss his experience working with analytical teams and analytical team topologies.
Information Products – What and Why
Join Shane GIbson and Tammy Leahy as they discuss what the Information Product pattern is and Why you would use it.
App Engine and Socket.IO
We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.