Its 2019 – Isn’t the Data Warehouse dead?
Google BigQuery – Online Training resources
Automate The Dross
Automate repeating data tasks to free up your analysts.
Answer a Burning Question
Get a repeatable answer to your most important business questions.
Replace Legacy Data Warehouse
Replace your legacy EDW with a SaaS based modern data platform.
SaaS Startup
Utilise AgileData.io as a data solution underpinning your SaaS product, without having to develop a data platform yourself.
Customer 360 View
Get a single view of all your customers, understand how and when they engage with you.
Manage Master Data
Centralise your data into a single data repository to enable you to manage data as a true asset.
Migrate Legacy Data
Migrate data from your legacy systems to your new system.
Why we founded AgileData
Why we chose Google Cloud as the infrastructure platform for AgileData
What problem(s) does AgileData solve?
Buy, Build or Lease
Micro Actions, ensuring a little bit of Magic Happens Here everytime
Why assumptions are just that
Reducing Manual Effort, Everywhere, Every-time
Virtual cXo
Provide your customers with simply magical access to their data, to enable you to help them make data driven decisions.
3 types of product features
The “Killer” Feature
Agile DataOps
AgileData >>> Modern Data Stack
Whats the hottest new data thing in 2022 — Data Mesh or Metric Store
Abracadabra! Unravel the Mysteries of Data Catalog
Data catalogs are comprehensive inventories of an organisations data assets, helping data analysts and information consumers to quickly find, understand, and utilise relevant information. They foster collaboration, maintain data governance, and ensure compliance.
Data Mesh 4.0.4
Social Good
Helping not-for-profit and social enterprises make data driven decisions
Data Mesh Podcast – Finding useful and repeatable patterns for data
What is Data Lineage?
Data Lineage, mapping your way to magic
Join Shane and Nigel as they discuss data lineage, what it is, why people want it and what it can actually be used for.
Lisa Crispin – Holistic testing
Join Murray Robinson and Shane Gibson in a conversation with Lisa Crispin on Holistic Testing. Build quality in. Continuous testing for continuous delivery.
Flying the airplane while building it – Jan Sheppard
Join Shane and Blair as they chat with Jan Sheppard about the experience of building a new agile way of working while still ensuring you are delivering data and analytical value to your stakeholders.
Bruce Taylor – Working with offshore teams
Join Murray Robinson and Shane Gibson in a conversation with Bruce Taylor about working with offshore teams in developing countries. Why do people send their work offshore? Cost or talent? Is it cheaper to send work offshore or not?
Kyle Griffin – Uncertainty, complexity and change
Analyst vs Analytics Engineer – Benn Stancil
Join Shane and guest Benn Stancil as they discuss the difference between the age old analyst role and the new emerging role of analytics engineer (amongst a few other interesting things)
LeSS with Bas Vodde
Join Murray Robinson and Shane Gibson in a conversation with Bas Vodde on LeSS. LeSS is true Scrum at Scale not just at the team level.
Adopting the AgileData Way of Working – Sam Sewell
Join Shane and Blair as they chat to Sam Sewell on his experience adopting the AgileData way of working as part of building a new agile team.
Mary and Tom Poppendieck – Lean software development
Join Murray Robinson and Shane Gibson in a conversation with Mary and Tom Poppendieck about Lean and Software Development. Organizations have queues because they don’t care about the customer.
Agile and flow with Staffan Noteberg
Join Murray Robinson and Shane Gibson in a conversation with Staffan Noteberg about Agile and Flow. Agile is about adapting to unexpected change.
Shopify
AgileData delivers beautiful dashboards for your Shopify data
Google Analytics
Store your Google Analytics data for more than 14 months, and we throw in a few gorgeous Dashboards to boot.
NZ Census Data
Get access to New Zealand Census data in a way that is so easy you will never have to access the Stats NZ website again.
agile 2 with Raj Nagappan
Join Murray Robinson and Shane Gibson for a conversation with Raj Nagappan on using Agile 2 to create awesome products.
A Data Engineer an Agile Coach and a Fish walk into a bar…
This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users
Team Skills vs Roles
Join Murray Robinson and Shane Gibson in a conversation about effective agile product teams. T-shaped people. Should everyone be a developer? Are specialised roles useful?
“Serverless” Data Processing
Using a manifest concept to run data pipelines
Agile and Analytics – Shaun McGirr
Join Shane and special guest Shaun McGirr as they discuss the combining of agile ways of working with analytics teams.
Erik de Bos – Agile is a paradigm shift
Join Murray Robinson and Shane Gibson in a conversation about Agile as a paradigm shift with Erik de Bos. Most organisations are slow moving bureaucracies defined by authoritarian hierarchies, rules and contracts. Agile is a completely new way of thinking and a new vocabulary which makes new concepts and information available.
3 agile architecture things
Join Shane and guest Brian McMillian as they discuss the art of architecture in an agile data world. We discuss 3 things: 1. the 4x approach2. data vault3. everything is code
Catalog & Cocktails Podcast – agile in the data domain
Early in 2022 Shane Gibson was lucky enought to talk to the Catalog and Cocktails podcast crew about agile in the data domain. Watch or listen to the episode.
Todd Lankford – Agile anti patterns
Join Murray Robinson and Shane Gibson in a conversation about Agile anti patterns with Todd Lankford. Patterns are common solutions to common problems.
Product leaders with David Pereira
Join Murray Robinson and Shane Gibson in a conversation with David Pereira about Product Leaders. Successful Product Leaders don’t write requirements; they define the problem, set a goal and empower the team to get the job done.
Agile Security
Join Shane and guest Laura Bell as they discuss how you can eat the security elephant in an agile way.
The magic of Agile Data teams vs Agile Software teams – Lynn Winterboer
Join Shane and special guest Lynn Winterboer as they discuss the core differences between teams that leverage an agile way of working while managing data versus creating software.
AgileBI – Raphael Branger
Join Shane and guest Raphael Branger as they discuss combining agile ways of working with the world of data and business intelligence (BI).
Agile Data Testing Techniques, a magical list of testing TLA’s
Join Shane and Nigel as they discuss the many notions you should understand to start your journey on the DataOps and automated data testing path.
AgileData Patterns
Join Shane and Nigel as they discuss what agile data patterns are and how you can apply them to speed up the delivery of data to your stakeholders.
Jurgen Appelo – Designing an agile organisation
Join Murray Robinson and Shane Gibson in a conversation with Jurgen Appelo on how to design an organisation to support self management, continuous innovation and human experience. Jurgens unFix model supports gradual change, dynamic teams, and recognises that managers play an important role
Tom Gilb – Software metrics
Join Murray Robinson and Shane Gibson in a conversation about Software Metrics with Tom Gilb. Tom Gilb inspired the authors of the Agile manifesto with his innovative incremental and scientific approach to Software Engineering. In this podcast we talk about Evo and Value Agile.
Data Products
Join Shane and guest Eric Broda as they discuss data products.
Debbie Levitt – UX design
Join Murray Robinson and Shane Gibson in a conversation about UX design with Debbie Levitt. UX Design is research based information architecture and interaction design that requires deep expertise.
Scaling data teams – Tammy Leahy
Shane Gibson chats to Tammy Leahy about how she helped scale the data teams she leads.
Ben Ford – The overlap between modern military ideas and agile
Join Murray Robinson and Shane Gibson in a conversation about the overlap between modern military ideas and Agile with Ben Ford.
Scott Ambler – Disciplined agile toolkit
Join Murray Robinson and Shane Gibson in a conversation with Scott Ambler about the Disciplined Agile Toolkit
Tony Ulwick – Outcome driven innovation
Join Murray Robinson and Shane Gibson in a conversation about Outcome-Driven Innovation with Tony Ulwick. Most products fail in the market because they don’t meet customers’ needs better than the alternatives
Three Agile Testing Methods – TDD, ATDD and BDD
In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.
Assessing agile capability
Join Murray Robinson and Shane Gibson in a conversation about assessing the agile capability of individuals, teams and partners.
ELT without persisted watermarks ? not a problem
We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured
The art of data story telling – Kat Greenbrook
Join Shane and Blair as they chat to Kat Greenbrook on her experience doing and training teams on Data Story telling.
Prototype to Production – Gerhard DeBeer
Join Shane and Blair as they chat with Gerhard DeBeer on moving prototypes to production in an agile way.
Simon Wardley – Wardley maps
Join Murray Robinson and Shane Gibson in a conversation with Simon Wardley about Wardley maps. Wardely maps are a discussion tool that surfaces assumptions about competitive landscapes.
The Focus Podcast – Agile Data Governance Patterns
Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.
Leading agile data at scale – Chris La Grange
Join Shane and Blair as they chat with Chris La Grange about leading agile data teams in large organisations.
Innovation iteration
Join Shane and Blair as they chat amongst themselves on their experience running innovation iterations.
Sandy Mamoli – Building great teams
Join Murray Robinson and Shane Gibson in a conversation with Sandy Mamoli about building great teams. Why do some teams perform well and others perform poorly.
The Magic of Serverless
Join Shane and Nigel as they discuss what serverless actually means, and the value of leveraging serverless components within your data platform.
Matthew Skelton – Team Topologies
Join Murray Robinson and Shane Gibson in a conversation with Matthew Skelton about Team topologies.
I can write a bit of code faster
App Engine and Socket.IO
We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.
Maturing product owner capability
Join Shane and Blair as they chat amongst themselves on maturing the product owner capability.
A DataOps and Data Science Journey
Join Shane as he chats to Hamish and Liam in a pub in the middle of the North Island about their journey to adopting a way of working that combines DataOps with Data Science.
Analysts can model democratising data modeling
In 2022 Shane Gibson was lucky enough to present “Analysts can model democratising data modeling” at the Knowledge Gap Conference
Watch the presentation.
A selection of practical agile patterns when using Data Vault
In 2021 Shane Gibson was lucky enough to present “A selection of practical agile patterns when using Data Vault” at the Knowledge Gap Conference
Chris Butler – agile product management
Join Murray Robinson and Shane Gibson in a conversation with Chris Butler about agile Product Management at Microsoft, Facebook and Google. Three core things, that product managers should do.
Information Products – What and Why
Join Shane GIbson and Tammy Leahy as they discuss what the Information Product pattern is and Why you would use it.
Agile contracting and planning
Join Murray Robinson and Shane Gibson in a no-nonsense discussion on agile and waterfall.
Agile initiation and planning
Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we talk about our experience with planning agile initiatives. How to do just enough planning just in time. We discuss the team you need, user stories, UX concepts and technical architecture, how to develop a release plan and estimate time and cost in a 2 to 4-week timebox. We also talk about the common traps, pitfalls and bad patterns we see in planning agile initiatives.
AgileData refactoring
Join Shane and Nigel as they discuss what refactoring is and why you should always plan to refactor your data, code and platform from day one.
Is scrum broken?
Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we discuss Ron Jeffries’ recent article “Can scrum be fixed?” We talk about Ron’s concerns with the Scrum Industrial complex and Developers issues with Scrum. We talk about what we like and don’t like about Scrum and how the gaps we’ve experienced can be overcome.
Evolutionary design & agile architecture
Join Murray Robinson and Shane Gibson in a no-nonsense agile discussion. In this podcast, we talk about evolutionary design and the problem with a narrow focus, planning at different time horizons, big blocks and small blocks and the importance of having flexible roadmaps that change as we learn. We also talk about centralised teams vs experts in teams, how the role of the coach is to build the team’s capability and how a coach should be a sports coach, not a therapist.
Product Management with Scott Baldwin
Join Murray Robinson and Shane Gibson in a conversation with Scott Baldwin about product management
Analytical team topologies – Ashwin Kamath
Join Shane and guest Ashwin Kamath as they discuss his experience working with analytical teams and analytical team topologies.
Bryan Finster – continuous delivery
Join Murray Robinson and Shane Gibson in a conversation with Bryan Finster about Continuous Delivery and SAFe. Continuous delivery improves quality, productivity, value, and developers’ lives
Information Product Canvas
Join Shane and Tammy Leahy as they discuss the Information Product Canvas, what each area of the canvas holds and why you would want to collect this information.
This is the second in a series of podcast episodes that deep dives into the Information Product pattern.
Data Layers, the magic of the right data in the right place
Join Shane and Nigel as they discuss what data layers and how we apply data layers as a core part of the AgileData.io Agile-Tecture.
Agile project management and leadership with Willem-Jan Ageling
Join Murray Robinson and Shane Gibson as they engage in an insightful conversation with Willem-Jan Ageling on agile project management and leadership.
Understand the importance of a goal-oriented leader who supports and empowers their team, and the common anti-patterns to avoid.
Christian Crumlish – the intersection between UX design and product management
Join Murray Robinson and Shane Gibson in a conversation with Christian Crumlish about the intersection between UX Design and Product Management.
Adam Thomas – product survival metrics
Jeff Gothelf – Lean UX and Sense and Respond
Join Murray Robinson and Shane Gibson in a conversation with Jeff Gothelf about Lean UX and Sense and Respond. Lean UX explains how to do UX design and product development in a highly agile way.
How To Bring Agile Practices To Your Data Projects
Jonathan Hall – implementing continuous delivery
Build Data Products Without A Data Team Using AgileData
ProductOps with Anabela Cesario
The lean product development playbook with Dan Olsen
agile projects
The language of data is not so natural
Continuous discovery with Teresa Torres
Mission command with Eric Lopez
Observability, Tick
agile innovation
AgileData WoW Q&A – Hamish Gray and May-Lyn Hu
Kanban and the Collaboration Equation with Jim Benson
Agile and Product – Justin Bauer
Myth: using the cloud for your data warehouse is expensive
The Life of a Awesome Product Owner – Hiria Te Rangi
Thoughts of a new Scrum Master – Juan Dione
Tips for a New Scrum Master – Blair Tempero
Data Architecture as a Service (DAaaS)
TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.
Pattern Libraries for knowledge work with Al Shalloway
UX Design in empowered product teams with Donna Spencer
The problem with sales led development with Rich Mironov
How to make better decisions using Ammerse with Jonathan Crossland
Agile-tecture Data Factory
Defining a Data Architecture is a key pattern when working in the data domain.
Its always tempting to boil the ocean when defining yours, don’t!
And once you have defined your data architecture, find a way to articulate and share it with simplicity.
Here is how we articulate the AgileData Data Agile-tecture.
Conceptually Modeling Concepts, Details and Events in AgileData
Join Shane and Nigel as they discuss how and why we define a conceptual model of Concepts, Details and Events in AgileData and how we map these to a physical Data Vault model.
Bring back business analysis with Howard Podeswa
The art of Business Agility – Andy Cooper
The agile brand has been destroyed by con men and clowns with Brett Maytom and Michael Kusters
Data Observability Uncovered: A Magical Lens for Data Magicians
Data observability provides comprehensive visibility into the health, quality, and reliability of your data ecosystem. It dives deeper than traditional monitoring, examining the actual data flowing through your pipelines. With tools like data lineage tracking, data quality metrics, and anomaly detection, data observability helps data magicians quickly detect and diagnose issues, ensuring accurate, reliable data-driven decisions.
5E’s
As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.
We outline what each of these are, how they are different to each other and how to charge for delivering them.
Observability – Raj Joseph
Fundamentals of product management with Roman Pichler
Free Google Analytics 4 (GA4) online courses
Data Lineage Patterns – Tomas Kratky
Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians
Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.
What really happened at Spotify with Brendan Marsh
Anatomy of a Data Product
A graphical overview of the components required for a Data Product
The Enchanting World of Data Magicians: Marketing Analytics vs. Product Analytics
Marketing Analytics involves analysing data from various channels, such as social media, email, and websites, to assess the performance of marketing efforts.
Product Analytics focuses on understanding and improving user experience and satisfaction with digital products or services.
DataOps: The Magic Wand for Data Magicians
DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.
Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians
Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.
Unveiling the Definition of Data Warehouses: Looking into Bill Inmon’s Magicians Top Hat
In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.
Data Warehouse Technology Essentials: The Magical Components Every Data Magician Needs
The key components of a successful data warehouse technology capability include data sources, data integration, data storage, metadata, data marts, data query and reporting tools, data warehouse management, and data security.
Cloud Analytics Databases: The Magical Realm for Data
Cloud Analytics Databases provide flexible, high-performance, cost-effective, and secure solution for storing and analysing large amounts of data. These databases promote collaboration and offer various choices, such as Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse Analytics, each with its unique features and ecosystem integrations.
Scrum Anti Patterns with Stefan Wolpers
AgileData Cost Comparison
AgileData reduces the cost of your data team and your data platform.
In this article we provide examples of those costs savings.
Shane Gibson – Making Data Modeling Accessible
The Enchanting World of Data Modeling: Conceptual, Logical, and Physical Spells Unraveled
Data modeling is a crucial process that involves creating shared understanding of data and its relationships. The three primary data model patterns are conceptual, logical, and physical. The conceptual data model provides a high-level overview of the data landscape, the logical data model delves deeper into data structures and relationships, and the physical data model translates the logical model into a database-specific schema. Understanding and effectively using these data models is essential for business analysts and data analysts, create efficient, well-organised data ecosystems.
Data Consulting Patterns with Joe Reis
Join Shane Gibson as he chats with Joe Reis on his experience in building and running a successful data and analytics consulting company.
Leadership support for becoming an agile organisation
Modern Product Management with Pawel Huryn
Join Murray Robinson and Shane Gibson as they chat with Pawel Huryn about product management. What it is, and isn’t? How to motivate teams and what makes a product succeed. We discuss the difference between product management, which focuses on discovery and project management, which focuses on delivery. And how to do both at the same time in one empowered product team. We talk about how you can become a product manager and the best product management tools to use. Join us as we uncover valuable insights to help you succeed in this critical role.
Metadata-Driven Data Pipelines: The Secret Behind Data Magicians’ Greatest Tricks
Metadata-driven data pipelines are the secret behind seamless data flows, empowering data magicians to create adaptable, scalable, and evolving data management systems. Leveraging metadata, these pipelines are dynamic, flexible, and automated, allowing for easy handling of changing data sources, formats, and requirements without manual intervention.
To whitelabel or not to whitelabel
Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.
From the Wild West to Agile and Beyond with Jim Highsmith
Join Murray Robinson and Shane Gibson as they chat with Jim Highsmith, one of the authors of the Agile Manifesto. Take a journey through the history of software development, from the wild west to structured methods and the emergence of the agile movement.
The patterns of Data Vault with Hans Hultgren
Data as a First-Class Citizen: Empowering Data Magicians
Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.
New Google Cloud feature to Optimise BigQuery Costs
This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.
Can you be an agile project manager with Jem Jelly
Join Murray Robinson and Shane Gibson as they delve into a comprehensive discussion on agile project management with certified scrum trainer, Jem ‘Jelly’ D’jalel.
Amplifying Your Data’s Value with Business Context
The AgileData Context feature enhances data understanding, facilitates effective decision-making, and preserves corporate knowledge by adding essential business context to data. This feature streamlines communication, improves data governance, and ultimately, maximises the value of your data, making it a powerful asset for your business.
Fast Answers at Your Fingertips: Unveiling AgileData’s ‘Ask a Quick Question’ Feature
Immerse yourself in the magical world of data with AgileData’s ‘Ask a Quick Question’ capability. Perfectly designed for data analysts and business analysts who need to swiftly extract insights from data, this capability facilitates quick data queries and rapid exploratory data analysis.
Unveiling the Secrets of Data Quality Metrics for Data Magicians: Ensuring Data Warehouse Excellence
Data quality metrics are crucial indicators in a data warehouse that measure the accuracy, completeness, consistency, timeliness, and uniqueness of data. These metrics help organisations ensure their data is reliable and fit for use, thus driving effective decision-making and analytics
The patterns of Activity Schema with Ahmed Elsamadisi
Magical plumbing for effective change dates
We discuss how to handle change data in a hands-off filedrop process. We use the ingestion timestamp as a simple proxy for the effective date of each record, allowing us to version each day’s data. For files with multiple change records, we scan all columns to identify and rank potential effective date columns. We then pass this information to an automated rule, ensuring it gets applied as we load the data. This process enables us to efficiently handle change data, track data flow, and manage multiple changes in an automated way.
The Hitchhikers guide to the Information Product Canvas
Working with executives with Peter Lam
The Magic of Customer Segmentation: Unlocking Personalised Experiences for Customers
Customer segmentation is the magical process of dividing your customers into distinct groups based on their characteristics, preferences, and needs. By understanding these segments, you can tailor your marketing strategies, optimize resource allocation, and maximize customer lifetime value. To unleash your customer segmentation magic, define your objectives, gather and analyze relevant data, identify key criteria, create distinct segments, profile each segment, tailor your strategies, and continuously evaluate and refine. Embrace the power of customer segmentation and create personalised experiences that enchant your customers and drive business success.
The challenge of parsing files from the wild
In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.
Amazons product development process with Ryan Lysne
How can data teams use Generative AI with Shaun McGirr
Product thinking and user story mapping with Jeff Patton
Join hosts Murray Robinson and Shane Gibson in a conversation with Jeff Patton, an advocate of agile and lean ways of working and author of the book “User Story Mapping.” In this episode, we focus on the concepts of product thinking and user story mapping.
Recoding America with Jennifer Pahlka
Layered Data Architectures with Veronika Durgin
Shane Gibson and Veronika Durgan discuss layered data architecture, data management, and the challenges of integrating software engineering with data analytics. They advocate for the ELT (Extract, Load, Transform) approach over traditional ETL methods and emphasise the importance of understanding data provenance to increase trust. The hosts also discuss the concept of data lakes and the idea of a “data lakehouse,” merging file storage with cloud compute. The conversation concludes with the importance of defining data layers and their policies, the value of automation in data handling, and the need for clear data governance.
AgileData Newsletter #34
To white label or to not white label
Concept of Team Design
Data Consulting Patterns with Joe Reis
Willem Jan Ageling – Agile Project Management and Leadership
Data Versioning
AgileData Newsletter #35
Magical Plumbing for Effective Change Dates
Common Team Design Patterns
The Patterns of Data Vault with Hans Hultgren
Pawel Huyrn – Product Management
Custom Sync
Latest AgileData Magician Guides
ADI
Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches
Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!
AgileData Newsletter #36
The Challenge of Parsing Files from the Wild
An AgileData Guide to Information Product Canvas
The Patterns of Activity Schema with Ahmed Elsamadisi
Pawel Huyrn – Product Management
Natural Language Rules Output
Latest AgileData Magician Guides
Designed in NZ
AgileData App UX Capability Maturity Model
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
Upgrading Python: A Plumbing Adventure in the Google Stack
In the ever-evolving world of AgileData DataOps, it was time to upgrade the Python version that powers the AgileData Platform.
We utilise micro-services patterns throughout the AgileData Platform and a bunch of Google Cloud Services. The upgrade could have gone well, or caused no end of problems.
Read more on our exciting plumbing journey.
AgileData Newsletter #33
The HitchHikers Guide to the Information Product Canvas
Bring Back Business Analysis with Howard Podeswa
New Google Cloud Feature to Optimise BigQuery Costs
Data Lineage Patterns
G60S – Google Sheets
Bring Back Data Modeling
Building a vibrant community with Scott Hirleman
In the episode of the AgileData podcast, Shane Gibson chats with Scott Hirleman, the founder of the data mesh community.
They delve into the nuances of cultivating and sustaining thriving communities.
The duo touch upon the broader patterns that can be applied to both external and internal communities within organisations, and the essence of being agile and responsive to the community’s evolving needs.
Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design
Reducing the complexity and effort to manage data is at the core of what we do. We love bringing magical UX to the data domain as we do this.
Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?
That magic is not always possible in the first, or even the third iteration of those features.
Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.
This post outlines what that maturity model is and how we apply it.
AgileData Newsletter #37
The Hitchhikers Guide to the Information Product Canvas
Information Product Canvas – Google Slide Template
How Can Data Teams use Generative AI with Shaun McGirr
Roman Pichler – Product Management
AgileData App Colours of the Catalog
Pulp Data Fiction
Empowering teams to take initiative with Jose Corella
Join Murray Robinson and Shane Gibson as they dive into a discussion with Jose Corella on how leadership principles from the US Air Force can find applications in civilian organisations.
Attribution Model Patterns with Yorgos Moschovis
Shane Gibson and Yorgos Moschovis discuss the multifaceted world of marketing attribution. They explore the complexity of tracking customer behaviour across online and offline channels, the impact of privacy regulations, and various attribution models.
Demystifying the Semantic Layer
The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.
Data Asset, Data Product, Data Service?
The Art of Action with Stephen Bungay
Join Murray Robinson, Shane Gibson, and special guest Stephen Bungay to discuss how military principles like “Mission Command” can be applied to manage teams effectively in uncertain conditions.
Are you delivering drills, holes or outcomes?
Essence with Ivar Jacobson
Join Murray Robinson and Shane Gibson as they chat with Ivar Jacobson about Essence; a standard to develop patterns and pattern libraries that is being used by Scrum, SAFE, Lean and Kanban organisations. We discuss object oriented software engineering, use cases, UML, RUP, Essence and AI assisted method coaching. Join us to hear from one of the founders of the software engineering industry.
Tesla and SpaceX with Joe Justice
In this episode, we talked to Joe Justice about how Tesla and Space X have developed a new operating model based on self-organizing teams that continuously discover, deliver, and improve to achieve a thousand year goal. It’s like the most radical combination of open space agility, OKR’s, Continuous Discovery, Continuous Delivery and Continuous Improvement, you can think of. Whatever you think of Elon Musk, these companies are way ahead of most other companies and accelerating further. Tune in for a fascinating discussion about how Tesla and Space X work.