Blogs

Because sharing is caring

NZ Scaleup AgileData achieves Google Cloud Ready – BigQuery Designation

In the data domain you typically have to balance between building the right thing and building the thing right.

The days of being able to spend 6 months or a year on “Sprint Zero” creating your data platform have gone.

One team I worked with called it “building the airplane as you fly it”

Here are 5 patterns I have seen data teams adopt to help them do this.

Product Management with John Cutler
Product Management with John Cutler

Join Murray Robinson and Shane Gibson as they discuss product management with John Cutler, famous for his newsletter The Beautiful Mess.

We dive into the purpose of product management as a driver of profitable growth. And we discuss how to use the north star framework to align your organisation around common goals. We examine how to use product roadmaps to communicate and prioritize plans and discuss how critical it is to be an open, honest, and effective leader. And finally we look at whether agile is critical or useless in product management. Join us as we uncover the insights and best practices product managers need to be more effective.

UX Research with Nick Fine
UX Research with Nick Fine

Join Murray Robinson and Shane Gibson as they discuss UX research with Dr. Nick Fine. Nick explains what to expect from a UX researcher, how to do user research well, and the challenges that researchers have to overcome to obtain valid results. Whether you’re a product manager, designer, or developer this episode is packed with actionable insights, real world examples and expert tips to enhance your understanding of UX research and its role in creating high value, digital products and services.

The Vanguard Method with Ibrar Hussain
The Vanguard Method with Ibrar Hussain

Join Murray Robinson and Shane Gibson as they discuss the Vanguard method with Ibbi Hussain, one of the founding architects of this revolutionary approach to organisation change. Ibi describes how the Vanguard method uses systems thinking, service design, and occupational psychology to make profound improvements to call center performance. Join us as we navigate through the philosophy of change, the importance of leadership. And practical insights into implementing sustainable improvement within organisations.

Episode 100 – A retrospective
Episode 100 – A retrospective

Welcome to our 100th episode.

In this episode, we talk about what we’ve learnt from our guests about agile, product development and leadership. We discuss what the podcast is about and why we started it. We unpack our critical take on fake agile and discuss agile leadership and product development. Join us as we take a deep dive into the insights, experiences and conversations that have shaped our podcast ovr the last 99 episodes. And here’s to another hundred episodes of cutting through the nonsense and bringing you the advice you need to succeed.

Business Agility with Evan Leybourn
Business Agility with Evan Leybourn

Join Murray Robinson and Shane Gibson as they chat with Evan Leybourn from the Business Agility Institute about business agility. We discuss how to define and measure business agility, examples of agile organisations, and the struggle with traditional bureaucratic models. We delve into the different types of leadership, the nature of internal cultural and political changes, and the challenges faced by organisations in adopting and implementing agility.

Innovation and disaster with Ron Westrum
Innovation and disaster with Ron Westrum

Join Murray Robinson and Shane Gibson as they chat with Ron Westrum about how organisational culture drives innovation and disaster. We discuss generative, bureaucratic and pathological cultures, using real-world examples from NASA, Boeing and others. This episode highlights the importance of leadership integrity, openness, honesty, the free flow of information, and a mission-focused approach to achieving organisational goals. Listen for insights on identifying and promoting a generative culture, dealing with bad news, the role of technical maestros, and much more.

Introducing Hai, AgileData 2024 Data Intern
Introducing Hai, AgileData 2024 Data Intern

I’m Hai, a name that intriguingly means “hi” in English. Originally from Vietnam, I now find myself in Australia, studying Data Science and embracing an internship at AgileData.io. This journey is not just about academic growth but also about applying my knowledge in practical, impactful ways. Join me as I explore the blend of technology and community, aiming to make a difference through data.

AgileData App

Explore AgileData features, updates, and tips

Network

Learn about consulting practises and good patterns for data focused consultancies

DataOps

Learn from our DataOps expertise, covering essential concepts, patterns, and tools

Data and Analytics

Unlock the power of data and analytics with expert guidance

Google Cloud

Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows

Journey

Explore real-life stories of our challenges, and lessons learned

Product Management

Enrich your product management skills with practical patterns

What Is

Describing data and analytics concepts, terms, and technologies to enable better understanding

Resources

Valuable resources to support your growth in the agile, and data and analytics domains

AgileData Podcast

Discussing combining agile, product and data patterns.

No Nonsense Agile Podcast

Discussing agile and product ways of working.

App Videos

Explore videos to better understand the AgileData App's features and capabilities.

Demystifying the Semantic Layer
Demystifying the Semantic Layer

The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.

Attribution Model Patterns with Yorgos Moschovis
Attribution Model Patterns with Yorgos Moschovis

Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.

Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design
Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Building a vibrant community with Scott Hirleman
Building a vibrant community with Scott Hirleman

Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.

AgileData App UX Capability Maturity Model
AgileData App UX Capability Maturity Model

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches
Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

Layered Data Architectures with Veronika Durgin
Layered Data Architectures with Veronika Durgin

Dive into the Agile Data Podcast with Shane Gibson and Veronika Durgin as they explore the intricacies of layered data architecture, data vault modeling, and the evolution of data management. Discover key insights on balancing data democratisation with governance, the role of ETL processes, and the impact of organisational structure on data strategy.

How can data teams use Generative AI with Shaun McGirr
How can data teams use Generative AI with Shaun McGirr

Discover the transformative impact of generative AI and large language models (LMS) in the world of data and analytics. This insightful podcast episode with Shane Gibson and Shaun McGirr delves into the evolution of data handling, from manual processes to advanced AI-driven automation. Uncover the vital role of AI in enhancing decision-making, business processes, and data democratization. Learn about the delicate balance between AI automation and human insight, the risks of over-reliance on AI, and the future of AI in data analytics. As the landscape of data analytics evolves rapidly, this episode is a must-listen for professionals seeking to adapt and thrive in an AI-driven future. Stay ahead of the curve in understanding how AI is reshaping the role of data professionals and transforming business strategies.

The challenge of parsing files from the wild
The challenge of parsing files from the wild

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.

The Magic of Customer Segmentation: Unlocking Personalised Experiences for Customers
The Magic of Customer Segmentation: Unlocking Personalised Experiences for Customers

Customer segmentation is the magical process of dividing your customers into distinct groups based on their characteristics, preferences, and needs. By understanding these segments, you can tailor your marketing strategies, optimize resource allocation, and maximize customer lifetime value. To unleash your customer segmentation magic, define your objectives, gather and analyze relevant data, identify key criteria, create distinct segments, profile each segment, tailor your strategies, and continuously evaluate and refine. Embrace the power of customer segmentation and create personalised experiences that enchant your customers and drive business success.

Magical plumbing for effective change dates
Magical plumbing for effective change dates

We discuss how to handle change data in a hands-off filedrop process. We use the ingestion timestamp as a simple proxy for the effective date of each record, allowing us to version each day’s data. For files with multiple change records, we scan all columns to identify and rank potential effective date columns. We then pass this information to an automated rule, ensuring it gets applied as we load the data. This process enables us to efficiently handle change data, track data flow, and manage multiple changes in an automated way.

The patterns of Activity Schema with Ahmed Elsamadisi
The patterns of Activity Schema with Ahmed Elsamadisi

In an insightful episode of the AgileData Podcast, Shane Gibson hosts Ahmed Elsamadisi to delve into the evolving world of data modeling, focusing on the innovative concept of the Activity Schema. Elsamadisi, with a rich background in AI and data science, shares his journey from working on self-driving cars to spearheading data initiatives at WeWork. The discussion centers on the pivotal role of data modeling in enhancing scalability and efficiency in data systems, with Elsamadisi highlighting the limitations of traditional models like star schema and data vault in addressing complex, modern data queries.

Amplifying Your Data’s Value with Business Context
Amplifying Your Data’s Value with Business Context

The AgileData Context feature enhances data understanding, facilitates effective decision-making, and preserves corporate knowledge by adding essential business context to data. This feature streamlines communication, improves data governance, and ultimately, maximises the value of your data, making it a powerful asset for your business.

New Google Cloud feature to Optimise BigQuery Costs
New Google Cloud feature to Optimise BigQuery Costs

This blog explores AgileData’s use of Google Cloud, specifically its BigQuery service, for cost-effective data handling. As a bootstrapped startup, AgileData incorporates data storage and compute costs into its SaaS subscription, protecting customers from unexpected bills. We constantly seek ways to minimise costs, utilising new Google tools for cost-saving recommendations. We argue that the efficiency and value of Google Cloud make it a preferable choice over other cloud analytic database options.

Data as a First-Class Citizen: Empowering Data Magicians
Data as a First-Class Citizen: Empowering Data Magicians

Data as a first-class citizen recognizes the value and importance of data in decision-making. It empowers data magicians by integrating data into the decision-making process, ensuring accessibility and availability, prioritising data quality and governance, and fostering a data-centric mindset.

The patterns of Data Vault with Hans Hultgren
The patterns of Data Vault with Hans Hultgren

In a compelling episode of the Agile Data Podcast, Shane Gibson invites Hans Hultgren to explore the intricacies of Data Vault modeling. Hans, with a substantial background in IT and data warehousing, brings his expertise to the table, discussing the evolution and benefits of Data Vault modeling in today’s complex data landscapes. Shane and Hans navigate through the foundations of Data Vault, including its core components like hubs, satellites, and links, and delve into more advanced concepts like Same-As Links (SALs) and Hierarchical Links (Hals). They highlight how Data Vault enables flexibility, agility, and incremental development in data modeling, ensuring scalability and adaptability to change.

To whitelabel or not to whitelabel
To whitelabel or not to whitelabel

Are you wrestling with the concept of whitelabelling your product? We at AgileData have been there. We discuss our journey through the decision-making process, where we grappled with the thought of our painstakingly crafted product being rebranded by another company.