Blogs

Because sharing is caring

NZ Scaleup AgileData achieves Google Cloud Ready – BigQuery Designation

In the data domain you typically have to balance between building the right thing and building the thing right.

The days of being able to spend 6 months or a year on “Sprint Zero” creating your data platform have gone.

One team I worked with called it “building the airplane as you fly it”

Here are 5 patterns I have seen data teams adopt to help them do this.

AgileData App UX Capability Maturity Model
AgileData App UX Capability Maturity Model

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches
Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

Layered Data Architectures with Veronika Durgin
Layered Data Architectures with Veronika Durgin

Dive into the Agile Data Podcast with Shane Gibson and Veronika Durgin as they explore the intricacies of layered data architecture, data vault modeling, and the evolution of data management. Discover key insights on balancing data democratisation with governance, the role of ETL processes, and the impact of organisational structure on data strategy.

Recoding America with Jennifer Pahlka
Recoding America with Jennifer Pahlka

Join hosts Murray Robinson and Shane Gibson in a conversation with Jen Pahlka, founder of Code for America and author of "Recoding America". They discuss the current challenges faced by governments in providing efficient digital services and how agile product...

How can data teams use Generative AI with Shaun McGirr
How can data teams use Generative AI with Shaun McGirr

Discover the transformative impact of generative AI and large language models (LMS) in the world of data and analytics. This insightful podcast episode with Shane Gibson and Shaun McGirr delves into the evolution of data handling, from manual processes to advanced AI-driven automation. Uncover the vital role of AI in enhancing decision-making, business processes, and data democratization. Learn about the delicate balance between AI automation and human insight, the risks of over-reliance on AI, and the future of AI in data analytics. As the landscape of data analytics evolves rapidly, this episode is a must-listen for professionals seeking to adapt and thrive in an AI-driven future. Stay ahead of the curve in understanding how AI is reshaping the role of data professionals and transforming business strategies.

The challenge of parsing files from the wild
The challenge of parsing files from the wild

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.

AgileData App

Explore AgileData features, updates, and tips

Network

Learn about consulting practises and good patterns for data focused consultancies

DataOps

Learn from our DataOps expertise, covering essential concepts, patterns, and tools

Data and Analytics

Unlock the power of data and analytics with expert guidance

Google Cloud

Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows

Journey

Explore real-life stories of our challenges, and lessons learned

Product Management

Enrich your product management skills with practical patterns

What Is

Describing data and analytics concepts, terms, and technologies to enable better understanding

Resources

Valuable resources to support your growth in the agile, and data and analytics domains

AgileData Podcast

Discussing combining agile, product and data patterns.

No Nonsense Agile Podcast

Discussing agile and product ways of working.

App Videos

Explore videos to better understand the AgileData App's features and capabilities.