Blogs

Because sharing is caring

Introducing Hai, AgileData 2024 Data Intern

I’m Hai, a name that intriguingly means “hi” in English. Originally from Vietnam, I now find myself in Australia, studying Data Science and embracing an internship at AgileData.io. This journey is not just about academic growth but also about applying my knowledge in practical, impactful ways. Join me as I explore the blend of technology and community, aiming to make a difference through data.

Defining self-service data
Defining self-service data

Everybody wants self service data, but what do they really mean when they say that.

If we gave them access to a set of highly nested JSON data, and say “help your self”, would that be what they expect?

Or do they expect self service to be able to get information without asking a person to get it for them.

Or are they expecting something in between.

I ask them which of the five simple self service patterns they want to find, which form of self service they are after.

There are 3 strategic / macro data use cases
There are 3 strategic / macro data use cases

I often ask which of these three macro data use cases the Organisations believed were its priorities to achieve their business strategy:

Providing data to Customers
Supporting Internal Processes
Providing data to External Organisations

Each of these three strategic / macro data use cases come with specific data architectures, data work and also impact the context of how you would design your agile data ways of working.

Building the Data Plane while flying it
Building the Data Plane while flying it

In the data domain you typically have to balance between building the right thing and building the thing right.

The days of being able to spend 6 months or a year on “Sprint Zero” creating your data platform have gone.

One team I worked with called it “building the airplane as you fly it”

Here are 5 patterns I have seen data teams adopt to help them do this.

2024 the year of the Intelligent Data Platform
2024 the year of the Intelligent Data Platform

AI was the buzzword for 2023 and it will continue to be the buzzword for 2024.

I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.

Ask AI
Assisted AI
Automated AI
Adopting these patterns moves a data platform from being a manual data platform, towards a data platform that can do some of the data work for you.

An Intelligent Data Platform.

Knowledge Graphs with Juan Sequeda
Knowledge Graphs with Juan Sequeda

Dive deep into the world of knowledge graphs with Juan Cicada on the Agile Data Podcast, hosted by Shane Gibson. Explore key insights from Juan’s journey in computer science, his pivotal role in semantic web development, and the transformative power of knowledge graphs in data integration. Discover how these technologies are reshaping the landscape of data management and the exciting future prospects with the advent of Large Language Models (LLMs). Tune in to understand the practical applications, challenges, and the future of knowledge graphs in enterprise data strategy.

AgileData App

Explore AgileData features, updates, and tips

Network

Learn about consulting practises and good patterns for data focused consultancies

DataOps

Learn from our DataOps expertise, covering essential concepts, patterns, and tools

Data and Analytics

Unlock the power of data and analytics with expert guidance

Google Cloud

Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows

Journey

Explore real-life stories of our challenges, and lessons learned

Product Management

Enrich your product management skills with practical patterns

What Is

Describing data and analytics concepts, terms, and technologies to enable better understanding

Resources

Valuable resources to support your growth in the agile, and data and analytics domains

AgileData Podcast

Discussing combining agile, product and data patterns.

No Nonsense Agile Podcast

Discussing agile and product ways of working.

App Videos

Explore videos to better understand the AgileData App's features and capabilities.

Data Storytelling with Kat Greenbrook
Data Storytelling with Kat Greenbrook

Explore the Art of Data Storytelling with Kat Greenbrook on the Agile Data Podcast. Dive into Kat’s transformative journey from aspiring vet to a data storytelling expert, and discover the power of the ABT (And, But, Therefore) narrative framework in conveying compelling data insights. Uncover common pitfalls and learn crucial differences between data visualization and storytelling. Enhance your business communication skills with practical tips and insights from ‘The Data Storyteller’s Handbook.’ Perfect for professionals in data analytics, business intelligence, and anyone keen to master the art of turning data into impactful stories.

The Art of Data: Visualisation vs Storytelling
The Art of Data: Visualisation vs Storytelling

Data visualization is like painting with data, using charts and graphs to make trends and patterns easy to understand. It’s great for presenting data objectively.

Data storytelling weaves a narrative around data, adding context, engaging emotions, and inspiring action. It’s perfect for persuading stakeholders.

Ways of Working with Scott Ambler
Ways of Working with Scott Ambler

Join Shane Gibson on the Agile Data Podcast for an enlightening conversation with Scott Ambler, an IT and Agile expert. Delve into Scott’s journey from pioneering programmer to data architecture and Agile methodologies. Discover the evolution of Agile data, the importance of adapting ways of working, and the pitfalls of best practices. Learn valuable insights into continuous improvement, team dynamics, and the complexities of data quality in today’s fast-paced IT landscape. Don’t miss this episode for an in-depth exploration of Agile data and its impact on IT projects and processes.

Demystifying the Semantic Layer
Demystifying the Semantic Layer

The semantic layer is your mystical bridge between complex data and meaningful business insights. It acts as a translator, converting technical data into a language you understand. It works through metadata, simplifying queries, promoting consistency, and enabling self-service analytics. This layer fosters collaboration, empowers customization, and adapts to changes seamlessly. With the semantic layer’s power, you can decipher data mysteries, conjure insights, and make decisions with wizard-like precision. Embrace this enchanting tool and let it elevate your data sorcery to new heights.

Attribution Model Patterns with Yorgos Moschovis
Attribution Model Patterns with Yorgos Moschovis

Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.

Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design
Understanding Concepts, Details, and Events: The Fundamental Building Blocks of AgileData Design

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Building a vibrant community with Scott Hirleman
Building a vibrant community with Scott Hirleman

Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.

AgileData App UX Capability Maturity Model
AgileData App UX Capability Maturity Model

Reducing the complexity and effort to manage data is at the core of what we do.  We love bringing magical UX to the data domain as we do this.

Every time we add a new capability or feature to the AgileData App or AgileData Platform, we think how could we just remove the need for a Data Magician to do that task at all?

That magic is not always possible in the first, or even the third iteration of those features.

Our AgileData App UX Capability Maturity Model helps us to keep that “magic sorting hat” goal at the top of our mind, every time we add a new thing.

This post outlines what that maturity model is and how we apply it.

Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches
Unveiling the Magic of Change Data Collection Patterns: Exploring Full Snapshot, Delta, CDC, and Event-Based Approaches

Change data collection patterns are like magical lenses that allow you to track data changes. The full snapshot pattern captures complete data at specific intervals for historical analysis. The delta pattern records only changes between snapshots to save storage. CDC captures real-time changes for data integration and synchronization. The event-based pattern tracks data changes triggered by specific events. Each pattern has unique benefits and use cases. Choose the right approach based on your data needs and become a data magician who stays up-to-date with real-time data insights!

Layered Data Architectures with Veronika Durgin
Layered Data Architectures with Veronika Durgin

Dive into the Agile Data Podcast with Shane Gibson and Veronika Durgin as they explore the intricacies of layered data architecture, data vault modeling, and the evolution of data management. Discover key insights on balancing data democratisation with governance, the role of ETL processes, and the impact of organisational structure on data strategy.

How can data teams use Generative AI with Shaun McGirr
How can data teams use Generative AI with Shaun McGirr

Discover the transformative impact of generative AI and large language models (LMS) in the world of data and analytics. This insightful podcast episode with Shane Gibson and Shaun McGirr delves into the evolution of data handling, from manual processes to advanced AI-driven automation. Uncover the vital role of AI in enhancing decision-making, business processes, and data democratization. Learn about the delicate balance between AI automation and human insight, the risks of over-reliance on AI, and the future of AI in data analytics. As the landscape of data analytics evolves rapidly, this episode is a must-listen for professionals seeking to adapt and thrive in an AI-driven future. Stay ahead of the curve in understanding how AI is reshaping the role of data professionals and transforming business strategies.

The challenge of parsing files from the wild
The challenge of parsing files from the wild

In this instalment of the AgileData DataOps series, we’re exploring how we handle the challenges of parsing files from the wild. To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant. These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it. If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution. This rigorous data processing ensures smooth and efficient data flow.