Blogs

Because sharing is caring

5 core Data Collection Patterns

At AgileData, delivering our Fractional Data Service has revealed the diverse challenges of integrating data from varied organisations, industries, and systems. To scale effectively, we’ve adopted five core data collection patterns based on our “Define it Once, Reuse it Often” (DORO) principle:

1. Push
2. Pull
3. Stream
4. Share
5. File Drop

These patterns are supported by a toolkit of tested technologies like Dataddo, Meltano, and Google services, allowing us to solve new data challenges quickly. Our approach ensures flexibility and scalability, always starting with the question: Push, Pull, Stream, Share, or File Drop?

Innovation and disaster with Ron Westrum
Innovation and disaster with Ron Westrum

Join Murray Robinson and Shane Gibson as they chat with Ron Westrum about how organisational culture drives innovation and disaster. We discuss generative, bureaucratic and pathological cultures, using real-world examples from NASA, Boeing and others. This episode highlights the importance of leadership integrity, openness, honesty, the free flow of information, and a mission-focused approach to achieving organisational goals. Listen for insights on identifying and promoting a generative culture, dealing with bad news, the role of technical maestros, and much more.

Introducing Hai, AgileData 2024 Data Intern
Introducing Hai, AgileData 2024 Data Intern

I’m Hai, a name that intriguingly means “hi” in English. Originally from Vietnam, I now find myself in Australia, studying Data Science and embracing an internship at AgileData.io. This journey is not just about academic growth but also about applying my knowledge in practical, impactful ways. Join me as I explore the blend of technology and community, aiming to make a difference through data.

Unmanaged with Jack Skeels
Unmanaged with Jack Skeels

Join Murray Robinson and Shane Gibson as they chat with Jack Skeels about agile for digital and marketing agencies. Jack advocates for reducing the managerial overhead in organisations to promote productivity and improve communication. We discuss common hurdles in agile implementations. The value of a manager as a sports coach. The importance of scoping the work. Over managing and how real change must start from the top. This episode is full of practical advice and critical insights into organisation dynamics. This conversation is especially useful for those seeking to implement agile in a more holistic, efficient, and outcome oriented way in a service provider.

Defining self-service data
Defining self-service data

Everybody wants self service data, but what do they really mean when they say that.

If we gave them access to a set of highly nested JSON data, and say “help your self”, would that be what they expect?

Or do they expect self service to be able to get information without asking a person to get it for them.

Or are they expecting something in between.

I ask them which of the five simple self service patterns they want to find, which form of self service they are after.

OKR’s with Adrian Howard
OKR’s with Adrian Howard

Join Murray Robinson and Shane Gibson as they discuss objectives and key results with Adrian Howard.

Adrian explains how OKR’s can be used as a strategic instrument for aligning goals within your organisation.  We walk through common pitfalls and misconceptions emphasizing the importance of using OKR’s focused on outcomes rather than outputs. We discuss the potential for using OKR’s with vendors. The idea of paired metrics. And how OKR’s can serve as indicators of underlying problems within your organisations culture and structure.

The Heart of Agile with Mike Leber
The Heart of Agile with Mike Leber

Join Murray Robinson and Shane Gibson as discuss business agility and the heart of agile with Mike Leber. 

We talk about how the heart of agile liberates you from rigid process frameworks by focusing on collaboration, delivery reflection and improvement.  We talk about how the agile industrial complex has turned agile into a heavyweight waterfall process.  And how innovation is happening outside the traditional agile bubble.  And lastly, we talk about agile leadership. If you’re interested in getting to the heart of agile and exploring business agility.

The state of product management with Jason Knight
The state of product management with Jason Knight

Join Murray Robinson and Shane Gibson as they chat with Jason Knight on the state of product management. 

Jason shares his insights and experiences on the responsibilities of a product manager, the evolving nature of the role. Differentiating buyer and user features the importance of testing and validating ideas. Supporting cross-functional teams and fostering a mindset of continuous improvement.  Jason shares insights from his experiences and discussions with thought leaders and provides advice for those navigating the product management field.

AgileData App

Explore AgileData features, updates, and tips

Network

Learn about consulting practises and good patterns for data focused consultancies

DataOps

Learn from our DataOps expertise, covering essential concepts, patterns, and tools

Data and Analytics

Unlock the power of data and analytics with expert guidance

Google Cloud

Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows

Journey

Explore real-life stories of our challenges, and lessons learned

Product Management

Enrich your product management skills with practical patterns

What Is

Describing data and analytics concepts, terms, and technologies to enable better understanding

Resources

Valuable resources to support your growth in the agile, and data and analytics domains

AgileData Podcast

Discussing combining agile, product and data patterns.

No Nonsense Agile Podcast

Discussing agile and product ways of working.

App Videos

Explore videos to better understand the AgileData App's features and capabilities.

Agile and Product – Justin Bauer

Join Shane Gibson as he chats with Justin Bauer on his experience combining the worlds of agile and product in a data driven company.Guests Justin BauerShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google...

Observability, Tick
Observability, Tick

TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels...

DataOps: The Magic Wand for Data Magicians
DataOps: The Magic Wand for Data Magicians

DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.

Information Product Canvas

Join Shane and Tammy Leahy as they discuss the Information Product Canvas, what each area of the canvas holds and why you would want to collect this information.

This is the second in a series of podcast episodes that deep dives into the Information Product pattern.

ELT without persisted watermarks ? not a problem
ELT without persisted watermarks ? not a problem

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured