The Challenge of Parsing Files from the Wild

In this post we discuss how we handle the challenges of parsing files from the wild.

We liken our data processing to a water system where each file is a new source of water.

To ensure clean and well-structured data, each file goes through several checks and processes, similar to a water treatment plant.

These steps include checking for previously seen files, looking for matching schema files, queuing the file, and parsing it.

If a file fails to load, we have procedures in place to retry loading or notify errors for later resolution.

This rigorous data processing ensures smooth and efficient data flow, akin to well-managed water in a plumbing system.

Read more at AgileData.io

An AgileData Guide to Information Product Canvas

We are writing an Agile Data guide for business analysts, product managers, data analysts, data engineers, data leaders and agile coaches struggling to not boil the ocean and striving to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them

We would love to invite you to be part of our book launch.

Leave us your email address and we will keep you updated as we get ready to go!

Read more at AgileData WoW

The Patterns of Activity Schema with Ahmed Elsamadisi

In this episode of the AgileData Podcast, we are joined by Ahmed Elsamadisi to discuss the patterns of activity schema.

The activity schema is a data modeling pattern for data modeling and transformation.

Some takeaway’s from the podcast:

• Traditional data modeling techniques are often difficult to use and maintain, they often lead to data warehouses that are complex and difficult to query

• The activity schema is a new data modeling paradigm that is positioned to simplify data modeling and make data warehouses more agile and scalable.

• The activity schema models all data as a single time series table, which makes it easier to answer any data question using a single query pattern.

• The activity schema is not a silver bullet. It is still important to have a good understanding of your business data and to use the activity schema in a way that makes sense for your specific needs.

• If you are looking for a new data modeling paradigm that can help you get more value from your data, the activity schema is worth considering.

Listen to the AgileData podcast or read the transcript at AgileData.io

Pawel Huyrn – Product Management

Join Murray Robinson and Shane Gibson as they engage in a conversation with Pawel Huryn about the ins and outs of product management. Learn what product management entails and how it differs from project management, with a focus on discovery versus delivery.

Key topics include:
🔸 Motivating teams for product success
🔸 Balancing discovery and delivery in empowered product teams
🔸 Transitioning into a product management role
🔸 Effective product management tools

Tune in to uncover valuable insights that will help you excel in the critical role of product management and understand how to effectively manage and empower your teams.

Listen to the No Nonsense Podcast or read the transcript at agiledata.io

Natural Language Rules Output

Understanding how data has being transformed is often complex.

We aim to make it simple.

Displaying a Natural Language version of the transformation is another step in our journey to reducing the complexity of managing data in a simply magical way.

Watch on YouTube

Latest AgileData Magician Guides

We are constantly adding new user guides to our doc site.

Here are the latest magician guides:

• Catalog – Tile Detail – Reviews

• Catalog – Tile Detail – Data Search