Because sharing is caring
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Describing data and analytics concepts, terms, and technologies to enable better understanding
Myth: using the cloud for your data warehouse is expensive
TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents. Is it too good to be true? Yes AND No. You can run a cloud platform for a low low cost, but its will take...
Agile and Product – Justin Bauer
Join Shane Gibson as he chats with Justin Bauer on his experience combining the worlds of agile and product in a data driven company.Guests Justin BauerShane GibsonResourcesSubscribe | Apple Podcast | Spotify | Google...
AgileData WoW Q&A – Hamish Gray and May-Lyn Hu
Join Shane Gibson, Hamish Gray and May-Lyn Hu as they talk through some of the experiences they have had in their organisation applying agile and data together in a new way of working.Guests Hamish Gray May-Lyn Hu...
TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels...
DataOps: The Magic Wand for Data Magicians
DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.
The language of data is not so natural
TD:LR The dream is we can just point the machine at our data, ask our question and get a useful answer. With ChatGPT we are closer than we have ever been, but we are not there yet,When Nigel and I first started...
Build Data Products Without A Data Team Using AgileData
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of...
How To Bring Agile Practices To Your Data Projects
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain. Listen to the episode or read the transcript....
Information Product Canvas
Join Shane and Tammy Leahy as they discuss the Information Product Canvas, what each area of the canvas holds and why you would want to collect this information.
This is the second in a series of podcast episodes that deep dives into the Information Product pattern.
Analytical team topologies – Ashwin Kamath
Join Shane and guest Ashwin Kamath as they discuss his experience working with analytical teams and analytical team topologies.
Information Products – What and Why
Join Shane GIbson and Tammy Leahy as they discuss what the Information Product pattern is and Why you would use it.
App Engine and Socket.IO
We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.
I can write a bit of code faster
TD:LR To get data tasks done involves a lot more than just bashing out a few lines of code to get the data into a format that you can give it to your stakeholder/customer. Unless of course it really is a one off and...
The Focus Podcast – Agile Data Governance Patterns
Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.
ELT without persisted watermarks ? not a problem
We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured
Three Agile Testing Methods – TDD, ATDD and BDD
In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.
Scaling data teams – Tammy Leahy
Shane Gibson chats to Tammy Leahy about how she helped scale the data teams she leads.
Join Shane and guest Eric Broda as they discuss data products.
Using a manifest concept to run data pipelines
TD:LR … you don’t always need to use DAGs to orchestrate Previously we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data...
“Serverless” Data Processing
TD:LR When we dreamed up agiledata.io and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and...
A Data Engineer an Agile Coach and a Fish walk into a bar…
This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users