TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic...
Blog Articles
Observability, Tick
TD:LR Data observability is not something new, its a set of features every data platform should...
The language of data is not so natural
TD:LR The dream is we can just point the machine at our data, ask our question and get a useful...
Build Data Products Without A Data Team Using AgileData
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of...
How To Bring Agile Practices To Your Data Projects
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain. Listen to the episode or read the transcript....
App Engine and Socket.IO
TD:LR We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this. Bringing...
I can write a bit of code faster
TD:LR To get data tasks done involves a lot more than just bashing out a few lines of code to get the data into a format that you can give it to your stakeholder/customer. Unless of course it really is a one off and...
The Focus Podcast – Agile Data Governance Patterns
TD:LR Early in 2022 I was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Listen to the episode, watch it or read the transcript. Listen Watch Read Aldo Rall: Hi, and...
ELT without persisted watermarks ? not a problem
TD:LR We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging...
Three Agile Testing Methods – TDD, ATDD and BDD
In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing. Test Driven Development (TDD) Acceptance Test Driven...
Using a manifest concept to run data pipelines
TD:LR … you don’t always need to use DAGs to orchestrate Previously we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data...
“Serverless” Data Processing
TD:LR When we dreamed up agiledata.io and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and...