AgileData DataOps Articles

App Engine and Socket.IO

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

ELT without persisted watermarks ? not a problem

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

Three Agile Testing Methods – TDD, ATDD and BDD

In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.

Using a manifest concept to run data pipelines

TD:LR … you don’t always need to use DAGs to orchestrate Previously  we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data pipelines at This post I’m going to...
Agile DataOps

TD:LR Agile DataOps is where we combine the processes and technologies from DataOps with a new agile way of working, to reduce the time taken and increase the value of the data we provide to our customers What's in a...