AgileData DataOps

Because sharing is caring

Myth: using the cloud for your data warehouse is expensive

Myth: using the cloud for your data warehouse is expensive

TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents.  Is it too good to be true?  Yes AND No. You can run a cloud platform for a low low cost, but its will take engineering effort  to get there. Myth: using the...

Myth: using the cloud for your data warehouse is expensive

TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents.  Is it too good to be true?  Yes AND No. You can run a cloud platform for a low low cost, but its will take engineering effort  to get there. Myth: using the...

Observability, Tick

TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels like every other influencer in the data space is...

App Engine and Socket.IO

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

ELT without persisted watermarks ? not a problem

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

Three Agile Testing Methods – TDD, ATDD and BDD

In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.

Using a manifest concept to run data pipelines

TD:LR … you don’t always need to use DAGs to orchestrate Previously  we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data pipelines at agiledata.io This post I’m going to...

“Serverless” Data Processing

TD:LR When we dreamed up agiledata.io and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and what that has to do with Apples. Is serverless even...

A Data Engineer an Agile Coach and a Fish walk into a bar…

This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users

AgileData >>> Modern Data Stack

TD:LR AgileData's mission is to reduce the complexity of managing data.  A large part of modern data complexity is selecting, implementing and maintaining a raft of different technologies to provide your "Modern Data Stack".  At AgileData we have already done the hard...

Agile DataOps

TD:LR Agile DataOps is where we combine the processes and technologies from DataOps with a new agile way of working, to reduce the time taken and increase the value of the data we provide to our customers What's in a name Everything it seems. I remember years ago...

Observability, Tick

TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels like every other influencer in the data space is...

App Engine and Socket.IO

We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.

ELT without persisted watermarks ? not a problem

We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured

Agile DataOps

TD:LR Agile DataOps is where we combine the processes and technologies from DataOps with a new agile way of working, to reduce the time taken and increase the value of the data we provide to our customers What's in a...

AgileData Product

Blogs about the AgileData Product

Consulting

Blogs about running a data consultancy

DataOps

Blogs about DataOps

Data and Analytics

Blogs about Data and Analytics

Google Cloud

Blogs about Google Cloud

Journey

Blogs about the AgileData Journey

Product Management

Blogs about Product Management

Resources

Links to Resources that are useful

What Is

Blogs that describe Data Things

All Blogs

Subscribe to our newsletter

We will email you whenever we share new content, no spam, pinky promise

Let me read it first.