A Data Engineer an Agile Coach and a Fish walk into a bar…

Stop me if you’ve heard this one before !

A Coach and an Engineer

I’m a Data Engineer with a passion for all things cloud, in particular the Google Cloud Platform. Together with my co-founder Shane, who is as far from a Data Engineer as you can get, we embarked on a journey to democratize data and remove the friction that is inherent where data is involved.

We like to say“simply magical data”, because your data experience should be fun … not scary, or expensive, or require large teams of specialists to‘do technical stuff’for you.

Along the way we adopted ADI (short for agiledata.io) the fish to bring some SaaS to our platform (software as a service) 😉

Proven Patterns

We believe that the tried and proven engineering patterns that have been used for the last 25 years when delivering data projects could be baked (watch for my use of ‘recipes’ later in this series) into a platform that customers could use ‘as a service’ for their data — getting the benefit of a cloud scale platform with the complexity taken care of .. magically.

When I say patterns, I’m talking about…. Patterns to move and ingest data. Patterns to model data. Patterns to insert and update data. Patterns to augment data. Patterns to backup and archive data etc

And us Engineers all use patterns. Every, Single, Time.

So, why not build those patterns into a platform so any user could onboard their own data and get value straight away ? no waiting for me to choose a service, build a pipeline, version the code, deploy the service, move the data, load the data, build and publish your models…. you get the point.

And that’s how the idea for agiledata.io came about.

How we built AgileData.io

This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users

How we leverage a serverless first architecture, using BigQuery as our powerhouse for data processing, Cloud Spanner as the swiss army knife backbone to the product, Cloud Functions for making stuff happen, and Cloud Pubsub messaging so all the moving parts can talk to each other.

Why Google Cloud Platform?

Great question !

Given we started from scratch, we needed to choose an ecosystem to build upon, and that ecosystem had to meet our idealistic aspirations:

• The services we used would be serverless — they would start up, do stuff, cost us money, shut down, stop costing us money. Fullstop.

• There would be no adoption friction to using a service — i could say, “we need a messaging service now”, and the platform would have one and we could start using it with no integration issues or learning curve as it would“just work”as expected with no hidden surprises or unforeseen challenges.

So, we kicked the tyres of the big players in the cloud space. We setup a few user accounts, subscribed to some services and tried to use them.

As a professional in this space, I’m embarrassed to say, I didn’t get very far with some of the offerings … they just didn’t make sense to me. The consoles were overly complicated, the getting started guides were confusing, and they just had too many hoops and steps to get through to get results.

Then we tried Bigquery…. and it just worked, damn did it work !!

Bigquery was quickly followed by Cloud Functions, Code Repository, Cloud Spanner, App Engine, Cloud Build, Container Registry …. the list goes on.

We now had a platform, and a dream … next time, Serverless Data Processing