Agile DataOps

TD:LR

What’s in a name

Everything it seems.

I remember years ago working with a financial organisation (ok it was a bank) and the team were tasked with building out a data mart that held customer data to support a digital channel initiative.

One of the key things we had to provide was a flag for customers that were active. Even in those days I  was using our “who does what” discovery approach and so we had already identified the concept of a customer and its associated rules.

But when we got to defining the rule for Active Customers we hit a problem, the Marketing, Finance and Risk teams all had different definitions for active customers.  For marketing it was pretty much if you had an account and weren’t dead you were an active customer.  For finance (from memory) it was if you had made an active transaction in an account in the last 3 months.  And I can’t remember the rule from the Risk team, but I do remember it was based on the regulatory rules provided by the Reserve Bank here in New Zealand.

We had a couple of goes at facilitating a common rule for Active Customer, but quite rightly each team needed their own rule to be applied to meet their business outcomes.

So in the end we created three flags, Active Marketing Customer, Active Finance Customer and Active Risk Customer.  We then suggested the digital channel team picked the flag that met their business outcome the best.

So that leads me on to the subject of DataOps

xxxxOps

We are seeing a plethora of people adopting the DevOps approach to the things they do and therefore the rise of xxxxOps.  For example SecOps, DevSecOps, MLOps and of course DataOps.

I recently saw a slide from a presentation Shaun McGirr did describing MLOps.  It said: 

My immediate reaction was where is the people part in that definition, its missing.

And that is because when we talk about DataOps we talk about it being more than just DevOps for data.

DataKitchen is one of the most vocal companies on the new world of DataOps. In their article DataOps is NOT Just DevOps for Data, they state:

Jon Loyens from Data.World also has an interesting article on the subject, What is DataOps?  In this article he states: 

Both these lines of thinking aligns with my experience, introducing an agile way of working is key when moving to a DataOps paradigm.  Adopting patterns and approaches from our DevOps cousins is important, but not the only things data and analytics teams need to adopt. 

Many many many definitions

If you google DevOps you will tend to find a raft of fairly consistent definitions, something along the lines of:

But if you google definitions of DataOps you will find a raft of varying definitions including these:

It is widely recognised that Andy Palmer was the first to coin the term DataOPs in his 2015 Tamr article From DevOps to DataOps

Ok lets see some other definitions:

And it wouldn’t be right to have quotes without a couple from the analysts:

PHEW! thats confusing

When there are lots of different definitions for the same thing, just like the banking teams I mentioned earlier, each person or group will tend to find the definition that resonates with them the most, or is the one that will help them achieve their desired business outcomes.

To date the definition I tended to use the most is from the team at DataKitchen:

And this is because like me they believe we need more than just better Development and Operations processes to make the adoption of DataOps successful.

To me DataOps is the combination of a bunch of proven patterns and approaches which are adopted by a data and analytics team, and these include patterns and approaches from:

• • • DevOps
    • Scrum
    • Lean
    • Data Science
    • Data warehousing
    • Statistics
    • And many many more

A thing by any other name ….

I think the trap we have fallen into is the same one as my active customer example.  We are trying to use one term to describe multiple things, each which are subtly different.

So from now on I am going to make a few changes to the way I describe things.

I am going to start by referring to DataOps as DevOps for data.  Just like Shauns slide which describes MLOps in a way that matches the DevOps definition, it makes the definition clear and unambiguous.

But if we are going to implement DataOps, we should also change the way we work in other areas at the same time.  And a great way to make that change is to adopt agile patterns and approaches from our agile cousins.

Agile DataOps

As much as I hate it when people coin a new “buzz word” I am going to do just that (forgive me) and start talking about “Agile DataOps”.  Just like Marketing Active Customer, I think adding one more word to the name, provides a lot more clarity on how it can be described

The team at DataKitchen have nailed what it takes behave in an Agile DataOps way with their Manifesto, if you haven’t read it I suggest you do.

I think that Data Mesh, Data Vault 2.0 and other names that seem to be appearing as a raft of new data approaches are all about combining ways of working (people processes) and xxxxOps (technology processes).  Something I need to do more reading and thinking about, before I comment further.

But there is more

As an aside one of the more interesting quotes for xxxxOps I have heard is:

Keep making data simply magical