The DataJourney Manifesto
After working for years with many data analytic teams using different technologies, we’ve come to believe that reducing errors and defects in the insight production process is the key to success. We’ve been shamed and blamed by our customers for problems with the data we did not cause, trapped with existing data processes we don’t understand, and sat in dread every morning, waiting for something to break in our data, reports, models, or other customer deliverables. We are tired of the stress and wasted productivity needed to find problems deep within the system. We want a method that enables us to observe our data’s complicated paths to avoid problems and errors, and customer frustration.
22 Principles, plus one. Select one to jump to it.
I-XI
Principles
-
I
Define and Know What Should Be
At any time, in data analytic systems, know what should be, what is, and the exact difference between the two.
-
II
Make Hope Infrequent
Hoping your data systems work in production is not a strategy. It’s a recipe for failure.
-
III
Customers Finding Problems Is Not OK
Customers finding problems in your data analytics is unacceptable. Find problems before your customer does.
-
IV
Don’t Trust Your Data Providers
Some data providers are on top of their game, while others barely fulfill an unwanted task. Either way, providers make mistakes. Get used to it. Protect against it. Use it as an opportunity for improvement.
-
V
Don’t Assume What Worked Last Week Will Work Today
Your team is constantly changing the code and configuration in your data estate. Make sure it is still working.
-
VI
Find The Problem Fast
Finding the exact source of the problem — whether in raw data, integrated data, models, reports, servers, software, and/or code — is half the battle.
-
VII
Perfect Data Quality Is Not a Cure-all
Even with perfect initial data quality, many other things can still go wrong.
-
VIII
Avoid Manual Quality Testing Like The Plague
Completely automate the testing of your data and tools.
-
IX
Your Data Production Is A Factory
Heed the lessons of Toyota, Lean, and Deming. Every tool in the system — data ingestion, transformation, database, predictive model, and visualization — is a workstation on that assembly line.
-
X
Complicated Data Architectures Need NASA’s Mission Control
Our modern data architectures are built on performative complexity. Your data architecture has many “little boxes,” each of which can fail.
-
XI
Follow DataOps Principles
Let the ideas in dataopsmanifesto.org guide your data team during development and production.
XII-XXII
We Need A New Idea: The Data Journey
-
XII
The Data Journey… Is The Expectation Layer
Data Journeys represent the expectations of all the myriad paths data takes from source to the insight value you deliver to your customer.
-
XIII
The Data Journey… Observes, But Does Not “Run”
Data Journeys track and monitor all levels of the data stack, from data quality validation to servers, software, code, costs, and utilization. It’s the “digital twin” of complicated batch and streaming data architectures. Data Journeys hold expectations and don’t “run” anything.
-
XIV
The Data Journey… Alerts In Real Time
A Data Journey supplies real-time statuses and alerts. With this information, you can know if everything ran on time and without errors and immediately identify the parts that didn’t.
-
XV
The Data Journey… Goes Across And Down
Data Journeys define the process lineage for the many complex elements that deliver insight. It covers components “across” your toolchain and “down” your technology stack, including logs, messages, run status, metrics, data validation tests, and other information from your data estate.
-
XVI
The Data Journey… Groups Components
A Data Journey has many components. Paraphrasing Anna Karenina: all happy, error-free Data Journeys are alike; each unhappy Data Journey is broken in its unique way. Data Journeys find the “unhappy” component quickly.
-
XVII
The Data Journey… Trusts But Verifies
“Trust, but verify” is an old Russian proverb. Trust comes from monitoring every component in your Data Journey, then verifying the data that touches it. Test, validate, and look for anomalies at every step.
-
XVIII
The Data Journey… Shares Schedules
Your production schedule is public property — share widely. Use it to eliminate silos between the engineers who built the components of the Data Journeys, the operators who run them, the customers who use them, and the managers who get yelled at if there is a problem.
-
XIX
The Data Journey… Learns From Production History
Each instance of a Data Journey provides history and evidence to find the root cause of a defect, help your team improve, and share evidence of improvement in production errors and unmet SLAs.
-
XX
The Data Journey… It Can Be A Business Workflow, Too
An instance of a Data Journey usually represents the batch or streaming technical steps used to create value from data. However, some Data Journey instances represent a business workflow in “the real world.” A particular customer may use that Data Journey to check on the status of that process.
-
XXI
The Data Journey… Lowers Deployment Risk
Use the Data Journey to find the impact of regressions during development. You can’t ship code to production based on manual or static analysis. Use the Data Journey to help automatically regression test your code in development to find the impact of changes.
-
XXII
The Data Journey… Reduces Errors And Drives Productivity
Your data analytics team productivity drops when they spend time finding and fixing problems in production. Unidentified errors in your Data Journeys cause costly business mistakes, erode the trust of your customers, and may have a compliance risk.
Start With Quality First
A Data Journey is only as good as the tests at each checkpoint. It can tell you a step ran late, but only a test can tell you the numbers inside are wrong. So start with automated data tests: they’re the cheapest step, and your customer feels them first. The DataOps Quality Manifesto lays out how, in 24 principles. Start with open-source DataOps TestGen.
Join the people who have already signed the Manifesto
Add your name to the data engineers, analysts, and data leaders who would rather find the problem before their customer does.
Put the manifesto to work with open source
DataOps Process Observability
Monitor every Data Journey from source to the dashboards that depend on it. Catch errors, late arrivals, and bottlenecks across every tool before your customer does.
Get Process ObservabilityDataOps TestGen
Profile your data and generate the quality tests automatically. Full coverage in minutes, with no hand-written rules to maintain.
Install TestGen