What it really means to make data-driven decisions
Gathering as much correct data as possible, constantly reviewing decisions and being able to tell signal from noise.
Startups are often proud to say that they do.
But in my experience, that’s not the whole truth. Sure, decisions are made based on data, but mostly the obvious ones and based on data that’s available at the time.
It’s difficult to recognize what data you need to collect. It takes knowing what other data is relevant. And it takes a lot of effort to analyze all of it thoroughly.
Furthermore, you also need to continuously review decisions as more and better data becomes available.
Finally, it also takes experience to tell when a change in indicators is a signal and not variance.
You must make decisions based on sufficient correct and complete data. Often at least one of these factors is missing and frequently more than one.
Having sufficient data is difficult. That’s why you should track more than you think you need.
Correct has two meanings. First, correct data means calculating it correctly and to a sufficient degree of accuracy. Secondly, it also means that you’re looking at the correct numbers.
Complete data means that you can see the whole picture and all the pieces that are affected.
An important skill that comes into play here is knowing what numbers are important and how they relate. For example, you need to understand the relationship between order count and basket size.
It also means following up the initial decision with constant monitoring. Based on how things are progressing, you need to make frequent adjustments and compare them to the baseline.
It’s a lot of work.
Recognizing how important this is and doing it constantly, consistently and well makes companies.
The Great Britain cycling team that dominated the olympics really drove this point home. They embraced a philosophy of marginal gains where they would make any slight improvements. For instance, they used a 3D printer to make a custom holder for the cycling computer for one of their team members.
Early-stage startups, however, aren’t expected and do not have the bandwidth to take it to this extreme. The closer you are to it, though, the better off you’ll be.
By constantly going through this process, you will learn how to tell when signals are strong enough and when they’re just variance.
The main issue is that with small data sets even seemingly large shifts in data carry little meaning.
One way to counteract this is to track several related but not dependent numbers and look for consistent changes.
- If you’re a marketplace, it means constantly tweaking your promotion mechanisms and coupons based on weighted basket size.
- If you’re selling things, it means constantly tweaking the prices of products.
- It means having tiered pricing and adjusting it based on historical order quantity
- If you’re a butcher, it means constantly adjusting the placement of your products and adjusting your promotions
- It means deciding whether to keep a popup that collects subscriber emails not through gut feeling but by comparing your email lists’ conversion rates and comparing them to the drop-off due to visitors closing your site when the popup is shown
- Don't overreact to weak signals
It makes you build poorly thought out and potentially wrong things.
- Measure What Matters by John Doerr
How to do OKRs properly
Suggest an improvement to this page (email@example.com)