think with data

5 Questions to Ask Before Putting Your Faith In Data

You receive a report. It has charts and decimal figures. Someone you trust has signed off on it. And you have approximately ninety seconds to decide if you’ll trust it. Most leaders will. Not because they’re careless but because challenging a figure can feel like opening a can of worms. Challenging the figures can seem like you’re questioning the accuracy of the source. No one wants to hold up a meeting. Good leaders don’t take it for or against. They know to ask a few questions in a way that will help everyone in the room learn. You won’t need any qualifications in statistics. You’ll just need to ask the following 5 questions.

5 Questions Every Business Leader Should Ask Before Trusting Data

1. Where has this come from and who owns it?

All figures on a slide will have started somewhere. It’ll have been typed into a spreadsheet, recorded by a machine or drawn from somewhere else in a system. At some point, someone will have made decisions. Where has the figure come from? Who inputted it, verified it and who would you contact if you thought it looked wrong? It can be surprising how often no one will know. The figure may have come from a spreadsheet that has been copied several times. A field may have been left blank and defaulted. Or units may have changed between teams. Someone may have a figure in tonnes and someone else in kg. They’ve just been added together. If there’s no owner of the figure, no one will be verifying it. It may not be inaccurate but if it is, no one will be responsible for noticing.

2. What is this compared to?

This is the most useful question you can ask in business. It only has three words! “I’ve had an 8% increase in sales.” Compared to what? Last month, last year, targets, another region? An 8% increase is brilliant if the market decreased but terrible if all competitors increased by 20%. Numbers don’t exist in isolation. They’re only useful when compared to something else. Usually, there won’t be a comparison. This is where the positive spin on the numbers will be. If the result is being shown against last quarter, it may look very different against last year. If they show an increase during the festive period, they’re claiming a breakthrough but this happens every year. Always ask for a comparison. If there’s no baseline, ask for it. Even better, ask what the worst comparison would be and verify that too. This is an easy way to keep the analysis honest.

Read More: What Is Data Fluency? Why It Matters for Every Business

3. What is missing?

Every data set will only give you a snapshot of the real world. The danger is what could be missing.

If you’ve done a customer satisfaction survey and received positive feedback, who completed it? If it was emailed out, maybe only the engaged customers responded. Those who left quietly never saw it. If you’ve piloted a new feature and it ‘worked’ with 40 users, were those your most enthusiastic power users? Was 40 even enough? How many people/records is this based on? How were they selected? Who is missing? Even if there isn’t a large sample size or it’s biased, it can look just as confident. The chart won’t get smaller to show how little data there is behind it.

4. What else could this be?

When two things move together it can feel like one caused the other. This is how our brains work. Many expensive decisions are made in this way The revenue went up this month when we launched our new campaign. We’ll give the credit to our campaign. It may be deserved but maybe this month was a seasonally busy month, the competitor ran out of stock or we had a price change in the same week. If we haven’t considered why this may be, we’re making an assumption that the campaign worked. You don’t need to be a statistician. Just ask what else changed around this time. If the presenter has a good answer, you’ve learnt something. If they look confused then you’ve also learnt something. It’s better to know now than to double your budget. A good habit to get into is to be cautious of results that are exactly what you hoped for. The more flattering the result is to your plan, the more you should question it. This isn’t cynical, it’s simply that we’re very good at hiding our wishful thinking in real data.

5. What will this change and what will change my mind?

This flips the question back to you. What will this actually change? If you can’t honestly say it will change anything, it’s just flair. You could have used that time doing something else. There are many dashboards with figures that everyone is interested in but provide no information. What would make me change my mind? Try to answer this before seeing the data. If you can’t think of anything that would change your mind then this analysis won’t make a difference. It’ll just be flair. This isn’t data driven. It’s using data as an excuse.

Tend to get better results from leaders who ask these questions out loud. People stop looking to prove the boss right and will look for the truth.

Risks Of Asking The Questions

Asking the 5 questions can feel like an interrogation. This leads to the team becoming fearful of being unsure.

A few tips to avoid this:

Ask genuinely. If you ask “I just want to make sure I understand” it will have a very different feel to “I don’t believe it.” Ask the questions on positive figures. If you only ask about the negative figures, people will see this as a punishment. Ask the most relevant 1 or 2 questions. You’ll rarely need all 5 for every figure. If it’s a regular update, you may only need to ask about the baseline. If it’s a big investment, ask all 5 questions. Be honest. If the leader is asking “what does this axis start at?” Everyone else will feel it’s ok to be curious. Thank them. If you reward people for finding issues, they’ll tell you about them sooner. If asked well these questions will speed up your team. Less decisions will need to be reversed.

Read More: What Happens When Data Is Inaccurate?

Why Is This A Team Habit?

While having one clever leader asking good questions can help, it won’t scale. They can’t be in every meeting. Many decisions will be made two levels below.

Organisations that manage their data well will have these questions as a habit. The analyst will ask them before sending the report. The manager will ask them before presenting. The new member of staff will feel comfortable asking “compared to what?” in their first week. They become part of the conversation.

This is what we mean by Think With Data at Ankashram. We don’t really focus on tools. We focus on helping people throughout your business feel comfortable challenging assumptions, critically analysing evidence and making decisions together. We deliver interactive sessions and the most common thing we hear is “I didn’t realise I could ask that.”

You can. And it’s probably the most useful thing you can do before trusting any numbers.

The 5 Questions

  1. Where has this come from and who owns it?
  2. What is this compared to?
  3. What is missing?
  4. What else could this be?
  5. What will this change and what will change my mind?

Print them off, pin them up or just remember them. Next time you get a report, ask one of the questions. You may be surprised by how much people have to say.

Read More: Why Your Business Dashboard Might Be Misleading You

Frequently Asked Questions

Q1. Why should business leaders challenge data rather than trust their analysts?

Good analysts will want to be challenged. Even the most careful people will have messy data sources, unclear definitions and small sample sizes. By asking clever questions you’re helping to highlight issues that no one else can see.

Q2.What’s the most important question to ask?

Probably “what is this compared to?” Without a baseline (target, previous period or benchmark), you won’t know if it’s good, bad or average.

Q3.Do I need technical knowledge?

No. All the questions are straightforward. They’re asking about where the data came from, comparing it to something, what’s missing from your sample, what else could this be and what will change. No knowledge of stats or software needed.

Q4. How can I ask these questions without putting my team on the defence?

Ask genuinely. Ask the same questions on positive figures. Make it clear that finding an issue is valued. When people realise finding an error is rewarded, they’ll start feeling like they’re working as a team.

Q5.How many should I ask?

It depends on the situation. If it’s a regular update, you may only need to ask about the baseline. If it’s a big investment or strategy decision, ask all 5. The aim is to create a habit.

Q6.How does Ankashram help organisations develop this habit?

We use our Think With Data approach and bespoke workshops to help people across departments feel comfortable challenging assumptions, critically analyse evidence and make decisions together.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *