“So you know this search report, the one we’ve been basing business decisions on for three years? It wasn’t set up correctly.”
The report my colleague was showing me was one that we had inherited and it had served as a significant influencer of business decisions for several years. My stomach felt queasy. How many business decisions had been based on this misinformation? How many well-intentioned hard-working people doing their best to make the business run better and to deliver better experiences to their users had relied on this report?
I’ve seen this kind of mistake many times and it’s a good example of why knowing things about your business can be so difficult. Often, what we think is knowledge is really misinformation.
Here are some examples:
Something breaks and you don’t realize it.
Your engineers make a back-end update to your website or app, maybe they refactor some code or maybe a software update cause your analytics tool to disconnect from the data source.
The wrong person set up your analytics.
Maybe you don’t have an analytics expert on staff and so a confident intern or your social media manager does their best to install a tracking script and create a property but they just don’t totally know what they’re doing. Analytics tools can be extremely complicated, their interface and functionality changes constantly, and documentation on how to use these tools can be lacking, hard to find, or contradictory.
The wrong person is interpreting the data for your team.
Even with a perfect analytics setup, knowing how to interpret the numbers is extremely difficult and fraught with a multitude of ways to get it wrong. We’ll go into this in greater depth in the future but, for now, suffice it to say that humans are not naturally good at turning the information found in charts into actionable, meaningful, insights.
No one thought to measure in the first place.
This one drives me crazy. Unless the right person was in the room at the early stages of your project’s development, it’s very likely your team forget to make a robust plan that accounts for the questions the product team will need to be able to answer to validate assumptions and improve over time and the questions your management team will be asking.
The list of ways your data game can be impaired is much longer. Every month that goes by in which your product is in the wild but the relevant metrics are not being tracked is a month of permanently lost learnings that you can’t get back. This all applies whether you’re considering digital analytics or measuring physical products and experiences.
We’ll go into a much more detailed explanation of the solutions to these problems in future posts but here are three recommendations you can take away today:
1. Make an analytics strategy.
What is the current state of your analytics environment? What is your desired state? What are the major decisions we need to make to pursue that desired state?
2. Make a plan for analytics governance.
Who owns your analytics strategy? Who governs your analytics tools and teams? Who is paying for what? How do teams choose what tools they are using such that they are empowered but your larger organization is not wasting money on duplicate purchases, unnecessarily expensive tools, and lack of interoperability?
3. Get outside eyes on your situation.
You don’t know what you and your team don’t know. It can take a lot of experience and expertise to even know what questions to ask. And we’re often just too close or in the weeds to catch everything. Getting an objective expert who can dig around in your analytics and ask fresh questions can help a lot.
Seeing that business operators and their teams are empowered to do their work meaningfully is really my central aim in the work I do. To me, strategy is about meaning-making. Work is more fun and inspiring when it all makes sense. Using data well can help with this but the numbers have their limitations and so do the people trying to work with them. Having a healthy sense of skepticism and wariness about our confidence in being “data-driven” is actually a great place to start getting better as an organization.