There’s a certain sense of security that comes from working with numbers. When it comes to the human brain, we think that if we can quantify an idea, we can be sure of it. 💯
It’s one of the reasons marketers tack numbers onto their products and campaigns, like when you see ads that claim:
4 out of 5 dentists prefer this toothpaste
or
this soap is 99.44% pure
Numbers feel like a sure thing – even if they aren’t.
Fundamentally, those warm and fuzzy feelings are all in your head because the world, particularly as it relates to business and technology, doesn’t have the attention span for stability and certainty. So, if we’re going to stay nimble and ahead of the curve, we’ve got to learn to embrace the unknown. 🫂
Managing risk means making friends with uncertainty 🤝
Census CEO and co-founder, Boris Jabes, sat down with Jeremiah Lowin, founder and CEO of Prefect, to talk about the difficulties of navigating an unknown future.
When Jeremiah began his career as a computational statistician, it was in the world of high finance. In 2007, as the world’s banks teetered on the brink of collapse, Jeremiah was working in risk management at a hedge fund. The conversations there and then, he said, were not theoretical.
“Every single day there was a true, real, and consequential application for the work we were doing,” he said. “The more data we had, the faster we could extract insights, and the better we were at our jobs. [The Lehman Brothers bankruptcy] made an enormous impression on me about how to think about uncertainty and how to work with the knowledge that things will go wrong even when it’s impossible to say how.”
Although everyone knew the bankruptcy was coming for weeks before the Lehman Brothers officially filed, no one knew exactly when, why, or how, Jeremiah said. So, leading up to the announcement, his firm sat down and worked through every scenario they could think of and assigned market likelihood to each. 💸
When it finally happened, Jeremiah remembers how calm the traders were in the midst of the catastrophe because they had rehearsed their reaction even before they knew exactly what they would be reacting to.
The moral of the story? Probability (and consequently preparation) is everything.
“I love the idea of statistics as a tool for exploring and quantifying uncertainty,” he said. “Without ever having to choose one story that would have to be correct, we built up an intuition of behaviors within our collective experience.”
Businesses love to talk about return on investment, not risk of investment, Boris said. It makes sense to talk in terms of outcomes when we’re sure the outcomes will happen. When definite outcomes are unknown, it makes more sense to talk about the many possible paths we could take and still get to the right neighborhood. 🗺️
“If the outcome is, ‘The button will have a four-pixel corner radius in this color,’ there’s only one way to satisfy that,” he explained. “But if the outcome is, ‘The user discovers this feature’ – now there’s creativity. There’s license and freedom to explore uncertainty and guide the team’s activities to the right place. That’s why I believe in really strong mission statements to guide decision making.”
In finance, surmounting uncertainty and gauging risk is all part of a day’s work. In other industries, though, it’s been historically difficult.
Why? Preparing for uncertainty can seem like you’re standing still and not making any real progress. In reality, however, it’s this preparation that actually propels you forward in the long run. 🚀
“Executives love moving forward. Engineers love moving forward,” Jeremiah said. “Analysis is either going to confirm the way forward, or it’s going to bring up uncertainty. And if you’re not a statistician, that can seem idiotic because you don’t like thinking about uncertainty.”
Data, in particular, can create an illusion of certainty where none actually exists. Anyone who has ever worked in marketing knows the pain of marketing attribution:
If a customer buys a product through an affiliate link, is that the channel that gets credit for the sale?
Or is it the ad they clicked on last week, the social media post they watched a few days ago, or the blog post they read yesterday? 🤔
Even when you have very rich data, the connection between the data and the thing you’re trying to quantify may be tenuous at best.
“You tend to forget your data is just one lens of the actual thing you’re trying to tell a story about,” Jeremiah said. “For any number, you can always find someone to tell the opposite story. Numbers are ultimately just lenses into behavior. The question is: Can you get people to buy into the story you’re telling?”
Precision paints you into a corner
People feel like if they can quantify something precisely enough, they’ll have an ironclad argument. Despite the so-called certainty mirage, the precision you once loved can become a prison as quickly as a protector.
“When you have an industry that is easy to quantify you rapidly lose sight of any nuance in interpretation,” Jeremiah said. “Every time I see someone comparing a cryptocurrency return to an equity return, I die a little bit because no one makes capital allocation decisions without acknowledging the inherent risk. You can’t compare those two numbers and claim any intellectual honesty.”
When looking at the numbers, people tend to draw their own conclusions and tell whatever story they believe the data will support. You’re better off telling a broad story you think is likely to be correct than narrowing in on a pet theory you hope is correct, Jeremiah said.
It can feel good to declare something with 15.62% confidence, but if the difference between 15% and 16% makes no meaningful impact on the business, you’re better off dropping the extra decimal places and embracing the wiggle room that comes with a little uncertainty.
“This is a challenge, especially for young companies,” he said. “You need to motivate people, to assure your customers, to draw the market to your product. So, you make these huge declarations. But you better know exactly what you’re talking about because you’ve left yourself very little room for error.”
Data teams can help our companies embrace the unknowable
The prime directive of a business intelligence team is to accurately depict reality, Boris said. Its main responsibility is to give executives certainty about where the business is and what it does. Giving those teams reign to explore uncertainty – to find areas of under- or over-investment – could reveal uncomfortable truths and opportunities for massive impact. 💥
“If you are in a purely solution-driven culture, it is harder to undo acceptance of a proposal than it was to get the acceptance in the first place,” Jeremiah said. “This is why I believe really strongly in cultures that embrace uncertainty or saying, ‘I don’t know, but if I’m wrong, here’s how we’ll know.’ If you give me a path to understanding how we can tell when it’s not working, I can move forward a lot faster.”
Managing risk and uncertainty are not part of the traditional purview of data teams – but a data team that can rescue its executives from always having to be right and precise can be a powerful force. When you start to recognize that uncertainty applies context to the data, you allow for a deeper understanding of the data (and, ultimately, the business).
It’s difficult to go from being 100% certain you are right to swallowing the bitter pill that you were wrong, Boris said. To prevent that unpleasant feeling – and the bad decisions that may have preceded it – it’s better to acknowledge you can never be 100% sure of anything.
“We all live and die by our dashboards. They’re useful, but they’re not enough,” Jeremiah said. “There has to be someone who takes the responsibility of drawing the inference and vision and storytelling from those dashboards. The data team can weave a sort of rich, contextual tapestry of what is happening in the world and how a company plays into it.”
Want to learn more? You can catch the full conversation between Boris and Jeremiah below, or on your favorite streaming platforms. 🎧
Then head on over to The Operational Analytics Club to join the conversation around this and many other data best practices. ✨