Data and algorithms - how far can it take us?

The formalisation of data and algorithms has recently been the purview of computer science. And of course, the application of data and algorithms, in particular from the tech sector, has become essential to how we solve problems in our daily lives. We use data and algorithms to help us decide what movie to watch, or how to get directions in an unfamiliar city. There's even a book called Algorithms to Live By.

In a work setting we're also always applying data to algorithms. Some settings are simple, such as deciding which of your past clients you should reconnect with this week based on how long it's been since you last spoke. And we've all had experience gathering data and using the pros-cons lists algorithm to make business decisions. Or, there's its more complicated cousin, Cost Benefit Analysis. In a lot of our Government and policy work this is the go-to algorithm for weighing up, in monetary terms, the cost and benefits of a new project, intervention, or policy.

But are there limits to the problems we can solve and the decisions we can make by simply throwing an algorithm at a bunch of data? In his book Wild Problems, Russ Roberts suggests that there is a class of problems - "wild problems" - for which the data-plus-algorithm approach doesn’t work. Problems like where to live, what type of career to have, and whether to have children. A key challenge with wild problems is not just the difficulty of knowing what decision to make, but the difficulty in comprehending or understanding what the possible outcomes of a decision would actually mean to us. Who really knows what it's like to have children before having them, or what it's like to live in Brisbane before actually moving there? 

A pros-cons list, or a Cost Benefit Analysis, has its limits. It assumes we actually know and understand the costs and benefits. For wild problems we don't.

Of course, most decisions we face are not wild problems. The decision of whether to eat Japanese or Indian for dinner is not a wild problem. But for some of the "simpler" decisions we make there are opportunities to take inspiration from wild problems by being open to unexpected costs and especially being open to unexpected benefits. By making decisions that cultivate serendipitous opportunities, we can increase our exposure to the potential of unexpected benefits.

Consider the decision of whether to attend a particular conference. The data might suggest the cost of being out of the office for two days outweighs the benefit of learning about those latest developments. After all, you can just read about it all online. But the pros-cons algorithm doesn't take into consideration the possible unexpected benefit of meeting someone for the first time who opens the door to a one-in-a-million new business opportunity for you.

Data and algorithms work well for patterns that are regular. Situations where history is a reasonable predictor of the future and situations where you've seen and measured the costs and benefits before. And this is the case for so many decisions in life and work, hence the utility of data and algorithms. But these are their limits. When making decisions, remember that data can’t predict the unexpected. Instead, embrace ways to increase your potential for unexpected benefits. You never know.