Trust the Math?

In 2021, The New York Times ran a story titled "This Book Is Not About Baseball. But Baseball Teams Swear by It." The book featured was Thinking, Fast and Slow by Daniel Kahneman.

In his book, Kahneman demonstrates through decades of psychological research how our judgments are prone to predictable biases and errors. Through a series of experiments, he showed that we tend to overrate our quick assessments of small samples, anchor to our first impressions, and search for evidence that confirms what we already believe. For an industry heavily focused on evaluating players and making decisions with incomplete information, it’s clear why these insights have become embedded in MLB front offices.

In many ways, that's been a productive development for the industry. Front offices have adopted analytical frameworks to reduce the influence of human bias and make more objective decisions throughout their organizations. Clubs that embraced an analytics-forward approach have gained an edge over those that did not.

However, as front offices have become increasingly reliant on statistical analysis in their decision-making, I've noticed a subtle shift: model outputs are sometimes treated as final verdicts rather than useful starting points. 

Projection models are designed to produce the best estimate of a player's true talent level based on historical data. Each model improvement brings us closer to understanding a player's ability and their potential contributions to winning baseball games. But no matter how sophisticated they become, models are still simplifications of reality; they are not reality.

Yet within the walls of a front office, it's easy for this distinction to become blurred. Once that distinction begins to fade, the impact can be felt throughout an organization.

A room full of smart baseball people can drift toward a model's conclusion even when, an hour earlier, they believed the opposite. Batter-pitcher matchup recommendations that are intended to help inform a manager's in-game decisions gradually become perceived as the final call despite reasonable counterarguments from staff. Scouts or front office personnel can feel pressure to walk back their evaluations once it becomes clear they don't align with a player's model grade. 

It’s not hard to see why this happens.

Model-backed recommendations are often more difficult to challenge than traditional forms of scouting and evaluation because they appear objective and grounded in mathematics. The process itself is viewed as rigorous, and the industry has largely accepted it as the modern way to make decisions. There is professional security in aligning with the consensus.

The downside of this dynamic, though, is not simply the risk that your model was poorly built or that it happens to be wrong, but also a much more subtle one: the erosion of your staff's willingness to voice their opinion. When your people sense that the model output is implicitly viewed as the final call, over time, the range of perspectives openly shared begins to narrow. This cycle can lead to a gradual stifling of organizational intellect. For an executive, it’s a dangerous place to be: making decisions with less context and less information than you realize. 

This concern isn't just a hypothetical one. Theo Epstein, arguably the most successful executive in the analytics era, recently cautioned on the Dirt From the Dugout podcast that the industry’s current state is “a little bit out of balance” when it comes to its use of analytics:

"Data is important and analytics are important…you can use those tools to help you make good predictions about how players are going to perform in the future. But if it's done at the exclusion of the human element and…getting to know other people, understanding what makes them tick, developing a connection with them…and putting them in a position to succeed, then it's not worth it."

Theo's caution is slightly different from mine, though related. His is that overreliance on analytics can come at the expense of the human element and an organization's ability to get the most out of its people. I agree with his conclusion. My issue is adjacent–when model outputs are treated as the definitive answer rather than what they actually are: useful estimates. 

Accepting a model's output with little room for question isn't an exercise in analytical thinking. It is a form of outsourcing judgment, which is the opposite of intellectual rigor.

To be clear, this is not an argument against an analytical approach to organizational strategy. Teams that aren't effectively leveraging data in scouting, player development, roster construction, and in-game strategy will fall behind those that are. The caution is about the authority we give to model outputs, and the costs that come with it. 

In my experience, the best executives use models as inputs to a decision, not substitutes for judgment. They challenge the assumptions behind their models, understand what information is and isn't captured, while building an environment where disagreement is welcome, even when it doesn't align with the math. 

Far easier said than done. Models are one of the most powerful tools we've built for understanding complex systems. Making sense of what is signal and what is noise in one of the highest-variance sports is no small feat. Having the right tools to aid in doing so is crucial. It's just not everything.

One of my chief takeaways from Thinking, Fast and Slow is that we are often more confident in our intuition than the evidence justifies. In MLB front offices, it seems we have become appropriately skeptical of our intuition. In some corners of the industry, however, I worry we have mistaken statistical analysis for certainty and have replaced one source of overconfidence with another.