Dynasty, in Theory: Everything Is Selection Bias

Why selection bias infects everything interesting and how to spot it.

Adam Harstad's Dynasty, in Theory: Everything Is Selection Bias Adam Harstad Published 09/28/2024

© PETER PEREIRA/The Standard-Times / USA TODAY NETWORK

There's a lot of strong dynasty analysis out there, especially when compared to five or ten years ago. But most of it is so dang practical-- Player X is undervalued, Player Y's workload is troubling, the market at this position is irrational, and take this specific action to win your league. Dynasty, in Theory is meant as a corrective, offering insights and takeaways into the strategic and structural nature of the game that might not lead to an immediate benefit but which should help us become better players over time.

A Very Useful Razor

My favorite thing about this column is that it affords the opportunity to write about things that ostensibly have nothing to do with fantasy football. Most of my favorite pieces are just ruminations on whether it's better to be a specialist or a generalist; why measuring things is so dangerous; how we can choose how we view things not based on what is true, but on what is useful; when it's acceptable and when it's unacceptable to be correct; why comparisons make no sense (but are worth making anyway); and how we fool ourselves into believing what we see is all there is.

I often find that the best way for me to understand fantasy football is to first understand everything. And as far as "understand everything" goes, there are few tools as useful as a razor. No, not that kind, but a philosophical razor, a tool to efficiently narrow down possible explanations or avoid unnecessary actions.

Occam's razor is, of course, the best-known ("The simplest explanation is usually the best one"), but there are plenty of other terrific examples-- Popper's falsifiability principle ("if it's not falsifiable, it's not scientific"), the Sagan standard ("extraordinary claims require extraordinary evidence").

Sherlock Holmes has made his mark ("once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth"), as has Albert Einstein ("make things as simple as possible, but no simpler"). And I've always been partial to Hanlon's razor: "Never attribute to malice that which is adequately explained by stupidity".

But my favorite is a bit more personal, my most-used problem-solving tool and my own humble contribution to the genre:

What is Selection Bias, and Why Does it Ruin Everything?

Humans are pattern-finding machines. We see trends in small samples and we extrapolate them out to larger samples. It's something of a superpower-- all of science and technology grew out of our ability to discern patterns in the world around us.

It can also be our downfall. Selection bias is when there are issues with our smaller sample that render it unrepresentative and unfit for extrapolation.

It can take a myriad of forms, but here's an example of the classic error in football: from 2016 to 2022, Ezekiel Elliott had 41 games where he carried the ball 20 or more times and the Cowboys were 31-10 in those games, a 76% win rate. In all other games, Dallas was 39-34, a 53% win rate.

If we were simply to extrapolate from Dallas' 31-10 record, we might say that they should have given more carries to Ezekiel Elliott. But there's a problem. When Dallas trailed by 14 or more points during that span, they called run plays on just 27.8% of their snaps. When they led by 14 or more points, they called runs on 60.9% of their snaps.

Dallas didn't win those games because Elliott got 20 carries; Elliott got 20 carries because Dallas won those games. In this example, having a lead is a confounder, or some third factor that influences both of the variables we're looking at. Big leads lead to wins, and big leads also lead to carries.

This form of selection bias is everywhere; virtually every time someone notices a correlation, a confounder is likely to blame. For instance, an increase in per-capita ice cream consumption correlates with an increase in per-capita instances of drowning. Does ice cream cause people to drown? Of course not. When the weather is hot, more people buy ice cream and more people go swimming; the more people are swimming, the more drownings there will be.

Let's Get Weird

Hidden confounders (sometimes referred to as "lurking variables") are the most basic and well-known forms of selection bias, but there are much wilder and woolier examples out there.

For instance, did you know that married couples with children consistently report being less satisfied with their marriages than their childless peers? More than that, for couples without children, marital satisfaction remained steady over time, while for couples with children, the longer they were married the less satisfied they were, on average.

Reader, would you guess that this is actually because of selection bias? (Of course you would; my audience is known to be exceptionally sharp. This is, of course, itself a product of selection bias.)

The reason there are so few married couples without kids who hate each other is not because couples without kids never fall out of love, it's because... couples who hate each other and don't have kids all get divorced. (The phrase you might have heard is not, after all, "staying together for the golden retrievers".)

(Is this another example of a confounder? Nope. A confounder would be if some mysterious third factor influenced both having kids and being miserable-- feel free to insert a joke about tequila here. In this case, the bias goes in the other direction. Having kids and being miserable both influence some unstated third factor-- in this case, divorce rates. This form of selection bias is known as conditioning on a collider. Statistics is hard!)

If you study students at a college, you will find a negative correlation between their SAT score and their high school GPAs. Is this proof that smart kids are lazier? Of course not; this is a form of selection bias known as Berkson's Paradox that happens when you restrict the range of the sample. Students with terrible grades and terrible test scores didn't get into this college. Students with phenomenal grades and phenomenal test scores got into a better college. The end result is a mix of students with good grades and bad scores and students with good scores and bad grades.

Already a subscriber?

Continue reading this content with a 100% FREE Insider account.

By signing up and providing us with your email address, you're agreeing to our Privacy Policy and Terms of Use and to receive emails from Tennessee.

Here's another weird one. In certain cases, rates of drug use will correlate with the percentage of a population that is devoutly religious. Does this mean the devoutly religious use more drugs? Quite the opposite. But high rates of drug use often lead to tragic deaths, and when faced with such a tragedy, many people turn to religion for comfort.

In this case, the relationship that appears at the population level (more religious people suggests more drug use) is the exact opposite of what is true at the individual level (being more devoutly religious reduces the chances of drug use). When relationships that are true at the population level are different than relationships at the individual level, this is known as ecological fallacy.

Selection bias is one of the drivers of the replication crisis in science. One proposed solution was for scientists to pre-register their hypotheses so they couldn't gather their data, see what it said, and then form a hypothesis to "test" afterward.

In perhaps the most maddening example of all, though, preregistration has itself fallen prey to selection bias; more than half of pre-registered studies still hadn't published five years after their proposed end date-- presumably the half that didn't find what they expected to find.  

Any More Examples from Sports?

Is the "when (running back) gets 20 carries the team wins Y% of the time" canard the only example of selection bias ruining otherwise cromulent football analysis? Reader, it is not. Perhaps you've seen some of the "4th down models" out there that use historical success rates to suggest whether teams should go for it, attempt a field goal, or punt on fourth down.

We had a fun example this week as a popular model tweeted that the Cowboys should definitely punt the football as they instead sent Brandon Aubrey out to casually nail a 60-yard field goal. 

Many joked that the model lacked a "Brandon Aubrey" adjustment and was therefore flawed. (And indeed, Aubrey is in the midst of one of the greatest stretches we've ever seen from a kicker.) But the problem isn't just Aubrey.

In 2015, Benjamin Morris wrote the definitive history of kicker performance in the NFL, noting that kickers improved virtually linearly with time, showing no sign of nearing a "skill cap". And then he discussed the implications of this finding for 4th down models. Morris found that a popular model of the day used estimates of kicker quality roughly equivalent to what would have represented league average in 2004.

Then he produced a chart to illustrate how the model changes if you updated it to use 2015-caliber kickers, instead. The current model recommended kicking a field goal in just 50% of situations between the 15 and 40 yard line. Assuming better kickers pushed that to recommending a field goal in 83% of such situations. The model had fallen victim to selection bias.

This is the tip of the iceberg. Our 4th down models overestimate conversion rates because teams are more likely to go for it on 4th-and-inches than on 4th-and-a "long" one.

On the other hand, they also underestimate conversion rates because bad teams attempt more fourth downs than good teams.

And the same bias exists for 2-point conversion rates.

Soccer and Hockey suffer from the opposite bias; "expected goal" models overestimate how likely an average shooter is to score because most shots in the sample do not come from average shooters, but from good ones.

In basketball, shooting percentage overrates low-volume shooters and underrates high-volume shooters because high-volume shooters are forced to take more tough shots (later in the shot clock, against more and better defenders on average).

You'll sometimes see analysts dismiss a player as a "garbage time performer", suggesting they're only producing their numbers in situations where defenses aren't trying their best. But typically the players who have the most production in garbage time are the players whose teams spend the most time in garbage time.

In football, one running back can average more yards per touch than another... despite the other running back averaging more yards per carry and more yards per reception. This is known as Simpson's paradox and is yet another example of how statistics is hard.

Not Everything Is Selection Bias

Some analysis is careful and clever and manages to sidestep the many pitfalls waiting to bring it low. I don't genuinely believe that everything is selection bias. But whether a framework is true is secondary to whether it is useful.

(Also, note: the proposal is that everything interesting is probably selection bias. But this is itself selection bias! The fact that something is interesting makes it more likely that it is flawed because surprising things are more likely to be interesting and also more likely to be mistaken.)

Treating every interesting fact I encounter as if is likely a result of selection bias has helped me to avoid a tremendous amount of bad analysis over the years. Just as importantly, it has helped me find plenty of good analysis; if I search my hardest for selection bias and I can't find anything, there's a better chance that the analysis is sound.

I think if you hone your own selection-bias-detectors, you'll likewise find yourself quicker to dismiss fallacious claims and embrace those that are true-- in football and in every other area of your life. To that end, I have a few last interesting observations for you to sharpen your razors on.

  • Among receivers drafted in the second round, there is a negative correlation between athleticism and career success. Therefore, athleticism doesn't matter for receivers.
  • In chess, the longer a player thinks before a move, the more likely he or she is to make a serious blunder. Therefore, chess players should just go with their gut.
  • Quarterback runs are more likely to convert on 4th down than running back runs. Quarterback runs also improve a team's position more (as measured by expected points added, or EPA), and are substantially less likely to lose yardage. Assuming injury rates were not a factor, teams should run more often with their quarterbacks and less often with their running backs.
  • Similarly, passes to running backs are less efficient on average than passes to wide receivers. Teams should throw more passes to their receivers and fewer to their backs.
  • Players average more points per game in weeks you bench them than in weeks you start them. (This is absolutely true, in case you were wondering.) This means we are bad at playing matchups and shouldn't even try.
  • Deep passing attempts and yardage are down, but deep passing efficiency remains incredibly high. Therefore, it is easy to throw deep in today's defensive environment.
  • #2 receivers average more separation on their targets than #1 receivers. Therefore, #2 receivers are actually better than #1s. 
  • Late-round rookie receivers average as many yards per route run (a gold-standard efficiency metric for WRs) as early-round rookie receivers. It is silly to spend early picks at receiver because you can get players who are just as good late.

If you can spot the flaws in all of those arguments, you're well on your way to more easily separating fact from fiction in your daily life. It is only after we know what is true that we can decide what to do about it.

 

Photos provided by Imagn Images
Share This Article

More by Adam Harstad

 

Dynasty, in Theory: Do the Playoffs Matter?

Adam Harstad

Should we include playoff performances when evaluating players?

01/18/25 Read More
 

Odds and Ends: Divisional Round

Adam Harstad

Examining past trends to predict the future.

01/17/25 Read More
 

Odds and Ends: Wild Card Weekend

Adam Harstad

Examining the playoff futures and correctly predicting the Super Bowl winner.

01/10/25 Read More
 

Dynasty, in Theory: Evaluating Rookie Receivers

Adam Harstad

Revisiting this year's rookies through the lens of the model

01/09/25 Read More
 

Dynasty, in Theory: Consistency is a Myth

Adam Harstad

Some believe consistency helps you win. (It doesn't.)

01/04/25 Read More
 

Odds and Ends: Week 18

Adam Harstad

How did we do for the year? Surprisingly well!

01/02/25 Read More