The irrationality of NFL play-callers

I recently read two papers that both essentially conclude (based on different aspects) that NFL play callers are not rational (or more specifically, not rational and risk neutral - an important point I'll return to at the end of the post). Recall that a rational decision-maker weighs up the costs and benefits of a decision, and when faced with mutually exclusive options (such as choosing which play to run in an NFL game), they should choose the option with the greatest net benefit (benefits minus costs).

The first paper (by Jonathan Hartley, an MBA student at the Wharton School at the University of Pennsylvania) looks at play-callers' choices between an extra point attempt and a two-point attempt following a touchdown. A rational and risk-neutral play-caller should choose whichever play provides the greatest expected benefit (expected number of points). In this case, Hartley found:

Between 2002 and 2014, the extra point conversion rate was 99.2% (out of 7738 attempts). As the average two point conversion rate remained 0.475, the expected points from a two-point conversion remains 0.95 below the automatic 0.992 points...

Over 2 seasons since the implementation of the new rules [increasing the distance the extra point try is attempted from], the extra point conversion rate has fallen from 0.992 to 0.95. Moreover, the total number of 2 point conversion attempts per season has nearly doubled...

In other words, when the NFL changed the extra point to being attempted from a greater distance (thereby making it more difficult), the expected value of an extra point try fell from 0.992 to 0.95 points. The expected value of a two-point conversion remained steady at 0.95 points. So, a rational and risk-neutral play-caller should now be indifferent between an extra point attempt and a two-point conversion. However, as Hartley shows in the paper, most teams still attempt very few two-point conversions, even those teams that have a history of success at them. The paper itself is pretty rough, but I wish the MBA students here could do this sort of work!

The second paper, by Noha Emara (Rutgers), David Owens (Haverford College), John Smith (Rutgers), and Lisa Wilmer (Florida State), is forthcoming in the Journal of Behavioral and Experimental Economics (ungated earlier version here), and looks at serial correlation in play-calling. Serial correlation occurs when you have a time series (like a series of plays) and where each observation in the time series is related (positively or negatively) to the observation or observations earlier in the time series. Obviously, an NFL offensive play-caller wants to call players in a random way - what we call a mixed strategy. Mixed strategy is particularly important in sports - think of the choice of where to serve in tennis, or where to shoot a penalty or which way to dive as a goalkeeper in soccer (see here or here for more on this). If an NFL offensive play-caller doesn't effectively randomise their play calling, then the defence can potentially exploit some prior knowledge of the play about to be called.

Humans are rubbish at trying to create random series, and indeed that's what Emara et al. found, based on their dataset of more than 200,000 plays from the 2000-2012 NFL seasons:

...the previous pass variable is negative and significant in each specification. This provides evidence that, even after controlling for down, distance, field position, and other observables, play calling exhibits significant negative serial correlation. The Previous pass-Previous failure interaction estimate is negative significant in both of the specifications where it appears, suggesting that play calling becomes even more negatively serially correlated following a failed play.

To translate, play-callers are significantly more likely to call a running play after a previous passing play, and to call a passing play after a previous running play, than would be expected if they were selecting plays randomly. And on top of that, if the previous play was a failure (e.g. if it lost yards), then they are even more likely to change the play type on the following play.

To make things worse, Emara et al. find evidence that teams would be better off if they ran more plays that were the same as the previous play:

We find that a rush following a rush gains 0.14 more yards than a rush following a pass. We also find that a pass following a pass gains 0.21 more yards than a pass following a rush. Estimates are more pronounced when we also control for whether the previous play was a failure. We find that a rush gains 0.24 more yards more following a failed rush than following a failed pass. Also, a pass gains 0.34 more yards following a non-failed pass than following a non-failed rush.

In summary, we find evidence that the efficacy of a play, as measured by yards gained, increases if it follows a play of the same type.

The results is even stronger on second down plays, but not so much for third down plays. However, the take-away message, like that of the first paper, is that play-callers are not being purely rational.

However, there is a caveat here. If we think that, based on this evidence, that play-callers should be calling more two-point conversions and switching up play types less often, then we may be forgetting that there is also a wider game at play here. If play-callers are risk averse, then this affects their decision-making. The two-point conversion may have the same expected value as an extra point attempt, but it is riskier (see also this post on NBA three-pointers from last week), so a risk averse play caller may avoid the two-point conversion more than the simple comparison of expected values would suggest.

But what about the play-callers in the second paper? Emara et al. have thought about this, and this is what they offer:

Perhaps teams feel pressure not to repeat the play type on offense, in order to avoid criticism for being too “predictable” by fans, media, or executives who have difficulty detecting whether outcomes of a sequence are statistically independent. Further, perhaps this concern is sufficiently important so that teams accept the negative consequences that arise from the risk that the defense can detect a pattern in their mixing.

Making play calls that the fans think are predictable (but which are actually more random) may make the play-caller themselves at risk of losing their job (or at least, of looking like they are doing a poor job). So, play-callers may attempt to make their play calls look more random by switching (from run to pass or vice versa) more often than they should, even though this is actually less random and costs the team in terms of yards gained per play.

The question is, now that these trends are known, will any team want to exploit them?

[HT: Marginal Revolution, here and here]