Wednesday, November 25, 2009

The NFL and How To Lie With Statistics

(In case you need a refresher on what happened, here's a summary.)

This blog post is a response to this article: Defending Belichick's Fourth Down Decision.

For the record, I disagree with that article and I think that it's a great example of how to mislead with statistics.

A number of points:

First, using the probability collected from the league as a whole is a fallacy of false analogy. The problem is right there in front of us. The probabilities are based on the league as a whole. The false analogy here is thinking that just because the numbers crunched for all teams who have attempted a 4th and 2 will apply to the Patriots. It's reducing the complexity of football down to a mere throw of the die. The oversimplification is so drastic and untenable. There is absolutely no way you can characterize a complicated situation like that by looking at the statistics of the league as a whole. Yet people read the article and believe the logic behind it.

It also misleads about what Belichick's decision process. In the end, only Belichick knows what went on in his mind. But to imply that Belichick churned the numbers in his head and made this call based on it is a total red herring. The message was loud and clear, especially to Belichick's defense. Coach Belichick did not trust his defense. Period. Do you think he would have made the same decision if he had his defense from 2002?

Secondly, the numbers used are very dubious. To quote the article:

With 2:08 left and the Colts with only one timeout, a successful 4th-and-2 conversion wins the game for all practical purposes. A conversion on 4th-and-2 would be successful 60 percent of the time. Historically, in a situation with 2:00 left and needing a TD to either win or tie, teams get the TD 53 percent of the time from that field position.

There are a couple of things wrong with this.

  • The article qualifies the 53% with the fact that this was taken from the sample space of situations with 2 minutes left needing a TD to win or tie. But the 4th down statistic is taken without such a qualification. This already skews the numbers we need. This leads me to suspect that the 60 percent number was cherry picked to make the case stronger. Alas, I don't have any way to confirm this.

  • The 4th down probability of 60% is probably too high. First of all, this takes into account both runs and throws. Given that the Pats chose to throw, it was likely that their actual probability was lower. Add to that the fact that the Colts are playing at home with the crowd on their side. Add to that the Colts have a defense that's unbeaten at this point in the season. Throw in the unquantifiable factors. The Colts defense took it personally as a "diss" by Belichick. Add to that the fact that the Pats defense morale will take a hit, not just for this game but for the rest of the season.

So, let's recalculate the probabilities using some revised numbers. The two key probabilities here are the probability of converting the 4th down and the probability of the Colts scoring from that field position.

Here's the table if you change that probability:

60% -> 79% WP
50% -> 74% WP
40% -> 69% WP
30% -> 63% WP

And that is assuming that the Colts, at home can only score 53% of the time from the 34 yard line with the game on the line.

So let's change that value assuming that the 4th down conversion is at 50% (giving the Colts an advantage at home).

53% -> 74% WP
60% -> 66% WP
70% -> 62% WP

So if we assume a coni flip probability for 4th down conversion, and give Manning a 70% probablity of converting from that field position, then the probability of a Pats win is only 62%.

Since as we already noted that the numbers quoted in the article are NFL league-wide, then it's pretty safe to assume that the actual probability for both are higher than what is quoted in the article.

It's closer to a coin flip. It's certainly not the 79% winning probability that the article would like you to believe that the Pats had.

IMNSHO, it's almost 100% certain that Belichick screwed up this game and the rest of the season big time.

No comments: