Revisiting 2016: How well would my probabilistic model have performed?

24 July 2019

Updated 4 July 2020 to correct a bug in the nationwide electoral college simulation.

People frequently ask me whether I predicted Trump winning the 2016 election, and I tell them no, that my model gave Clinton a very comfortable victory. But then I mention that the same model predicted all 50 states in 2012.

Then I explain to them about how after 2016, I completely redesigned my model and how it incorporates uncertainties, etc., etc. So then the natural question to ask is, well, then how would it have done on the 2016 election?

Good question.

Especially because of the fact that the redesign happened because of 2016, I was curious myself as to how it would have measured up if it had been run in 2016.

So here's the answer.

Methodology: I ran the model on the polling data from 2016 on 15 battleground states along with ME-2 and NE-2. A weaker model was used on MN and NM due to absence of sufficient good polling data. The remaining 33 states and DC were assumed to be safely Democratic or Republican according to their partisan lean. So overall, my model was partially run on the overall election, which is probably what is going to happen when I run it on the presidential election next year, since few states have enough polling data for my model to work. Here were the results:

The following table compares the results of my 2018-OLS and LAD models to FiveThirtyEight's Polls-Only model and the RealClearPolitics average, with the actual result shown in the final column.

So even this model shows Clinton winning over 300 electoral votes. However, this is a closer prediction than what my old model said in 2016, which gave Clinton 347 electoral votes.

However, this is only half of the story. The other half of the story is the probability of the result:

The remaining 0.7% is the possiblity of a 269-269 tie.

The median result was Clinton winning 292-246.

The most likely result was Clinton winning 348-190, which is quite similar to what my old model predicted.

So these results now look a lot closer to FiveThirtyEight's election forecast, which gave Clinton a 71.4% chance of winning.

To compare this forecast with FiveThirtyEight and RealClearPolitics, I computed the total loss for each of the outcomes across these 15 states. Here, loss was defined as negative log-likelihood (NLL) for the probabilistic models and sum of squared errors (SSE) for the margins. For those who don't have a background in statistical modeling, remember that smaller loss is better.

Now let's compare our probabilistic forecast to the other ones listed on the New York Times's Upshot. The following table shows the NLL losses on the probabilistic predictions of various published forecasts. The data is collected from here.

Note that the columns of these two tables may not add up perfectly due since these presented numbers are rounded.

First, let's take a look at the NLL losses. They were the largest in Wisconsin, Pennsylvania, and Michigan, as we expected. We see here that the OLS model did relatively similar to (though slightly better than) FiveThirtyEight in these 15 states. The LAD model did much worse, but still beat all of the other predictions, including the Princeton Election Consortium.

Now consider the SSE losses. The models missed the most in Iowa, Michigan, Ohio, Pennsylvania, and Wisconsin all around. However, we also see something else in the SSE losses - both the OLS and the LAD models outperformed FiveThirtyEight significantly, though neither did as well as a simple average of the polls done by RealClearPolitics. However, one should bear in mind that the RealClearPolitics average includes lower quality polls that tend to oversample Republicans, and it's merely coincidence this election that the Republican significantly outperformed the polls. This phenomenon did not repeat itself in the 2018 midterms.

So here are the key takeaways:

1. The 2018-OLS would have performed similarly to FiveThirtyEight's polls-only forecast in terms of predicting the likelihood of a candidate winning the 2016 election.
2. Both the 2018-OLS and LAD models outperformed FiveThirtyEight's polls-only forecast in terms of predicting the margin of victory in the 2016 election.
3. So as we could see, none of the models showed Pennsylvania, Michigan, and Wisconsin all going to Trump. In fact, one could argue that it would have been almost mathematically impossible for an unbiased model to foresee this result just from the polling data alone, which showed Clinton leading in all three states. The bottom line is that a good model should not have shown that Trump would have won these states, but should have shown that he could.

We'll have to find out in 2020 whether or not this was a one-off occurrence, or whether my model holds in its ability to compare against the best election forecasts out there.