2018 Midterms: How well did we do?

7 August 2019

Modified 6 October 2019 to update for 2020-NN model and a bug in the Governors NLL formula.

These retrospectives are meant to dive into the numbers and see how well we did, particularly in comparison to some other models. Here, loss functions are used to compare different models. For our purposes, loss is defined as negative log-likelihood (NLL) for the probabilistic models and sum of squared errors (SSE) for the deterministic margins. For those who don't have a background in statistical modeling, remember that smaller loss is better.

Summary

In the Senate, my models outperformed FiveThirtyEight's Lite and Classic models. However, in the gubernatorial elections, FiveThirtyEight's models did slightly better than mine in the probabilities and in overall margins, though deterministic models did slightly better in most states with a lot of polling data. Also keep in mind that my probabilistic models require more polling data than FiveThirtyEight's models do.

2018 Senate - Probabilities

Probabilistic models are evaluated by negative log-likelihood.

State	2018-OLS NLL	2018-LAD NLL	FiveThirtyEight Classic	FiveThirtyEight Lite
Arizona	0.49	0.49	0.49	0.46
Florida	1.37	2.12	1.22	1.28
Indiana	0.81	0.77	1.27	1.24
Michigan	0.05	0.01	0.04	0.03
Missouri	0.54	0.49	0.84	0.77
Montana	0.23	0.12	0.27	0.34
Nevada	0.23	0.04	0.56	0.57
New Jersey	0.02	0.00	0.06	0.06
North Dakota	0.01	0.00	0.31	0.11
Tennessee	0.21	0.09	0.22	0.23
Texas	0.01	0.00	0.24	0.21
TOTAL	3.98	4.12	5.52	5.29

State	2020-NN NLL	FiveThirtyEight Lite NLL	FiveThirtyEight Classic NLL
Arizona	0.57	0.46	0.49
Connecticut	0.00	0.00	0.00
Florida	1.39	1.28	1.22
Indiana	0.78	1.24	1.27
Massachusetts	0.00	0.00	0.00
Michigan	0.02	0.03	0.04
Minnesota	0.00	0.00	0.00
Minnesota	0.12	0.09	0.08
Missouri	0.55	0.77	0.84
Montana	0.23	0.34	0.27
Nevada	0.24	0.57	0.56
New Jersey	0.03	0.06	0.06
North Dakota	0.02	0.11	0.31
Ohio	0.09	0.02	0.03
Pennsylvania	0.01	0.03	0.03
Tennessee	0.15	0.23	0.22
Texas	0.10	0.21	0.24
West Virginia	0.04	0.15	0.13
Wisconsin	0.02	0.02	0.02
TOTAL	4.36	5.62	5.82

2018 Senate - Margins

Deterministic models are evaluated by sum of squared errors.

State	2018-OLS SSE	2018-LAD SSE	2012 SSE	2018-M SSE	FiveThirtyEight Classic SSE	FiveThirtyEight Lite SSE	RealClearPolitics SSE
Arizona	1.69	1.96	2.56	1.69	0.36	0.25	10.89
Florida	9.61	13.69	10.24	10.24	10.89	10.24	6.76
Indiana	54.76	46.24	64	79.21	92.16	79.21	43.56
Michigan	0.81	5.29	22.09	6.76	23.04	18.49	3.61
Missouri	26.01	23.04	29.16	36.00	50.41	42.25	29.16
Montana	1.44	4.84	0.49	0.81	2.89	0.16	0.04
Nevada	3.61	0.36	5.29	9.00	15.21	16.00	25.00
New Jersey	1.21	24.01	2.56	13.69	0.81	0.36	0.01
North Dakota	0.09	1.69	0.01	1.00	40.96	5.76	4.00
Tennessee	34.81	23.04	29.16	14.44	29.16	37.21	31.36
Texas	18.49	12.25	16.81	19.36	5.29	7.29	17.64
TOTAL	152.53	156.41	182.37	192.20	271.18	217.22	172.03

State	2012 SSE	2018-M SSE	FiveThirtyEight Classic SSE	FiveThirtyEight Lite SSE
Arizona	2.56	1.69	0.36	0.25
Connecticut	1.00	15.21	1.96	1.21
Florida	10.24	10.24	10.89	10.24
Indiana	64.00	79.21	92.16	79.21
Massachusetts	0.09	0.64	1.69	2.56
Michigan	22.09	6.76	23.04	18.49
Minnesota	0.01	3.61	6.76	13.69
Minnesota	3.61	6.76	0.81	4.41
Missouri	29.16	36.00	50.41	42.25
Montana	0.49	0.81	2.89	0.16
Nevada	5.29	9.00	15.21	16.00
New Jersey	2.56	13.69	0.81	0.36
North Dakota	0.01	1.00	40.96	5.76
Ohio	10.24	21.16	26.01	30.25
Pennsylvania	5.29	4.41	2.25	1.44
Tennessee	29.16	14.44	29.16	37.21
Texas	16.81	19.36	5.29	7.29
Virginia	0.64	0.09	0.01	2.56
West Virginia	27.04	14.44	18.49	20.25
Wisconsin	0.04	0.04	4.41	1.96
TOTAL	230.33	258.56	333.57	295.55

2018 Governors - Probabilities

State	2018-OLS NLL	2018-LAD NLL	FiveThirtyEight Lite NLL	FiveThirtyEight Classic NLL
Arizona	0.00	0.00	0.01	0.01
Connecticut	0.47	0.29	0.25	0.24
Florida	1.44	2.15	1.68	1.48
Georgia	0.66	0.44	0.39	0.39
Michigan	0.06	0.05	0.08	0.07
Nevada	0.43	0.33	0.64	0.66
Tennessee	0.04	0.03	0.01	0.02
TOTAL	3.10	3.28	3.05	2.87

State	2020-NN NLL	FiveThirtyEight Lite NLL	FiveThirtyEight Classic NLL
Alaska	0.23	0.40	0.37
Arizona	0.01	0.01	0.01
California	0.00	0.01	0.01
Colorado	0.12	0.12	0.05
Connecticut	0.31	0.25	0.24
Florida	1.51	1.68	1.48
Georgia	0.59	0.39	0.39
Iowa	0.77	0.86	0.85
Massachusetts	0.00	0.00	0.00
Michigan	0.04	0.08	0.07
Minnesota	0.12	0.10	0.07
Nevada	0.39	0.64	0.66
New Hampshire	0.33	0.27	0.15
New Mexico	0.04	0.07	0.06
Ohio	1.59	1.09	0.90
Oklahoma	0.15	0.33	0.15
Oregon	0.32	0.36	0.19
Pennsylvania	0.00	0.01	0.01
South Dakota	0.35	0.81	0.46
Tennessee	0.01	0.01	0.02
Texas	0.00	0.00	0.01
Wisconsin	0.26	0.35	0.52
TOTAL	7.13	7.85	6.69

2018 Governors - Margins

State	2018-OLS SSE	2018-LAD SSE	2012 SSE	2018-M SSE	FiveThirtyEight Lite SSE	FiveThirtyEight Classic SSE	RealClearPolitics SSE
Arizona	0.64	0.01	0.49	0.81	1.69	1.44	1.96
Connecticut	0.81	14.44	2.89	16.81	3.24	4.84	2.56
Florida	12.96	22.09	14.44	11.56	22.09	21.16	16.00
Georgia	1.69	0.25	0.49	0.16	0.81	0.64	2.56
Michigan	0.16	0.01	1.00	0.25	0.25	1.44	2.25
Nevada	6.76	3.24	3.61	4.41	14.44	15.21	22.09
Tennessee	73.96	106.09	31.36	20.25	37.21	47.61	44.89
TOTAL	96.98	146.13	54.28	54.25	79.73	92.34	92.31
Without TN	23.02	40.04	22.92	34.00	42.52	44.73	47.42

State	2012 SSE	2018-M SSE	FiveThirtyEight Lite SSE	FiveThirtyEight Classic SSE
Alaska	21.16	2.25	15.21	20.25
Arizona	0.49	0.81	1.69	1.44
California	43.56	73.96	42.25	27.04
Colorado	3.61	2.25	0.64	15.21
Connecticut	2.89	16.81	3.24	4.84
Florida	14.44	11.56	22.09	21.16
Georgia	0.49	0.16	0.81	0.64
Iowa	11.56	16.00	16.81	16.00
Kansas	30.25	30.25	16.00	33.64
Maryland	26.01	12.96	28.09	16.81
Massachusetts	33.64	54.76	36.00	0.16
Michigan	1.00	0.25	0.25	1.44
Minnesota	6.76	6.25	8.41	1.69
Nevada	3.61	4.41	14.44	15.21
New Hampshire	0.36	2.89	1.21	4.00
New Mexico	51.84	51.84	26.01	23.04
Ohio	70.56	39.69	44.89	33.64
Oklahoma	49.00	82.81	57.76	24.01
Oregon	4.41	1.21	4.84	0.16
Pennsylvania	9.00	1.96	24.01	17.64
South Dakota	3.61	1.96	19.36	0.81
Tennessee	31.36	20.25	37.21	47.61
Texas	16.81	22.09	12.96	12.96
Wisconsin	3.61	7.84	6.76	0.25
TOTAL	440.03	465.22	440.94	339.65