Detroit Tracking Poll

by: Grebner

Tue Mar 17, 2009 at 01:45:51 AM EDT

May 6 Postmortem:  Maybe I need to find a new line of work.  Polling in Detroit is always a little dicey, but this election should have been easy.  Only 94,000 people voted (compared to 92,000 in the February's primary) and we had their names before we drew the sample.  So we were calling the right people.

In the primary we used the same recorded voice, the same wording, the same selection process.  And our results lined up almost perfectly with the actual vote total.

So why were we off by fully seven percent in the general election?  There was enough data (from almost 2000 completed interviews) to plot an apparent shift from Bing toward Cockrel over the period of two months.  The calculated margin of error was either four or five percent, depending on the assumption used, so we missed it by more than a hair.  Two other polls, using different methods, came to the same conclusions, and were just as wrong.

Honestly, I'm baffled.  I can't even think of a plausible excuse, although I hope to invent one eventually.  Here are my first efforts:

Theory #1: There was a late shift toward Bing,driven by endorsements from Hendrix, Jesse Jackson, and both major papers.  Problem:  why didn't this last-minute surge show up in the polling, which continued untiil the night before the election?  Why did Bing win among absentee voters, whose ballots were cast three and four weeks ago?  [UPDATE: It turns out Cockrel carried the absentees by several thousand votes, while Bing won the election day voters by roughly 7000.  Early media reports were completely wrong.  This correction adds weight to the late-shift hypothesis.]  Still, this deus ex machina was the answer put forth by Bernie Porn, at EPIC-MRA.  

Theory #2:  The election was racially charged, and people were reluctant to reveal their true preferences.  This theory almost makes sense, since Bing appeared to be draw disproportionate support from younger black voters, and our recorded voice was obviously non-black.  In 2005, Kwame's support was much larger than any poll picked up, apparently because some voters were reluctant to admit they planned to vote for him.  Problem: there was not trace of such an effect in the primary, and unlike 2005 (which had 210,000 votes) we had no turnout surge whatever.  Our polling showed Bing with a healthy lead in March, which gradually seemed to slip away - but then reappeared on election day.  Why would our 90,000-odd voters suddenly become shy about admitting they were voting for Bing?

Theory #3:  Something weird was going on through the entire process.  As I noted during the polling, our number of completes dropped dramatically from the primary and into March.  Since the samples were identical, and we called at the same times of day, using the same wording and voice, it simply made no sense that the completion rate should drop by 30%.  And on election day, a huge number of people simply didn't show up - people whom my model was counting on.  Were those mainly Cockrel supporters?  In other words, did something somehow create a huge number of lukewarm Cockrel supporters, who were barely able to press a number in response to my polling, but not to get off their rears on election day?  It should be possible to test this theory when we get the records next month.  It doesn't seem obviously implausible, but the only thing in its favor is that it ties everything together.  There's no particular reason to believe it's true.

Oh - the Charter Revision passed 79% to 21%, compared to my final projection of 72%. In the case of a ballot proposal, as I've said repeatedly, polling isn't generally accurate anyway, so I don't mind missing by seven points.

  Individual Days' Results          Four-panel moving average

             Bing  Cockrel  total           Bing%    Cockrel%   period

 5/4        39     39          78            45          55          4/27 - 5/4

 5/2        35     38          73            45          55          4/25 - 5/2
 4/29      40     47          87            44          56          4/22 - 4/29
 4/27      35     58          93            46          54          4/20 - 4/27
 4/25      43     44          87            47          53          4/18 - 4/25
 4/22      37     51          88            48          52          4/15 - 4/22
 4/20      47     41          88            48          52          4/13 - 4/20
 4/18      31     43          74            47          53          4/11 - 4/18
 4/15      49     44          93            50          50          4/8 - 4/15
 4/13      37     48          85            49          51          4/7 - 4/13
 4/11      39     38          77            52          48          4/4 - 4/11
 4/8        45     43          88            52          48          4/2 - 4/8 
 4/7        46     44          90            55          45          3/31 - 4/7
 4/4        45     38          83            57          43          3/28 - 4/4
 4/2        39     34          73            57          43          3/25 - 4/2
 3/31      46     27          73            56          44          3/23 - 3/31 
 3/28      55     40          95            53          47          3/21 - 3/28
 3/25      51     46          97            56          44          3/18 - 3/25
 3/23      50     44          94            56          44          3/16 - 3/23
 3/21      51     53        104
 3/18      69     30          99
 3/16      53     45          98

 TOTAL  982   935       1917

             YES    NO       total            YES%       NO%          period

 5/4         54    16          70              72           28          4/27 - 5/4

 5/2         37    15          52              73           27          4/25 - 5/2
 4/29       57    14          71              73           27          4/22 - 4/29
 4/27       49    31          80              69           31          4/20 - 4/27 
 4/25       55    15          70              74           26          4/18 - 4/25
 4/22       61    21          82              71           29          4/15 - 4/22
 4/20       53    30          83              69           31          4/13 - 4/20
 4/18       52    11          63              69           31          4/11 - 4/18
 4/15       49    25          74              66           34          4/8 - 4/15  
 4/13       50    24          74              66           34          4/7 - 4/13 
 4/11       32    22          54              65           35          4/4 -  4/11   
 4/8         52    24          76              68           32          4/2 - 4/8  
 4/7         59    30          89              67           33          3/31 - 4/7
 4/4         45    26          71
 4/2         41    14          55
 3/31       41    23          64

 TOTAL   787   341       1128

March 16:  I've launched another round of robo-polling for the Detroit Mayoral Special General Election - I'll post new data three times a week.

But I'm mainly planning to use the data to deliver a very simple introduction to poll analysis.  I'll pitch it at the level of people who read the margin of error listed in newspaper polls and take them at their word.  If you already know that squaring a correlation coefficient yields the proportion of variance explained, you're welcome to nitpick me,  but you're not going to find anything new. 

Obviously, 98 responses doesn't give us much to work with.  The usual response would be to throw up our hands and say "It's too close to call." and leave it at that.

But let's ask - why is it too close to call?  What measure do we use?  When we look at a simple head-to-head race, we get a very good approximation of "the margin of error" by comparing the difference between the candidates' totals (in this case, 53 minus 45 or 8) to the square root of the sum of the candidates' responses (45+53 = 98, whose square root is about 9.9).  Notice we're using the actual number of responses, NOT the percentages in this calculation.

To get a "statistically significant" difference, we want to see the difference be at least 2.0 times the "standard error" which is the 9.9 we calculated above.  You can think of the standard error as (more-or-less) the "average error" that results from this sample size.

Since the actual difference we saw between Bing and Cockrel was 8 responses, and "on the average" we'd see a difference of 9.9 responses in a poll of 98 people, what we see here is too small to mean much at all.  Which is exactly what you'd expect, just looking at how small the sample was.  

In a couple of days, with twice as much data, the standard error and the "margin of error" will both shrink by about 30%.  Since the candidate percentages will change as well, we may see Bing take a real lead - or we may find out that it's a dead heat.

The one thing we can say, even with such a small sample is that Cockrel certainly doesn't have a large lead in the race right now - but for all we know he may have a small one.

March 18:  Interesting results from today's calls:  Bing did MUCH better than yesterday, enough to pull into a clear lead.

Two days ago, using a fairly crude analytic method, we concluded that the difference between the two candidates was far short of statistical significance.  Today, partly because we have twice as large a sample, but mainly because today's sample more heavily favored him, Bing can be seen to have a clear lead.

Bing's lead is now 47 votes, which is 3.3 times as large as the estimated standard error (square root of 197 results, or 14.0) which should happen less than one time in a thousand through chance alone.  We should always be wary of taking a particular surprising result and applying standard statistical tests, if the surprise was the cause of our attention.  But I'm not worried in this case, given I had announced my plan to report the results here as they become available, as well as the fact the result is much larger than bare statistical significance (1.96 times the standard error.)

We might worry that one night's results might be influenced by some specific transient factor that specifically favored one of the candidates (for example, the television schedule might have made some group in the population more likely to be at home, or more reluctant to interrupt their viewing to answer the phone.)  Because we didn't see such effects in the primary, that seems somewhat unlikely.  In any event, the next set of results - which I expect will also favor Bing - will probably clarify things.

At this moment, it seems fair to say that Bing appears to have a clear lead.  Of course, we don't know whether that lead will hold up over the course of the campaign and as each candidate is tested by events and the media.  But Bing does have a lead.

A very interesting statistical question is whether the two nights' results are DIFFERENT from one another.  That is, is the fact Bing got 54% one night and 70% two days later, too big a swing to be caused by chance alone?   Admitting - again - there's the danger of "cherry-picking" interesting results, let's test the DIFFERENCE between the two nights's results.

There's a very nice method of comparing the results reflecting two candidates and two nights.  First we make the results into a 2x2 table:

69 30 

54 48 

We calculate what is called an "odds ratio" by multiplying the numbers on one diagonal (69 times 48) and dividing by the numbers on the cross-diagonal (30 times 54). If Bing had gotten the same percentage each night, this number would be very close to 1.0, but for this table, the actual ratio is 2.04.  We now take the "natural log" of that ratio, to find the "log-odds" or "logit".  (Don't worry, none of this will be on the test.  You can find the natural log by typing ln(69*48/(54*30)) into Google's search screen - Google's calculator is case-sensitive.  You should get 0.71)

The larger the log-odds, the clearer that something has really changed from one night's calls to the next.  The standard error of the logit is simply the square root of the sum of the reciprocal of the cell counts.  In this case sqrt(1/69 + 1/30 + 1/54 + 1/48)  or 0.30.  (Again, Google will give you the answer, if you're careful about parentheses and using all lower-case letters.)  

Dividing the table's log-odds (0.71) by the standard error (0.30) yields 2.42, which is pretty strong evidence that the two nights' results really were different. I'm personally skeptical, because I don't know what could have caused that change in a period of two days.  In any event, if Cockrel doesn't do dramatically better soon, he'll be out of the race.

To sum up, if we want to analyze this 2x2 table:

    A  B

    C  D

We calculate the log-odds by:   ln(A * D / (B * C))

And the standard error by:  sqrt( 1/A + 1/B + 1/C + 1/D)

If the log-odds is more than 1.96 times as large as the standard error, we can say "It's significant at the 0.05 level."

March 21:  Today's lesson is that if you conduct small numbers of surveys, on different days, you'll see a lot of fluctuation, and it won't be possible to tell what's a real change and what's just statistical jitter. 

Even knowing this, two days ago I was convinced by Bing's extremely strong one-day showing that he had a clear lead over the incumbent.  Today's results serve to remind me to reserve judgment.

Since Bing's cumulative lead (45) is well over twice as large as the square root of the total number of responses (sqrt(337) = 18.3) it's still clear that Bing is leading, but it no longer looks like a blow-out.

Or - using the method I prefer:

ln(191/146) / sqrt(1/191 + 1/146) = 2.44

which says the same thing.

Starting Tuesday, I'll post 4-day moving averages, which are a much more reliable indicator of real change than looking at single day results.

March 23: Nothing much to report.  

One interesting thing is that I'm only completing about 100 calls out of the 1000 I call each day, compared to an average of about 125 in the primary.  Since the sample is very similar to the one I used in the primary, and the wording and voice are identical, it suggests that the candidates aren't really connecting with people.

I would have expected the number of completed interviews to rise, not fall, as the voters' choice became more focused.  But what seems to be happening is that the people who previously prefered one of the other candidates are being left cold by the choices they face now.

I'm sure the turnout in May will be higher than it was in the primary, but maybe it won't be as big an increase as I had thought.

 March 28:  It's not completely clear, but today's results shift the balance toward believing that Bing has a small-ish lead.  The four-panel moving average actually shifts toward Cockrel, because Bing's one blow-out result gets dropped from the calculation, but if we had used either three or five days' results, Bing would have a significant lead.

I was asked to add a question about the Detroit City Charter reform proposal, which shares the ballot with the mayoral special election.  I was tempted to refuse, because of my belief that phone polls are completely inappropriate for predicting the outcome of ballot proposals.  (As I've written earlier, the only method that seems to work involves casting a dummy written ballot.)  But it occurred to me that my mission here isn't to predict the outcome of an election, but to demonstrate polling technique.  And what better method than to run two simultaneous polls, one for the candidates (which will probably predict the outcome within four or five percentage points) and one for the proposal (which will probably be way off.) So - in the interest of scholarship - I'll begin reporting the results of the ballot question, probably by Wednesday evening.

March 31:  A second consecutive good day for Bing.  Using my simple assessment tool, over the past four days Bing has led 202-157, and 

ln(202/157)/sqrt(1/202+1/157) = +2.37 SE's.

which is a pretty good lead in my book.  Today's anomaly is that we only completed 73 usable interviews, although the sample was indistinguishable from the previous samples, and the script was almost identical.  Whatever it means, it certainly doesn't suggest either candidate is setting the city on fire.  (That's probably an unfortunate phrase, but so be it.)

Also, as promised, I've begun to poll attitudes toward the Charter Revision question.  Based on very small numbers, from just one day, it did very well: Yes-41 to No-23.  (Using the same technique above,  it's at +2.22 SE's.)  But how a proposal fares in a robo-call poll has no predictable relation to how the same voters will eventually vote when they see it on the ballot.

April 2: We appear to be settling into a clear pattern: Bing has a moderate, but substantial lead, with day-to-day fluctuations which are likely just due to sampling variations.  The most intriguing fact is the low rate of completion, which appears to be DROPPING as the election approaches, rather than rising.

As far as the Charter Revision question, if I were doing this for a paying client, I'd be tempted to tell them to stop polling and devote their money to something more useful.  Even though we have only 119 responses, it's clear that there's overwhelming public support - to the extent people understand the question they're being asked.  Obtaining more data (which we'll do, in spite of its uselessness) will narrow the uncertainty of the estimate, but it won't make this poll any better a predictor of the final outcome.  Today, we're at 68% +/- 9%.  A week from now, if we happen to be at 72%, or 62%, what difference will it make?  We can say that likely voters like the sound of the proposal, but will they like the sight of it?

April 4:  Another day, another confirmation of what we already saw.  Notice how the individual days' results jump around, because of their small-sample jitter, while the moving average is steady as a rock.

In just under three weeks, we've collected 800+ responses, which give Bing an average of 56%.  From this data, there's no evidence of any trend over time; that is, the overall average seems to be just as valid a predictor of the outcome as the most recent four panels.  That's basically what we saw in the primary: that the campaigning and each day's news don't seem to affect any actual votes.  If that's a correct reading, it's bad news for Ken.

One interesting tidbit, which I mentioned previously, is the apparent decline in the number of responses to the poll.  I don't think there's any reason connected with the poll itself to explain the drop.  All the samples were drawn at the beginning, and they're non-overlapping and statistically identical.   The same recording is used, with the same set-up parameters.  We call at the same times of the day.  And we're getting the same results as far as candidate preference.  So how do we analyze the drop-off in interest?

This sort of random surprise arises frequently in polling - a pattern attracts our attention that we hadn't expected, and so we don't have any pre-planned method of analysis.  The first thing to do is to be skeptical;  because we are only looking at it because it seems weird, it's likely to be pure coincidence.  But we don't have to reject every unexpected pattern;  we just need to set a higher standard before we accept it.  Does it continue to appear AFTER our attention is drawn to it?  Does it disappear if the data is analyzed using a different technique?  Does it reach not just the regular 2-standard-error threshold, but even 3-standard-errors?  Does the pattern seem logical, now that we've had a chance to consider possible causes?

In this case, the falloff in completion rate is only about 2.4 SE's, using various modes of analysis.  It will be easy to see over the next four weeks whether the pattern is maintained, or not.  It seems consistent with the much larger number of completions we saw during the primary, when we placed the same number of calls, used a very similar script, and targetted a very similar universe.  In the primary, every single panel returned 103 to 143 responses.  From the special general, the range has been 73 to 104 - and declining.

The message may be that nobody much cares about the outcome, and that the turnout won't be as large as we might have guessed. 

 April 7:  No news at all.  I'm not even going to bother bumping it to the front page.

April 11: Another day which undermines Bing's lead without disproving it.

April 13:   There's (maybe) something happening here - what it is ain't exactly clear.  Today, Cockrel actually out-polled Bing for the first time in three weeks.  Averaged with small Bing leads in each of the three previous panels, that moves our four-panel average to a tiny (and non-significant) lead for Cockrel.  We are left with mud.

Certainly, we can't take this as evidence that Cockrel is "really" leading.  But there's also no reason to believe that Bing is still leading.  Looking only at the last four panels, which break in Cockrel's favor, 173-to-167, all we can say is that each of the candidates is somewhere between 45% and 55% in a head-to-head race, and that our sample isn't large enough to say more.

But if we look at the larger collection of data, from all thirteen days, we need to ask whether Bing is slipping.  Not whether he's trailing, because we don't have evidence of that, but whether the lead he showed in the beginning has now either diminished or gone.  On that question, the evidence is ambiguous; depending on exactly which comparison we examine, the change is either barely statistically significant, or slightly below the significant level.  Rather than pick one comparison as the "right" one, I just repeat that the election is still three weeks away, and we'll have more data soon.

If we ran a much larger survey, say harvesting 300 completes per day, we'd be able to answer these questions definitively.  But if we ran a larger survey, and we were still paying for it out of our own pockets, even the better data might not seem like a bargain.  Information costs money.

Even if we can't draw any hard conclusions, Cockrel ought to be happy about these results, and they should worry Bing. 

April 15:  More fog.  A better showing for Bing, which merely underlines that we don't know if we've seen a slight trend toward Cockrel, or if we're just staring at statistical noise and imagining that we're seeing a pattern.  Once we perform the calculations and realize that we're between one reality and another, statistical tests don't really offer a map for getting out.  More data may help, but as of today, it's possible to make one analysis showing that Bing holds a slight, steady lead and an equally valid one that shows Cockrel is gaining on him. 

An objective pollster, looking at this data, should report:  "I dunno.  Hard to say.  Maybe Bing has a slight lead.  Maybe Cockrel is gaining on him.  Maybe not." 

On the Charter Amendment front, the situation is the normal one for cheapie polling:  one side has a large stable lead, and spending more money to get better data would be an utter waste.  If anybody wonders what the public thinks of the proposal, they really can't be concerned whether it's supported by 65% versus 70%.  As I've said many times, the important question is how they feel when they READ the ballot language, which is likely to bear only a very rough relation to what they answer to a telephone survey.

April 18: Finally, NEWS!   Cockrel's strong showing today finally makes clear that the race is shifting in his favor.

To be more precise, Cockrel was clearly losing when I started surveying, last month.  Now, it's a reasonably close race, and he's doing better than he was, but it's not possible to say from this data whether either candidate has a small lead (say, 55% to 45%).

The reason I feel confident in saying the race has changed is the result of a simple calculation:  I compared the first five panels with the most recent five panels.  (I ignore the midddle five panels for this comparison - no matter what they show, they don't make much statistical difference when we're looking for a trend.) 

Thus, in the first five panels, Bing led 274-to-218.  The totals from the most recent five panels were 201-to-216.  (Note that the first shows a statistically significant Bing lead, while the latter does NOT show a significant Cockrel lead.) 

Using my favorite statistical technique, the logit of the table is

Ln(274*216/(218*201)) = 0.301

The SE of the logit is

Sqrt(1/274 + 1/216 + 1/218 + 1/201) = 0.134

So the shift in the log-odds is about 2.25 SE's, which makes me feel good about it.  So does the fact that using different collections of dates doesn't dramatically change the result.  In other words, the evidence of a shift is "robust", that is, it stands up even if we use different analytic technique.

In other news, the absentee ballots are clearly being returned much faster than during the primary - only a week after being mailed, 15000 have already been returned to the Clerk. 

April 20: New data, but no new conclusions.  Maybe in a few days I'll start looking at crosstabs between the responses and underlying demographic data.  Or maybe I'll be too lazy - we'll see.  

April 22:  Nothing to report except that Cockrel had another good night, adding evidence that Bing's lead is either slipping away or already gone.  21000 absentee ballots have already been returned.