Kevin Marshall's America
Last night someone linked me to a story about same-sex marriage and in particular a recent poll showing that a slim majority of New Yorkers now opposed it.
I hit the brakes when I read that last part. For several weeks, independent polling organizations from throughout the State showed that not only did a majority of New Yorkers support same-sex marriage or marriage equality (whatever term floats your boat), but the support had been steadily trending upward for some time now. What changed in a week?
I looked into it and found out: nothing. Because the poll was commissioned by anti-same-sex marriage advocates the National Organization for Marriage, and the poll it self was conducted using some very questionable methods that I was taught to steer clear from when I briefly worked with a polling organization. Most damning of all, the organization only polled 302 people and tried to portray it as a state-wide consensus.
302. Out of all of New York State.
I wasn’t surprised, since there have always been polling organizations that play fast and loose with their methods to achieve desired results. I was surprised, though, that so many had fallen for it.
Polling, unfortunately, gets a bad rap. I have a unique perspective in that I spent one summer interning for the Director of Polling at the Siena Research Institute many years ago. I learned a lot from the Director, a guy named Joe Caruso who had formerly worked on Hillary Clinton’s successful Senatorial campaign. It also opened my eyes to how shockingly accurate polling could be, particularly when it came to elections. They didn’t just accurately predict winners, but also the margin. In one gubernatorial poll, when the “undecideds” were split in half and distributed evenly to the two candidates, it called the result to the exact percentage point.
One of the first things I learned on the job was that size matters…sample size, that is. In order to get an accurate number, pollsters need to ensure that the number falls within an acceptable threshold. Additionally, in order to maintain integrity and ensure fairness, any polling organization with any sense of credibility will go above and beyond to ensure the most accurate results possible. There were also other variables that needed to be accounted for: demographics, age, and geographic area needed to be accurately represented. So, for instance, if we had eight hundred responses and six hundred of them were from people under the age of 25 and all from the metro New York City region, you kept going until a balance was achieved. It was a meticulous system and there were complicated formulas which escape me now (and baffled me at the time – I wrote the press releases and helped provide analysis of the results since math was never my strong suit).
The end result was that, at minimum, we would poll no less than 800 people if our pool was likely registered voters and 1,000 if it was a poll of the general population. So for a professional polling organization to only poll 302 likely voters on a social issue is more than just skewed, it borders on fraudulent.
There’s also a matter of the wording. It didn’t ask if people supported or opposed same-sex marriage. Rather, it asked “Do you agree or disagree that marriage should only be between a man and a woman?” Many of you are probably asking “well, what’s the difference? Isn’t that just arguing semantics?”
Yes and no. If this were a conversation between you and I, in private, then it wouldn’t matter. That’s because for the most part, if we were having a private discussion we’d have some sort of candor and be comfortable enough with our views to honestly express them. However, in a line of questioning from a stranger, anything that might load it is going to skew the results.
Think of it this way: essentially, the real responses aren’t to the full questions, they’re to everything after “do you support or oppose” or “Do you agree or disagree”. If I ask “do you support or oppose same-sex marriage” you’re more likely to give me an honest answer, because the important phrase is “same-sex marriage.‘” That’s what you hear and react to. If I ask “Do you agree or disagree that marriage should only be between a man and a woman,” what they’re reacting to is the statement “marriage should only be between a man and a woman.” That’s heard as a statement or proclamation. The way we’re wired, we’re more likely to agree with the assertion of a stranger over the phone in order to avoid any potential awkwardness or discomfort. But if that same stranger is simply saying “same-sex marriage,” the tone of the question is neutral and the person will be comfortable giving their opinion no matter where they fall on the issue, because it is asked without any loaded terms or prequisites.
What also stuck out to me was the breakdown of the age of the respondents of this poll:
7% 18-39 years old
15% 40-49 years old
31% 50 ² 59 years old
38% 60+ years old
9% NO RESPONSE
Only 7% of the population under the age of 40. New York, apparently, is Bizarro Logan’s Run.
In all seriousness, it goes back to the two fundamental flaws of this poll: only asking 302 individuals and only sampling from likely voters. When conducting a poll to project results of an election or referendum, it makes logical sense to only poll likely registered voters, but when asking opinions on controversial legislation – especially social issues – a poll should sample the entire population. Politicians will cater to the segment of the population most likely to vote, which is usually older. But if they have any real expertise in their field, they’ll tell you the danger of only going to that pool on an issue like same-sex marriage. Social issues can, and have in the past, create and mobilize new voters, even if it’s just for a single election cycle.
So when you read a news article that cites a poll result, the first thing to look at is who did the poll. The first red flag will be that the actual organization conducting the poll isn’t mentioned in the lead (and especially if it isn’t at all). Then you need to look at what’s being asked, and who. Because when it comes to polling, the important thing isn’t the numbers. It’s what’s behind it.
Share it!
31 Responses to New poll claims New Yorkers oppose same-sex marriage, but is it true?
Leave a Reply
STALK ME
In some ways, Kevin is being too easy on the poll. The age breakdown shows that the number of respondents over the age of 60 is over 5 times the number in the 18-39 age bracket. The fact that the poll only samples voters doesn’t explain that much of the weirdness of that age distribution; I’m not sure even the small sample size does. In the 2008 elections, the number of voters in that younger age bracket still outnumbered those in the 60+ group, even if not by as large a factor as in the general population. That 5+ ratio of old to young is just so out of line that there must have been some other issue with the polling methodology than just sampling likely voters. (I’ve looked at similar issues in some of the other polls showing majority support for same-gender marriage, and the demographics in those polls match the demographics of the general population pretty well.)
Dave - Agreed. I left it out of the article itself, but I have a strong suspicion they made their calls during the working day (9am to 5pm).
i couold care less if they get married, however, i do think they should be allowed just like everyone else, cuz they work, they pay taxes, they put money into the economy, they eat, drink, n goto the bathroom, like every other human… the only difference is their sexual preference, and im not the person thats gonna go in to their bedroom and tell them how to do it… they are HUMAN like me, and you, they deserve the same rights…
50% of all stats are 100% made up….
Love the Logan’s Run referance…
what they were really selecting for wasn’t “likely voters”, but instead “likely anti-gay voters” and still they could barely get to 50%. it must really suck to be NOM.
It should also be noted that 67% said they were married, and another 17% said they’re single but previously married (i.e. divorced, separated, whatever). That means 84% of people polled are either married or were married, and therefore are (most, at least) straight and enjoy(ed) the benefits of “traditional” marriage, thereby taking for granted a right that so many others either never availed themselves of or are outright denied.
As comparison, according to statjump.com, 50.1% of New Yorkers are married, making NYS 50th on the list in the country:
http://www.statjump.com/lists/married-population-dp2c35ts.html
Yup. Gay marriage is still boring to blog about. Let’s switch it up Kev.
For the Troy Record/Gramercy Communications poll on the Troy mayoral race, they needed a sample size of at least 400 for the results to be statistically significant (They received more than 1,100), and that was just for one city. So, yeah.
Clarification – Asked in an e-mail but I’ll answer here. In regards to this:
A person asked how far a polling organization goes to achieve that balance. I wrote this piece this morning before work and didn’t want to include ALL the details lest it become too monotonous. But in short: what will typically be done is an organization will shoot for, say, 850 responses. X number of responses could come from one group – whether it’s in NYC, under or over a certain age, etc. – until that segment could no longer be polled because it would not be representative and would not be in proportion with what it is in the general population. So if they already got the number they needed from NYC, they’d no longer call that area, etc.
Regardless of what some politicians and judges might say, marriage is a biological relationship that can only exist between people of different sexes. Men and women are not only different anatomically, they have different biochemistries and scientific studies indicate their brains function differently. Research has established that male bodies produce chemicals (pheromones) that women benefit from. http://serendip.brynmawr.edu/bb/neuro/neuro03/web2/kshiner.html
Research on female pheromones tend to focus on how they attract men, but studies indicate men benefit biologically from marriage.
http://www.themedguru.com/20110128/newsfeature/marriage-good-mens-physical-womens-mental-health-86143466.html
Marriage binds men and women together as a means of facilitating reproduction not so they can just have someone to live with.
Governments have traditionally recognized and protected marriage to encourage people to produce new members for society not to give any special right to heterosexuals.
What’s a “special right” in this instance?
Reasonmclucus – It should be noted, too, that the scientific evidence you presented doesn’t really relate to marriage. Marriage is a societal construct, whereas you’re talking about the act of heterosexual organisms reproducing sexually. As for men benefitting biologically from marriage, is it just marriage to a woman that would be biologically beneficial?
Others have pointed out that’s not always the case, but they’re wrong, because it’s actually not the case at all. Reproduction can and is facilitated in humans, along with many other biological organisms, without two exclusive partners co-habitating. It can be, but is not always, innate, but rather habitual.
Reasonmclucus, that’s actually more of a reason to promote same sex marriage because we no longer need nor want excess population growth.
I am married without children as are many of my friends. Is my marriage less valuable to you because I haven’t popped out babies? If that is indeed what you believe,I am very offended by that.
As a woman, my purpose in life is NOT to make babies; it’s whatever I chose it to be.
@Reasonmclucus — Procreation is not a prerequisite to marriage, and actually it will proceed just fine without government support. And you have totally ignored the importance of strong families to society when it comes to raising children … something of greater value than procreation and something that is as true of families headed up by same-gender couples as those headed up by straight couples. And while what you say about biochemistry and brain function has some validity when it comes to straight men and women, there are parallel things that could be said about homosexual men and women. So that argument might backfire on you. The major mental-health organizations all support same-gender marriage as a good thing for the couples, any kids they might have, and for society in general.
@10 – isn’t that basically the same argument as elderly people or sterile people shouldn’t be allowed to get married?
Reasonmclucus-It’s interesting that you quote scientific studies yet completely ignore the science of gender itself. One in every 2000 humans is born of indeterminate gender. http://www.babymed.com/congenital-malformation/indeterminate-sex. Statistically, that breaks down to about three million people on the planet.
Shouldn’t that cause you to rethink your argument?
I would let zebras marry dolphins if it meant we could move past this issue and get to fixing REAL problems, e.g. how come I can’t get no Tang no more?
Apologies if this appears as a double post- Governments have traditionally recognized and protected marriage to encourage people to produce new members for society not to give any special right to heterosexuals.
Actually, heterosexual couples already have these rights reasonmclucus. What we need is acceptance for homosexual couples who have been around since the Egyptian times, just under the radar, who now wish EQUAL rights, not special rights. sigh… I feel like a broken record. Marriage is a LEGAL contract. When the ceremony is performed by a religious cleric he/she is actually, out of convenience, performing two tasks. 1- the legal one for the government binding a couple together. Your marriage license! Historically performed by religious clerics, yes, before separation of Church and State. The deal was sealed with an os- kiss swearing fealty to each other. 2- the religious sacrament. Think about it. Where do you go for a divorce? A court of LAW. The Church has nothing to do with this proceeding at all! Only those who are religious, also seek something separate and from the Church- an annulment or something similar. The GOVERNMENT has NOTHING to do with this at all! Come on folks… The Church is SEPARATE from the state and folks keep blurring this fact! Who care about the DNA make up of the couple. As far as I am concerned they only need love each other and have a marriage license. If the Church does not wish to marry two women, fine. The women just need a Justice of the Peace and a beautiful park or City Hall. Same with two men. Let there be peace in our world and lots of loving individuals rather than hate, intolerance, and continued lack of respect for the feelings of others.
Hold on here! let’s look at the age old proverb:
FIRST COMES LOVE,THEN COMES MARRIAGE, THEN COMES THE BABY IN THE BABY CARRIAGE….
case closed.
Hal, maybe you’re just too old for “poon tang” :o)
And some adoption agencies allow homosexual couples to adopt : P It is done as a single person and not a couple. One person in the couple adopts the child and bingo! A same sex couple has adopted a baby!!! So it can also fit the old proverb. Or, they hire a surrogate, ect… also making it fit the proverb. Loopholes are such fun!
Ok Pat,
FIRST COMES LOVE…THEN YOU ADOPT A CHILD, THEN YOU RUN OVER TO VERMONT AND GET MARRIED…OR MAYBE ny, DEPENDING ON HOW THE POWERS THAT BE DECIDE ON YOUR RIGHTS…
Doesn’t flow as well… ;)
@Tony Barbaro, #22, as a New Yorker, I take offense to your argument against the economic prosperity of our state. When weddings go to Vermont, the money spent on those weddings goes to Vermont as well. And this is coming from a former Vermonter.
@Tony Barbaro
Traditions are not inherently “right”.
Up until 1865, people in the South could have made arguments that, well, hey, slavery had been going on for thousands of years, so therefore there’s nothing wrong with it.
I’ve received three robocalls this week from these people.
They claim they’re conducting a poll, but when I say “no” i
in response to their loaded question, they end the call
without asking me my age or trying to get any other
information. By the way, I’m 77 and have no problem
with same-sex couples getting married.
JayK, I think more money is spent in Vermont shopping in Manchester or skiing at Jay, Bromley, Stowe, Killington, etc, than at homosexual receptions…
@ #19, Tony, your dead on accurate with that one.
Put away your torches and pitchforks guys..just trying to lighten the mood….
@Jakester, So we should work harder on our skiing industry instead? Either way, it does not make economic sense to encourage any money to leave our state. In the end, the economic arguments are just icing on the cake. Civil rights, equality, family values and freedom of religion are much more pertinent issues supporting same-sex marriage.
Oh! Oh! You guys!
I’m visiting my grandmother, and she tells me that NOM has been calling her repeatedly for this supposed survey, much to her irritation.
Just now, I answered the phone and lo and behold, it was Auto-Brian Brown.
I answered “yes” to the “between a man and a woman” question just to see what would happen.
Well surprise, surprise, they don’t follow the line of questions that they published with their “study” at all.
ABB then says something like “THANK YOU! For standing up for traditional marriage! as you know… blah blah blah… something about traditional marriage being threatened… Would you be willing to contact your New York State senator to urge them to vote no on the Same-Sex Marriage legislation?”
No.
“Would you be willing to to further support traditional marriage in the form of a monetary contribution to National Organization for Marriage.”
NO.
“Thank you for your time, for more information, visit blah blah blah, NOM’s website…”
As if it wasn’t transparent enough how illegitimate this study was… at the heart of it, it’s just NOM’s way of harassing and scaring (mostly elderly) people into giving them more money. How desperate.
And THIS is the poll that Sen. Diaz and Michael Long are using to justify voting down the measure, in disdain for all other legitimate polls that say otherwise. Foolish.
Wow, Victoria! That’s disgusting. I’ve been wondering what the procedure was since the post days ago on Libby’s blog. Thanks for sharing that!