• No Purpose: A lot of market research asks the wrong questions.  Companies often follow a standard template and ask generalized questions.  Instead, clearly articulate your Guiding Research Questions and only ask questions that give you answers you can use.

• Not Doing Good Secondary Research: Many companies fail to look at existing available research.  There’s no need to pay to conduct your own primary research if there’s existing data.  Do a thorough search before designing your own research plan. Make sure to check dates and validity of the information.

• Wrong sample: Asking the right questions of the wrong respondents will yield irrelevant results.  Be sure you think about whom you should talk to.  Make sure that your online market research company recommends the optimal sample to answer your key research questions.  The right questions to the right people will yield usable and valuable information.

• Wrong research tool: Many companies start the research process by saying, “I want to do a survey,” or “I want to do a focus group.”  Instead, consider which research tool will best accomplish your goals within your budget.  Sometimes a combination of methods works best.

• Insufficient Sample Size: Be careful about drawing inappropriate conclusions from research.  Qualitative research is excellent for probing and delving into details, but it’s not good for extrapolating to large populations.  Quantitative research must be based on a random sample that is large enough for extrapolation.

• Overspending: Shop around for a market research firm that understands that good research takes into account your objectives, your time frame and your budget.  By weighing all three, you can arrive at an optimal budget.

• Improper Analysis: It’s easy to look at the results of market research through biased eyes.  This is especially tricky with qualitative research. Choose the right analytical tools and look for statistically significant findings from quantitative research and common key insights from qualitative.

SSI compared a 20-minute survey to a shorter version to test for fatigue effects and the impact on response quality. Both sliding scale and open-ended questions were tested, and the study was conducted first in 2004 and replicated in 2009.

In all cases, response completion and engagement decreased with the long survey, according to the SSI white paper. Author Pete Cape concluded, “If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and responsive respondents.”

Market Tools, however, concluded that there are a number of design variables that lead to respondents’ rating of the survey, rate of abandonment, the first incidence of “speeding” through the survey, and the percentage of pages sped through. They concluded that while survey length is a good predictor of most respondent engagement measures, there is wide variation in the design variables that are influential in respondent engagement.

Engagement is driven by a complex interaction among design variables. There is no fixed maxim about survey length that applies in all cases.

Questionnaires need to be as short as possible while still accomplishing its objectives.  But, survey design factors also play a key role and can mitigate respondent fatigue and abandonment.

Both white papers are available online, SSI’s here and Market Tools’ here.

1. Recruit well. Just as with traditional groups, you’ll need to over-recruit to account for no-shows, and you’ll need to compensate participants at the same level as with traditional groups.

2. Prepare your moderator’s guide thoroughly and know it well. You’ll be typing and thinking at the same time, so you won’t have a lot of time to glance at your guide.

3. Prepare long descriptions and/or links ahead of time in a text document. Test them to make sure they work and be ready to cut and paste them into the discussion. Practice this ahead of time to avoid fumbling.

4. Feel comfortable with your facility before you begin. Make sure it’s simple to use. Bells and whistles may be fun to look at, but as a moderator, you want to make sure that your facility works quickly and simply and that there’s someone on staff to offer assistance.

5. Arrive early at the facility to greet early arrivals. Acknowledge their arrival and let them know that you’ll be beginning soon. As each participant arrives, be sure to acknowledge him or her. If you don’t, the participant can be insecure about being in the right place at the right time.

6. Set expectations. At the beginning of the focus group, make the participants feel comfortable with the online format. Tell them that spelling and grammar are not important. You’re looking for honest opinions.

7. Set participant ground rules. Tell participants that it’s okay to agree or disagree with one another, but ask them to be sure to answer all the questions from the moderator.

8. Learn participants’ names and keep track of what each person is saying. Respond to individuals by name. This is extremely important! If you don’t do this, you will lose people from the group discussion. If you want everyone to respond, be sure to say this. Remember that you won’t have the physical presence of the participants and visual cues to keep people involved, so you have to keep track of them and use names to assure participation.

9. Ask everyone to answer a question at once. Moderators often begin traditional focus groups by going around the table, asking each person to answer one at a time. In an online group, you can achieve the same effect by asking everyone to respond at once. Tell participants they need not wait for others to type in their answers. Both moderator and participants will see each person’s response as they finish typing, and dialogue can follow.

10. Be prepared for less continuity in the conversation flow than with traditional groups. Differences in typing speeds combined with a lack of physical presence means that some participants may spend a longer time than others answering a question. Their responses may come once you’re already on to another topic. In essence, a good online moderator has to be skilled at handling two or three conversations at once.

11. Develop excellent keyboard skills and a great memory. Some moderators find it tough to type and remember names and conversation at the same time. This takes practice, so you may want to do some mock focus groups before you do the “real thing.” Observing an expert moderator is also very helpful.

12. Make the focus group conversational, “chatty,” and elicit the personalities of the participants. Use colloquial expressions. Use “smiley faces” and other Internet symbols and phrases, but be sure to explain shortcut phrases the first time to use them, i.e. LOL (laughing out loud). Failure to do this makes some participants uneasy that they are not as Internet savvy as other participants, and this can reduce participation.

13. Keep track of participants. If you haven’t heard from someone in awhile ask, “Mary, are you still with us?”

14. Practice makes perfect. It’s often a good idea to hire an experienced online moderator for your first few groups. By observing, you’ll quickly learn the “tricks of the trade.” Good Luck!

Consider the following hypothetical newspaper excerpt:

An October poll of 800 registered voters found that, if the election were held that day, Candidate-X would beat Candidate-Y, 55% to 40%.  While Candidate-X held a 15 point lead, his numbers have “slipped” from a September poll that showed Candidate-X at 58% and Candidate-Y at 39%.

Although data are reported this way all the time, something in the above paragraph is incorrect.  What is it?  Simply put, there is no statistical basis for claiming that Candidate-X’s lead has “slipped” at all.

Why?  Because the margin of error for a poll of eight hundred people is plus or minus 4 percentage points, and Candidate-X’s September-to-October difference fell within that 4-point margin.  So, statistically speaking, there was no change between September and October.

From another angle… if you asked a question from this poll one hundred times, ninety-five of those times the percentage of people giving a particular answer would fall within 4 points of the percentage that gave that same answer in this poll.  Statisticians refer to this as a confidence interval (the “ninety five out of a hundred” referring to the 95% confidence limit).

And here’s another very important way to think about it:  For every 20 times you repeat a poll, one of those times you will get an answer that is completely wrong (because that poll was the one in twenty where the results fell outside the margin of error).

The bottom line:  Never place all of your faith in a single survey.  Only by looking at numerous surveys will you have the most accurate picture.

When graphed, normally distributed data form the classic bell curve. Per the above example, this horizontal axis would show calories consumed while the vertical axis would show how many people eat x calories).

Of course, not every data set’s curve looks this perfect. Some are flatter, some are steeper, and some have means that lean to either side. But all normally distributed data resemble this basic shape.

What the standard deviation tells us is how tightly data are clustered around the mean. When values are tightly clustered in a steep bell curve, the standard deviation is small. When values are spread apart in a flatter curve, the standard deviation is larger.

Graphically, one standard deviation (the red area) away from the mean (the center vertical line) represents about 68 percent of the people. Two standard deviations away (the red area plus the green area) accounts for about 95 percent. And three standard deviations away (the red, green, blue areas) accounts for about 99 percent.

If the above curve were flatter and more spread out, the standard deviation would have to be larger to account for 68 percent of the people. That’s how the standard deviation tells us how spread out the values are from the mean.

If you were comparing test scores across school districts, for example, the standard deviation would tell you how diverse each district’s scores are. Let’s say District-A has a higher mean test score than District-B. Does this mean that kids in District-A are really smarter? Perhaps not.

Because a bigger standard deviation means that more kids scored toward one extreme or the other, a few follow-up questions might determine that District-A’s mean scores skewed higher because the State sends “gifted and talented” kids there. Or that District-B’s mean scores skewed lower because “mainstreamed” special education students were sent there. As you can see, the standard deviation can reveal a less obvious but highly relevant layer of information.

The standard deviation also can help you to evaluate the merit of highly publicized research studies. For example, in a study that claims to show a relationship between eating spinach and building muscle mass, a large standard deviation might suggest that such claims are not valid as they first appear.