Today I will continue the topic “what makes a good questionnaire”. Last time I shared the first part of my “Good Survey Guideline” checklist, including the tips for survey language, length, focus and meaning. Find more tips below!

The Principle of Measurement (Part 2)

Assumed Knowledge: An interesting academic found that people tend to express opinion about things they don’t know at all. To avoid confusion and make-up answers, you should not assume that respondents know the concepts mentioned in the questionnaire. All statements should be carefully composed to avoid making assumptions about respondent knowledge of a specific issue or topic.

Word Choice: Similar to language, when designing a questionnaire, you should take into account the subtleties of language. Be careful of using synonyms and antonyms. Do NOT use double negatives.

Order: Be aware of the fact that the meaning of any concept can be altered by a preceding concept. Watch out for the order effects – pretest whenever possible.  As for the response categories (e.g. Strongly Disagree, Disagree, Feel Neutral, Agree, Strongly Agree ), make it from negative to positive.

(List to be Continued…)

This is the second post of my new series: Market Research Methods 101. From now on in this series, I’ll introduce specific market research method as well as some relevant examples and cases I’ve done. Across the methods I’ll discuss,there are some general terms representing the basic, logic and magic of scientific market research methods.They can help you make sure your research is techniquely correct. The three crucial and most-mentioned terms are Validity, Reliability and Level of Measurement. They will be the topic of my next three posts in Market Research Methods 101. Today, let’s begin with Validity.

Instead of displaying various definitions, I’d like to share with you a simple but inspiring example that helped me understand what validity is. You’ll find this example quite engaging because it is an issue that almost all the girls (possibly also guys) care about in their entire life: Weight!

Weight Scale Example helps you understand what Validity is!

Let’s assume (well, probably it is true for you : -b) that you measure your weight 3 times a day by a bathroom weight scale. In the morning, the weight shows that you are 100 pounds (Hooray!). At noon, you weigh yourself again with the same scale: still about 100 pounds (plus or minus 2 or 3 depending on whether you do it before or after lunch). In the evening, you stand on the scale the third time and again you find yourself about 100 pounds. Now you may be pretty sure your REAL weight is 100 pounds. But, wait a minute, are you 100% right? Probably not. What if you go to a gym and weigh yourself with another scale and find you are 110 pounds? Are you still sure about you are 90 pounds? I guess you might want to find a third scale to measure your weight again. If it tells you 110, you may start to question your own bathroom weight scale and think there might be  something wrong with it.

So, how does this relevant to validity? In market research, when we want to measure or evaluate some issue or topic, a common mistake is that researchers may simply assume the measurement they use (e.g. the questions in the survey, interview or focus group) can indeed measure the issue or topic they are supposed to measure.  Back to the weight scale example, it’s just like you assume that your bathroom weight scale can accurately measure your real weight at first. However, you didn’t check if the number shown by the scale is actually “your weight”, not something else. This “dangerous” assumption, most often without our awareness, led you to a wrong conclusion about what your weight is. This kind of situation happened in market research, too. Say, we would like to research students’ satisfaction of a professor in a course. If we simply reply on the change of how many students chose this course, we probably won’t get an accurate evaluation. What if the course is a required one and only one session is available for the students? Then though the professor is not doing a good job, there will still be a lot of students register for it. A more appropriate approach to measure students’ satisfaction might be directly ask their attitudes and opinions toward the course and the professor’s performance.

It might be correct, but not valid.

Now do you understand what validity is? To summarize, in the context of market research, validity is to which a test measures what it claims to measure (the real issue or topic). It means that the items on the test represent the entire range of possible items the test should cover and it covers nothing other than those. It is crucial since you can consider it as the essence of a research project. If you would like  to go east but are actually running to the west, then how can you finally get to your destination?

I hope my weight scale example help you gain a good understanding of validity. If you find it interesting and want to know more, please follow my following post. The weight scale will continue and I’ll talk about Reliability in the next post!

To answer yesterday’s question, why is the marriage between “analytics” and “social” so powerful?, first of all, we need to go back to the essence of market research for a while. Yes, it’s all about solving problems, to be specific, solving the clients’ problem by providing them with relevant customer insight and intelligence. It’s worth noting that all three companies mentioned above are relying on the online context and all sort of relevant to social media. Social media is such a powerful tool to gather people’s ideas, attitudes and behaviors precisely. With keywords searching tools like twitter hashtag, researchers can easily identify relevant customer groups or target audiences, analyze their posts to find the pattern and make interpretations accordingly . To some extent, it simplifies the traditional sampling process, and more importantly, it dramatically accelerates the pace of market research since getting real-time data is no longer a dream.

Admittedly, the field of social data analysis and big data analytics is far from maturity and there are still space for its improvement. Another incorrect impression might be  this approach can totally replace the traditional market research relying on surveys, interviews, focus groups, etc.  In the panel, DJ emphasized that social data analysis serves as an effective assistant for traditional market research, but it’s definitely not omnipotent. In my opinion, its first weakness is the demographic limit. Although the Internet and social media become increasingly popular, it’s still dominated by the young generation. For a market research project whose target is the old, it is dangerous to solely rely on the online social data. In addition, I don’t think currently we’ve found an effective way to match the social data about attitudes and the big data about behavior. Twitter and Facebook are good at telling us what people are thinking and sharing; Google AdWords and Google Analytics did a good job to show what people are searching; While Amazon and other big e-commerce giants know what people really buy. The three parties separately dominate one  important process of consumer intelligence: Awareness and Interest, Attention and Retention, Purchase and Repeated Purchase. If we could get these three kinds of data into one Super databases, then the market researchers can finally complete the logic of cause and effect. If the so-called “customized recommendation” can base on data generated by this dynamic, then it will become the REAL customization. It makes me feel so excited even only thinking about the potential power it possesses! However, it raised up the ethical issue of customer security and privacy. We want the valuable data and information, but we don’t want the customers feel intrusive and insecure.

Do you agree with me? Are you planning to develop your career in the emerging innovative market research industry? I’d love to hear your thoughts!

Today is an exciting day. Yes, first of all, it’s the election day. But for me, even not considering the election, it’s still exciting because today I found new meaning and prospect of market research as a career!

From a career panel held in a class today, I got the chance to hear three young, promising professionals who used to study at BU COM, and now working in market research and communication industry, sharing their work experience and understanding of their jobs and the industry.  DJ serves as a Social Data Strategiest at Crimson Hexagon (a startup company providing social media analysis and big data analytics software and services); Chris started his career as a Community Associate at Communispace (a market research firm that works with online consumer insights communities); while Rachel is currently a Brand & Buzz Coordinator at HubSpot (a leading company providing inbound marketing software and services). Before diving into more information about their jobs and companies, I’d like to share two interesting numbers with you: 1. None of the three companies existed 15 years ago; 2. None of the three professionals’ job title existed 5 years ago. That’s the reason I name this post as “it(market research) is an innovation machine”. When the traditional market research analytics marries with “social” (no matter it is social media data, online communities or online marketing), it injects  strong energies and allows uncountable possibilities to the market research industry. The merger has turned the industry into an innovation machine.

Why is this merger so powerful? If you want to know the answer, please follow my post “Market Research as a Career: It’s an Innovation Machine! (2)” tomorrow!

Forrester Research, A Global Research & Advisory Firm

In this summer, I was really lucky to intern at Forrester Research, one of the most reputable global research and advisory firms. This is my first market research experience out of school, inspiring me how market research is like as a career. Honestly, it was not an internship that I would say is very enjoyable (you might find some reasons below in this post), but it didn’t change my dream of being a future market researcher and recently I realized that what I learned at Forrester is even more that I imagined.  I’m planning to share all the valuable experiences I had during those 3 months little by little in my future posts, making them into a series called “Market Research as a Career”. In today’s very first post for this series, I’d like to begin generally and share with you some of the keywords (or phrases) of market research as a career.

Forrester and Me

Challenging, Intense, but Exciting. I don’t need to explain how many advantages of having your first internship at a big name firm in a field, but believe it or not, there are also a small number of disadvantages, especially in the market research industry. The main reason is that it is a super challenging and intense profession. And the firms, especially big firms, always have high expectations on their employees, even on the interns like me. That is because, as an industry of great intelligence and power, market research firms are  “required” by the clients to always be the best. So, it’s normal that you feel stressful and unadaptable at the beginning of your market research career. When the first time I saw the schedule of my supervisor (Mr. TJ Keitt, Senior Analyst on CIO Research Team) is full from 8:30 a.m. to 6:00 p.m., I really had no idea how he would handle it.  I saw him immersed himself in the endless interviews for new research reports, inquires by current customers and request for proposals (RFP). However, he enjoyed it, so did other analysts on our team I observed. Market research as a career has its magic: be the first one to identify new trends that might change an industry, be the hero (or heroine) who save your clients out of terrible situations, be the one who is respected by thinking logically, critically and comprehensively. Nobody can ever say that is not exciting!

Communication is the KING. There is no doubt market research is a teamwork. As I talked in the previous post, it’s all about solving problems, a market research project always has a long process (10-12 steps) and it is very continuous and inter-related. In industry, due to time and budget limit, it is impossible for one single team or department to handle all the steps. Survey design, sample selection, data analysis would be assigned to different departments, or even be outsourced. Though the tasks is in the charge of different teams, the results must be logical and consistent. Therefore, effective communication to other teams is the king for working at a market research firm.

Attention to details. This is quite self-exploratory. Market analysts always work with huge datasets and complex data analysis. Even an small error in data entering will make the output nothing but garbage.

Aggressiveness makes a good researcher. I was quite reversed during my internship at Forrester. When you a newbie surrounded by super smart analysts in a meeting, it’s too hard to bravely speak out the ideas in your mind! However, you have to! TJ told me that in a market research firm, almost no promotion is caused solely by how long you have been working. The “shortcut” to become a experienced and respected researcher is to voluntarily ask for various kinds of work. In other words, be aggressive.

Like my keywords for market research as a career? What are yours?

Aside  —  Posted: January 30, 2013 in Marke Research as a Career
Tags: , , ,

Let’s continue Jessie’s “Good Survey Guideline” checklist today. This is the third also the last part of this list. Find more tips below!

The Principle of Measurement (Part 3)

Hypothetical: Hypothetical measure is a description of a fictitious situation in which you are asking the individual to place themselves. The longer the description and the further removed it it from the reality of the respondent, the more is the burden you imposing on the respondent. So try your best to keep it short and practical.

Open-Ended: An open-ended measure is when the researcher does not provide the respondent with a set of response categories. The advantage of open-ended questions: 1) It does not restrict the respondent to predetermined categories; 2) It is suitable when the total number of potential response categories is very large and/or response categories are not fully known by the researcher; 3) It is suitable when providing response categories can influence some aspect of the respondent’s answer or knowledge; 4) It is suitable for exploratory research.

Recall: It refers to the instances when the question is asking respondents to remember beliefs, attitudes, knowledge, and/or behavior that they held or in which they engaged at a prior date. Recall can introduce burden onto the respondent.For enhancing recall, provide reference points and prime the respondents whenever possible.

Neutral Stance: It refers to an instance when the individual does not have a clear direction in his or her stated opinion/attitude. It should be included when you want to ensure that the data would be indicative of how respondents truly felt about a certain aspect of the product.

 

Questionnaire is the most common and powerful self-report-based measurement instrument. However, since human beings are extremely complicated, it’s really hard to pull survey participants’ real attitudes and thoughts out of their minds. I saw several readers of my previous posts mentioned that they would like to know more about the secret sause of designing a good questionnaire. I’m planing to generalize the tips I knew into a checklist. Next time when you are asked to develop a questionnaire, you can refer to my list and optimize your surveys. Since the list is pretty long, I’m going to cut it into three parts. Today Let’s begin with part 1 of Jessie’s “Good Survey Guideline List”.

The Principle of Measurement (Part 1)

Language: The questions and statements on a questionnaire should be straightforward in a spoken language style. The simpler, the better. No slang.

Length: Questions should be kept short except where lengthy statements were necessary to emphasize meaning.

Focus: Each question should be phrased to encompass only one dimension of the theme at a time. You should avoid including any explicit or implicit double-barreled statements.

Meaning: When there is a key concept that might be understood differently by various respondents, you should introduce the concepts with the narrowest meaning possible to clearly express it to all respondents.

(List to be Continued…)