Archive for the ‘Market Research Methods 101’ Category

The “Market Research Methods 101” series is back! Today I’ll introduce the last but not least of “The Three Musketeers” in the market research world: Level of Measurement. If you have read my posts for the first two “musketeers”, you should know that I love the weight scale example (hope you also like it). It helped me explain the concepts of validity and reliability and will help you and me understand what level of measure ment in just a minute. But before that, let’s review what validity and reliability are.

Validity: the weight scale is accurately showing the weight, not showing it in a wrong way, not showing anything else

Reliability: you stood on the scale 3 times, and you got the same number of your weight 3 times.

So how can we understand level of measurement by the same example? In real life, some girls can be really strict with their weight. They feel horrible by gaining mere 1 pounds. Therefore, they would like to measure their weight as accurately as possible. They don’t want the scale simply show it’s about “100 pounds”. They want to clearly know whether it’s exactly “100” pounds or it’s actually “102” pounds. Here comes level of measurement, which shows how accurately we are measuring a concept. In market research, there are 4 levels of measurement: Nominal, Ordinal, Interval, and Ratio. Let me take the measurement of income as an example to further explain the 4 levels. I’ll first give your the measurements, so you can try to find the differences by yourself before I introduce them in detail.

Nominal: Is your family’s annual income over 50,000?   A. Yes    B. No

Ordinal: How do you describe your family’s annual income status?  A. Low-income  B. Middle-income  C. High-income

Interval: Ratings on a continuous 1-10 scale (1=very low and 10=very high), where do you think your family’s annual income fall in?

Ratio: Approximately, what is your family’s annual income? _________

Here I borrowed Dr. Philip Hodgson‘s approach to explain the 4 levels of measurement. Each level of measurement is characterized by its properties. Nominal measurement has just one property:CLASSIFICATION. Ordinal measurement has two properties: CLASSIFICATION and ORDER. Interval measurement has three properties: CLASSIFICATION, ORDER and EQUAL INTERVALS. Ratio data has four properties: CLASSIFICATION, ORDER, EQUAL INTERVALS and TRUE ZERO. Because the higher levels of measurement contain more properties and more information, they permit a wider variety of interpretations. For this reason, as a market researcher, we should always keep the following rule in mind: Always assign the highest permissible level of measurement to a given set of observations.

Today I will continue the topic “what makes a good questionnaire”. Last time I shared the first part of my “Good Survey Guideline” checklist, including the tips for survey language, length, focus and meaning. Find more tips below!

The Principle of Measurement (Part 2)

Assumed Knowledge: An interesting academic found that people tend to express opinion about things they don’t know at all. To avoid confusion and make-up answers, you should not assume that respondents know the concepts mentioned in the questionnaire. All statements should be carefully composed to avoid making assumptions about respondent knowledge of a specific issue or topic.

Word Choice: Similar to language, when designing a questionnaire, you should take into account the subtleties of language. Be careful of using synonyms and antonyms. Do NOT use double negatives.

Order: Be aware of the fact that the meaning of any concept can be altered by a preceding concept. Watch out for the order effects – pretest whenever possible.  As for the response categories (e.g. Strongly Disagree, Disagree, Feel Neutral, Agree, Strongly Agree ), make it from negative to positive.

(List to be Continued…)

This is the second post of my new series: Market Research Methods 101. From now on in this series, I’ll introduce specific market research method as well as some relevant examples and cases I’ve done. Across the methods I’ll discuss,there are some general terms representing the basic, logic and magic of scientific market research methods.They can help you make sure your research is techniquely correct. The three crucial and most-mentioned terms are Validity, Reliability and Level of Measurement. They will be the topic of my next three posts in Market Research Methods 101. Today, let’s begin with Validity.

Instead of displaying various definitions, I’d like to share with you a simple but inspiring example that helped me understand what validity is. You’ll find this example quite engaging because it is an issue that almost all the girls (possibly also guys) care about in their entire life: Weight!

Weight Scale Example helps you understand what Validity is!

Let’s assume (well, probably it is true for you : -b) that you measure your weight 3 times a day by a bathroom weight scale. In the morning, the weight shows that you are 100 pounds (Hooray!). At noon, you weigh yourself again with the same scale: still about 100 pounds (plus or minus 2 or 3 depending on whether you do it before or after lunch). In the evening, you stand on the scale the third time and again you find yourself about 100 pounds. Now you may be pretty sure your REAL weight is 100 pounds. But, wait a minute, are you 100% right? Probably not. What if you go to a gym and weigh yourself with another scale and find you are 110 pounds? Are you still sure about you are 90 pounds? I guess you might want to find a third scale to measure your weight again. If it tells you 110, you may start to question your own bathroom weight scale and think there might be  something wrong with it.

So, how does this relevant to validity? In market research, when we want to measure or evaluate some issue or topic, a common mistake is that researchers may simply assume the measurement they use (e.g. the questions in the survey, interview or focus group) can indeed measure the issue or topic they are supposed to measure.  Back to the weight scale example, it’s just like you assume that your bathroom weight scale can accurately measure your real weight at first. However, you didn’t check if the number shown by the scale is actually “your weight”, not something else. This “dangerous” assumption, most often without our awareness, led you to a wrong conclusion about what your weight is. This kind of situation happened in market research, too. Say, we would like to research students’ satisfaction of a professor in a course. If we simply reply on the change of how many students chose this course, we probably won’t get an accurate evaluation. What if the course is a required one and only one session is available for the students? Then though the professor is not doing a good job, there will still be a lot of students register for it. A more appropriate approach to measure students’ satisfaction might be directly ask their attitudes and opinions toward the course and the professor’s performance.

It might be correct, but not valid.

Now do you understand what validity is? To summarize, in the context of market research, validity is to which a test measures what it claims to measure (the real issue or topic). It means that the items on the test represent the entire range of possible items the test should cover and it covers nothing other than those. It is crucial since you can consider it as the essence of a research project. If you would like  to go east but are actually running to the west, then how can you finally get to your destination?

I hope my weight scale example help you gain a good understanding of validity. If you find it interesting and want to know more, please follow my following post. The weight scale will continue and I’ll talk about Reliability in the next post!

Let’s continue Jessie’s “Good Survey Guideline” checklist today. This is the third also the last part of this list. Find more tips below!

The Principle of Measurement (Part 3)

Hypothetical: Hypothetical measure is a description of a fictitious situation in which you are asking the individual to place themselves. The longer the description and the further removed it it from the reality of the respondent, the more is the burden you imposing on the respondent. So try your best to keep it short and practical.

Open-Ended: An open-ended measure is when the researcher does not provide the respondent with a set of response categories. The advantage of open-ended questions: 1) It does not restrict the respondent to predetermined categories; 2) It is suitable when the total number of potential response categories is very large and/or response categories are not fully known by the researcher; 3) It is suitable when providing response categories can influence some aspect of the respondent’s answer or knowledge; 4) It is suitable for exploratory research.

Recall: It refers to the instances when the question is asking respondents to remember beliefs, attitudes, knowledge, and/or behavior that they held or in which they engaged at a prior date. Recall can introduce burden onto the respondent.For enhancing recall, provide reference points and prime the respondents whenever possible.

Neutral Stance: It refers to an instance when the individual does not have a clear direction in his or her stated opinion/attitude. It should be included when you want to ensure that the data would be indicative of how respondents truly felt about a certain aspect of the product.

 

Questionnaire is the most common and powerful self-report-based measurement instrument. However, since human beings are extremely complicated, it’s really hard to pull survey participants’ real attitudes and thoughts out of their minds. I saw several readers of my previous posts mentioned that they would like to know more about the secret sause of designing a good questionnaire. I’m planing to generalize the tips I knew into a checklist. Next time when you are asked to develop a questionnaire, you can refer to my list and optimize your surveys. Since the list is pretty long, I’m going to cut it into three parts. Today Let’s begin with part 1 of Jessie’s “Good Survey Guideline List”.

The Principle of Measurement (Part 1)

Language: The questions and statements on a questionnaire should be straightforward in a spoken language style. The simpler, the better. No slang.

Length: Questions should be kept short except where lengthy statements were necessary to emphasize meaning.

Focus: Each question should be phrased to encompass only one dimension of the theme at a time. You should avoid including any explicit or implicit double-barreled statements.

Meaning: When there is a key concept that might be understood differently by various respondents, you should introduce the concepts with the narrowest meaning possible to clearly express it to all respondents.

(List to be Continued…)

From the comments of my previous posts, I noted that it is not the basic process, but the seemingly complicated methods and techniques that stem people from “loving” market research. So, I decided to begin a series “Market Research Methods 101” to introduce market research techniques and methods I learned and practiced. Hopefully, I can find a simple and entertaining way to help you at least have a knowledge of the common market research methods, if you have not already become a fan or even an expert.

With respect to primary market research, generally, there are two kinds of methods: qualitative and quantitative. Qualitative market research methods include focus groups, in-depth interviews, content analysis, ethnography, evaluation and semiotics, etc. While quantitative methods are related to statistics, all the way from percentages, multiple regression to factor analysis. The graphic from the article Qualitative Vs. Quantitative Research- When to use which can serve as a cheat sheet when we need to make the decision of which type of methods to use for a particular research project.

Before diving into specific methods, which I plan to do in my future posts of this series, I’d like to first clarify some misunderstanding of qualitative and quantitative research methods.

1. In the context of market research, though these two categories possess different approaches, they share the same ultimate goal: exploring issues, understanding causes and effects, and finally, solving the problems.  When you hesitate about which kind of methods to use, always remember to go back to the origin and ask yourself: What problem(s) I would like to solve? What are my research questions? And what methods are practical to apply?

2.  Whether data is involved is not the attribute to differentiate qualitative and quantitative methods. Both of the two categories use data for measurement. However, qualitative methods reply on unstructured data while quantitative methods prefer structured data. The essential difference between qual. & quan. is generalization, meaning whether the research findings are representative and projectable to the general population. Basically, qualitative research is on a case-by-case base, so the results are always limited to specific contexts; while quantitative research mostly aims at identifying a model or even a theory based on great amount of data, therefore, its results can be utilized in a broader context.

3. There are no good or bad methods. There are no so-called “best” methods. There are only most appropriate methods. In my opinion, the debate of which kind of methods surpass the other is meaningless. Going back to the graphic above, we can find that these two types of methods both have expertise that its counterpart doesn’t have. In industry, it is rare that a market research project sole relies on one type of methods.

For example, assume you want to research people’s attitude about a new ice-cream flavor that Ben&Jerry plan to launch shortly. First of all, you may want to hold focus groups to hear about people’s opinion about ice-cream generally as well as their specific opinion on this new flavor. From the focus group, you might find out what worked and what didn’t work, and then you wonder if it is the same case in the general public, especially Ben&Jerry’s customers. That’s where a survey comes in. Based on the findins from the focus group, we can conduct a survey among a large group of people (usually above 500) to gather data and information for quantitative analysis. From the analysis, you may find specific patterns of people’s ice-cream consumption, but you might not understand the reason behind some of the patterns. At this stage, a series of post focus groups can help you probe those specific questions and concerns. For the sake of fun, I call this kind of approach “hamburger research approach” (pre-qualitative+quantitative+post-qualitative). It is quite common in the real market research industry because it enables business decision makers to gain maximum consumer insights and reduce the uncertainty to minimum extent.

Hope you find this introduction helpful. If you have specific market research methods that you’d like me to discuss in my future posts, please feel free to let me know!

The series of Market Research Methods 101 continues! Today we are going to discuss another basic but crucial concept: Reliability.

Before we head to today’s theme, let’s review the relevant concept, validity, which I introduced in my last post. In the context of market research, validity means that when you want to measure a specific topic or idea, you are exactly measuring what you are supposed to measure and do not include anything else. Going back to my favorite weight scale example, if you are 100% sure your bathroom weight scale can accurately shows your weight, so we can say the scale is valid. Then how can the scale example shed light on what reliability is? Here we go! Let’s assume you keep the good habit of measuring your weight 3 times a day by your bathroom weight scale. In the morning, the weight shows that you are 100 pounds; at noon, it says 80 pounds; in the evening, you stand on the scale the third time and this time you found your “weight” is 120 pounds. If so, do you think your bathroom weight scale is reliable?

Of course not. We know that, regularly, it’s impossible for one’s weight to fluctuate dramatically within a day. Therefore, the three weight giving by the scale should be consistent. If not, then we can tell it is lack of reliability. With respect to market research methods, since currently most of market research still relies on the self-report approach and we all know human beings are extremely complicated and sometimes capricious (well, it hurts but it’s true), it is especially important to evaluate whether the answers to a survey by a respondent is reliable. Well, the core question is, HOW?

Actually, the answer is already in the weight scale example. An easy way to identify lack of reliability is to see whether a respondent’s reply to the same question is consistent. When designing a survey, for a specific concept that we would like to measure, we can intentionally set multiple-item measures at different places in a questionnaire. Therefore, after we collect all the data and information, we are able to see the consistency of a respondents in terms of a specific idea. However, when design multiple-item measures, you don’t want to simply repeat the same question with different words. Remember, human beings are not as constant as rocks, every single detail on a survey will influence their minds, which may prevent them from giving real answer to a question. Imagine you are answering a questionnaire asking you the same questions three times in different words, how do you feel? Probably not very happy. You might think it’s a waste of your time and may start to make up answers or even stop taking the survey.

The ideal form of multiple-item measures is to have 3-5 questions asking about the same topic without the respondents’ awareness that those questions are asking the same topic. To be honest, it is extremely hard to achieve. Nevertheless, we do have techniques and tips to make it happen. In my future posts, I’ll introduce those guidelines little by little. If you are interested in this topic, please follow or RSS my blog so you can receive my updates in the first time.

Do you understand what reliability is after reading this post? Now we know both what validity and reliability are. Can you tell the difference between them? The target picture below can help you review and  differentiate the two concepts.

In the next post, I’m going to introduce the last but not lease basic measurement concept that I think is crucial, level of measurement. It’s coming soon!