Posts Tagged ‘validity’

This is the second post of my new series: Market Research Methods 101. From now on in this series, I’ll introduce specific market research method as well as some relevant examples and cases I’ve done. Across the methods I’ll discuss,there are some general terms representing the basic, logic and magic of scientific market research methods.They can help you make sure your research is techniquely correct. The three crucial and most-mentioned terms are Validity, Reliability and Level of Measurement. They will be the topic of my next three posts in Market Research Methods 101. Today, let’s begin with Validity.

Instead of displaying various definitions, I’d like to share with you a simple but inspiring example that helped me understand what validity is. You’ll find this example quite engaging because it is an issue that almost all the girls (possibly also guys) care about in their entire life: Weight!

Weight Scale Example helps you understand what Validity is!

Let’s assume (well, probably it is true for you : -b) that you measure your weight 3 times a day by a bathroom weight scale. In the morning, the weight shows that you are 100 pounds (Hooray!). At noon, you weigh yourself again with the same scale: still about 100 pounds (plus or minus 2 or 3 depending on whether you do it before or after lunch). In the evening, you stand on the scale the third time and again you find yourself about 100 pounds. Now you may be pretty sure your REAL weight is 100 pounds. But, wait a minute, are you 100% right? Probably not. What if you go to a gym and weigh yourself with another scale and find you are 110 pounds? Are you still sure about you are 90 pounds? I guess you might want to find a third scale to measure your weight again. If it tells you 110, you may start to question your own bathroom weight scale and think there might be  something wrong with it.

So, how does this relevant to validity? In market research, when we want to measure or evaluate some issue or topic, a common mistake is that researchers may simply assume the measurement they use (e.g. the questions in the survey, interview or focus group) can indeed measure the issue or topic they are supposed to measure.  Back to the weight scale example, it’s just like you assume that your bathroom weight scale can accurately measure your real weight at first. However, you didn’t check if the number shown by the scale is actually “your weight”, not something else. This “dangerous” assumption, most often without our awareness, led you to a wrong conclusion about what your weight is. This kind of situation happened in market research, too. Say, we would like to research students’ satisfaction of a professor in a course. If we simply reply on the change of how many students chose this course, we probably won’t get an accurate evaluation. What if the course is a required one and only one session is available for the students? Then though the professor is not doing a good job, there will still be a lot of students register for it. A more appropriate approach to measure students’ satisfaction might be directly ask their attitudes and opinions toward the course and the professor’s performance.

It might be correct, but not valid.

Now do you understand what validity is? To summarize, in the context of market research, validity is to which a test measures what it claims to measure (the real issue or topic). It means that the items on the test represent the entire range of possible items the test should cover and it covers nothing other than those. It is crucial since you can consider it as the essence of a research project. If you would like  to go east but are actually running to the west, then how can you finally get to your destination?

I hope my weight scale example help you gain a good understanding of validity. If you find it interesting and want to know more, please follow my following post. The weight scale will continue and I’ll talk about Reliability in the next post!

Advertisements

The series of Market Research Methods 101 continues! Today we are going to discuss another basic but crucial concept: Reliability.

Before we head to today’s theme, let’s review the relevant concept, validity, which I introduced in my last post. In the context of market research, validity means that when you want to measure a specific topic or idea, you are exactly measuring what you are supposed to measure and do not include anything else. Going back to my favorite weight scale example, if you are 100% sure your bathroom weight scale can accurately shows your weight, so we can say the scale is valid. Then how can the scale example shed light on what reliability is? Here we go! Let’s assume you keep the good habit of measuring your weight 3 times a day by your bathroom weight scale. In the morning, the weight shows that you are 100 pounds; at noon, it says 80 pounds; in the evening, you stand on the scale the third time and this time you found your “weight” is 120 pounds. If so, do you think your bathroom weight scale is reliable?

Of course not. We know that, regularly, it’s impossible for one’s weight to fluctuate dramatically within a day. Therefore, the three weight giving by the scale should be consistent. If not, then we can tell it is lack of reliability. With respect to market research methods, since currently most of market research still relies on the self-report approach and we all know human beings are extremely complicated and sometimes capricious (well, it hurts but it’s true), it is especially important to evaluate whether the answers to a survey by a respondent is reliable. Well, the core question is, HOW?

Actually, the answer is already in the weight scale example. An easy way to identify lack of reliability is to see whether a respondent’s reply to the same question is consistent. When designing a survey, for a specific concept that we would like to measure, we can intentionally set multiple-item measures at different places in a questionnaire. Therefore, after we collect all the data and information, we are able to see the consistency of a respondents in terms of a specific idea. However, when design multiple-item measures, you don’t want to simply repeat the same question with different words. Remember, human beings are not as constant as rocks, every single detail on a survey will influence their minds, which may prevent them from giving real answer to a question. Imagine you are answering a questionnaire asking you the same questions three times in different words, how do you feel? Probably not very happy. You might think it’s a waste of your time and may start to make up answers or even stop taking the survey.

The ideal form of multiple-item measures is to have 3-5 questions asking about the same topic without the respondents’ awareness that those questions are asking the same topic. To be honest, it is extremely hard to achieve. Nevertheless, we do have techniques and tips to make it happen. In my future posts, I’ll introduce those guidelines little by little. If you are interested in this topic, please follow or RSS my blog so you can receive my updates in the first time.

Do you understand what reliability is after reading this post? Now we know both what validity and reliability are. Can you tell the difference between them? The target picture below can help you review and  differentiate the two concepts.

In the next post, I’m going to introduce the last but not lease basic measurement concept that I think is crucial, level of measurement. It’s coming soon!