Asking customers to either rank or rate a series of attributes has been a common practice of market researchers for years. Unfortunately, these traditional methods of research have some drawbacks and may not produce the results researchers and business teams are looking for. In the first part of our two-part series, we will discuss the pitfalls of traditional surveying methods and a solution to these pitfalls.

The Pitfalls of Traditional Surveying Methods 

Although rating and ranking methods of market research provide valuable insights, there are multiple drawbacks to these surveying methods.

According to QuestionPro, “researchers have found out that rating questions are susceptible to user scale bias, scale meaning bias, and lack of discrimination.” This includes the issue of “straight lining”, which is where a respondent will provide the same rating for all the attributes listed. This can pose a problem for the researcher as it makes it difficult to determine the order of importance of each attribute.

For example, let us assume you are designing a new medical device for physicians and you want to know which product attributes you should focus your resources on. If you send a survey to your customers asking them to rate your list of attributes, you will most likely find that they think all the attributes are important. For instance, having a product that delivers results fast and is portable may both be important. But if both are rated of equal importance, how can you determine which attribute you should invest more resources on? A traditional rating scale cannot answer this question.

In addition, rating scales are also relative to the respondent and often fall victim to bias as one respondent’s definition of “excellent” may not be the same as another’s definition, making it difficult to truly determine the value of each score.

Another traditional method, ranking questions, are a great way to determine the relative importance of attributes and eliminates the biases found in rating scale definitions. However, they too are not without their shortcomings. Unfortunately, with ranking, you can only have respondents rank a limited number of attributes at a time or you risk survey fatigue. This makes it difficult to truly understand levels of importance for a long list of attributes.

Ranking also makes it challenging to determine the size of the difference between two items. If we revisit our medical device example and ask the respondents to rank each of the product attributes instead, we may find that having a portable product is ranked number one and having the results delivered fast is ranked number two. But how significant of a gap is there between the two attributes? Are the attributes almost of equal importance or will physicians pick portability over fast results every time? We cannot answer these questions by simply ranking the attributes.

Maximum Difference Scaling 

Maximum Difference (MaxDiff) Scaling is a type of surveying technique used to understand the likes and dislikes of respondents. Often referred to as best-worst scaling, this surveying technique asks respondents to choose their most and least important attribute from a list. This process is then repeated multiple times with each attribute being showed an equal number of instances.

According to Select Statistical Consultants, the “goal is to rank the attributes in terms of importance to customers on a common scale so comparisons and trade-offs between them can be made.” This allows researchers to have a clear understanding of the order of importance for each attribute.

If we go back to our medical device example, by using the MaxDiff scaling method, the researcher will have a sure idea of which attributes of the product are most important to physicians and the size of the difference between each attribute. This way they can optimize their resources and build a device that will achieve maximum return on investment.

MaxDiff is also easy for respondents as they simply are choosing their most and least important attribute from a short list. In addition, because the task only needs to be repeated a few times, it is more practical and less fatiguing for respondents. MaxDiff also eliminates rating scale bias and forces respondents to make a choice, yielding more reliable results.

In Our Next Article…

We can all agree that it is critical to talk and listen to our customers, but the research methods we choose can have a major impact on the validity and quality of the data we collect. Although traditional rank and rate research methods provide valuable insights, MaxDiff scaling can go beyond what these two methods offer and address some of their pitfalls.

As we continue this discussion in our next article, we will look at where companies can apply MaxDiff scaling and what are some of the latest innovations in this space.