אבי בניטה

The even scale paradigm

 

A classic dilemma when building a questionnaire is whether to use an even or an odd answer scale. Those in favor of the even scale argue that it prevents the answerers to “drift to the middle”... as if there was something wrong with it.

 

In our opinion, the choice of the middle is not drifting, but a legitimate choice with the same meaning as any other choice. Actually, the fact that the even scale “forces” the answerer to be pro or contra seems to us the opposite of the reality where it is possible not to take sides but remain indecisive.

 

As a result, it seems that the odd scale enables the answerer to reflect in a more reliable and natural way the range of its feelings.

​

​

The 10 points scale paradigm

 

Another dilemma related to the scale of the answer is connected to its span. Many of those dealing with surveys tend to use a 10 levels scale, believing it has a higher sensitivity than the more common 5 levels scale.

 

The main disadvantage of the 10 levels scale is that it is not possible to attach an explicit wording to each category, therefore one cannot be sure that the person that answered 7 is less satisfied than the person that answered 8, a fact that might severely impact the reliability of the results.

 

Moreover, most of the world benchmark publications are on the 5 levels scale. The above reasons and many others (such as the difficulty of presenting a 10 categories distribution), make us recommend the 5 points scale.

​

​

The "average everything" paradigm

​

The typical questionnaire is divided into many logical topics, and each topic consists of many questions. In the organizational climate questionnaire, such an issue may be “the working environment" that may include questions related to the workstation, office maintenance, ergonomic aspects and maybe also the quality of the meals and the relationships with the roommates.

 

Therefore it looks natural to average all the questions included in the topic and creates an average general score for the “working environment”. This practice is completely erroneous from the statistical point of view since it “mixes oranges with tomatoes” and leads to a highly unreliable result.

 

It is not possible to give here a detailed explanation of the problem embedded in this approach, let's say only that is related in general to the fact that questions with a low correlation between them, that is to say, that do not represent the same content world, are not to be averaged. As a rule, preliminary checks must be conducted before deciding to bring several questions to the same dimension. The most popular check is Cronbach alpha and factorial analysis coupled with content analysis and experience.

​

​

The "statistical significance" paradigm

 

One of my preferred paradigms! When I present results in different categorizations (let's say by gender), there will always be someone in the audience asking if the difference between the results has statistical significance. Indeed, something from their studies has subsisted…

 

Our unequivocal recommendation is to stop being worried by this academic question, and to change it to the question: "is the difference meaningful from the managerial point of view?" This is not the place to explain why the significance issue not always has an added value in the analysis of a survey, but the rule is that a meaningful difference from the managerial point of view is likely to be statistically significant, while a statistically significant difference is not forcefully meaningful from the managerial point of view.

 

There are special cases when the significant issue is important (controlled experiments for example), but it is recommended to better focus on managerial meaningfulness.

​

​

 The "Percent favorable" paradigm

 

When analyzing a survey, a decision must be taken as to the leading metric indicator to use. A common measure is an average (on of a 5 points scale), but many companies prefer to use the measure “the percent of the persons with a favorable attitude”.

 

For example, when analyzing the general satisfaction question, the average measure will show a value such as 3.65 (on a 5 points scale) while the percentile measure will show 72% (the percent of the satisfied or very satisfied).

 

Both of the measures are legitimate and meaningful. Our claim against the use of the percent indicator is on that it transforms artificially a 5 points scale into a 2 points scale (as if the scale was 1- satisfied, 0- other) and one may ask why a 5 points scale has bee used in the first place?

 

Statistically, this involves the loss of a great deal of information held on the broader scale that is much better represented by the average.

 

It may be claimed that the average does not reflect the distribution variance, but this is true for the percent measure also.