Demystifying Survey Statistics

Leanne Buehler, Ph.D., Principal Science and Innovation AdvisorEmployee EngagementLeave a Comment

The role of HR is changing in many ways. One major shift is the use of HR analytics for data-driven decision making. After working with HR for many years, I have lost count of the number of clients who have expressed their lack of affection for statistics (and that’s putting it mildly!). But numbers do not have to be scary! In fact, they can be extremely informative when you have a basic understanding of their meaning and guidelines for interpretation. Here are brief descriptions of some commonly used statistics that come up in the use of surveys.

Mean and Standard Deviation.
The mean of a survey score is simply the average of all of the survey responses. This is a very common statistic and easy to understand, however, it is important to understand the limitations of the mean.

The mean is simply an indication of the central point of the data but does not tell you about the spread of the data. Take the example below where people responded to a survey using a five point scale (5 = Strongly Agree, 3 = Neither Agree or Disagree, 1 = Strongly Disagree). Sample A and Sample B both have a mean of 3, however, when you look closer at the data you see that the spreads are very different. In sample A, everyone responded with a 3. In sample B, people responded in two extremes, either very positive or very negative. This distinction is important because the way you would interpret and take action on the data is very different (changing a consistently neutral experience vs. understanding the reason behind dichotomized extremes).

One way to understand the spread of the data is through the statistic of the standard deviation. The standard deviation tells you how much spread there is around the mean. A low standard deviation (Sample A) means that the data points cluster around the mean. A high standard deviation (Sample B) means that the data points are more spread out.

Rather than relying on the standard deviation, I prefer to see a visual representation of how the data are distributed. You can glean similar insight through the use of a bar chart that represents percent favorable, neutral and unfavorable scores. In most cases, I find this is more informative than using the mean at all.

Normative Percentile.
On many surveys, benchmark data are reported in terms of a percentile ranking. A percentile ranking is an indication of where you fall in a list of numbers lined up from highest to lowest. This number is helpful because it lets you know how you compare to others.

Let’s look at the example of using percentiles to understand the height of children. When the doctor indicates that a child is in the 40th percentile, she is saying that the child is taller than 40% of all children and that 60% of all children taller more than the child. The higher the percentile, the taller the child is compared to others.

In the case of engagement surveys, a percentile of 75% means that the organization scored better than 75% of all other organizations and that 25% of organizations scored higher.

As a word of caution, while normative percentile rankings give you a sense of comparison, they fail to account for context. Going back to the example of height, the percentile does not account for important factors such as socio-economic status, race, height of parents, or the fact that the child aspires to be a basketball player.

We see the same challenges when comparing survey statistics for organizations. While some survey companies try to reduce the contextual variables by slicing norms by industry or number of employees, it is virtually impossible to create a sample of data of organizations that are truly comparable. For example, organizational strategy, economic factors, labor market, compensation strategy, tenure of employees, etc. all can have an impact on employee engagement scores. Therefore, putting too much weight on norms is ill-advised and can lead to inaccurate conclusions. The best use of norms is to get high-level context and then turn your focus on what will help the organization be most successful given its specific strategy and challenges.

Statistically Significant.
This term describes the likelihood that a result or relationship is caused by something other than random chance. In statistics, a result is considered significant not because it is important or meaningful, but because it is unlikely to have occurred by chance alone.

So what does this mean and why should you care? Let’s take an example from an employee engagement survey. Say you have 1,000 employees and scores on an engagement survey improved from 50% favorable to 52% favorable. You want to know – Is this improvement meaningful, or could it just be chance? Statistically speaking, we cannot say with confidence that the two point improvement happened for any reason other than chance. Scores would have to improve to 54% favorable for us to say the change happened for a reason beyond chance.

An important component of statistical significance is sample size. The bigger the sample size, the less movement it takes for the change to be significant. For example, with a sample size of 10,000 employees, it only takes a change from 50% favorable to 52% favorable to be statistically significant.

Understanding statistical significance is important to how you interpret and use survey results. For example, you may be tempted to “punish” a department for being five percentage points lower than another department, however this difference may not be statistically significant (and could be due to chance).

Correlation is often used as a method to determine what survey items are the biggest drivers of employee engagement. Correlation is a measure of the strength of the relationship between two variables. When two things are highly correlated, you can expect that as one thing goes up, so does the other. Take the classic example of height and weight. In general (but not always), we expect that someone who is taller also weighs more.

The numerical representation of a correlation ranges from -1.0 to 1.0. A correlation of 1.0 is a perfect correlation. As one thing goes up one unit, the other variable also goes up one unit. In social sciences, it is very rare to see a perfect correlation. Take this example from an engagement survey in which we are interested in understanding what has the biggest impact on whether or not employees feel valued. Looking at the correlations, we see that Recognition is most strongly related to feeling valued. An understanding of the relationship between the items can be very useful in terms of narrowing down where to focus change efforts.

The caveat with correlation is that it does not mean causation. Take the classic example of the apparently high correlation between shark attacks and ice cream sales. Does this mean that shark attacks cause people to buy ice cream? No! The reason for this relationship is that they both tend to be higher during warmer summer weather. Correlation does not equal causation.

In the engagement example above, while Recognition and Feel Valued are highly correlated, we cannot statistically conclude that recognition makes people feel valued (although common sense would suggest that is likely the case).

Being aware of statistics and what they can tell you is an important aspect of correctly using employee survey data. Such data can be extremely powerful in spurring organizational change, but without care, it can lead to the wrong conclusions and set you off course.

Written by Dr. Leanne Buehler, VP Consulting Services, Newmeasures