This morning I was listening to a podcast, which included the host commenting on the results of a recent research study. The research findings included a number of things high performing companies were more or less likely to do or use, with one particular finding being this:
High performing companies are 29% more likely to use in the moment feedback.
The findings are based on a survey of employers, with a seemingly quantitative bend on defining high performing companies. The full research study is not yet available, so part of me wanted to wait until seeing the full details before commenting on the findings or methodology. But I’ve noticed a shortcut that I think is often used to talk about survey findings – I have used it myself, and need to be more careful.
We need to be more aware that survey results contain what the respondents said, not necessarily what actually happens. You can ask the same survey question to the same person on a different day and get a different response. Questions are always subject to some degree of interpretation. And the response from one person within a company may not reflect what another might say. So for all those reasons, a bit more precise statement of the above finding might be:
Survey respondents from high performing companies are 29% more likely to say their company uses in the moment feedback.
Some will argue this is needless, adding length and creating potential confusion among the reader. I would argue it identifies a potential source of data bias. Not to mention I’m not sure a company “uses” feedback – people give feedback, not companies. I would certainly report that my company encourages in the moment feedback (is there a company that discourages in the moment feedback?), but I am 100% certain I can find a manager within the company who isn’t great at it.
The key here is to realize that surveys typically capture perceptions, naturally flavored by the person being surveyed, and those perceptions may not reflect reality. As long as we interpret survey results for what they are, we’re fine. But if confuse perception with reality, we run the risk of drawing the wrong conclusions.