What do performance evaluations really measure?
Professor Steve Scullen’s research recently inspired a global company to rethink its approach to peer feedback.
When your boss sits you down for your annual performance review, you hope the feedback honestly reflects your performance. But extensive research by Steve Scullen, professor of management and international business at Drake University, suggests the evaluation may say more about your boss than it says about you.
The research was first published 15 years ago, but it still has profound impact on the way business is done. Scullen and his research colleagues analyzed the performance ratings of nearly 5,000 managers who had each been evaluated by six coworkers. Raters sometimes agreed, but often differed in the scores they assigned to the same colleague. The researchers found that 65 percent of the variance in ratings was attributable to personal idiosyncrasies and bias of individual raters, and only 25 percent to actual employee performance.
“In other words, one colleague might give you low marks on ‘strategic planning,’ but that doesn’t mean that another one would come to the same conclusion,” Scullen says. “Much of that variation stems from raters’ diverse perceptions of the importance of strategic planning, of their own skills toward strategic planning, and other personal factors that might have nothing to do with the quality of work you performed for those colleagues.”
The key findings recently inspired Deloitte, a global professional services firm with 65,000 employees, to significantly restructure its performance appraisal process. Deloitte’s ongoing initiative was highlighted as a “Big Idea” in the April 2015 issue of Harvard Business Review: https://hbr.org/2015/04/reinventing-performance-management.
“We’ve seen organizations trying to address the human dimension of performance appraisals in a variety of ways,” Scullen says. “Some try to address the problem through training; other organizations host ‘calibration sessions’ where they put raters together and have them talk about an employee, to reach more of a group consensus on performance. These are just a few of many approaches.”
But subsequent research shows that over-relying on annual evaluations can be costly and often ineffective for companies. That’s why Deloitte is streamlining their appraisal process in a way that de-emphasizes annual reviews in favor of more frequent, and more action-oriented, mechanisms of performance feedback. Team leaders are asked to evaluate their team members at the end of every project; or, for longer-term projects, once every quarter.
“Once-a-year goals are too ‘batched’ for a real-time world, and conversations about year-end ratings are generally less valuable than conversations conducted in the moment about actual performance,” wrote Marcus Buckingham and Ashley Goodall, co-authors of the recent Harvard Business Journal article, titled “Reinventing Performance Management.”
Deloitte’s new evaluations don’t invoke the typical rating scale of high-to-low (or effective to ineffective) but instead ask raters to say what action they would take related to their team member in the future—for example, whether the rater would award their colleague the highest possible compensation increase and bonus, if the money was coming out of the rater’s own pocket.
While it’s too early for the company to report definitively on the effectiveness of their undertaking, Scullen says it’s interesting because it dovetails with research that supports increased frequency of evaluations.
“It’s very important that supervisors talk to their employees often,” Scullen said. “Discuss performance—the good, and the not so good—as often as you can. Never surprise your employees at appraisal time with criticisms that should have been shared previously.”
Scullen says supervisors increase the reliability and value of evaluations if they set clear expectations up front, discuss the standards that will be used for assessing performance, establish a regular schedule for taking notes on an employee’s performance, and consistently listen to their employees’ perspectives on what is going well and not-so-well.
“If you do all of that,” he said, “the end-of-year performance review should be more of a formality than anything else.”
Scullen recommends the following considerations for employees who want the most meaningful performance review possible:
- Ask early on for a discussion of what your boss expects from you.
- Make regular notes about your performance. Your boss may or may not remember your accomplishments, so it’s important for you to keep stock of them.
- Communicate often and with diplomacy. Be sure you understand what your boss feels you are doing well, and what needs improvement. Without making excuses, be sure your boss understands any performance obstacles you are facing.
- Remind your boss of your accomplishments. When possible, diplomatic reminders may make it easier for your boss to appraise your performance.