What to measure, why, and howA Poppulo global survey showed IC professionals regarded measurement as hugely important yet half said it was the activity they spent the least time doing. Here we look at the what, the why and how of measurement, including outcome standards and organizational impact. The following relates to work by Sean Williams and Stacy Smith, leading members of a task force of industry professionals and academics, who have been working to establish a set of internationally accepted benchmark standards for measurement in IC.
1. What is measurement?
There are three steps:
- Research before the commencement of a communication initiative or campaign;
- Measurement on an ongoing basis during the campaign, and;
- Evaluation to assess the impact of the campaign.
2. Measuring outputs – so what?
For lots of internal communicators measurement is limited to outputs: the number of intranet stories posted, number of likes and shares, the number of active users of an ESN, and intranet visitors. But as Stacey Smith, Senior Council & Partner at Jackson, Jackson & Wagner says, measuring outputs is like measuring somebody on a bicycle, the number of times a person’s legs go round, the number of revolutions of the wheels:
3. Measuring outtakes
This is whether employees received, paid attention to, understood or retained particular messages/communications.
- Awareness: We cannot do anything unless people are aware of something. Awareness is whether an employee has heard an organizational message, issue or topic. Have they become aware of it? You’re not going to start any behavior or achieve any outcome without awareness; it’s simply impossible.
- Knowledge: Employees’ level of comprehension about the messages being conveyed. How much do they comprehend what we have communicated?
- Understanding: Employees’ ability to relate that knowledge to their work in a way that helps the organization meet its goals. How do they connect and really understand? So often employees will be aware and the knowledge is there but the understanding doesn’t relate.
- Relevance: How does that information connect to them? What’s in it for me? That’s the sticking point. Find a way to make it relevant for them or they’re not going to get it. We need to measure and think about relevance.
- Retention: Does it stick with them over time? Because if they are aware of it, they have knowledge of it, they understand it and they find it of relevance to themselves, but they don’t remember it, then we have a problem.
4. How to measure outtake standards
- Surveys: Go directly to the relevant audience and ask them questions about their understanding of the communication or campaign.
- Metrics on social: The level of shares, comments etc. Are the messages being talked about, commented on, and which are performing best?
- Content analysis of these comments: Are stakeholders getting the facts right, do the comments show that employees understand or don’t understand?
- Observation of behavior: If there’s an announcement or message and people are observed not complying with it then it obviously has not been effective.
It is not essential to do all of these measurements all of the time but it is certainly possible to do some of them relatively easily and quickly.
5. The outcomes standards for measurement
These are evidence of the changes to, or reinforcements of, the opinions, the perceptions, the values or the culture that we’re dealing with. Outcome standards to be measured, and their definition:
- Attitude: A way of thinking or feeling about a subject, organization or message.
- Advocacy: The employee’s discretionary effort and time to promote and defend the company’s products and services. “This is a great place to work, these are great products, you really need to come work here, etc.” How often are they going out saying this or are they saying 'ooh you wouldn’t believe what’s going on in my organization!'.
- Authenticity: The perception that an organization is transparent, honest and fair. So often we hear employees saying they don’t know what’s going on, that they’re not told the full story by management, etc. Measuring authenticity is a critical component in the whole piece of understanding an employee’s connection.
- Empowerment: Where employees have the information, rewards and power to take initiatives and make decisions to solve problems and improve performance. We ask employees to do more, to be creative, but if they don’t feel empowered to do so by the organization and its management then they are not doing what the company needs them to do. So we need to measure their sense of empowerment.
- Collaboration: A process of employees across different divisions or units across the organization coming together. The really problematic part of lateral relationships and communication: does the right hand know what the left hand is doing? Do they understand that they’re working together and not competing? It’s really important to measure because many of the breakdowns and problems and potential crises happen because there’s a lack of collaboration across the organization.
- Teamwork: Employees in the same unit coming together to achieve a common goal or objective under the leadership of the appointed manager.
- Discretionary effort: The amount of effort employees give to an organization, a team or a project above and beyond what is required. Are they doing more than they are asked to do? This a good behavioral measure. Are they going out of their way to do more and to give more to make that effort?
- Trust: A belief in the reliability, truth and integrity of the organization’s leadership decision-making and communication. Trust is something that’s built over time, and is a key underpinning of our understanding of what’s going on in an organization.
- Satisfaction: The extent to which employees are happy or content with their job or their work.
- Transparency: The willingness of the organization to share positive or negative information with employees in a timely fashion. How do employees view the transparency of their organization? It can be difficult to get things out in a timely fashion when various inputs, legal etc. are required. But the longer it takes to communicate the less sense of transparency there is.
- Fairness: The perception held by employees that the organizational processes that allocate resources and resolves disputes are impartial and just. Fairness is key, and measuring that sense of fairness so that we can take action on it is critical.
It’s notable that engagement is not on the list of standards. Sean Williams explains why:
6. Measuring outcome standards
- One-on-one interviews with key opinion leaders – not the power leaders – and focus groups
- Self-administered diaries: Where employees are asked to detail their daily activity in a journal for analysis and evaluation afterwards
- Observation of behavior: Power-sharing and teaming – can we see that this is happening; reduced complaints, etc
Regarding organizational behavior, for example, this can be really challenging in a large organization, but it doesn’t always have to be the whole organization that is observed, it can be portions or sections for pilot observation. For example, to see if there’s agreement on what managers say they are doing and what employees say the managers are doing.
One such limited observation of behavior in a section of a large organization found that up to 40% of managers were not holding the regular staff meetings they said they were holding!
7. Organizational impact
This is whether and how internal communication has influenced organizational performance. Organizational Impact Standards and their definition:
- Productivity: The quality and quantity of work output based on resources. How is communication impacting productivity?
- Innovation: Thinking differently and experimenting with new approaches, ideas or behaviors relating to the organization. Is the organization being innovative and how are communications influencing innovation?
- Continuous improvement: The process whereby employees offer small or large improvements to improve efficiency, productivity and quality of a product or process in a work environment.
- Reputation: The biggest one for communicators. Stakeholders’ – both internal and external – evaluation based on personal and observed experiences of an organization and its communications.
- Employee retention: The number or percentage of employees who remain employed.
- Safety: Employee freedom from physical and emotional harm, injury or loss.
8. Measuring organizational impact standards
- Economic value-added analysis – how is IC impacting the bottom line
- System metrics
- Performance reviews – is communications part of people’s job spec and how are we helping them become better communicators
- Independent research studies
- Attrition rates: Incident rates and severity indices – are there problems, accidents or reputational issues and are people leaving, or do they stay because they value the organization?
Note on surveys: the prevalence of surveys in people’s lives has resulted in survey fatigue in the workplace, so it’s important that when they are deployed they are conducted properly and, most importantly, followed through on afterwards. Whereas previously surveys were a big and onerous undertaking, it’s now possible to do smaller ones that can concentrate on specific areas of interest in the business or organizations.
Stacey Smith says that one of the biggest turn-offs regarding staff surveys is when nothing happens afterwards and employees don’t hear from management about the findings.
This section is based on a Poppulo webinar with Sean Williams, Assistant Teaching Professor at Bowling Green State University, Ohio, and Stacey Smith, Senior Counsel and Partner at Jackson, Jackson & Wagner, New Hampshire: What to measure, why and how – an ongoing issue in internal communication.