Conferences, journals, books and articles on measurement abound. Many assert various methods and hours of discussion take place among learning professionals. It sometimes seems as if we’re wandering about in search of the elusive “measurement” Holy Grail.
Having spent most of my career in a line position before coming to Motorola University, I have to admit that the learning measurement conundrum was quite a mystery to me. From where I came, measurements were based on the goals we wanted to achieve: customer satisfaction, costs, margins, earnings, innovative growth, productivity, defects, cycle time, sales, market share, waste, retention, etc.
Basically, it boiled down to three fundamental focus areas: your customers, your business operations and your talent. We used a very simple but purposeful approach called “goal, question, metric” (GQM) that NASA originated and Victor Basili perpetuated.
Measuring costs money. On a recent CLO panel, I referred to this as the “ROI of the ROI” or “ROI2” because a great deal of the learning community is fascinated with the concept.
Seeking to measure things that don’t need to be measured or are not convincingly measured is a waste of money. As Albert Einstein said, “Not everything that can be counted counts, and not everything that counts can be counted.”
What counts for your learning function? What counts for my organization might be different than what counts for yours — our organizational cultures might differ. We might be at different points in the evolution of our learning function, or we might have different business needs, speak a different language, have different target audiences, etc.
In other words, your goals might be different from mine. Thus, although I certainly will share some measures Motorola University uses, I’ll more seek to provide you a thought process or a measurement strategy by which to determine the right measures for your learning function.
Goals are the Most Critical
The most important question is: What are your goals? The answer will drive your measurement suite. Basili expresses goals in terms of:
In regard to objects, an organization’s success is based on many dimensions. At Motorola University, we have goals established in the three basic “object” areas: customer (at Motorola University, we say “client”), business operations and talent. These categories have the flavor of the balanced-score card approach, but they are not as comprehensive.
Think in Terms of Value, Not Worthiness
What is your purpose or motivation for measuring? Your goals should be positive and business-oriented. Too many times, I see learning function goals aimed at what I refer to as the “I’m not worthy” syndrome.
People say, “We have to prove our worth.” Think about that thought process — what are you conveying, that you’re not worthy?
We need to establish value — it’s a good business practice — and some of the “worthiness” thinking should be redirected more toward “value.”
You might find this a subtle difference in thinking, but it is quite critical because the way you position your thoughts will drive the words you use, the tone you convey, your ability to influence and the measures you ultimately select.
Have an Opinion
At Motorola University, we strive for highly satisfied clients from the perspective of the client, cost-effective operations from the perspective of our sponsors and the talent to make this happen from the perspective of our sponsors and management. These are very high-level goals, and they require further distillation based on your culture, clients, business goals, target audience and what is important in your company.
How do you distill this further? You could ask your stakeholders what your goals should be. Involving them is absolutely necessary but not sufficient — learning professionals are paid to be experts and understand their client base. As such, they need to have an opinion.
Going in with a blank piece of paper and asking what your goals should be reminds me of a waiter taking an order. Going in with a proposal to start the discussion, based on an understanding of learning and the business, demonstrates your value and establishes your credibility as a true business partner. Some find this a risky approach, which can be true — if you are off the mark considerably, you instantly will lose credibility. But with value and credibility come some risk. Are you a “value” thinker or a “worthy” thinker?
Do Your Homework
What is considered a highly satisfied client in your company? What goals has your company established for its customers’ satisfaction? This is a good place to start. It aligns your goal-setting approach with what is already credible and accepted in the organization.
At Motorola University, we use a point-value goal for our strategic learning interventions. We measure client satisfaction in terms of:
It is important to measure all three satisfaction parameters because learning solutions are subject to the same trade-off decisions as any other in today’s complex and dynamic business environment.
Next, what is considered a well-run business operation in your company? What goals does your company establish for its business operations? Do they focus on cost efficiency, productivity, quality, defects, cycle time or something else? Much also depends on the business model for your learning function. Your challenge will be to identify the analogous learning goal.
At Motorola University, we employ many business operation goals, including cost efficiency, quality and process goals that drive our ability to perform our function with fewer defects and fast cycle time. We have various service-level agreements with our outsource partners. Together, these operational goals support our client satisfaction goals for performance, schedule and cost.
This is a critical point — without operational goals that support client goals, it is uncertain whether the learning function “machine” is laser-focused on what is truly required. This is where some learning functions fall short. If issues occur with the output, you need to be able to trace them back to specific internal operations for improvement.
Finally, what is considered top talent in your company? What goals does your company use regarding talent?
At Motorola University, our business model is to employ a small set of highly skilled people to perform the strategic work and outsource the more tactical work.
A goal that is especially effective for driving the strategic ability of learning consultants is percent deviation of the client’s learning requirements as forecasted by learning consultants, before they have any type of assessment from the final set of learning requirements after an assessment is performed. Learning consultants who perform well to this goal have minimal deviation and, therefore, truly understand their client base.
Regardless of the goal area, it is also important to assess the company’s other support functions, for example, human resources, finance or legal. What goals are seen as credible and acted on positively? Consider the lessons learned by these other functions before proposing goals.
What Questions Address Whether the Goal is Being Met?
We all have mental models, the concept Peter Senge introduced years ago. When defining our measurement suite, we can run into serious problems if we and our stakeholders have differing mental models of our goals. For example, if we set a goal of 25 percent improvement in client satisfaction, how many ways could that be interpreted? If we set a goal for 4.2 client satisfaction, someone might wonder what exactly is satisfied.
Thus, jumping from a goal to a metric might result in misunderstandings across the stakeholders. Questions help all stakeholders understand one another’s points of view, come to a common understanding of what the goal means and what is needed to determine whether the goal is being met.
Consider a goal to improve cost effectiveness. Questions to drill down to a common understanding might include:
Imagine the discussion on this question. No doubt all stakeholders will walk away with a deeper understanding of what each of them values, and the cost-effectiveness metric will be focused on something all agree will have purposeful impact.
Consider a client satisfaction goal of 10 percent reduction in product defects. Questions to drill down to a common understanding regarding the learning component might include:
The client might answer, “I see software engineers who are new and unfamiliar with the product causing a number of defects. We’re at 50 defects after our first test pass. I think that if they had a way to become familiar before they have significant coding responsibility, we could decrease these defects by at least 25 percent.” The metrics would flow quite easily from this discussion, with everyone having a common understanding and expectation.
What Metrics Answer the Questions?
Following from the above example, the resulting client satisfaction metric might be very satisfied (decrease defects by 25-plus percent), satisfied (decrease defects by 24 percent to 20 percent), dissatisfied (decrease defects by 19 percent to 15 percent) and very dissatisfied (decrease defects less than 15 percent).
Why not leave it as a business impact measure instead of positioning it as a client satisfaction scale? If you want to roll up your scores, they need to be apples and apples. Business impact measures will vary, depending on the specific business goal, and it will be impossible to aggregate them for an overall learning function measure. At the end of the day, the business impact measure for your function is whether the client is satisfied with the result.
Your Holy Grail Versus The Holy Grail
I have provided some examples from Motorola University and some additional perspectives as well. These are based on the goals of a fairly mature corporate university, the language our stakeholders use, and the perspectives and measures that are meaningful in our culture. Depending on your learning function and environment, your measures might be different. But with any luck, we will have the same thought process to arrive at our measurement approach: GQM. The single learning measurement Holy Grail does not exist, but your Holy Grail exists for a particular point in time.
Corinne Miller is director of Motorola University. She can be reached at firstname.lastname@example.org.