Learning management systems have become the mainstream technology underlying corporate training and the migration to e-learning. Today with PeopleSoft, Oracle, Siebel and IBM, as well as Saba, Docent and dozens of others in the market, companies have many
by Josh Bersin
December 1, 2003
But just as companies are getting their arms around learning management systems and implementations, another even more exciting application is just around the corner: training analytics. Just like sales and marketing analytics systems were developed as an outgrowth of customer relationship management (CRM) systems, training analytics systems are becoming a new category to complement the LMS.
Measurement has always been a challenge in the training industry—according to ASTD’s 2002 industry survey, companies spend between 1 percent and 4 percent of total payroll on training. This translates to between $300 and more than $1,500 per employee per year. Imagine how a training analytics system could show you the specific programs, initiatives and content that drive ROI from this investment.
The solution is here. Training analytics is a new breed of application that gives companies a systematic, reputable way to measure the activity, efficiency, compliance and effectiveness of training. We see a trend toward an exciting new solution that will give you the ability to really measure training ROI and correlate training back to business metrics in your corporation.
The Challenge: What to Measure?
The first challenge companies face when asked to measure training is determining what to measure. The Kirkpatrick model gives you a framework, but not really any techniques or solutions. In consulting with many companies, we find three critical categories in training measurement: efficiency, effectiveness and compliance.
Efficiency: How efficiently are we utilizing our training investment? What is our real cost per student per hour? Who is consuming our training programs, and how efficiently are we delivering to them? Is our migration to e-learning and other Web technologies saving us money? If so, how much? Across our e-learning programs, are we driving down costs? How can we get more output for our existing investment? And if we do need to cut, where should we cut?
“We are a metrics-driven company,” said Mark Sullivan, director of training technologies for Honeywell. “In order to manage the training process, we have to set benchmarks and strive to exceed them year over year. Cost-effectiveness is perhaps the first and most important because we know we are essentially an overhead function.”
Effectiveness: What is the impact of our training investments on a program-by-program basis? Are we getting the right enrollment and completion rates we need? Are the students really satisfied, and does satisfaction vary across programs, instructors and audiences? For any given program, how does it correlate to solving the business problem we set out to solve? Can we measure business impact on a person-by-person, group-by-group or division-by-division basis as a result of our major investment in a training program? If so, what have we learned that can help us improve over time?
Whitney Shelley, director of training and development for Kinko’s, said, “In today’s results-oriented business climate, it’s more important than ever for trainers to demonstrate to management the value and return on investment we bring to the organization. This means creating programs and initiatives clearly linked to overarching business objectives—and having effective tools in place to measure their success.”
Compliance: This is simple, but may be the most important measure of all. Many learning programs are government or company mandates. At Honeywell, Six Sigma is a CEO mandate across all manufacturing divisions. Engineers must complete Six Sigma training and become certified within six months. “Honeywell is a metrics-driven company,” said Sullivan. “Our business is based on ‘precision results’—and for precision results we need ‘precision training.’ Our CEO has a mandatory initiative for 20,000 engineers to become Six Sigma certified by year-end. Six Sigma defines how we measure quality. We have to establish benchmark training measures just like we establish benchmarks for product quality.”
In the health-care industry, the government mandates training to comply with the Health Insurance Portability and Accountability Act (HIPAA). Managers must complete sexual harassment training in order to prevent law suits.
How do we drive this process? We must measure compliance day-by-day, week-by-week and give the line-of-business managers the information they need to make sure that every employee is in compliance before the deadline passes.
The Training ‘Dashboard’ or ‘Balanced Scorecard’
Back in the mid-1990s, Nolan and Norton pioneered the concept of a business “scorecard” (often called the “Balanced Scorecard”), which would set specific metrics by which a company could measure itself. These scorecards monitor specific business metrics that are important to the company.
Some of the typical scorecard measures of efficiency might include:
- Training budget as a percentage of revenue.
- Training budget per employee.
- Training budget per offering.
- Revenue per training program or per customer.
- Cost of training per student hour or student day.
Some typical scorecard measures for effectiveness could include:
- Enrollment rates and completion rates on a program-by-program basis.
- Scores and certification rates across groups or programs.
- Improvements in sales, reduction in turnover or improvement in quality tied to specific training programs.
Scorecard measures relating to compliance include:
- Completion rates for mandatory programs.
- Certification acquisition rates.
- Percent certified in each business unit.
- Risk of falling out of compliance.
With a training analytics system, these dashboards are easy to create. The dashboard is designed to help you monitor and distribute the measures that your company uses.
Measure Training Consistently
The ASTD 2002 industry survey found that only a third of companies profiled try to measure learning effectiveness. ASTD found that 12 percent or less try to measure job and business impact.
Why is this? In research with more than 30 training organizations in 2002, we found that the biggest reason companies do not measure training more rigorously is that they do not have the experience, tools and infrastructure to do so. It is not lack of interest or importance, but rather a lack of tools and expertise.
LMS systems have built-in reporting today, but as you will see, reporting and analysis are different problems to solve. Built-in reporting provides the ability to create an ongoing, systematic measurement process for training.
Cost-Justifying Measurement
If you consider the total training investment per person in an organization, how much should a company spend on measurement? One percent? Five percent? Ten percent? According to the most recent ASTD survey of best practices, most companies spend 40 percent to 50 percent of their training dollars on content development, 8 percent to 10 percent on infrastructure and the remaining on salaries and cost of facilities.
Measurement infrastructure should be included in the CLO’s budget. Think about allocating a small but fixed percentage of the budget to providing the tools and systems to measure the effectiveness of the overall investment. If a company spends 50 cents of every training dollar on content, why not spend 2 to 3 cents to implement or purchase the tools to measure the effectiveness of the content? Spending $2 to $10 per learner on a training analytics system can create a systematic, repeatable approach to measuring the ROI and compliance of training.
In today’s economy you must cost-justify every new investment—and measurement is no exception. How do you cost-justify the tools and procedures to measure training? By identifying the business impact and risk of not training.
“If we are out of compliance on critical safety training, we can face large fines,” said Sullivan. “These costs justify our investment in the infrastructure to manage and carefully measure our training results.”
“The investment in analytics is far outweighed by the risks of inadequate training,” said Shelley. “Our success hinges directly on our team members’ understanding of our products, services, operations and policies. Given the nature of our business, team members must be thoroughly trained in legally sensitive areas, such as customer confidentiality, non-disclosure and copyright, and we must be able to track the deployment of that training.”
Measurement to Complement the LMS
The advent of training analytics changes this equation. Training analytics is a systems solution that enables you to efficiently measure your learning investments in a repeatable, scalable, actionable way.
Analytics requires data. The data you need today resides in your LMS. If you have implemented some kind of LMS that houses enrollments, fees, completion data, scores and certifications, then you have the basis for an analytics system. It is important to make sure that your LMS and e-learning content adheres to AICC and SCORM standards, so that you can rely on the completion and score data from e-learning and webinar courses.
Most learning management systems have built-in reports. Although reporting tools are important, they alone will not give you the solution you need.
According to Webster’s, the word “analyze” means to divide a complex whole into its parts or elements. “Analyze” suggests separating or distinguishing the component parts of something (such as a substance, a process, a situation) so as to discover its true nature or inner relationships.
Analysis is not the same as reporting. It is important to understand this. A report is the result of an analysis. Before you can create a report, you have to dive into the information to understand what it is you want to understand. From there you can create a report to view that information regularly.
In order to create dashboards, benchmarks and measures, you will need information that is not located in your LMS today. The analytics solution will create computed measures that give you business insights—this information is derived from information in your LMS.
Designed for Business People, Not Programmers
The concept of analytics is to provide a software solution that lets a normal human businessperson (not a Ph.D. statistician pr SQL programmer) understand what’s going on in their training and e-learning operations. The solution should answer basic business questions, such as:
- How much did something cost?
- What were the components of the cost?
- Who took or completed a learning offering?
- What can we do to improve it?
The training analytics solution gives different users the information they need to make decisions.
Once you understand the business questions needed, the next issue to address is making it easy to find the information you need. In our research we have identified three categories of analytics users (information consumers). Depending on their jobs, they use information for different purposes. (See Figure 1.)
As you decide how to proceed, remember that these three groups—executives, line managers and training executives—need slightly different views of information. Executives typically want dashboards or charts. Line managers typically need tabular reports and charts designed around their audiences and programs. Training executives and managers need the ability to slice, dice, drill and filter to find just the information they need.
See Figure 2 for a diagram of the basic architecture of a training analytics system.
E-Learning Standards
What you find when you implement such a system is that information must be complete. Since most companies are migrating 10 percent to 50 percent or more of their training to e-learning, more and more data is available now than ever before.
A well-designed e-learning program will tell you enrollments, student hours, completion rates, scores and even dropout rates. In order to get this information, however, it is more important than ever that your content adhere to AICC or SCORM standards. Only if your content adheres to these standards will you get the tracking information you need to measure and analyze your training investments.
Business Intelligence Tools
The training analytics solution is really nothing more than an application-focused business intelligence solution—and your company already has many of the tools you need. Companies like Cognos, Business Objects, SAS and others have provided these tools for years. Once you have the basic infrastructure in place, these tools will give you the reporting, filtering, viewing and charting information you need to view and distribute information.
Getting Started
The most important thing is to start now. The first and most important step that you as a CLO must go through is establishing the critical measurement issues you want to address.
In most companies, there are one or two business-critical training measurement issues that must be addressed first. For you they may be compliance-related, they may be efficiency-related, or maybe you need to cost-justify your investment in e-learning. What you must ask yourself is, “What should we be measuring that will really increase the value and business impact of our learning operations?”
Once you have clearly established these few measurement challenges, you can start building or buying the solution.
Several vendors have started to ship turnkey analytics solutions, such as Docent, Saba and Knowledge Advisors. Docent’s and Saba’s systems integrate directly with those companies’ respective proprietary learning management systems, so they are an excellent fit if you use those systems. Knowledge Advisors’ system focuses more on program measurements and can both deliver surveys and analyze results.
Each of these systems gives you an out-of-the-box analysis that will let you quickly understand the activity, efficiency and effectiveness of your training.
In many companies, the IT organization already has a business intelligence or data warehousing team that knows how to implement these solutions. If they have a plan in place, you can work with them to understand the costs of building such a system yourself.
Many companies will build their own analytics solutions. If you have an IT organization that has any experience with data warehousing, they will see this as an easy-to-implement new internal system. A great example of such a solution is a system built by Pep Boys to measure usage and compliance with field sales e-learning programs. This system was implemented using Hyperion (a well-known business intelligence product) and is used by all line managers to help them make sure that employees complete their standard online training programs.
We are still in the beginning stages of this market. But from our experience with companies implementing these solutions, we are beginning to understand the metrics. A good rule of thumb is that you should budget $4 to $6 per learner in your learning operation toward the development and implementation of a training analytics system.
Benefits to the Enterprise
In a recent study we did with IDC, we asked 30 companies, “How many of you would like to spend more time and money measuring your training investments?” The answers were enlightening. (See Figure 3.) Clearly there is a demand for more analysis of training today. Companies want to “close the loop” between training investments and business results. The time has come for training analytics.
Josh Bersin is president and CEO of Bersin & Associates (www.bersin.com), a leading provider of research and consulting services in e-learning technology and implementation. Josh is the author of the upcoming “The Blended Learning Book” and “The Training Analytics Handbook.” E-mail Josh at jbersin@clomedia.com.