The National Security Agency (NSA) and its military arm, the Central Security Service (CSS), have two primary operational objectives: protect information systems in the United States and gather intelligence from a vast array of foreign communications. To accomplish these goals, the NSA/CSS performs cryptography, or the practice of devising and using ciphers, and cryptanalysis, which involves cracking the codes of adversaries’ agencies and institutions. The surveillance of these transmissions provides vital data for national security that is delivered to decision-makers at all levels, from the President and Congress down to tactical military commanders. Because of the secretive nature of its work, the organization cannot divulge specific details, such as how many employees it has or what funds it receives and spends in a year’s time. However, the NSA/CSS maintains that if it were ranked alongside private corporations in terms of dollars spent, floor space occupied and personnel employed, it would fall somewhere in the top 10 percent of the Fortune 500.
The NSA/CSS’ workforce is divided about evenly between civilian and military employees, including analysts, engineers, physicists, mathematicians, linguists, computer scientists, researchers, security officers and data-flow experts. The organization’s Associate Directorate for Education and Training (ADET) is tasked with developing and delivering learning programs for all of these, as well as personnel within associate establishments. “We’re responsible for the training of cryptologists globally,” said Deborah M. Wharff, Ph.D., chief of learning analytics and strategies and a senior manager at ADET. “Cryptologists can be part of the military services, they can be part of other government agencies, and they can be part of other countries. Our throughput on an annual basis to our formal courses is about 100,000 people. Of that 100,000, of course, you’ll have repeat employees. We also provide training for our customers. We also have the DOD (Department of Defense) customers, who are working DOD-specific missions, who we train. It helps with our value chain, if you will.”
In her role, Wharff controls the resource allocation—money, people and space—related to ADET’s core functions, which include education and training requirements processes, performance consulting efforts, learning needs assessment and evaluation, and all educational research. She also manages performance metrics, which are reported to the NSA/CSS’ director, Lt. Gen. Michael V. Hayden of the U.S. Air Force. “My organization is responsible for all faculty and staff development for ADET,” Wharff said. “We manage an adjunct faculty program—more than 80 percent of our courses are taught by adjunct faculty. We’re responsible for consulting on all instructional strategies, design and development on formal and informal learning events. We are responsible for standards, quality and relevancy of all of our programs. All of the activities in the organization that I lead underpin all of the products and services that we deliver to our corporation.”
ADET maintains 62 educational departments, which range from mathematics, engineering and other hard sciences to soft skills like customer services and briefing delivery. “Within those curricula, we average between 20 to 50 courses, and they’re mostly aligned by basic-, intermediate- and advanced-level skills,” Wharff said. “There are some curricula that are very focused on our core cryptological competencies: intelligence analysis, signals analysis, language, networking and telecommunications. Our more advanced courses are geared toward problem-solving, decision-making, innovation and research, and provide us with new capabilities to enhance processes and development of our technologies and tools. In any given year, we will offer 1,000 courses, and maybe two to three offerings of each of those.
“We’re rolling out a lot of new courses,” she added. “They’re all focused on cryptology and other disciplines that are core to our business. We’re doing this to keep current with our environment and what’s going on in the world today. As the political situation changes, our training and education programs have to change. We always have to keep our analysts abreast of all the current technologies. In any given year, we’re rolling out 20 to 30 new courses, which might span from one day to 14 days.”
Wharff said ADET is in the process of moving the organization toward a more blended approach to learning, which will involve greater utilization of technical modalities. At present, NSA/CSS employees have access to more than 2,000 computer-based courses. “We’re part of a DOD consortium that’s called Viewport, and it’s moving toward a government intelligence-wide training system, where these courses will be available on what we call our ‘external’ Internet,” she explained. “We compartmentalize the courses that are classified, but most of the courses are unclassified. The courses that are classified are only available on the (NSA/CSS) intranet.”
ADET has worked with some U.S. universities to help NSA/CSS use technology as an enabler for its courseware. The idea is that within five years, every new course will have some technical components, although technology would never be the sole delivery modality used, Wharff said. “Because we still have a significant number of people who have learned to learn in face-to-face events, we’ll continue to have face-to-face activities, but those programs that require face-to-face will be enhanced by technology. Collaborative tools will allow us to do action-based learning. They’ll work on problems virtually. That’s our goal: to start to morph our more traditional courses into blended approaches.”
To track the effectiveness and efficiency of NSA/CSS learning programs, Wharff and her team use the balanced scorecard, Jack Phillips’ ROI model and Kirkpatrick’s four levels of evaluation. “As our strategic goals and objectives change, our scorecard changes, and we’ve just added some new strategic goals to our plan,” she said. “We have nine objectives, and we’re measuring seven of those right now. From an internal process perspective, we’re looking at time-to-market and customer satisfaction.”
For example, Wharff explained that the organization is currently using standards to reduce course development time. “I’ve implemented a performance metric to reduce that by 25 percent over the next six months, so we’re tracking our time to market,” she said. “In 2006, we have a phased-in approach, where all of 1,000- and 2,000-level courses will be technology-enabled. We’ll be measuring the reduction in human interface needed to teach, so that we can put more resources into our 3,000- and 4,000-level courses. We’re also looking at the ability to measure the impact of learning on job performance. Right now, we’re reviewing all of our tests and measures of learning in key curricula areas. Any curricula area in which we’re doing focused needs assessments and identifying gaps, we’re looking at the solution sets to make sure the existing sets are aligned with the results of that needs assessment. We’ve tightly linked our needs assessment process with our evaluation process.”
ADET uses Level 1 questionnaires to project the transfer of knowledge from the classroom over to the job. Following completion of courses, ADET asks learners to anticipate what content they will use in their work, how much they’ll use and whether the time spent away from the job offset the time spent in the training. The findings from these surveys were generally very favorable, Wharff said. Among participants who attended mission-related courses and reviewed them after the fact, about 78 percent said that they had used 50 percent to 90 percent of what they learned on the job. In addition, 89 percent reported they were satisfied with the improvement in their skills as a result of the training.
The employees’ superiors were pleased with the results of the classes as well. “Of the four courses we evaluated at Level 3, 84 percent of the supervisors reported increased productivity from the training relative to the time (participants) spent away from their jobs,” Wharff said. “One hundred percent of the supervisors were satisfied with the improvement of skills.”
These metrics also serve to expose any systemic impediments to enhanced performance when learners go back to their jobs. “We mandate some training, especially when it’s related to emerging technologies,” Wharff said. “We had a mandated program of three courses, and the Level 3 evaluation showed that the middle course, which was a tools-based course, had no evidence of improved performance once (learners) got back to the job. Barriers showed that these students, due to a governance issue, were not getting access to those tools in a timely way, so those skill sets were atrophying before they were getting a chance to use them. We ramped down the throughput on that one course for several months while we were able to modify not only the environment—so they would have access to the tools on the desktop—but also modify the course to make it more job-specific.”
ADET apparently does its job well: Five organizations in both the public and private sector have benchmarked against NSA’s learning and development practices in the past year, Wharff said. Additionally, a CLO from another government agency wanted to shadow Wharff to see how ADET integrates performance consulting with its requirements process, and how those have been coupled with needs assessment and evaluation processes to create a loop of feedback.
Part of the reason behind ADET’s success is that it receives a great deal of support from the top of the NSA/CSS, particularly from its director, Lt. Gen. Hayden. “It’s outstanding,” Wharff said of the relationship between Hayden and ADET leaders. “He meets regularly with our associate directorate of education and training. Lt. Gen. Hayden is a strong advocate of education and training, as any military commander would be. As such, he has significantly boosted the infrastructure of the associate directorate for education and training through money and people. He highly values evaluation and assessment as an enabler to ensuring high-quality and relevant training for the workforce.”
Another component of ADET’s achievements is undoubtedly the skill and enthusiasm that Wharff brings to her own role within the NSA/CSS. “I view my work as a calling,” she said. “I feel strongly that we should all be very self-directed in our learning in the context of our vision and goals for ourselves. We must always be inquisitive and exploring the world around us, and looking for ways that we can enhance our own learning. It’s not just a utility perspective of what we do in our job, but how we benefit the world. It’s more altruistic than just coming to work and doing a job. It’s how we fit in this world through the work we do.”
–Brian Summerfield, firstname.lastname@example.orgFiled under: Learning Delivery, Measurement, Technology