Assessment and Evaluation Case Study
Client: Federal Aviation Administration's Aircraft Certification Service (FAA AIR)
Area of Focus: Program Assessment and Evaluation
Contract Vehicle: OPM TMA
Abstract
FAA needed to evaluate the effectiveness of required training courses for their safety inspectors. FAA engaged Carney to evaluate how well the coursework aligned with the stated objectives, and how well the participants applied their courseware learning on the job months after training.
Background
The Federal Aviation Administration's Aircraft Certification Service (FAA AIR) was tasked with training the staff who conducted safety inspections on the aircraft. Safety inspectors review airplanes at three separate stages: when the plane is being designed, when it goes into production, and when it is ready for service. Because new data is always being collected and new tools developed, FAA AIR is constantly refining its courses to keep inspectors up-to-date with best practices. Evaluating the effectiveness of the courses is a critical task for the agency. The stakes are extremely high for inspectors since people’s lives literally depend on them to perform their jobs meticulously.
Challenges
The challenges were to determine if the objectives were accurately covered by the course materials, as well as, how well learners retained and used the material covered in class when they returned to their jobs.
Carney Solution
Carney observed a series of courses to determine how objectives were being met and identifying objectives that were not fully covered. Carney evaluated instructional strengths and weaknesses, gaps, redundancies, update requirements, and structure for the courses.
In addition, Carney conducted "transfer of training" evaluations to determine how well learners performed on the job post-training. Carney partnered with a company called ILG which created a detailed impact map, or a visualization of the effect a training course has on an organization. To create the impact map, ILG conducted interviews of training participants and their supervisors six to nine months after the training. By waiting to conduct interviews, Carney could ascertain how much of the knowledge that the course was intended to transfer was actually being recalled and or used by the employees in their jobs.
After the observations and evaluations, Carney provided recommendations for improving the courses which included prioritizing and refining the learning objectives, refining the overall flow of the courses, increasing the degree of learner involvement and interactivity, and updating selected content.
Results
For some courses, urgent changes were made quickly based on recommendations, but for other courses recommended changes resulted in a major revision of the course.
Delivering Remarkable Client Experience
FAA AIR has since engaged Carney multiple times to observe and evaluate additional courses. Currently, Carney is doing major revisions to two large courses.
Carney – A trusted partner
Carney, founded in 1994, provides government and military clients with innovative knowledge management solutions that accelerate workforce performance. Carney's products and services help agencies accomplish mission-critical objectives and improve readiness.
Through best practices and a thorough examination of the challenges facing military and government leaders, Team Carney identifies new ways to capture, control, and transform knowledge through competency analyses, knowledge transfer strategies, and assessment evaluations.
Carney strives to provide remarkable client experiences by delivering quality performance solutions at the speed of need.