The Strategic Highway Safety Plan Evaluation Process Model Overview

Transportation safety reached an important milestone in October 2007 when all 50 States and the District of Columbia completed development of Strategic Highway Safety Plans (SHSP). Since then, dramatic road safety improvements have occurred across the nation, including a decrease in fatalities and serious injuries. These improvements may in part be related to changes resulting from the implementation of data driven, multidisciplinary, collaborative, results-oriented SHSPs.

Today all States are implementing SHSPs, and some are asking questions about SHSP effectiveness, e.g., which elements work well and which do not meet expectations. An organized approach to evaluation or an Evaluation Process Model (EPM) helps answer these important questions.

Elements of SHSP evaluation already are in place in many States; however, additional benefits can be realized by organizing and institutionalizing these elements into a comprehensive program evaluation. Evaluation strengthens SHSP efforts by:

• Demonstrating the SHSP’s contribution to transportation safety;
• Uncovering challenges in prioritizing or implementing programs and strategies;
• Determining progress in meeting transportation safety goals and objectives;
• Validating emphasis areas and strategies, or revealing the need to revise them;
• Identifying opportunities for greater efficiencies and improvements to the SHSP;
• Confirming the need for a comprehensive, data driven approach to safety; and
• Underscoring the need to prioritize transportation safety resources and funding.

History and Background

To address the devastating human and economic consequences of traffic crashes, the American Association of State Highway and Transportation Officials (AASHTO) published an SHSP in 1997 and encouraged States to develop data-driven, evidence-based SHSPs addressing the emphasis areas in the AASHTO plan. Some States subsequently developed SHSPs using the AASHTO plan as a model. In 2005, Federal legislation (23 U.S.C. §148.) required States to develop SHSPs.

The Federal Highway Administration (FHWA), together with other safety partners, provided SHSP development guidance in A Champion’s Guidebook to Saving Lives (Champion’s Guidebook) and implementation guidance in the SHSP Implementation Process Model (IPM). The next logical step after development and implementation is to evaluate SHSP process and performance to determine if limited safety resources are deployed efficiently and effectively.

The information in the EPM builds upon and is supported by current evaluation methodology, model practices, and processes identified through in-depth interviews in six States, technical review and comment by FHWA safety experts, the knowledge and experience of the EPM development team, and lessons learned from a multi-State pilot test.

EPM Purpose

Why Should States Use the EPM?

Safety improvements depend on a program of data driven priorities and proven effective strategies. Evaluation helps States achieve such a program by analyzing SHSP process and performance and determining whether current activities deserve enhancement, revision, or replacement.

Evaluation is intended to take the place of trial and error, guesswork based on anecdotal evidence, and intuition. The Highway Safety Improvement Program (HSIP) requires regularly recurring SHSP evaluation to ensure the accuracy of data and proposed strategies. The EPM can help States address this evaluation requirement as well.

The EPM provides guidance on evaluation methods, including when, where, and how to use them. It will help States answer basic questions of evaluation, such as:

1. What are we trying to do?
2. How well are we doing it?
3. How can we improve?

Not every element described in the EPM will apply to every SHSP, but all SHSP leaders and managers can benefit from considering the information and recommendations. Regions, localities, tribes, and others also may find the EPM useful for evaluating safety plans.

When Should a State Use the EPM?

The EPM’s primary purpose is to support a “point in time” program evaluation of the SHSP process and performance. The specific timing of SHSP evaluation is determined by the State, but must take into consideration Federal requirements as well as the needs and circumstances of the State.

The EPM is applied more successfully if certain elements of evaluation are ongoing and are instituted early in the SHSP process. For example, tracking the implementation status of strategies and setting performance goals and objectives are evaluation practices typically initiated early in the SHSP process and then monitored regularly to gauge progress. Practices such as these support the needs of SHSP evaluation and theEPM.

Who Should Use the EPM?

The EPM’s intended audiences are SHSP leaders, such as safety program managers/coordinators, transportation safety planners, and other traffic safety professionals.

How is Program Evaluation Different from Project Evaluation?

Program evaluation (In this document, the terms program evaluation and evaluation are used interchangeably. Project evaluation will be specified as such.) looks at the overall SHSP. It can identify where SHSP and emphasis area goals are met or unmet and point toward likely strengths or shortcomings, i.e., the failure to implement certain strategies or the identification of strategies not having the expected effect. A project-level evaluation examines the effectiveness of specific projects, such as replacing a four-way intersection with a roundabout, improving a pedestrian crossing, etc. Please refer to the Highway Safety Improvement Program (HSIP) Manual and/or the National Highway Traffic Safety Administration’s (NHTSA) The Art of Appropriate Evaluation: A Guide for Highway Safety Program Managers for additional information on project-level evaluation.

Is Program Evaluation Worth the Effort?

SHSP program evaluation takes time and effort, but it is essential for continued improvements. The EPM is designed to help simplify and provide direction for achieving the most useful results. Sometimes program managers are reluctant to conduct evaluations because they fear potential fallout from negative results; however, both positive and negative results can lead to positive change. Successful SHSP implementation relies on being open to continual feedback and adjusting the SHSP as needed. Evaluation provides this feedback and helps identify successful, as well as ineffective strategies, make warranted course corrections, and enhance the return on safety investments.

EPM Components

The EPM contains background information, recommended actions, checklists, and self assessment questions for addressing four basic evaluation components: planning, process evaluation, performance evaluation, and the use of evaluation results. The recommended actions are steps designed to help States prepare for and conduct an SHSP evaluation, and the self assessment questions help identify successes, challenges, and opportunities in the SHSP itself or in the implementation process.

The appendix contains worksheets for Chapters 2 and 3. The worksheets are intended to provide a deeper review of the self assessment questions, document evaluation results, and record follow up action items to maintain or improve SHSP process and performance.

Planning for Evaluation – Getting Started

Chapter 1 addresses how to prepare for an evaluation and describes various planning methods and steps required to organize the evaluation. If an SHSP evaluation plan already is in place, Chapter 1 can be reviewed for ideas to strengthen it.

Process Evaluation – Getting on the Right Track

Chapter 2 is intended to help States evaluate SHSP management methods, including the strength of the leadership structure; whether the development process was sufficiently data driven, evidence-based, multidisciplinary, multimodal, and collaborative; and the degree of alignment among agency priorities. This examination will help leaders and managers identify opportunities to improve the overall SHSP process.

Performance Evaluation – Measuring Outputs and Outcomes

Chapter 3 provides methods for evaluating SHSP outputs and outcomes. Outputs measure the degree to which SHSP strategies and action plans are implemented. Outcomes measure the degree to which SHSP goals and objectives are met.

Using Evaluation – The Focus is Results

Chapter 4 offers methods for interpreting, applying, and sharing the evaluation results to improve the SHSP process and performance.

Chapter 1 >>