The importance of evaluation in road safety education

Based on an interview with Professor Frank McKenna for Safety Education: Summer 2011 by Jenny McWhirter

Frank McKenna is well known to RoSPA – he is a member of RoSPA's road safety committee and a regular speaker at RoSPA road safety congresses. He is also known professionally for his work on hazard perception, which led ultimately to the implementation of the hazard perception test as part of driver testing.

As a psychologist Frank McKenna began working on road safety almost 30 years ago. He recalls, with some irony, that his interest arose 'by accident' when he was offered a job for at the Applied Psychology Unit at the University of Cambridge, for which he had no previous experience.

It seems that that lack of experience was the best possible start as he began his research with no prior assumptions about what individual factors influenced road accident involvement. A review of the literature at the time revealed a lack of evidence for what works in preventing road accidents and that interventions were based on 'little more than superstition'.

Professor McKenna had two other advantages at this early stage in his career: regular contact with road safety practitioners to whom he is often asked to speak, and communication with victims and families of victims with whom he has often shared conference platforms. "It is profoundly unnatural to outlive one's children, and yet this is what happens in families where a young person is killed in a road accident," he said.

Throughout his career these two motivations: scientific curiosity and the desire to reduce road casualties, especially among the young, have sustained him and made him one of the most influential figures in road safety in the UK today.

Frank sparked a debate with his 'think piece' Education in Road Safety: Are we getting it right? RAC Foundation Report 10/113 (2010) which focuses on effectiveness in road safety education. Frank defines education as the communication of knowledge: from "one to many" and distinguishes this from training, which is skills based. This distinction may seem odd to teachers of Personal Social and Health Education which aims to develop young people's knowledge, skills and attitudes.

Frank acknowledges this dilemma and suggests that educators need to be clearer about their aims and objectives when planning an intervention. This would make it much easier to establish if we are "getting it right".

He also stresses that much of what he says about road safety education applies to other health concerns such as drug education and to public health interventions in general. The reference list at the back of the article attests to this since it draws on published studies from across the whole of public health.

So what are Frank's key messages for road safety practitioners?

"First do no harm". This is an important ethical principle in medicine as well as public health, but also has practical and financial implications. We can't afford to waste resources on interventions which do not work, or which may make matters worse.

An example of interventions which could contribute to making matters worse is one which contributes to a misperception of the social norm. Interventions which highlight the frequency of unsafe driving by young people can lead to the perception that most young people drive recklessly, speed, drink and drive and do not wear their seatbelts. A moment's reflection will reveal this is not the case.

Rather, Frank says, we should focus on the outcomes for victims and their families. Even a minor collision can have serious health and financial consequences which, if they could be prevented, would benefit individuals and society as a whole.

The next key message relates to planning interventions. It is not enough to want the world to be a safer place. We need to plan our interventions on the best possible evidence available. Currently that evidence is very sparse (across the whole of public health, not just road safety). In the absence of evidence we need to focus on theory.

Unfortunately too few practitioners (e.g. road safety officers, fire and rescue officers, teachers) have the opportunity to study public health theory in any depth. Most road safety practitioners are professionals in their own sphere: fire and rescue officers are experts in saving lives, teachers are experts in their subject and in understanding how children learn at different ages and stages.

However, Frank points out that 'expertise does not generalise'. We can be experts in one area but well-meaning amateurs in another. Of course it can be difficult to acknowledge this when we are fired up with emotion generated by the death of a young person in a road accident.

It is also important to recognise that power is not the same as expertise. Seniority in an organisation, or politically, does not automatically confer an understanding of how to prevent road injuries. Having an understanding of public health theory would help policy makers, funders, managers and practitioners plan more effective interventions.

This leads Frank to his next key message: We need to be clear about what we are trying to achieve. The overall goal of road casualty reduction may not be achievable through road safety education, so we should not claim that it is. Rather, we should focus education interventions on education outcomes.

While we may not have much evidence that road safety education works, this is not the same as saying that road safety education does not work. Theory suggests that under some circumstances road safety education can work. We should be looking for interventions which have this potential and describe the aims and objectives as clearly as possible.

In these times of austerity the next key message may be the most important – and the most difficult to communicate. We must evaluate what we do, to find out if it is effective – and of course we must learn from the evaluation which means publishing our findings as widely as possible, even if those reveal that we did not get it right.

Frank admits that his own work has not always been rigorously evaluated. The introduction of the hazard perception test is a good example. When asked what would be needed to prove that the introduction of this test would help new drivers to be safe, his proposals were not followed up. "We need to change the culture so that interventions are properly planned and that evaluation is included in that planning." Again, policy makers, funders and managers need to lead the way by expecting evaluation.

So what is the answer to the big question: Can road safety education work? Here Professor McKenna is at his most hopeful. Road safety education can work in two possible ways:

Directly from knowledge gain and/or attitude shift.

This is theoretically possible but we don't yet have good enough evidence to support this.

Indirectly, as a way of enabling other approaches, such as legislation or the use of engineering solutions.

There may be better grounds for pursuing this approach. This could be the means by which education has contributed to the changes in our behaviour with respect to a wide range of health improvements in recent years, including reductions in smoking, increased use of seatbelts and installation of smoke alarms. Perhaps the aims of road safety education should be to contribute to long term cultural change, rather than road casualty reduction.

What is clear is that we can't go on as we are, relying on outdated beliefs, poorly articulated aims and objectives and overly ambitious outcomes. Frank McKenna's paper should make us all stop and think – can we get it right?

Frank McKenna (2010) Education in Road Safety: Are we getting it right? RAC Foundation Report 10/11.

RoSPA has created the Evaluation hub and E-valu-it online tool to help individuals and organisations to evaluate their work.