Process evaluation and lessons learnt

A key aspect of process evaluation is to compare actual versus planned implementation. With reference to the Logic Model, this means evaluating the intervention Inputs and Outputs.

This is a result in its own right and helps to explain Impact results. 

This may be a simple comparison of actual vs planned activity (outputs). For example, there were 24 School Educational Visits compared to a plan of 20. Or there were 50 roadshows instead of a plan for 100. 

A similar approach can be adopted for evaluating inputs, such as checking actual expenditure versus planned expenditure.

Many interventions aim to reduce trespass by changing peoples’ knowledge, attitudes, perceptions and/or behaviours. Typical examples include educational and mass media programmes. An evaluation aim can therefore be to measure changes in peoples’ knowledge, attitudes, perceptions and/or behaviours.

This case study provides an example of evaluating changes in outcomes. 

If before and after measures are available, this allows a change in peoples’ knowledge, attitudes, perceptions and/or behaviours to be measured. 

For example:

• Before School Educational Visits 8% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”
• After School Educational Visits 95% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”

In this example, the change is very high. For smaller changes, a statistician could test the significance of changes. 

This assessment could, resource permitting, be extended by re-measuring knowledge (for example) one year later as a test of retention. 

A common approach is the use of thematic analysis.

Thematic analysis involves identifying common topics, ideas and/or issues, and sub-themes.

If a large amount of feedback is acquired, analysis can be completed using tools such as Nvivo. This requires training and validated experience.

The thematic analysis can help explain impact results by grouping feedback around the following types of questions:

• Was it the right intervention for the type of trespass location and trespass motivation?
• How well was it planned and resourced?
• Was it implemented as intended?
• Were there any unexpected issues?
• What reasons do stakeholders give for the impact of the intervention?

Evaluation question: What influenced the impact of the intervention?

Theme 1: effectiveness of patrols

The first theme of stakeholder feedback concerned the effectiveness of patrols in deterring trespass. The sub-themes were:

• The effectiveness of patrols and how this is dependent on their frequency, concurrence with peak trespass times of time and visibility. Cited by 10 BTP respondents.

• The durability of impact and how this is said to be related to the number of months that patrols are carried out. Cited by 5 BTP respondents.

Theme 2: safety and feasibility of patrols.

A second theme, cited by 8 BTP respondents, related to the safety and feasibility of patrols. The sub-themes were:

• Access to track side spaces and

• Risk of being struck by trains.