Process evaluation and lessons learnt
-
Actual vs planned inputs and outputs
A key aspect of process evaluation is to compare actual versus planned implementation. With reference to the Logic Model, this means evaluating the intervention Inputs and Outputs.
This is a result in its own right and helps to explain Impact results.
This may be a simple comparison of actual vs planned activity (outputs). For example, there were 24 School Educational Visits compared to a plan of 20. Or there were 50 roadshows instead of a plan for 100.
A similar approach can be adopted for evaluating inputs, such as checking actual expenditure versus planned expenditure.
-
Outcomes
Many interventions aim to reduce trespass by changing peoples’ knowledge, attitudes, perceptions and/or behaviours. Typical examples include educational and mass media programmes. An evaluation aim can therefore be to measure changes in peoples’ knowledge, attitudes, perceptions and/or behaviours.
This case study provides an example of evaluating changes in outcomes.
If before and after measures are available, this allows a change in peoples’ knowledge, attitudes, perceptions and/or behaviours to be measured.
For example:
- Before School Educational Visits 8% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”
- After School Educational Visits 95% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”
In this example, the change is very high. For smaller changes, a statistician could test the significance of changes.
This assessment could, resource permitting, be extended by re-measuring knowledge (for example) one year later as a test of retention.
-
Analysis of interviews and workshops
A common approach is the use of thematic analysis.
Thematic analysis involves identifying common topics, ideas and/or issues, and sub-themes.
If a large amount of feedback is acquired, analysis can be completed using tools such as Nvivo. This requires training and validated experience.
-
Group analysis around evaluation questions
The thematic analysis can help explain impact results by grouping feedback around the following types of questions:
- Was it the right intervention for the type of trespass location and trespass motivation?
- How well was it planned and resourced?
- Was it implemented as intended?
- Were there any unexpected issues?
- What reasons do stakeholders give for the impact of the intervention?
-
A hypothetical example of a thematic analysis result
Evaluation question: What influenced the impact of the intervention?
Theme 1: effectiveness of patrols
The first theme of stakeholder feedback concerned the effectiveness of patrols in deterring trespass. The sub-themes were:
- The effectiveness of patrols and how this is dependent on their frequency, concurrence with peak trespass times of time and visibility. Cited by 10 BTP respondents.
- The durability of impact and how this is said to be related to the number of months that patrols are carried out. Cited by 5 BTP respondents.
Theme 2: safety and feasibility of patrols.
A second theme, cited by 8 BTP respondents, related to the safety and feasibility of patrols. The sub-themes were:
- Access to track side spaces and
- Risk of being struck by trains.