Menu

Process evaluation and lessons learnt

Process evaluation may involve comparing planned versus actual implementation, delivery of outcomes and lessons learnt.
  • Actual vs planned inputs and outputs

    A key aspect of process evaluation is to compare actual versus planned implementation. With reference to the Logic Model, this means evaluating the intervention Inputs and Outputs.

    This is a result in its own right and helps to explain Impact results. 

    This may be a simple comparison of actual vs planned activity (outputs). For example, there were 24 School Educational Visits compared to a plan of 20. Or there were 50 roadshows instead of a plan for 100. 

    A similar approach can be adopted for evaluating inputs, such as checking actual expenditure versus planned expenditure.

  • Outcomes

    Many interventions aim to reduce trespass by changing peoples’ knowledge, attitudes, perceptions and/or behaviours. Typical examples include educational and mass media programmes. An evaluation aim can therefore be to measure changes in peoples’ knowledge, attitudes, perceptions and/or behaviours.

    This case study provides an example of evaluating changes in outcomes. 

    If before and after measures are available, this allows a change in peoples’ knowledge, attitudes, perceptions and/or behaviours to be measured. 

    For example:

    • Before School Educational Visits 8% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”
    • After School Educational Visits 95% of 153 pupils knew “How far it takes for a train to stop in equivalence of football pitches”

    In this example, the change is very high. For smaller changes, a statistician could test the significance of changes. 

    This assessment could, resource permitting, be extended by re-measuring knowledge (for example) one year later as a test of retention. 

  • Analysis of interviews and workshops

    A common approach is the use of thematic analysis.

    Thematic analysis involves identifying common topics, ideas and/or issues, and sub-themes.

    If a large amount of feedback is acquired, analysis can be completed using tools such as Nvivo. This requires training and validated experience.

    Analysis of interviews and workshops diagram

  • Group analysis around evaluation questions

    The thematic analysis can help explain impact results by grouping feedback around the following types of questions:

    • Was it the right intervention for the type of trespass location and trespass motivation?
    • How well was it planned and resourced?
    • Was it implemented as intended?
    • Were there any unexpected issues?
    • What reasons do stakeholders give for the impact of the intervention?
  • A hypothetical example of a thematic analysis result

    Evaluation question: What influenced the impact of the intervention?

    Theme 1: effectiveness of patrols

    The first theme of stakeholder feedback concerned the effectiveness of patrols in deterring trespass. The sub-themes were:

    • The effectiveness of patrols and how this is dependent on their frequency, concurrence with peak trespass times of time and visibility. Cited by 10 BTP respondents.
    • The durability of impact and how this is said to be related to the number of months that patrols are carried out. Cited by 5 BTP respondents.

    Theme 2: safety and feasibility of patrols.

    A second theme, cited by 8 BTP respondents, related to the safety and feasibility of patrols. The sub-themes were:

    • Access to track side spaces and
    • Risk of being struck by trains.
Haven’t found what you’re looking for?
Get in touch with our Senior Safety Intelligence Analyst for further information.
Siona Vass
Tel: 020 3142 5485
X
Cookies help us improve your website experience.
By using our website, you agree to our use of cookies.
Confirm