Menu

Decide on evaluation measures

The evaluation measures should be feasible, be relevant to the purpose of the evaluation and align to the Logic Model.

The evaluation measures can be mapped to each stage of a Logic Model. This would help “trace” the relationship between inputs, outputs, outcomes and impacts. For example:

  • Input – a measure of the resource committed to developing and carrying out school educational visits
  • Output - a measure of the number of school visits performed
  • Outcome - a measure of change in student perceptions of rail trespass risk
  • Impact - a measure of change in number of trespass events involving young people 

If the evaluation aims to assess the process, this will require additional “measures” related to stakeholder feedback on the implementation process and a comparison of intended versus actual implementation. 

If there is a potential for unintended impacts, then measures of displacement and substitution may be required.  This is referred to in the Impact evaluation: Before and after comparison section.

More information and advice on evaluation measures is provided below.

  • Process

    Typical questions asked in evaluation

    1. Was the intervention selected on the basis of a review of the type of trespass, its causes and motivations?
    2. Did the design of the intervention follow recognised good practice?
    3. Has the intervention been implemented and maintained as intended? 
    4. What were the inputs and outputs of the intervention?
    5. What influenced the extent and quality of implementation?
    6. What is the perception of the intervention amongst people implementing it and / or receiving the intervention? 
    • Do they think it is practical and effective? 
    • Do they want to carry on engaging with the intervention?
    • How effective do stakeholders think the intervention is? Why is this?
    1. What could be done differently to increase the effectiveness of the intervention? 

    Typical data and information collection methods

    • Intervention planning documentation, such as description of the intervention and underpinning assessment.
    • Data on the volume of work done, such as how many patrols have been performed or how many school visits have been completed?
    • Interviews, questionnaire surveys and workshops with people implementing the intervention.
    • Interviews, questionnaire surveys and workshops with the target population, such rail passengers, residents and school pupils.
    • Case studies of the intervention
    • Observation or inspection of activities, such as observing an education session or inspection of a newly installed fence.
  • Impact

    Typical questions asked in evaluation

    1. To what extent has the number of incidents changed in the target location(s)?
    2. To what extent has the number of incidents changed (i.e. more so in the target location) when compared to other similar areas lacking this intervention?
    3. Is there evidence that the impact will be sustained?
    4. Was the intervention cost-effective?
    5. To what extent has there been a change in outcomes, such as attitudes, knowledge, capacity and behaviours?
    6. What influenced the impact of the intervention?
    7. Were there unintended impacts?

    Typical data and information collection methods

    • Incident data: Before and after comparison of incident data (SMIS) in the target location compared to trends in other areas (without the intervention).
    • Data on other factors, such as number of passengers, changes in train throughput and changes in local crime levels, which may have influenced the impact of the intervention.
    • Interviews, questionnaire surveys and workshops with people implementing the intervention.
    • Time and cost data: Cost of materials, amount of time involved in implementation
    • Schedule 8 payments data.
  • Lessons learnt

    Typical questions asked in evaluation

    1. What processes helped achieve success?
    2. What were the barriers to implementation and impact?
    3. What could be done differently to increase the effectiveness of the intervention?

    Typical data and information collection methods

    Same data as for process assessments.

  • Summative

    Typical questions asked in evaluation

    1. How does the cost-effectiveness of this intervention compare to other interventions?
    2. How feasible and worthwhile is it to implement this intervention elsewhere?
    3. What needs to be done to secure the legacy impact of this intervention?
    4. What could be done differently to increase the effectiveness of this type of intervention in the future?

    Typical data and information collection methods

    Same data as for process and impact assessments plus evaluation results for other interventions..

Haven’t found what you’re looking for?
Get in touch with our Senior Safety Intelligence Analyst for further information.
Siona Vass
Tel: 020 3142 5485
X
Cookies help us improve your website experience.
By using our website, you agree to our use of cookies.
Confirm