Impact evaluation: Before and after comparison
The reliability of a before and after comparison can be increased by applying good practice. These include:
Comparing changes in an intervention area to a similar area(s) that did not have the intervention, ideally chosen at random.
In an operational setting, it may not be possible to randomly chose an intervention and comparison area. For example, the intervention area may be chosen due to the occurrence of fatal incidents or a trespass reduction strategy. In this situation, a comparison area may be chosen “opportunistically” based on identifying the best available similar comparison area(s).
Implementing an intervention in a number of areas, and assessing before and after trends, to check that the measured impact is repeated, consistent and not a spurious change in recorded incidents.
In an operational setting, this may need to be achieved “opportunistically” by evaluating each instance of an intervention and pooling results.
Having a long enough before and after period to check that any fall in the After Period is not simply a fall back to a longer-term average number of incidents or due to seasonal trends.
The ability to check longer term trends depends on having a consistent and reliable data set going back over a number of years, such as 3 years or more.
Having a large enough count of incidents that the data is statistically robust.
In the context of rail trespass, many interventions are small scale and/or implemented in a relatively small area. This may cause the count of incidents in the intervention area to be, in statistical terms, low. As elaborated below, options including pooling data for a number of intervention areas, in order to boost the count of incidents.
The conduct of before and after analysis may require specialist support, such as from a research analyst. Where it is uncertain whether the incident data is robust, specialist advice can be sought from an analyst or statistician to check the power and reliability of the data.
Some of the potential challenges in achieving a valid before and after comparison, include.