Impact evaluation: Before and after comparison

28/08/2020

A common form of rail trespass prevention impact assessment is to compare the number of incidents between the period before and the period after an intervention has been implemented. The aim is to determine whether the number of incidents have fallen after implementing an intervention and, as data permits, to quantify the magnitude of reduction and the level of confidence that this reduction is due to the intervention and is not a chance coincidence. A before and after comparison may also determine if an upward trend in incidents has been slowed down, with a slower rate of increase after implementing an intervention.

The reliability of a before and after comparison can be increased by applying good practice. These include:

Comparing changes in an intervention area to a similar area(s) that did not have the intervention, ideally chosen at random.

In an operational setting, it may not be possible to randomly chose an intervention and comparison area. For example, the intervention area may be chosen due to the occurrence of fatal incidents or a trespass reduction strategy. In this situation, a comparison area may be chosen “opportunistically” based on identifying the best available similar comparison area(s).

Implementing an intervention in a number of areas, and assessing before and after trends, to check that the measured impact is repeated, consistent and not a spurious change in recorded incidents.

In an operational setting, this may need to be achieved “opportunistically” by evaluating each instance of an intervention and pooling results.

Having a long enough before and after period to check that any fall in the After Period is not simply a fall back to a longer-term average number of incidents or due to seasonal trends.

The ability to check longer term trends depends on having a consistent and reliable data set going back over a number of years, such as 3 years or more.

Having a large enough count of incidents that the data is statistically robust.

In the context of rail trespass, many interventions are small scale and/or implemented in a relatively small area. This may cause the count of incidents in the intervention area to be, in statistical terms, low. As elaborated below, options including pooling data for a number of intervention areas, in order to boost the count of incidents.

The conduct of before and after analysis may require specialist support, such as from a research analyst. Where it is uncertain whether the incident data is robust, specialist advice can be sought from an analyst or statistician to check the power and reliability of the data.

Some of the potential challenges in achieving a valid before and after comparison, include.

Avoiding false positive results
A cross cutting issue with before and after assessment is that it always possible that the number of incidents in an intervention area fluctuates up and down over a period of years. Care must be taken that the intervention was launched during a peak in incidents, and that incidents then fell back down to their longer-term average. This could give a false positive result, attributing a reduction in incidents to the intervention when it is in fact a coincidental fall in incidents.

This is illustrated in Figure 6. The year used as a Before has an average of 80 incidents per year, compared to 58 in the After year. This would give a large fall in the measured number of incidents. However, the average number of incidents in the After period is actually the same as the longer-term average of 58. The intervention simply coincided with a random fluctuation in the number of incidents.

In the context of trespass, an intervention may be implemented due to evidence of a rise in incidents. It is possible that the number of incidents rise and fall for no specific reason. This creates a risk that interventions are implemented when incidents peak, followed by a “natural” return to a lower number, especially if short time periods are compared.

A number of ways for reducing the risk of a false positive result include:

Pooling data for a larger number of areas, so that the random fluctuation in incidents is reduced;

Examining data for a longer before and longer after period, so that it can be checked that the measured reduction is not just a decline back to a longer-term average number of incidents;

Having similar comparison areas against which to compare trends.
Controlling for seasonal trends in incidents

National data indicates that trespass incidents are lowest in winter, rise in spring to a May peak and then declining. A hypothetical poor before and after comparison is shown in the figure below. The before period is the peak spring season. The after period starts with the seasonal decline from July onwards.

A before and after comparison needs to match the periods being compared to avoid a seasonal fall in incidents being wrongly attributed to an intervention.

If it is judged that only one intervention has impacted an area(s), then a simple before and after comparison with matched periods and robust incident data may be sufficient.
Impact Examples

Two examples of impact assessment can be downloaded here, both these examples illustrate an impact assessment using actual NDFU data to retrospectively assess targeted locations

Example 1: Coppermill Junction to Harlow

Example 2: Colchester to Hythe.
Isolating impact of coincidental interventions or changes

Comparing between intervention and other area(s)

It is possible that a reduction in incidents in an intervention area occurs due to other factors. An option is to compare changes in incidents in the intervention area(s) with another area(s) that does not have the intervention. This is termed having a “control” group(s). Ideally the control group has a similar residential population, similar socio-demographics, and similar types of stations and running lines.. The extent to which the control groups are similar or not should be determined and acknowledged in the impact evaluation.

The duration of the before and after periods need to be long enough to check that any fall in incidents in the “intervention” area is not a spurious return to a historical average number of incidents. This may require longer term (a number of years) data to be examined and used as a before period.

Isolating impact of coincidental interventions.

If there is more than one intervention impacting a location, it can be difficult to isolate the impact of each intervention. This may require assessing trends in other areas which do and do not have both interventions.

Changes in other factors

It is also possible that other factors change between the before and after period. For example, the adjacent land use may change, deprivation may change or the number of trains may change. Firstly, it Is important to spot significant coincidental changes in other factors that may influence the frequency of trespass. This can be achieved by consulting local stakeholders and open source data on, for example, local numbers of crime. If there has been a significant change in other factors, one option is to exclude the area from the evaluation and instead use an area(s) which has not experience coincidental changes.

The control of other factors can become complicated and advice from a specialist analyst nay be required.
Low numbers of incidents
If the number of incidents is, in statistical terms, low, this can make it difficult to confidently test changes in the number of recorded incidents.

Generally speaking:

Trends in tens of incidents in the before and after period will have a low power.

Trends in about one hundred incidents in the before and after period will have a moderate power.

Trends in hundreds of incidents in the before and after period will have a higher power.

If the number of incidents is low, then the possibility of pooling data for a number of intervention areas and/or using longer time periods should be considered.

Some qualitative guidelines on confidence in the data set are provided below

Very high - large dataset (hundreds of incidents), with long before and after period (years), randomly chosen (matched) comparison areas, clear trends in data.

High - large dataset (hundreds of incidents), with long before and after period (years), opportunistic (matched) comparison areas, moderately clear trends in data.

Moderate - before and after comparison using a matched seasonal data from previous years, comparison of one intervention area to another matched area, maybe moderate number of incidents (~100) in before period.

Low - before and after comparison using a matched seasonal data from previous years within one area, maybe moderate number of incidents (~100) in before period.

Very low - before and after comparison using a few months of data within one small area, maybe low number of incidents (dozens).

The “power” of a dataset can be tested statistically.

It is also possible for a statistician or analyst to provide a power calculation of the before and after incident data. A power calculation indicates the probability of being able to detect a difference between the before and after data.

Power ranges from 0 to 1, where 0 is zero power and 1 is maximum (perfect) power. As a guideline, a power of 0.8 (80%) or more is required. This means there is an 80% chance of being able to detect a change between before and after data.
Overcoming noisy and unclear trends
The number of incidents per month (or per quarter) may fluctuate a lot. A hypothetical example is given in the figure below. This can make it difficult to discern a clear change in the number of incidents after implementation. This may require either:

The use of a longer comparison time period and/or

Pooling of data from a number of locations, in order to achieve a clearer trend.

For example:

Data could be pooled for 20 stations that have had end of platform barriers installed, rather than assessing each station by itself.

Data could be pooled for 10 “hotspots” that had additional police patrols.

This may provide clearer trends from which an impact can be discerned.
Spotting displacement

It is possible that an intervention has “displaced” trespass outside of the intervention areas. A comparison of incidents before and after within the intervention area may show a fall but overlook the point that these have been displaced to adjacent areas. For example, installing security fencing on a section of running line may prevent trespassers walking the line there, but trespassers go to other unprotected sections of running line.

In order to check for this, the change in the number of incidents in adjacent areas should be checked. A hypothetical example is shown in the table below. In this example, the number of incidents in areas with newly installed fencing have fallen, but trespass has risen in adjacent areas without security fencing.

Another option is to plot incidents on a map for the before and after periods, and visually check for displacement.
Spotting substitution effects

Substitution may involve, for example, trespassers throwing stones at trains instead of vandalising line side equipment after security fencing is installed. If the evaluation only assessed vandalism it would overlook a rise in stone throwing. Table 16 below gives a hypothetical example of a 40% fall in vandalism coinciding with an 49% rise in line side stone throwing. The (hypothetical) intervention was a spring/summer targeted fence repair and clean-up programme.

A judgement is required of whether the trespass motivation may be associated with substitution. If this is possible, then the mixture of types of trespass before and after should be checked.