Corrections_Today_September_October_2020_Vol.82_No.5

NIJ Update

As an example, if the impact of a drug treatment program on the rate of drug relapse is being explored, the drug treatment (X) must be provided before the period of time that drug relapse (Y) is measured, there must be a relationship between receiv- ing the drug treatment program (X) and whether or not drug relapse (Y) occurs, and there cannot be varia- tion between those who do and do not receive the drug treatment (X) in other factors that affect drug relapse (Y) — such as a person’s internal motivation to change (Z). Establishing a good comparison group is at the heart of determining the causal impact of a program or policy. The goal is to compare those receiving the treatment program or policy of interest to a comparison group of those who look identi- cal to the treatment group, with the only difference being that one group receives the treatment and the other group does not. While various statistical options exist for establishing a comparison group, the RCT is the strongest design because it best establishes equally comparable groups on all known/measurable and unknown/ unmeasurable factors. It accom- plishes this by assignment of eligible individuals at random either to a treatment group (those who will receive the program or policy) or to a control group (those who will not receive the program or policy). By random assignment, the treatment and control group are essentially identical on all observed and unob- served factors. That is why the RCT design is so appealing, often referred to as the “gold standard” of evalua- tion methods.

complex. Performing these experi- ments correctly, with a reasonable level of credibility, requires a fair amount of statistics knowledge and experience. In other words, it is easy to get things wrong using quasi-ex- perimental methods. Third, it is fairly well documented that quasi-experi- mental methods tend to exaggerate the size of the effects found in criminal justice programs or policies. That distortion produces misleading results on what programs or policies work and by how much. 1 By random assignment, the treatment and control group are essentially identical on all observed and unobserved factors.That is why the RCT design is so appealing, often referred to as the “gold standard” of evaluation methods. It should therefore be suspected that to the degree evaluation research in correctional contexts relies on quasi-experimental designs, it is likely exaggerating the true impact of programs and policies. This may

Why not use a quasi- experimental alternative to the RCT? For evaluating the causal impact of a program or policy, the alterna- tive to an RCT is an observational study, in which the evaluator must rely on a retrospective look at the individuals who have already par- ticipated in the program or policy and try to identify a suitable com- parison group of those who did not participate. This approach is difficult because often program partici- pants self-select into treatment, or treatment is assigned based on prede- termined criteria. Differing outcomes following the treatment may be due to preexisting differences between the groups, as opposed to the treat- ment itself. This phenomenon is often referred to as selection bias. Evaluators can choose from a number of alternative statistical methods for creating comparable groups and addressing selection bias. These methods are referred to as quasi-experimental designs. For several reasons, however, none of these methods can produce estimates of the causal impact of a program or policy that are as unbiased and consistent as the estimates produced by an RCT design. First, quasi-experimental designs only allow the evaluator to address observed selection biases. Only the RCT can also address unobserved biases, such as differences in internal motivation to change. Second, even if very little unobserved selection bias existed, the statistical methods required to address observed biases in high-quality quasi-experimental designs are quite sophisticated and

Corrections Today September/October 2020 — 19