3. Implement and Determine the Effectiveness of a Program
to Improve Attendance or Reduce Drop-Outs

This kind of project requires two kinds of evaluation:

1. A formative evaluation to determine if the new program has been implemented.

2. A summative evaluation to determine if the newly implemented program has improved attendance or reduced drop-outs.

I   Formative Evaluation

The task here is to determine how well (or, indeed, if) the new program has been implemented. This is not something that can be taken for granted.

Thus, you will need to assemble credible, objective evidence (not simply your own judgments) on the following types of questions:

1. Does the program exist?
2. Is it operating as it is supposed to?
3. If not, what changes are needed to make it operational?
4. Are funds being appropriately spent?
5. Have the necessary books, materials, and equipment been purchased and made available to the teachers and students?
6. Has the program hired and trained competent staff?
7. Is the program serving the intended students?
8. Are they receiving the intended educational information and services?

This kind of evaluation will tell you whether the program is up and running and potentially capable of having an impact. It is essential to complete this type of evaluation before undertaking a summative, impact, or outcome evaluation. It makes no sense to evaluate the effectiveness of a program that does not exist-even though we regrettably see many instances of this.

II    Summative Evaluation

The two basic questions here are:

>Has there been an improvement in attendance or a reduction in drop-outs?

>Can any improvement that has occurred be confidently attributed to the new program rather than other factors?

There are two preferred evaluation designs for answering these questions:

(1)  Collect attendance or drop-out data on (1) a group of students, a class, or a school exposed to the new program and on (2) a comparable control group of students, a class, or a school that wasn't exposed to the program. If attendance or drop-out rate improvements are significantly greater in the treatment group, you can confidently conclude that these improvements were the result of the new program. Simply collecting pre- and post-program data on the treatment group without a control group will not do. Attendance or drop-out rates may have improved because of other conditions or factors besides the new program. For comparison, you need some measure of change under the conditions of non-treatment, i.e., a control group.

(2)  As an alternative, if a control group is not available, you can use what is known as the interrupted time-series design. If attendance or drop-out data are available for several years or periods prior to the introduction of the new program, and there is a marked improvement in these rates immediately following the introduction of the new program that is sustained in following years or periods, you can confidently conclude that the improvements were the result of the new program.

For more information on these two designs, see the section on Alternative Summative Evaluation Methods in the Short Course on Evaluation Basics, and references 1 (Campbell), 4 (Cook & Campbell), and 8 (Rossi & Freeman) in the Evaluation References.

Design Selections | Evaluation Support Home
Applied Research Center | EdD Major | Ed Leaders Home/ Basic Statistics