12. Improve the Achievement Scores of Low-Scoring Students

This kind of project requires two kinds of evaluation:

1. A formative evaluation to determine if the program for raising the students scores has been implemented.

2. A summative evaluation to determine if the newly implemented program has improved student achievement scores.

I  Formative Evaluation

The task here is to determine how well (or, indeed, if) the new program has been implemented. This is not something that can be taken for granted.

Thus, you will need to assemble credible, objective evidence (not simply your own judgments) on the following types of questions:

1. Does the program exist?
2. Is it operating as it is supposed to?
3. If not, what changes are needed to make it operational?
4. Are funds being appropriately spent?
5. Have the necessary books, materials, and equipment been purchased and made available to the teachers and students?
6. Has the program hired and trained competent staff?
7. Is the program serving the intended, low-scoring students?
8. Are they receiving the intended educational information and services?

This kind of evaluation will tell you whether the program is up and running and potentially capable of having an impact. It is essential to complete this type of evaluation before undertaking a summative, impact, or outcome evaluation. It makes no sense to evaluate the effectiveness of a program that does not exist-even though we regrettably see many instances of this.

II  Summative Evaluation

The two basic questions here are:

>Has there been an improvement among the low-scoring students on the measure of student achievement?

>Can any improvement that has occurred be confidently attributed to the new program rather than other factors?

The preferred evaluation design for answering these questions involves the use of a control group. Administer pre- and post-achievement tests to both a group of low-scoring students exposed to the new program and a comparable control group of low-scoring students who weren't exposed to the program. If the gains of the treatment group are statistically and educationally greater than those of the control group, you can confidently conclude that these gains were the result of the new program. Simply administering pre- and post-tests to the treatment or program group without a control group will not do. Scores may have increased because of other conditions or factors besides the new program. For comparison, you need some measure of change under the conditions of non-treatment. (See the section on An Example of the Most Common Pitfall in Evaluating Education Programs in the Short Course on Evaluation Basics.)

Caution: There is a special problem in this type of project where low-scoring students are selected to be the target population for the new program. It is referred to as "regression to the mean." The problem is that extreme scores are often unreliable and unstable. Practically, what this means is that if you select a group of low-scoring students and administer the test again in, say, a month, it is likely that the mean score for the group will have increased, regardless of whether the students participated in any new program. (The same is true for extremely high scores; the will typically decline.) The usual method for dealing with this problem is to use a different test for selecting the students than the one that will be used as the pre- post-measure of achievement gain.

For more information on this design, see the section on Alternative Summative Evaluation Methods in the Short Course on Evaluation Basics, and references 1 (Campbell), 4 (Cook & Campbell), and 8 (Rossi & Freeman) in the Evaluation References.

Design Selections | Evaluation Support Home
Applied Research Center | EdD Major | Ed Leaders Home/ Basic Statistics