|
includes/content/subnav.asp NOINDEX
|
Review of Research and Evaluation Related to Reading Recovery
A number of studies have explored the effectiveness of the Reading
Recovery intervention as well as the lasting effects of the program
for the children served. In addition, other factors that may
influence program outcomes have been studied. In this section,
information is provided through (a) summaries of published reviews
of Reading Recovery, (b) a report on replication data across 13
years of program implementation, (c) summaries of studies that have
explored Reading Recovery's effectiveness, and (d) a discussion
about subsequent performance of Reading Recovery children. The
section ends with suggestions to consider when reading or conducting
Reading Recovery research or evaluation studies.
Several published reviews of Reading Recovery that include
information about outcomes measures are available. Five are cited in
Table 2 (page 22). Responses to some of the critical aspects of
these reviews are included in Section 4 of this document.
- First-Year Outcomes Are Compelling: 13 Years of Data
As reported in Section 1, Reading Recovery replicates its effect
at the level of individual subjects. Evaluation data are collected
annually on each child served in the Reading Recovery program.
Results from the National Data Evaluation Center are compelling.
As reported by Lyons,51
Reading Recovery has used 2 types of replication methodology to
determine program effectiveness: systematic replication and
simultaneous replication. Repeatedly producing the same results with
different students across different settings increases confidence in
an intervention, providing substantial evidence of the effectiveness
of Reading Recovery tutoring. As shown in Table 3 (page 24), results
across 13 years of data collection demonstrate the consistency of
Reading Recovery outcomes across extensive replication
documentation.
The national evaluation data can be considered in two ways:
- From 1985 to 1997, a total of 436,249 children entered the
program. This total included children who moved during their
programs and children who had insufficient time to complete
programs before the end of the year. For this total group (who
were initially the lowest achieving), 60% met the criteria for
discontinuing.
- Reading Recovery served a total of 436,249 children from
1985 to 1997. Of that group, 313,848 had enough time to
experience a complete program. Of the children with complete
programs, 81% reached criteria for release or successful
discontinuing from the program.
| Table 2: Summary of Reading Recovery
Reviews |
|
SOURCE |
Preventing Early Reading Failure with
One-to-One Tutoring: A Review of Five Programs. Wasik &
Slavin (1993) Reading Research Quarterly, 28 (3), pp.
179-200. |
| PURPOSE |
To consider the effectiveness of 5 tutorial
programs from 2 perspectives: empirical and pragmatic |
SCOPE OF
REVIEW |
The authors reviewed quantitative and
qualitative research on five tutoring programs: Reading
Recovery, Success for All, Prevention of Learning
Disabilities, Wallach Tutoring Program, and Programmed
Tutorial Reading. |
CONCLUSIONS AND
RECOMMENDATIONS |
General Conclusions Across Programs:
1. Programs with the most comprehensive models of reading
(the most complete instructional interventions) have larger
impact than programs addressing only a few components of the
reading process. RR and Success for All include several
reading components.
2. Using tutors is not enough. The content of the program
and the instructional delivery may be important variables.
3. Using certified teachers obtains substantially larger
impact than using paraprofessionals.
Conclusions About RR:
1. RR brings the learning of many of the lowest achieving
students up to average-achieving peers.
2. Effects of RR are impressive at the end of implementation
year and effects are maintained for at least 2 years.
3. Results on evaluations of lasting effects are positive
but complex.
4. Only RR has attempted to assess implementation and its
effect on outcome data. |
|
COMMENTS |
Authors
raised some methodological issues about RR research and
about students served. They concluded that the rapidly
expanding use of RR throughout the US shows that the program
is practical to use. |
| |
| SOURCE |
Reading Recovery in the United States:
What Difference Does It Make to an Age Cohort? Hiebert
(1994) Educational Researcher, 23 (9), pp. 15-25. |
| PURPOSE |
To review and examine available data on RR's
effectiveness in American contexts, specifically as it
influences an age cohort |
SCOPE OF
REVIEW |
The author examined 3 types of data on RR:
(1) the longitudinal study in Columbus, Ohio (DeFord,
Pinnell, Lyons, Place, 1990); (2) the comparison study of
early interventions (Pinnell, Lyons, DeFord, Bryk, &
Seltzer, 1994); (3) Regional training center reports from
Ohio State University, University of Illinois, and Texas
Woman's University. |
CONCLUSIONS AND
RECOMMENDATIONS |
1. A high percentage of RR children can
orally read at least first grade text at the end of grade 1.
2. Once a program is in place, there is considerable
fidelity in the results.
3. Prominent elements of the RR program are identified as
characteristics of successful beginning reading instruction.
4. Weekly training sessions give teachers an unprecedented
amount of guided observation of students.
5. Data reviewed led the author to conclude that the effects
of RR on an age cohort are unconvincing.
6. When cost figures are calculated on the basis of success
levels of remaining students at grade 4, the cost per
successful student is higher.
7. The author recommended studies with more comprehensive
tasks that fully define the sample. She also called for
exploration of effects in low-income schools and with
second-language children. It was further recommended that
the underlying principles of RR should be explored with
consideration to applicability in student-teacher contexts
other than tutoring. |
| COMMENTS |
The author stated that data on
many aspects of RR implementation are inaccessible or
incomplete. She cited limitations of existing data.
A response to Hiebert's review was published in the
Educational Researcher, 25 (7), pp. 23-25 (Pinnell,
Lyons, & Jones 1996). Hiebert's response to the response was
printed on pp. 26-28 in the same publication. |
| |
|
| SOURCE |
Reading Recovery: An Independent
Evaluation of the Effects of an Early Instructional
Intervention for "At Risk" Learners. Shanahan & Barr (1995)
Reading Research Quarterly, 30, pp. 958-996. |
| PURPOSE |
To analyze the effectiveness of RR in 4
dimensions:
1. RR students' gains relative to gains of average- and
low-achieving students
2. Maintenance of learning gains after special instruction
has ended
3. Cost and benefits of the program
4. RR's influence on other instructional changes in schools |
SCOPE OF
REVIEW |
The goal of the authors was to offer a
thorough, systematic analysis of all available empirical
work on RR. They reviewed all published evaluations of RR,
and any available unpublished ones that included sufficient
basic information to allow meaningful analysis. When
possible to analyze data in a more precise and direct
manner, data were combined across studies. Overall,
consideration of existing research and evaluation studies
was largely qualitative. |
CONCLUSIONS AND
RECOMMENDATIONS |
1. RR brings the learning of many of the
lowest achieving students up to average-achieving peers.
2. RR is a robust program in terms of consequences for
student learning and replicability across sites.
3. RR has become a significant force in shaping the way we
view early literacy development.
4. After savings from lower retention rates and special
education services, per pupil annual expenditure is
approximately $3,200 to $4,000, with variation among
districts due to teacher salaries and benefits.
5. More research is needed on RR's impact on students'
classroom experiences and ways to reduce costs.
6. More rigorous research on the effects of RR is needed as
well as studies related to program refinements to enhance
learning or efficiency. |
| COMMENTS |
This review provided perhaps the most
comprehensive independent evaluation of RR up to the time of
publication. Authors cited both caveats and challenges for
consideration related to research and to practice.
Authors of a statewide study by Pinnell, Lyons, DeFord, Bryk,
& Seltzer (1994) responded to Shanahan and Barr's claim that
half the data from that study had been lost. Pinnell
explained in a letter to the editor of RRQ, Vol. 32
(1), 1997, p. 114, that only 5 of the 40 schools were
excluded and provided the rationale. Shanahan and Barr
responded to Pinnell in the same publication. |
| |
|
| SOURCE |
Ten Promising Programs for Educating
All Children: Evidence of Impact. Herman & Stringfield
(1997). Arlington, VA: Educational Research Service |
| PURPOSE |
To report information collected in a
three-year study conducted by the Johns Hopkins University
Center for the Social Organization of Schools designed to
answer 2 questions: (1) Are there specific programs or
restructuring designs that can enhance the learning of
students who are at risk of school failure? (2) What are
their key characteristics and what local conditions and
action are required to replicate those promising programs? |
SCOPE OF
REVIEW |
Authors examined 10 different nationally
known programs that were identified as holding promise for
educating disadvantaged children. They reviewed 13 studies
of RR effectiveness and collected observational evidence at
exemplar sites. |
CONCLUSIONS AND
RECOMMENDATIONS |
1. Expectations for RR are high, in part
because the program focuses on a small number of children.
The program has a reputation for producing strong,
quantifiable reading gains.
2. A potential problem noted in some sites was tendency to
blame or label the child when the strategy was not effective
for the student.
3. Districts should be prepared to address some unintended
consequences of the program including staff jealousies over
resources, lack of coordination, and unrealistically high
expectations for the program.
4. The consistently high fidelity of program implementation
across sites was an important aspect of RR.
5. The high-quality staff development model for RR is one of
the most important aspects of RR. |
| COMMENTS |
Authors commended the staff development
model:
"The intensity and the methods utilized by RR in training
and the insistence on high level of RR performance provided
an almost singularly attractive model for future staff
development efforts, regardless of program type. As schools
systematize and create more opportunities for serious staff
development, the thoroughness of the RR model seems to be
well worth emulating." (p. 86) |
| |
|
| SOURCE |
*Reading Recovery: A Summary of Research
Pinnell (1997) Research on Teaching Literacy Through the
Communicative and Visual Arts, Flood, Heath, & Lopp
(Eds.). A Project of the International Reading Association,
pp. 638-654. |
| PURPOSE |
To summarize what is known about RR and what
has been learned through research connected with the program |
SCOPE OF
REVIEW |
The author briefly describes RR and then
reviews research on program success, on teaching and
learning, on teacher development, and on program
implementation. Where they are available, sound critical
reviews are noted. |
CONCLUSIONS AND
RECOMMENDATIONS |
The review of research on program
effectiveness includes studies from New Zealand, empirical
studies in the U.S., replication studies, results in diverse
settings, impact of contextual factors, and studies from
Descubriendo La Lectura/Reading Recovery in Spanish. The
review of research on teaching and learning includes areas
such as studies of teacher behavior and student outcomes,
teacher-student interactions, and the impact of RR on
teacher learning. A brief review of limited implementation
studies is also included. |
| COMMENTS |
|
|
*Indicates that author is directly involved
with Reading Recovery. |
| |
| Table 3: U.S. Reading Recovery
children served, program children and percentage of program
children discontinued from 1984-1997 |
| Year |
Served2 |
Program3 |
Discontinued4 |
% |
| 1984-19851 |
110 |
55 |
37 |
67% |
| 1985-1986 |
230 |
136 |
99 |
73% |
| 1986-1987 |
2,048 |
1,336 |
1,059 |
79% |
| 1987-1988 |
3,649 |
2,648 |
2,269 |
86% |
| 1988-1989 |
4,772 |
3,609 |
2,994 |
83% |
| 1989-1990 |
7,778 |
5,840 |
4,888 |
84% |
| 1990-1991 |
12,605 |
9,283 |
8,126 |
88% |
| 1991-1992 |
21,821 |
16,026 |
13,499 |
84% |
| 1992-1993 |
36,443 |
26,582 |
22,109 |
83% |
| 1993-1994 |
56,077 |
40,493 |
33,243 |
82% |
| 1994-1995 |
81,220 |
57,712 |
46,637 |
81% |
| 1995-1996 |
99,617 |
71,193 |
59,266 |
83% |
| 1996-1997 |
109,879 |
78,935 |
65,551 |
83% |
| Totals |
436,249 |
313,848 |
259,777 |
81% |
| |
1Pilot
year: |
RR teachers were in
training. |
| |
2Served: |
Program children
and children who entered Reading Recovery but did not
receive a minimum of 60 lessons because they moved, were
absent for extended periods of time, or the school year
ended prior to completion of lessons. Column 1 is inclusive
of the subcategory Program Children, column 2 |
| |
3Program: |
RR children who
received a minimum of 60 lessons or were discontinued prior
to receiving 60 lessons. This definition of program children
changed beginning in the 1998-1999 school year. |
| |
4Discontinued: |
RR children who
were released from the RR program reading within average
band reading levels of the class. |
Data on children served by Descubriendo La Lectura (DLL), or Reading
Recovery in Spanish, are also impressive. During the 1996-1997
school year, 2,951 children were served in 50 sites across 9 states.
Of the total group of 2,951, including children who moved or had
insufficient time to complete the program before the school year
ended, 58% met the criteria for discontinuing. Of the 1,952 children
with an opportunity for a complete program, 81% were successfully
discontinued.
Stakeholders, then, have substantive information about Reading
Recovery program outcomes to use in decision making. What other
widely-disseminated programs can produce 13 years of data on every
child served in the program to document results?
The summary of studies in Table 4 includes information about
initial program outcomes as well as subsequent performance of
children served. Some of the studies also compare the effectiveness
of Reading Recovery with other interventions or modifications.
Following the table is a discussion of subsequent performance of
Reading Recovery children.
Because of its
record of high quality training, program integrity, and results,
prominent scholars who are not connected with the program have
commented on its effectiveness. Here are sample comments:
-
"These
criticisms aside, the effects of Reading Recovery are impressive
at the end of the implementation year and the effects are
maintained for at least 2 years."52
(Wasik & Slavin)
-
"The program
does incorporate several key features of a successful redesign
process. It has shaped its methods according to the results of
its own and other's research. It has tested and honed its
techniques through years of trials and refinements."53
(Wilson & Daviss)
-
"Evidence
firmly supports the conclusion that Reading Recovery does bring
the learning of many children up to that of their
average-achieving peers. Thus, in answer to the question 'Does
Reading Recovery work?' we must respond in the affirmative."54
(Shanahan & Barr)
-
"No other
remedial program has ever come close to achieving the results
demonstrated by Reading Recovery."55,
(Cunningham & Allington)
-
Another
important outcome of the study was that it showed that Reading
Recovery can be a highly effective intervention program.56
(Iversen & Tunmer)
Table 4:
Review of Reading Recovery Studies
Reading Recovery
Children Continue to Progress With Their Peers After the
Intervention
Long-term research is difficult because mobility of students means
that samples unavoidably shrink. Large resources are needed to
follow students. If samples shrink too much, it is unknown how well
the sample represents the population. In addition, systemic factors
such as subsequent instruction, implementation decisions, and
individual life circumstances intervene and affect student progress.
In spite of these difficulties, there is strong evidence that the
effects of Reading Recovery are long-lasting. Studies in New Zealand
were first to document the lasting effects of the program for
children.57 An Australian
study, not designed to look for a continuing effect on the progress
of individuals involved in Reading Recovery, discovered the effect
as a surprising outcome.58
Rowe, the researcher, studied the progress made in reading by
children (N=5,000) from school entry to grade 6 in 100 schools in
Victoria, Australia. By grades five and six, Rowe found that Reading
Recovery students were distributed across the same score range as
the general school population, and with fewer low scores.
Rowe's analysis provided evidence that Reading Recovery had "removed
the tail end" of the achievement distribution. Four to five years of
classroom and school influence made children who were "tail enders"
no different from the normal variability. At the beginning of their
years in school, they were clustered at the low range; by grades
five and six, that was no longer the case.
Several follow-up studies in the United States have confirmed Rowe's
finding that in later grades the scores of Reading Recovery children
more closely approximated the spread of scores in the random group.
For example, in one follow-up study59,
about 70 percent of the former discontinued children had scores
considered to be average or meeting passing criteria on 2
standardized measures of reading comprehension by their fourth-grade
year. Findings, consistent with the conclusions of Rowe as well as
Shanahan and Barr,60 show
that some Reading Recovery children remain tentative in their
literacy behaviors after the intervention but perform better at
higher grades.
It is important to remember that in these studies the children whose
initial assessments were at the tail-end of the population
distribution are being compared with children whose initial
assessments are assumed to represent a normal curve distribution.61
It would not be surprising if overall performance on standardized
literacy measures is lower for some former Reading Recovery children
than for the general population.
School districts adopt Reading Recovery to fulfill a responsibility
to teach all children. Many of these school districts conduct their
own inquiries into the subsequent progress of Reading Recovery
students. Most of these follow-up studies are not controlled studies
with random assignment and other features of experimental research.
However, the reality of school evaluation requires different rigor
than experimental study. These evaluations represent solid evidence
for school districts that Reading Recovery students, originally the
lowest achieving first graders, are achieving within an average
range and profiting from ongoing classroom instruction. These
results are replicated in site after site.
After any early intervention some children, initially doing well,
might remain vulnerable to life circumstances or poor subsequent
instruction. Despite a successful early intervention, a student
might remain vulnerable in many different ways. After all, one
cannot expect 30 to 50 hours of instruction, no matter how intensive
or accelerative, to be the only support a student has throughout 12
years of schooling.
Program quality, too, makes a difference. When programs are just
getting started, there may be scanty coverage, implementation
problems, or weak training; the long-term gains might not be as
robust as expected. The program needs to gain strength through
improved implementation and experience.
Claiming that the fading of learning gains is a persistent problem
for interventions in general, Rozzelle has suggested several
underlying causes, including the fact that change takes time.
Complex change takes even more effort and time to achieve quality
results. Rozzelle cautions schools to protect new programs during
the implementation stage, to monitor student progress, and to plan
for ongoing teacher training and accountability.62
One perspective might be to see early intervention as a first step
in a series of educational experiences and interactions. We do see
Reading Recovery as a first step in supporting a child, who for a
variety of reasons, does not make good progress in reading and
writing. Subsequent support need not be expensive and would
certainly include high quality, continuing support for classroom
teachers so that instruction is strong year after year, whatever the
particular method or approach chosen by the school district.
Success in the
early grades does not guarantee success throughout the school years
and beyond, but failure in the early grades does virtually guarantee
failure in later schooling. If there is a chance to prevent the
negative spiral that begins with early reading failure from the
start, then it seems necessary to do so.63
Considerations
When Reviewing or Conducting Research on Reading Recovery
When Reading Recovery programs are being evaluated, the
interpretation of the data obtained depends upon the characteristics
of the implementations. The characteristics of the implementations
sampled must be reported along with the characteristics of the
samples of children if we are to understand the results obtained.
"The essential conditions for the success of Reading Recovery in a
system lie in the coherence, the resourcing and the reach of the
support and quality assurance structures which are put in place for
its implementation."64
From many available evaluations it is clear that the following
factors influence the results found in evaluations: the age of the
implementation, assurance that the teachers were beyond their year
of training and current in their knowledge of the program, the level
of implementation in the schools (i.e. what proportion of children
who needed the program received a full program), whether the program
was running effectively (i.e. what proportions of the children
served reached one of the two positive outcomes of the program
described on pages 12-13), assurance that children received daily
lessons, at least descriptive comment on the quality of classroom
support across the years of implementation, and the support of
administrators and other stakeholders.
Readers of evaluations are advised to check original sources when
reading critiques of research to verify any errors in reporting. For
example, researchers in one study65
claimed that Clay had excluded some children from her original data.
Their claim was then repeated by others. However, Clay's 197966
publication provides documentation that no children were dropped
from her samples. In another example, reviewers67
reported that half of the data were lost in a statewide study in
Ohio.68 Responses from the
investigators69 revealed that
only 5 of 40 schools were excluded for reasons approved by an
outside research advisory board.
It is beyond the scope of this document to explore all the factors
to be considered when conducting or reviewing research and program
evaluations. However, we suggest that research and evaluation
studies related to Reading Recovery should be examined with
attention to their accuracy in reporting the original studies,
issues relating to features of design and methodology, and possible
biases and limitations of the findings. It is also important that
evaluators become familiar with the complexity of the program and
give due weight in evaluation to both positive outcomes.
|
