College Success Foundation
Achievers Scholars 10-Year Follow-up Study
The purpose of this report is to provide a summative evaluation of the impact of the College Success Foundation’s (CSF) implementation of the Washington State Achievers program on participants and on participating high schools. Grant funding for new scholarships and support programs in the Achievers high schools concluded in 2010 after 10 years, when the last scholarship recipients graduated from high school and entered into college. The funding for the scholarship recipients will continue through 2015 for the scholars currently in college. The BERC Group, Inc. conducted a retrospective study to 1) document program outcomes, 2) better understand the program components and contextual factors that influenced the success of the program, and 3) explore promising practices to improve college-ready high school graduation and college completion rates of low-income students. The following evaluation questions were posed for this study.
The participants for this study fall into three comparison groups: (1) Achievers Scholarship Recipients versus Non-Recipients, (2) Achievers High Schools versus Comparison High Schools; and (3) high and low performing Achievers High Schools and their matched Comparison Schools. Non-Recipients are students who applied for but did not receive the Achievers Scholarship. Researchers originally selected the Comparison High Schools in 2004 for a Comparison School Study in the 2004-2005 school year from a pool of 86 schools similar to the Achievers schools in enrollment, race and ethnicity, percentage of students receiving free or reduced lunch price lunch, and academic achievement. In consultation with the Gates Foundation, 16 Comparison Schools were identified from the pool of 86. Four Comparison Schools subsequently received the Bill & Melinda Gates Foundation Model Districts Initiative grant and were replaced with schools from the original pool.
Researchers collected quantitative data from all high schools. Researchers also conducted site visits at 10 Achievers High Schools, including the five schools with the most positive outcomes and the five schools with the least positive outcomes. To determine the school outcome rankings, researchers created a metric using a variety of measures including implementation ratings from CSF personnel, rates of improvement on high school graduation, college eligibility, and college directattendance (i.e., enrolling in college within the year following high school). In addition, researchers conducted site visits at six Comparison High Schools, including the matched schools for the top three and bottom three ranked Achievers High Schools.
EVALUATION QUESTION #1: To what extent is there a change in rates for college eligibility, college application, scholarship application, college attendance, college persistence, and college degree completion among Washington State Achievers and applicants not selected to participate in the program?
To answer Evaluation Question #1, researchers analyzed schools’ master schedules; transcripts; graduation rates; pre-college course taking patterns; and college attendance, persistence, and graduation data. Key findings from the quantitative analyses included:
EVALUATION QUESTION #2: To what extent is there a change in rates for college eligibility, college application, scholarship application, college attendance, college persistence, and college degree completion among high school and college participants in Achievers High Schools and the Comparison High Schools?
To answer Evaluation Question #2, researchers analyzed schools’ master schedules; transcripts; graduation rates; pre-college course taking patterns; and college attendance, persistence, and graduation data. Key findings from the quantitative analyses included:
EVALUATION QUESTION #3: To what extent and in what ways did the partnership with the College Success Foundation contribute to improved outcomes?
Results from the interviews and focus groups at all 10 Washington State Achievers high schools show participants agree they benefitted from their partnership with CSF. Reflecting on changes that have occurred since receiving the Achievers grant, staff members agreed they have more of a college focus, with a stronger college readiness agenda than prior to the grant. Some of these elements included adding an advisory program; implementing college awareness nights, FAFSA nights, and parent nights; offering the PSAT in schools; providing SAT/ACT support through Princeton Review; increasing rigor; and changing graduation requirements. The CSF College Preparatory Advisor (CPA) initiated many of these program elements. Furthermore, many schools implemented assignments that support students in preparing for the college application and scholarship process, such as writing essays in English class. Staff members credited the partnership with CSF as contributing to these programs. Students and staff members reported that CSF personnel and specific program elements helped create a college ready culture. Current college students reported they would like to have more information and updates from CSF staff rather than from the College Mentor Coordinator.
Staff members and students agreed that all program elements, in combination, helped contribute to improved outcomes. Among the program elements cited as most helpful were:
EVALUATION QUESTION #4: What features of the model are associated with higher rates of college eligibility, college attendance, and college degree completion?
Researchers analyzed the common characteristics of the top five improving schools. The results from the interviews and focus groups were qualitatively different than those from the bottom five. Several key characteristics were evident in the top five improving schools:
A number of contextual factors that slowed progress in the lower performing schools, and in many cases, it is simply the reverse of what helped schools improve. Some of these issues included limited staff buy-in, negative reactions to the Small Schools’ Initiative, lack of school level leadership, and lack of high expectations. While some of the highest performing schools shared some of these issues, they found ways to work around the issues.
EVALUATION QUESTION #5: What promising practices emerge around College Readiness?
Researchers analyzed the practices occurring at the high performing Achievers High Schools and the Comparison High Schools that appeared to be contributing to some of the improvements in getting students college ready. The practices included combinations of CSF program elements as well as school-wide support programs. These promising practices include building communities of studentswho provide support and camaraderie, providing access to dedicated college advisors in the high school, setting expectations for high school students around college application processes, providing structured support for the college application process, setting higher expectations for academic progress in high school, providing meaningful mentoring, providing effective college and career programs and resources in high schools, providing experiences to understand college and college life, providing funding opportunities for overlooked students, and having common beliefs and focus among high school staff members.
Access to a dedicated CPA was critical for many Achievers Scholars while they were in high school. The advisors provided information, resources, social and emotional support, and direction. Having an advisor focused solely on college and who could relate to the students clearly made a difference in students pursuing college. Some of the promising practices are described in more detail below:
The Achievers High School staff members believe the Achievers grant had a positive impact on their schools. Staff members believe that the grant and the support from CSF changed the culture of the school and the community. Staff members reported that the focus on college and scholarships forced teachers to address their own belief system about who can and should attend college, and within the community, there were increased expectations that the school prepare students for college.
The Achievers Scholarship created a legacy, both at the school and community levels, by increasing motivation and awareness. Current students with older siblings, cousins, and friends in college now visualize attending college, and staff members know they have to continue to support this vision. They also believe the community now expects this, and it is their responsibility to sustain the accessibility of college provided through the scholarships. The degree of sustainability varies across schools. Some schools continue to receive the Achievers Program without the scholarships, and at these schools, there have been few efforts to plan for sustainability. While several of the principals reported they would fund a College Coordinator position if the program goes away, they have not had to do this yet. At all schools, some program elements have sustained, such as advisory programs; college awareness nights, FAFSA nights, and parent nights; PSAT in schools; SAT/ACT support through Princeton Review; and the integration of writing college essays in English classes. While some of these elements existed before the Achievers Program, staff members agree they have improved upon these efforts and they are more systemic rather than isolated events. However, staff members at the schools without funding recognize that there are elements that they are not able to sustain. Nearly all schools have lost the CPA, who helped to maintain the constant college focus, while five schools continued receiving funding for their CPA. While some of the schools could not hire a College Coordinator, they have tried to reassign duties to a counselor or administrator intern to take on this role, at least part time.
Staff members at nearly every school suggested that sustainability could be strengthened if CSF would build the capacity of the staff to do the work, such as providing professional development for college issues. With training, staff members believe some programming, such as advisories, could be more powerful, and their counselors would have more information on the college process. In addition, counselors reported they would like more information on advising undocumented students. Staff members also suggested it would be helpful for the CPA to develop and leave a list of the resources they used and their contacts, so school personnel could follow-up with those resources.
*Data are only reported when there are greater than 10 students in a group. In particular, data for the Native American students are not reported because there are fewer than 10 students each year.
The BERC Group, Inc.
Duane B. Baker, Ed.D.
Candace A. Gratama, Ed.D.
Shawn D. Bachtler, Ph.D.
Kari M. Peterson, Ph.D
The BERC Group brings experience and an extensive knowledge base to any evaluation project.