School Improvement Assistance Program
Cohort 1 Evaluation Program
Executive Summary
|
The purpose of this report is to provide summative feedback to personnel at the Office of Superintendent of Public Instruction about the assistance provided to schools in Cohort 1 of the School Improvement Assistance Program (SIAP). Although summative in nature about the effects on Cohort I schools, the report is also designed to provide formative feedback to assist in the ongoing development of the assistance program, which is now in its fourth cohort. The School Improvement Assistance Program was primarily intended as a school level intervention. As such, it was expected to address various school level issues. Elements of the program, however, were designed to bring about greater district level involvement in the schools, and the intent was that the change efforts would result in improved student achievement. OSPI interventions, although focused on school level changes, attempt to coordinate district involvement and hopes to impact classroom practice. The major components of the School Improvement Assistance Program include: School Improvement Teams, Educational Audits, School Improvement Plans, Performance Agreements, and School Improvement Facilitators.
The evaluation utilized a multiple measures, mixed methodology approach. The evaluation design focused on four areas of inquiry:
In order to answer these questions, researchers gathered a variety of qualitative and quantitative data including: interviews at the 25 schools in Cohort 1; WASL and ITBS results; OSPI SIAP Staff, Elementary Student and Secondary Student Surveys; and OSPI documents and materials. In addition, a classroom observation protocol was utilized to enhance the evaluation. Overall, 746 people participated in evaluation activities that included interviews and focus groups. Classroom observations were conducted in 295 core academic classes. The intervention and support that OSPI provides to schools as part of the SIAP process has several distinct features, including a School Improvement Team (SIT), Educational Audits, Performance Agreements, Professional Development, School Improvement Plans (SIPs), and School Improvement Facilitators (SIFs). The benefit of the SIAP to schools was largely reliant upon the staff’s readiness to benefit from the support. The most important aspects of the support were the combination of clear governance, clear planning, and clear support (SIT + SIP + SIF = SIAP). If one aspect of the program had to be singled out as most critical, the effective SIF would rise to the top. School responses to support varied for a number of reasons. In most cases the schools quickly or eventually embraced the support and process. In others, however, schools began the process resistant and remained that way through much of the program. In some instances, district and school leadership changes added to the difficulty of maintaining full implementation of the SIAP support. School improvement efforts are intended to increase academic achievement. Many staff members felt that the SIAP made a difference and that an increase in academic achievement could be attributed to the program activities. Some of the program activities mentioned as making the biggest (perceived) difference included: aligning curriculum; creating a common focus; raising expectations for teachers, as well as students; and increasing accountability. However, some staff members were not willing to attribute improvement to the program and voiced concerns about the program. Concerns ranged from procedural problems to a lack of confidence that significant achievement gains could ever be made given the student populations the schools serve. Actual achievement data indicated that the baseline (2001) WASL and ITBS composite scores for the SIAP Cohort 1 schools were all substantially lower than the average composite scores of the comparison schools and the state. The Cohort 1 elementary schools exhibited the most striking growth, and exhibited significant gains over the comparison schools from 2001 to 2004 on the third grade ITBS and fourth Grade WASL composite scores. The Cohort 1 sixth grade ITBS scores were relatively flat and showed no difference in gain over time as compared to the comparison schools. The Cohort 1 seventh grade WASL composite scores showed an impressive gain, but the overall gain is not significantly different than that of the comparison schools. School climate and school culture are terms that are sometimes used interchangeably. Climate, in this case, is defined as the quality of the school environment that students and staff experience which affects their behavior. School culture is defined as a system of shared norms, values, and tacit assumptions held by members of the organization that hold a unit together and give it a distinct identity. Overall, there is a general sense that the SIAP activities have helped to enhance school climate and culture. School climate appears to have improved, largely due to changes in school culture. In some schools, the culture has changed by having higher expectations for teachers and students; creating a common focus; increasing collaboration and communication; establishing a supportive learning environment; and developing a professional learning community. In other schools, staff member said they were unable to change their school culture because of relational trust issues. Some also said they had an overwhelming sense of defeat because of challenging student demographics. These barriers create frustrations and significant emotional barriers to bring about a cultural shift. Researchers rated each school’s level of SIAP implementation and calculated gains in academic achievement from 2001 to 2004 in order to determine if there was a relationship between program implementation and achievement gains. It should be emphasized, however, that the nature of the project and the evaluation design do not allow causal inferences to be made between the SIAP program and student achievement gains. Based on the examination of both qualitative and quantitative data, it appears that schools that did a reasonable job of meeting the program requirements in a positive fashion were able to improve student academic achievement. The hallmarks of these schools were an acceptance of the SIAP program; a readiness and willingness to participate; and thoughtful use of the SIP process and the audits to address difficult issues. Additionally, these schools created adult learning communities that focused on professional development and had a special interest in improving teaching and learning. Furthermore,these schools built their capacity to change vis-à-vis the SIF rather than becoming reliant on the SIF. Finally, the district was included as a partner in the school level intervention, rather than carrying out the work in isolation from the district. The SIAP efforts are clearly sustainable but still susceptible to erosion of effect over time without ongoing support. For the most part, relationships between school and district leaders were strengthened due to more purposeful communication and leadership changes. In many cases, participating schools had other means of support in addition to the SIAP, but it was pointed out that the program helped schools align external resources in order to have a greater effect on achievement. Sustainability has more to do with future support than it does past success. For some districts, three years has produced focus and some academic improvement, but the eventual removal of the SIF at the end of the grant will be a critical test. Principals and administrators must assume greater leadership roles in the school as the SIF transitions away from the SIAP process. In addition to staff surveys, interviews, and focus groups, researchers conducted 295 classroom observations in the 25 schools to determine the extent to which “powerful teaching and learning” was present in the schools. In all but one component, the Cohort I schools exhibited fewer characteristics than the state, as described in the classroom observation protocol (TAOP). Researchers observed powerful teaching and learning 7% of the time in Cohort I schools as compared to the state average of 17%. The fact that Cohort I schools scored lower than the state was not surprising given the documented correlations between this type of teaching and academic achievement, the historically low academic performance of students in Cohort I schools, and the lack of intentional intervention around instructional practices. The SIAP model of support is unique in the United States. Many states have state sponsored accountability systems that focus on rewards, punishments, and takeovers. Washington State accountability amounts to comprehensive support for struggling schools. Findings from Cohort I schools were modest but encouraging. Continued attention should be given to subsequent cohorts to determine if there are higher levels of implementation and to determine if there is more evidence of impact on student achievement. |
Prepared ByThe BERC Group, LLC.
Duane B. Baker, Ed.D. Candace A. Gratama, M.S. Joan N. Clay, M.S. Stephanie R. Arington, M.S. Alan C. Bulger, M.S. Robert H. Fulton, J.D. Prepared ForOffice of Superintendent of Public Instruction
Old Capitol Building PO Box 47200 Olympia, WA 98504-7200 PublishedNovember, 2004
Program Evaluations
The BERC Group brings experience and an extensive knowledge base to any evaluation project.
|
