Navigation 101 Program Evaluation
Year Three Report
Executive Summary
|
The purpose of this Year Three report is to provide summative feedback to personnel at the Office of Superintendent of Public Instruction (OSPI) regarding evidence of implementation and impact of Navigation 101 in Washington State. The report is also designed to provide formative feedback to assist in ongoing program development.
The goal of the Navigation 101 program is to help students to become college and career ready. Administrators, teachers, parents, and students across the state acknowledge this sentiment as the primary mission of the program. Many school staff members recognize the impact the program makes on students and on overall school climate as the initiative strengthens from year to year. Similar to findings from the previous two years, researchers found variations in practices around function and duration of advisories, portfolio use and maintenance, conference format and frequency, student-informed scheduling, and evaluation procedures. In schools with long term programs in place, staff members identified three specific ways Navigation 101 has grown within their schools. These include the clarification and strengthening of program goals; an observed increase in staff buy in; and the shift from an ‘added on program’ to a program that is embedded, embraced, and crucial to the school’s culture. However, many schools with long histories of implementation admit to using mostly homegrown curriculum instead of the one provided by Navigation 101. Other reported strengths and weaknesses regarding the key elements were similar to previous years’ findings. Focus group participants identified barriers and challenges to implementing Navigation 101. While most challenges relate to the key elements, other identified barriers revolve around pressure school members face to implement multiple initiatives, challenges to find consistent advisory time, and, as found in previous year reports, struggles to effectively communicate with caregivers. During focus groups, some school representatives discussed the evolution of the program in relation to support provided. One of the changes mentioned by service providers was a more customizable menu of supports. However, stakeholders at some schools are still frustrated by the time and resources required to “jump through hoops” and are unclear to what extent they are allowed to go in their own direction. For schools that have been implementing the program for a long time, staff members perceive less benefit from the touch points, webinars and presentations, and more benefit from simple collaboration with peers. Several stakeholders suggested including more regional workshops and collaboration opportunities for staff members to get together and learn from tried and true practices at other schools. School staff members reported generally positive appraisals of the levels of service, and most stakeholders reported timely responses from OSPI and Envictus personnel. To assess evidence of impact, researchers analyzed transcripts; student assessment results; graduation rates; College Bound application rates; college attendance, persistence, and graduation data, pre-college course taking patterns; student and staff surveys, and student-led conference attendance and perception data. The data were compared to state date when available, and in most cases, additional analyses were conducted to determine differences based on years of implementation and self-reported level of implementation. There are some positive trends in the data. Generally, results showed that years of experience implementing Navigation 101 had some positive effects; however, these trends were not statistically significant, likely because of small sample sizes. A greater percentage of parents attend student-led conferences compared to traditional conferences, and perception data from parents, students and advisors were positive about the experience. In addition, many students sign up for the College Bound scholarship at the middle school level, suggesting that students have an awareness about the opportunities available to them and an interest in signing up for these opportunities. The results from the transcript analysis show there has been a decrease in the percentage of students meeting minimum course taking requirements to enter into a four-year college (transcript analysis). The percentage of graduates meeting WSAC requirements increased from 2008 to 2009 (42.9% to 44.8%), decreased in 2010 to 37.9%, and then increased each year to 41.4% in 2012. The decrease between the graduating class of 2009 and 2010 likely occurred because WSAC minimum requirements increased in the area of science. However, in 2012, results are approaching 2009 results prior to the change. These results show that a majority of students graduating from these schools cannot be admitted to college because of course deficiencies. It also shows that the graduation requirements at these schools, while meeting the state’s minimum requirements for a high school diploma, are not aligned with colleges’ admission expectations. In addition, it appears schools did not make the necessarily advising adjustments when requirements changed in 2010.This suggests a need for increased information to school personnel about impending changes so schools can adjust to increased requirements and adjust master schedules accordingly. An analysis of Gatekeeper Courses shows an increase in the percentage of students who take Algebra or higher in middle school, in the percentage of students who take advanced math in high school, and an increase in of those who take high school Chemistry from 2008 to 2012. In general, analysis of achievement data suggests that Navigation 101 grantees appear to be following a similar pattern to the state, with the Navigation 101 schools performing below the state average. However, in reading achievement at the elementary level, the gap between Navigation 101 schools and the state appears to be closing. When compared to the state, the rate of improvement in reading was higher for Navigation 101 elementary and middle schools than it was for schools making up the state average. The rate of improvement for Navigation 101 high schools was lower than the state. In math, the rate of improvement for Navigation 101 schools at all school levels was higher than for schools making up the state average. When analyzing the results based on year in the program and self-reported level of implementation, there was one statistically significant finding. Schools in Year 5 or more of the program had higher reading achievement improvement at the elementary school level than schools in other years. A positive trend in math improvement was evident at the elementary school level as well, based on years implementing the program, though the results were not significant. Analyses of graduation data show Navigation 101 schools are improving at a slower pace than the state. Previous results showed a trend that as the length of time implementing Navigation 101 increases, there tended to be a greater improvement in graduation rates. However, there were no significant findings this year related to years implementing the program or self-reported levels of implementation. It is notable, however, that Navigation 101 schools have a similar graduate rate compared to the state. This is significant considering that the grantee schools contain a higher percentage of students qualifying for free and/or reduced priced meals than schools making up the state average. Analyses of college attendance show the rate of improvement for Navigation 101 schools is slower than the state. Further analyses by years implementing Navigation 101 and self-reported levels of implementation were not significant. However, the data shows a clear trend that the longer the schools implement Navigation 101, the greater the increase in college direct rates. This is a positive finding. Finally, researchers analyzed student and teacher survey results for Navigation 101 schools for three school years: 2010, 2011, and 2012. Results on the student surveys showed statistically significant increases in five of the factors: High Expectations, In Depth Learning, Satisfaction 1, Satisfaction 2, and Navigation 101 Beliefs, compared to 2010. Results from the teacher survey also showed some positive changes. Investigating the means for each factor revealed that schools had higher mean teacher survey scores on three of the factors in 2011 and 2012 compared to 2010 (Standards Based Teaching, Personalization, Future Focus) and two factors had significantly higher scores in 2012 compared to 2010 (Quality of Education and Technology). An identified strong point and key driver of change within a school culture is the fact Navigation 101 is a foundational piece that supports a comprehensive guidance and counseling program. Many stakeholders suggest the comprehensive design of the program helps to create a structure that supports all students to become career and college ready. In some buildings, staff members further link Navigation lessons to support ideals introduced in the American School Counseling Association (ASCA) national model. In order to continue programmatic growth and success, we recommend stakeholders provide whole staff advisor training; customizable and differentiated lessons;continually refreshed curriculum; additional focus on military, trade, and community college tracks; consistent expectations for SLCs and portfolio; and clear and reasonable expectation for data collection. |
Prepared ByThe BERC Group, Inc.
Duane B. Baker, Ed.D. Candace A. Gratama, Ed.D. Sarah C. Brenner, M.Ed. Lisa M. (Goetz) Law, M.Ed. Kari M. Peterson, Ph.D. Prepared ForOffice of Superintendent of Public Instruction
Old Capitol Building PO Box 47200 Olympia, WA 98504-7200 PublishedNovember, 2012
Program Evaluations
The BERC Group brings experience and an extensive knowledge base to any evaluation project.
|
