STAR Protocol Report

Home > Resources > Reports > STAR Protocol Report

STAR Protocol Report
A Classroom Observation Tool

Executive Summary

This draft study describes the development of the STAR Classroom Observation Protocol and presents results from classroom observations over the last six years. Research shows that the elements of current Washington State Education Reform efforts call for reliable and valid instruments that measure Powerful Teaching and Learning (PTL) in schools. As such, a conceptual framework was identified based largely on the foundational components of the Teaching Attributes Observation Protocol (TAOP), other similar measurements, and an extensive literature review. As a result, The BERC Group developed the STAR Protocol to measure Powerful Teaching and Learning in classrooms and schools. The protocol is divided into five Essential Components: Skills, Knowledge, Thinking, Application, and Relationships. Three Indicators are organized around each Essential Component, comprising a total of fifteen Indicators. Approximately 95 strategies are listed as part of the protocol and describe how the Indicators are manifested.

Following a training period, classroom observations were conducted in 14,927 classrooms. Provisions were made for regular inter-rater reliability and agreement checks where two raters scored the same observation and rated the protocol independently. Findings show a high degree of consistency in the rating process. Additionally, the Kappa reliability coefficient for the protocol is .90 suggesting good reliability between observers on the Overall score for the protocol. The results of a factor analysis show that the STAR Protocol is a uni-dimensional instrument tapping into a single construct: Powerful Teaching and Learning.

The general findings of this study are that strong Powerful Teaching and Learning was clearly observable in 14% of classrooms lessons. An additional 35% of classroom lessons show some evidence of Powerful Teaching and Learning. The remaining 51% show very little or no evidence of Powerful Teaching and Learning. Data also suggests little change has occurred in Powerful Teaching and Learning over a six-year period, when examining first-time schools that have never been previously observed. Although a moderate increase of 12% was found, this indicates the process for the reform movement is moving at a slow pace. The analyses also show a positive contribution of PTL to student achievement beyond the effects of low income. Most notably, a unique contribution was found for PTL in predicting math achievement. About 7% of the variance in math achievement was explained by PTL. Only small contributions of PTL were found for reading achievement (1%) and science achievement (2%). A small contribution of PTL was also revealed in predicting writing achievement (1%), however in the opposite direction.

The findings in this study show limited progress toward reform movement efforts. Research shows that the culture of the school must be fundamentally changed before academic achievement for students is likely. Meaningful reform can only move forward through Second Order changes. Creating a new school culture where professional development centers on researched-based instructional practices appears to be the key to providing the best chance for all students to be successful. To this end, the STAR Protocol was designed to synthesize characteristics of reformed instruction. This instrument offers a way to push the reform movement beyond its current state by being used as a measurement for reformed teaching and as a tool for professional development.