Author ORCID Identifier

https://orcid.org/0000-0002-9106-0417

Year of Publication

2021

Degree Name

Doctor of Philosophy (PhD)

Document Type

Doctoral Dissertation

College

Education

Department/School/Program

Education Sciences

First Advisor

Dr. Kelly Bradley

Second Advisor

Dr. Margret Schroeder

Abstract

Rater-mediated classroom observation protocols are increasingly being used for teaching performance assessments, which makes identifying and controlling for various rater effects a central issue to ensure the rating quality. A series of validation studies under the classical test theory framework, including content validity, interrater reliability, and structure analysis, have been completed for the 16-item Mathematics Classroom Observation Protocol for Practices (MCOP2).

However, the MCOP2 data have never been investigated under the Rasch framework. Due to the methodological limitations of the CTT approach for rater-mediated assessments, it is imperative to examine the MCOP2 validity and reliability using the MFRM modeling technique to implement dimensionality analysis, item-level analysis, rater effects control, and ratee and rater ability level calibration.

To that end, two existing samples of the MCOP2 data were obtained and analyzed, where twelve raters were asked to rate 237 math classroom observations, using the MCOP2 classroom observation protocol. The data were analyzed under the MFRM framework, using Facets 3.83.3.

Results of the Facets analysis showed that both the MCOP2 subscales (i.e., Student Engagement & Teacher Facilitation) were valid, unidimensional, and highly reliable rater-mediated performance measures across raters, ratees, and study samples. However, rater-item bias analyses revealed a type of intra-rater inconsistency, where some raters tended to rate more severely than other raters on certain items while more leniently on some other items.

The overall findings are promising in that they provide systematic preliminary psychometric evidence for the viability of the MCOP2 protocol to be used for math teachers’ self-assessment and/or peer-assessment along with other designated raters in the future studies.

Digital Object Identifier (DOI)

https://doi.org/10.13023/etd.2021.158

Share

COinS