Author ORCID Identifier
Date Available
5-18-2021
Year of Publication
2021
Degree Name
Doctor of Philosophy (PhD)
Document Type
Doctoral Dissertation
College
Education
Department/School/Program
Education Sciences
First Advisor
Dr. Kelly Bradley
Second Advisor
Dr. Margret Schroeder
Abstract
Rater-mediated classroom observation protocols are increasingly being used for teaching performance assessments, which makes identifying and controlling for various rater effects a central issue to ensure the rating quality. A series of validation studies under the classical test theory framework, including content validity, interrater reliability, and structure analysis, have been completed for the 16-item Mathematics Classroom Observation Protocol for Practices (MCOP2).
However, the MCOP2 data have never been investigated under the Rasch framework. Due to the methodological limitations of the CTT approach for rater-mediated assessments, it is imperative to examine the MCOP2 validity and reliability using the MFRM modeling technique to implement dimensionality analysis, item-level analysis, rater effects control, and ratee and rater ability level calibration.
To that end, two existing samples of the MCOP2 data were obtained and analyzed, where twelve raters were asked to rate 237 math classroom observations, using the MCOP2 classroom observation protocol. The data were analyzed under the MFRM framework, using Facets 3.83.3.
Results of the Facets analysis showed that both the MCOP2 subscales (i.e., Student Engagement & Teacher Facilitation) were valid, unidimensional, and highly reliable rater-mediated performance measures across raters, ratees, and study samples. However, rater-item bias analyses revealed a type of intra-rater inconsistency, where some raters tended to rate more severely than other raters on certain items while more leniently on some other items.
The overall findings are promising in that they provide systematic preliminary psychometric evidence for the viability of the MCOP2 protocol to be used for math teachers’ self-assessment and/or peer-assessment along with other designated raters in the future studies.
Digital Object Identifier (DOI)
https://doi.org/10.13023/etd.2021.158
Recommended Citation
Niu, Chunling, "RATE TO MEASURE MATHEMATICS TEACHING: USING THE MANY-FACET RASCH MODELING TO REEVALUATE THE MATHEMATICS CLASSROOM OBSERVATION PROTOCOL FOR PRACTICES (MCOP2)" (2021). Theses and Dissertations--Education Sciences. 90.
https://uknowledge.uky.edu/edsc_etds/90
Included in
Educational Assessment, Evaluation, and Research Commons, Teacher Education and Professional Development Commons
