Event

TUJ Distinguished Lecturer Series: Analyzing Rating Data Using a Many-Facet Rasch Measurement Approach

Saturday, October 19th, 2019, 2:00 PM - 5:00 PM

Speaker: Carol Myford (The University of Illinois at Chicago, U.S.A.)

In performance assessment settings, raters use measurement tools such as rating scales or scoring rubrics to evaluate students' performances or products. Raters assign their ratings based upon information that they gather and synthesize as they examine each performance or product. However, by virtue of their being human, raters sometimes unavoidably introduce error (rater effects) into the assessment process. Data analysts can employ a many-facet Rasch measurement (MFRM) approach for analyzing ratings to detect and measure the impact of rater effects such as leniency/severity, central tendency, and halo effect on the ratings. Using the output from these analyses, administrators in charge of monitoring quality control for assessment systems obtain detailed, practical information about how various "facets" (e.g., students, raters, rating criteria) of their assessment systems are performing. They can use that information to help them determine to what extent their systems are under statistical control and to initiate meaningful changes in an effort to improve their systems.

This seminar will introduce participants to the many-facet Rasch measurement approach for analyzing ratings. Topics we will cover include the following:

• The role of raters and their influence in open-ended assessments
• Rater effects that can influence ratings
• Statistical approaches that researchers have used to analyze rating data and characterize rater performance
• The conceptual and mathematical foundations of many-facet Rasch measurement
• Group-level and individual-level questions that a MFRM analysis can answer
• Creating a data file for a MFRM analysis
• Preparing a specification file for a MFRM analysis
• Running MFRM analyses and reformatting output
• Creating judging plans
• Making sense of output from MFRM analyses
• MFRM anchoring procedures for equating
• Strategies for resolving disconnected subsets issues
• Conducting bias interaction analyses
• Hybrid MFRM models and their uses

Prerequisite Skills and Knowledge Needed for Seminar: Participation The target seminar participants are researchers, statisticians, data analysts, and advanced graduate students who want to learn how to use many-facet Rasch measurement for analyzing rating data. Having a basic understanding of the conceptual and mathematical foundations of Rasch measurement would be helpful but not required, as would prior experience using Rasch measurement analysis software (e.g., Winsteps).

Organization: Temple University Japan

Cost: free

Venue: Osaka Ekimae Bldg. 3, 21st Floor, 1-1-3-2100 Umeda, Kita-ku, Osaka 530-0001 (map)

Location: Osaka City, Osaka Prefecture, Japan

Contact Temple University Japan

Temple University Japan

Work phone: 03-5441-9800