SESSION 4A

Stephen Walker, UQ-ICTE
Addressing the challenge of rater bias in speaking and writing assessments using many-facet Rasch analysis

Abstract
Providers of university direct entry programs face the challenge of creating assessments which support academic integrity and provide validity evidence to meet TEQSA ELICOS Standards. This challenge is compounded when assessing speaking and writing skills because it involves human raters making subjective judgements, as they evaluate the language produced by students against rating scales. Research shows that even robust rater training and standardisation programs cannot eliminate rater bias, and that substantial differences between raters remain. Such differences can easily result in a student ‘passing’ or ‘failing’ an assessment!

This presentation will show how many-facet Rasch measurement – a model to investigate the interaction between test taker ability, task difficulty and rater characteristics such as harshness, leniency and consistency – can help direct entry program assessment of speaking and writing ability to meet these challenges by:

- identifying bias
- adjusting for differences
- providing feedback to raters and rater trainers
- delivering fair and valid outcomes for students

The presentation will exemplify the problem and outline several approaches for addressing it. It will allow time for audience questions and discussion on how these approaches could be applied in their own assessment contexts.

Bio
Stephen is Academic Manager - Assessment at ICTE-UQ where he is responsible for planning and overseeing the development and validation of English language assessment. He is also the Australia teacher representative for the Association for Language Testing and Assessment of Australia and New Zealand (ALTAANZ) and co-convenor of the English Australia Assessment SIG. His current interests include applications of IRT and RASCH measurement in assessment development, computer adaptive testing and restoring old land rovers.