SESSION 1A

Stephen Walker, UQ -ICTE
Lies, damned lies and statistical analysis

Abstract
For many teachers who are required to develop language tests, the time available is largely taken up with writing the test items themselves. Skilful and experienced item writers can learn over time to consistently produce a good proportion of effective items, and to identify potential problems with items during the test development process.

However, some information about the way a test item performs is deeply hidden and can only be discovered through statistical analysis. We can for example see how easy or difficult the test takers find an item or whether the stronger students get it right and the weaker ones get it wrong. Analysis can also reveal whether there is a 'wrong' answer which a lot of people choose, or if there is a ‘distractor’ in a matching task which no-one chooses. It can also show the extent to which items contribute to the test’s internal consistency by actually tapping into the trait we want to measure (e.g. reading ability) rather than some other irrelevant trait (e.g. general knowledge) . These aspects of test item performance help us to determine how well a test provides an accurate measurement of test takers’ abilities. In any testing situation, we want our assessments to be as valid, reliable, and fair as possible, but in high stakes test situations such as university admission gatekeeping, this is crucial.

This talk aims to provide a beginner’s ‘no maths’ introduction to the world of statistical item analysis.

Bio
Stephen is the academic manager responsible for the development of English language assessment at UQ-ICTE and also the Bridging English Program, an academic English pathway program for international students wishing to study at UQ. Stephen has been a member of the TESOL management team at UQ-ICTE since 2007. He is also Teacher Representative Australia for ALTAANZ and his work includes the design, development and validation of reliable language assessments.