The Office of Qualifications and Examinations Regulation (Ofqual) wants to understand whether there might be a role for AI in marking, the body announced in a blog last week.
To unravel the capabilities of this technology in grading A-levels and GCSEs, the regulator will launch an ‘AI competition’. Ofqual expects to receive several thousand student responses to an English language GCSE essay question, which will first be marked by humans. The responses will then form part of a competition in which qualified individuals and organisations will train the AI system to mark according to the training set.
The regulator notes that every AI system requires training, where it is fed a large number of examples alongside the ‘right answer’ (or the best human judgement), which it then attempts to replicate.
“It is important that the training examples use the ‘best’ human judgement, because AI systems are only as good as the data put into them,” Ofqual explains. “Therefore, in our study we will use the most senior markers, and each essay will be marked multiple times to ensure the marks do not reflect error.”
Following this process, the regulator will be able to test the systems on another set of questions for which the humans know the marks but the AI systems do not. Ofqual hopes the competition will help identify and refine the best practice in the field.
If there are genuine potential improvements, ways which might enhance marking quality, of course we want to know, so we can encourage the system to adopt such practices safely
Ofqual hopes the competition results will help them undertake further subsequent research; for example, modelling the impact of AI as a second marker or as a marker monitoring system.
“It is early days in terms of looking at AI in marking, but it is important to take some first steps on this in England, by beginning this exploratory research. If there are genuine potential improvements, ways which might enhance marking quality, of course we want to know, so we can encourage the system to adopt such practices safely. Similarly, we want to have a deep understanding of the potential risk in operating such technology in our high stakes examinations,” the Ofqual blog post notes.
The regulator is currently recruiting schools that are willing to participate, hoping to garner at least 3,000 student responses. Schools will receive examiner marks and annotations for each question to develop their teaching.
“This new project is exciting,” adds Ofqual. “Any future use of AI is likely to take some time and a lot of testing. We are not going to suddenly see AI being used at scale in making high profile qualifications overnight. But we hope in the very least this work will help the sector to think about other ways and innovations to improve marking quality and the delivery of qualifications more broadly to ensure each learner’s work gets the mark it deserves.”
If you or your school are keen to take part in the AI competition, email [email protected]
