Multiple Choice Question (MCQ) quizzes have long been part of many assessment plans. But with easy access to AI and its phenomenally rapid performance improvement, we are frequently being asked the same two questions: Is there any way we can develop AI-proof Canvas MCQ quizzes? And if not, why bother having quizzes at all?
While there is no practicable way to AI-proof a Canvas quiz without involving exam level invigilation, when delivered with a flexible student-centred approach, formative quizzes are still a really worthwhile way of giving students opportunities to practice and supporting their learning.
AI can answer difficult questions better than most people
Large language models (LLMs) such as ChatGPT or Claude are now outperforming humans in many assessment tasks. Benchmarks (sets of questions) used to assess these models’ performance show that they have improved at a very fast pace. “In just over two years, AI performance on GPQA Diamond [benchmark quiz of science topics] went from 39% (below non-expert human level) to over 94% (well above PhD expert level)”.
Students can use AI to “help” them do their quiz – from typing individual questions into an LLM chat prompt for an answer to using a purpose – built plug-in to complete the whole quiz. While there are tools such as Respondus Lockdown Browser designed to prevent students from accessing unauthorised sites, these tools can only be successful when used alongside other resource-intensive invigilation methods. At the University of Sydney, Respondus Lockdown Browser is only approved for securing assessments when the assessment task is managed by the Exams Office.
That leaves most Canvas MCQ quizzes best placed as open, formative assessments; that is, assessments for learning, as they are categorised in the Sydney Assessment Framework.
How do MCQ quizzes support learning?
MCQs can help activate the retrieval process to reinforce learning (testing effect), can stabilise marginal knowledge (knowledge that is stored in memory but is not accessible at a given moment), can lead to knowledge transfer, and with feedback can enhance long-term retention of knowledge. More recent research has shown that mastery quizzing, which is when students can retake quizzes until they achieve a perfect score, results in better performance in a final quiz than other test practice methods; this finding persisted when the investigators controlled for feedback and over longer periods.
But when students use AI to help them do the quiz, they gain little to no benefit from the exercise, and the whole activity can feel like a waste of everyone’s time. So, how do we encourage students to take advantage of the learning opportunities of MCQ quizzes?
What doesn’t work?
It is tempting to try to shape student behaviour by restricting quiz settings – limiting the number of attempts and the time available, and using settings that force students to answer one question at a time, and in a particular order. With marks at stake and limited supervision, these restrictions may discourage students from making a genuine attempt. They may instead try to maximise their marks by checking with an AI Agent for an answer or even using an AI plug-in to complete the whole quiz without reading a single question.
How can we set up quizzes for success?
Rather than using restrictions to try and fight AI assistance, we recommend encouraging students to take advantage of the learning opportunities of a genuine attempt, reminding them of the known benefits of practice, including the gain in final marks – and providing clear instructions and flexible quiz settings. We recommend designing quizzes with the following features:
- learning outcomes/concepts being checked, along with advice and guidance on course resources in the instructions
- no time limits – this reduces pressure and can accommodate students with technical issues or academic adjustments
- multiple attempts allowed, keeping the highest score to encourage students to have a go and learn from each attempt
- variations on each concept checking question so that each time a student takes the quiz they see a different set of questions – more learning opportunities
- all the questions shown at once so that students can scan through all the questions and answer them in any order
- feedback for each response (correct and incorrect), and allowing students to view their quiz responses and feedback after each attempt for self-assessment and learning
What have students told us?
In a panel discussion on AI and assessment, students showed awareness of the learning benefits of quizzes and of the temptations to use AI. One noted that having multiple attempts at weekly quizzes meant that students were not scared to try and fail, so that they could make one or more attempts and then do some revision before submitting their final mark, instead of relying on AI to complete the quiz. Another student mentioned that although regular low-stakes quizzes are a great way to see progress, students in their first semester may not see the value themselves. This was confirmed by another student who learned to break his over-reliance on AI the hard way, after using it to make his chemistry notes… “I walked into the final exam and knew I was finished”.
Reminding students of the learning opportunities of your quizzes and keeping the settings flexible will ease your workload while also reducing student stress, encouraging them to make a genuine attempt and gain the learning benefits.
AI can help educators with MCQs
AI agents aren’t just good at answering MCQs – they’re great at helping educators build them, saving the hours it used to take to write good questions. The USyd MCQ agent can draft complete questions from simple prompts, including plausible distractors and tailored feedback for both correct and incorrect responses, and produce multiple variations on each concept-checking question so students see a different set at every attempt.
Students can also use AI well here – to draft their own practice questions for review and revision.
Want to try this in your own teaching?
- Find out more about the USyd MCQ agent and register for a workshop
- Point students to the prompts to help you learn pages on the student-developed AI in Education Canvas site for using AI productively and responsibly
