In a large first-year economics unit, students kept asking for the same thing: more exam-style practice. Generative AI gave us a way to provide it.
Most of the early conversations about AI in assessment have centred on academic integrity. We were interested in a different question: could these tools actually help students learn?
This pilot drew on a familiar idea in university teaching: students learn more effectively when they have access to low-stakes practice and timely feedback before high-stakes assessment – a principle well established in research on classroom assessment and good feedback practice. Research on retrieval practice also shows that repeated testing can itself strengthen learning and long-term retention. In large classes, however, it can be hard to provide enough varied practice and timely feedback for every student – a constraint many of us will recognise. Recent UNESCO guidance on generative AI in education suggests that such tools may help extend these opportunities at scale, provided they are used in ways that are transparent, purposeful, and closely aligned with learning goals.
That was the thinking behind a pilot we ran in the first-year economics unit BUSS1040: Economics for Business Decision Making in Semester 2, 2025. BUSS1040 is a large-cohort unit, and student feedback has consistently pointed to a desire for more exam-style practice. The mid-semester assessment required students to work through multiple-choice and problem-based questions under timed conditions, so students were looking not just for more questions, but for more structured opportunities to practise in ways that felt relevant to the assessment.
Building an online tutor
To respond to this need, we used a custom-built version of ChatGPT to generate an extensive bank of multiple-choice questions, with more than one hundred questions per topic, based on course materials provided by the teaching team. The tool gave students access to additional exam-style questions, worked solutions, explanations, and automated scoring outside class.
In effect, it acted as a 24/7 online tutor.
In the image, we show a screenshot of what students see when they access the custom‑built ChatGPT.
The aim was not to replace lectures, tutorials, or existing revision materials. Instead, the practice bank was designed to complement them by giving students an additional, flexible way to practise, check their understanding, and engage more actively in preparing for assessment. The idea was simple: if students can access more practice on demand, with immediate feedback and explanation, they are better placed to identify gaps in their understanding and direct their own study.
We built the practice bank as a custom GPT within ChatGPT. We chose this approach over other AI platforms available at USYD, such as Cogniti, because ChatGPT was, at the time, more widely used and familiar to both students and the unit coordinator. We also plan to use Cogniti to develop a separate online tutor for this unit, which will allow us to compare the effectiveness of the two platforms.
Developing the practice bank involved training the chatbot using existing materials, including tutorial questions, online quizzes, and practice exam papers. We also used ChatGPT to generate additional questions and iteratively refined our prompts to ensure that the questions, answers, learning objectives, and difficulty levels were appropriate. The AI-generated questions and worked solutions were reviewed by the teaching team. Students accessed the chatbot through a link provided in Canvas.
If you would like to try something similar, there are a few practical steps you can follow to implement it in another context. Clearly identifying learning objectives, developing effective prompts, conducting iterative testing, and regularly refining the chatbot are all key to ensuring it provides meaningful and useful learning support.
What students told us
What they liked
To evaluate the pilot, we ran a post-mid-semester survey. There were 132 completed responses in the dataset, including 130 consenting responses. Of those, 97 students reported using the AI practice bank before the test, representing 74% of consenting respondents.
Among students who used the bank and rated its usefulness, the overall response was encouraging. A total of 65% rated it as moderately, very, or extremely useful, while 22% rated it as slightly useful and 13% reported that it was not useful. Use was also not just one-off: 72% of users reported returning to the bank more than once, suggesting that many students were incorporating it into their study routine rather than simply trying it once.
The open-ended comments help explain these patterns. Several students pointed to the value of having accessible, low-stakes practice with worked solutions. As one student wrote
I really like having practice exams… I also found comprehensive worked solutions helpful.
Taken together, the survey and comments suggest that students valued the practice bank primarily as a formative revision tool rather than simply as an AI novelty.
What needs improving
At the same time, the feedback also highlighted clear areas for improvement. Among students who did not use the bank, the most common reasons were lack of awareness and lack of time, rather than resistance to AI itself. This was echoed in the comments, with one student noting that the tool should have been “Available earlier!” This suggests that successful uptake depends not only on the quality of the resource, but also on how early and how clearly it is introduced in the semester.
A consistent message from both the survey and the comments was that students wanted closer alignment between the practice bank and the actual assessment. Survey responses showed strong demand for more exam-style questions with step-by-step worked solutions, followed by full-length timed mock exams and section-by-section drills. The open-ended comments reinforced this pattern. Students asked for “way more exam styled questions with step-by-step answers” and “a larger set of example questions.” Another student noted that “the accuracy of the answers could be improved, otherwise everything was good,” highlighting the importance of quality control as well as quantity.
What we learned as teachers
The pilot taught us as much about our own role as about the tool. A few lessons stood out:
- AI can scale practice opportunities, but it doesn’t remove the need for teacher judgement.
- The value lay in quality and alignment with the unit, not the volume of questions.
- Effort shifted from drafting every question to designing, reviewing and curating them.
- Students embraced the tool when it was framed as low-stakes, formative support — and introduced early.
AI can scale practice opportunities, but it doesn’t remove the need for teacher judgment.
Making AI useful, not just available
Overall, the pilot suggests that students were receptive to the idea of an AI-supported practice bank, but that they wanted it to be introduced earlier, expanded further, and aligned more closely with the style and difficulty of the exam.
The broader lesson for us is that the question is not simply whether AI should be part of learning design, but how it can be used in ways that are pedagogically purposeful. In our case, the most promising role for AI was not as a shortcut, but as a formative support tool: something that could provide students with more opportunities to practise, receive feedback, and prepare with confidence.
That, ultimately, was the main lesson from this pilot. The value of the tool did not lie in the technology alone, but in how well it was embedded within a broader teaching and assessment design.
Thinking of trying this?
If you’d like to set up something similar in your own unit, a few things helped us:
- Start with one topic rather than the whole unit.
- Introduce it early – ideally week one – so students have time to use it.
- Align the question style closely to the actual assessment.
- Build in time to check AI-generated solutions for accuracy.
