The rapid advancement of generative AI tools has brought about a new era in educational assessments. These tools, with their ability to create everything from textual content to voice simulations and visual media, invite us to refine our approaches to how we evaluate student learning. We are now familiar with the Secure (Lane 1) or Open (Lane 2) guidance. In consultation with our colleagues in the Faculty of Medicine and Health and across the University, we are mapping our assessments into the new categories. The short-term goal might be to consciously map and then design assessments with generative AI in mind. The long-term goal is to align our assessment practices with enduring principles that can securely stand while technological advancements grow. It can be helpful to zoom out and consider the core principles we might apply to this work, and an example of how they have been applied in the classroom last year.
Students in Year 4 Bachelor of Applied Science (Exercise Physiology) complete EXSS4021 Exercise in Multimorbidity, a cohort of 77 in 2024, in the Intensive September session as the final academic unit of their degree, followed by their final clinical placement, before entering the workforce as graduate Accredited Exercise Physiologists (AEPs). The first assessment task, a group-based assignment involving synthesis of a patient history and initial assessment data into a letter reporting back to the referring doctor, was redesigned to utilise AI tools to assist with this routine clinical documentation activity. This was supported by teaching content exploring the ethical and responsible use of AI in healthcare underpinned by the NSW Agency for Clinical Innovation Critical Intelligence Unit AI living evidence tables.
The core principles
The following core principles are already familiar to us and can help guide educators in the Faculty of Medicine and Health through thinking about changing assessment tasks. They are offered to be resilient and stand strongly as we adapt to our growing technological capabilities. The principles are: scholarly, human, authentic, and knowing your students, which can be remembered using the helpful acronym SHAK*.
Scholarly: How do you know what you know?
Scholarship is the pursuit of a deep understanding of knowledge—how it is obtained, structured, and validated—emphasising epistemology, logic, and evidence in the learning process. In assessments, good scholarship involves validating the origin of ideas and ensuring students can articulate their thought process coherently and convincingly. Adhering to high ethical standards, such as honesty in reporting findings, transparency in methodologies, and proper citation of sources, is crucial. Effective scholarship is demonstrated by students through critical thinking skills, a comprehensive understanding of their field, responsible engagement with sources, awareness of epistemological considerations, and the ability to construct logical, evidence-supported arguments. All of this is to be communicated clearly and convincingly, both orally and in writing. Students need to apply good evaluative judgement to determine what is good evidence, and this principle holds whether they are using academic peer reviewed articles, media resources or generative AI.
Lecture and tutorial activities in EXSS4021 Exercise in Multimorbidity encouraged students to examine ethical considerations for AI use in healthcare including informed consent, bias, and data security risks as well as clinician responsibilities and reporting requirements. Students explored AI tools marketed to allied health professionals such as AI scribes and tools automating clinical documentation, determining if they meet Australian data security and privacy requirements and comparing them to generic tools such as Microsoft Copilot or ChatGPT to produce a letter template.
The redesigned assessment task challenged students’ ability to appropriately synthesise a patient history and subjective and objective data into a concise but informative letter communicating their professional interpretation and treatment plan, utilising appropriate language and presentation elements. Students were required to consider the ethical and responsible handling of patient information if they chose to use an AI tool to produce their letter and ensure honesty and transparency by declaring the nature of any AI use. Interestingly, six of the twenty student groups chose not to use AI to complete the task, while the remaining groups used AI to generate a letter template, summarise the patient information and provide an initial interpretation, and/or refine language (six, five, and four groups, respectively). Five submissions, all using AI in some way, were graded ‘excellent’ for language and presentation, while one submission, also using AI, was graded ‘unacceptable’. Conversely, of the five submissions graded ‘excellent’ for their professional interpretation, only one used AI while four did not
Human: What is foundational to being human?
The human element of assessment focuses on the humanity of the student. This includes appreciating how students think, what they need, and how they express themselves. By centring our attention on the human element, we can create assessments that underscore the students’ engagement and their ability to bring something unique to their work. They might demonstrate their individuality through creativity, curiosity, connection, community, or choice. The human element involves networking, decision making, collaboration and explaining your thinking, often seen in teamwork as part of group projects. It might also present itself in assessment design which encourages students to explore different modalities which creatively express their understanding of course content in original forms.
As a group project in EXSS4021, students brought their experiences and opinions to collaborate with their peers in deciding on their approach to including or excluding the use of AI tools, informed by the preceding tutorial activities. As social learners they communicated with their peers, explaining their rationale, mirroring the assessment task of communicating to the referring health practitioner, displaying their professional judgements.
Authentic: How can we encourage real-world relevance and evaluative judgment?
Under this aspect, we consider how assessments can mirror practical situations and professional applications. This principle is twofold; it involves creating tasks that are directly tied to real-world experiences, and it also emphasises the importance of evaluative judgment—assessing the process itself and the ability to discern the quality of the work. By incorporating these elements, students can practice making informed decisions and critically evaluating their work and that of others, which are crucial skills in any discipline, and demonstrate their understanding of what they are being taught. In most professions, incorporating generative AI will be part of the daily work; in this context the use of generative AI is an authentic assessment.
In EXSS4021, the redesign took an assessment task that was already an authentic representation of a typical daily clinical task, and added the genuine challenge facing health and medical practitioners today: making an informed decision about which, if any, AI tool to use, to produce an effective communication that adheres to clinical reporting requirements.
As we see our professional accreditation bodies updating standards to incorporate the ethical use of AI, in terms of consent and patient data and data security, or interpreting patient data, our graduating students are in a unique position to drive the creation of policies and practices around the ethical use of AI in healthcare.
Know your students: What is the usual standard?
This is a principle that reminds us of the importance of personal understanding and connection with our students. Recognising their work patterns, performance levels, and individual characteristics helps us to identify when a piece of submitted work doesn’t seem to match their capabilities, which can be a sign of over-reliance on AI tools or students misrepresenting submitted work as their own. Where there is evidence of suspicious activity, these cases should be reported for academic integrity investigation.
The task in EXSS4021, recognises the stage of the students as pre-graduates about to enter the health workforce and relevance of the assessment task in preparing them for practice.
While using AI or not did not directly correlate to marks, it is interesting that within this small sample of students, using AI tools seemed to help improve language and presentation scores (all five submissions marked ‘excellent’ used AI in some way), while four of the five submissions marked ‘excellent’ for interpretation did not use AI. Student feedback showed that students enjoyed the authentic nature of the unit as a whole, and enjoyed their innovative learning about the use of AI in healthcare.
The importance of sharing ideas
The process of integrating generative AI into educational assessments is one of collective exploration. There are no absolute experts in this field yet, but by experimenting with new methods and sharing what we learn, we can start to build a comprehensive collective understanding. Open dialogue and collaboration among educators will be pivotal in crafting assessment strategies that are innovative, effective, and principled. Please let your educational peers know what you are trying, so the good suggestions can be shared, and the trials which have proven disappointing can be redesigned or avoided. This is a global disruption to education; we best manage it by working together.
By grounding our assessments in these familiar core principles, we can ensure that our educational practices not only keep pace with technological advancements but also remain deeply rooted in the principles that promote meaningful learning. It helps us see the continuance of good education principles even while we are required to do the difficult work of redesigning our assessment tasks.
* or use KASH, if you want to start with Know Your Students.
