{"id":20771,"date":"2024-04-18T11:27:20","date_gmt":"2024-04-18T01:27:20","guid":{"rendered":"https:\/\/educational-innovation.sydney.edu.au\/teaching@sydney\/?p=20771"},"modified":"2024-05-07T10:06:51","modified_gmt":"2024-05-07T00:06:51","slug":"five-myths-about-interactive-oral-assessments-and-how-to-get-started","status":"publish","type":"post","link":"https:\/\/educational-innovation.sydney.edu.au\/teaching@sydney\/five-myths-about-interactive-oral-assessments-and-how-to-get-started\/","title":{"rendered":"Five myths about interactive oral assessments and how to get started"},"content":{"rendered":"

What if there was a way to make assessments more engaging, reduce marking time, enhance academic integrity, and assess unit or course learning outcomes effectively? Danielle Logan-Fleming<\/a> and Popi Sotiriadou<\/a> from Griffith University have seen this work around the world and in many disciplines through the interactive oral (IO) assessment<\/a>. They recently shared their expertise in a series of workshops aimed at supporting quality oral assessment tasks at Sydney.<\/span><\/p>\n

Aren’t interactive oral assessments just oral exams?<\/h2>\n

Interactive oral assessments<\/span><\/a> are authentic, scenario-based conversations where students can extend and synthesise their knowledge to demonstrate and apply concepts.\u00a0<\/span><\/p>\n

Importantly, an IO assessment is not a question and answer interaction, not a scripted conversation, not a viva voce, not a presentation, nor an OSCE. As each student engages with the scenario, the conversation will unfold in a unique way, allowing the educator to personalise the assessment experience.<\/strong> Supported with a marking rubric, the assessment is standards-based and mapped against the course or unit outcomes.<\/span><\/p>\n

In one example demonstrated by Dani and Popi, a student studying the hospitality industry arrived at an IO ready to discuss options for a hotel chain\u2019s breakfast offerings. Previous assessments and learning activities had helped students develop expertise on the hotel chain\u2019s clientele, food and beverage offerings, and industry trends. The student then participated in the IO as an employee of the hotel arriving at a meeting to discuss breakfast offerings (buffet, a la carte, etc). The marker acted as a hotel manager and adapted the conversation to listen to the proposal, check for skills, and query information in a way that gave the student good opportunities to achieve high standards against a marking rubric.<\/span><\/p>\n

Dani and Popi advise that there are six key components to the IO that ensure the assessment works; they advise this formula is closely followed when designing IO assessments.\u00a0\u00a0<\/span><\/p>\n

\"Flowchart<\/a>
Six components to ensure interactive oral assessments work<\/figcaption><\/figure>\n
    \n
  1. Scaffolding and support<\/strong>: consider where in your unit you can introduce opportunities for students to practise<\/strong> the skills they will need for an IO assessment. These practice activities could focus on the topic of the assessment or the practice of a conversation itself. Some students will be unfamiliar with having to speak out loud about their ideas, having always been assessed in writing, so it is important to make sure students have practice speaking and using the language of the discipline. It’s also important to give students an example of an IO (that you create) and have them mark the example IO on a rubric.<\/span><\/li>\n
  2. Scenario-based<\/strong>: One of the exciting things about the IO is the opportunity for creativity in how you design the assessment<\/strong>. The scenario allows you to pick a context that is appropriate to your discipline and the level of expertise of your students. The scenario could be prescriptive or give leeway for students to select the topic or the focus of the assessment.<\/li>\n
  3. Aligned to program<\/strong>: consider how the IO is aligned to the rest of the program and how it can allow students to demonstrate how they extend and synthesise<\/strong> their knowledge.<\/li>\n
  4. Learning outcomes<\/strong>: the IO conversations are shaped by the learning outcomes it seeks to assess. These outcomes define the criteria in the rubric, which in term informs the prompts used by educators as they shape the conversation during the IO. These prompts are not pre-scripted questions, but rather sentence-starters<\/strong> that encourage students to address the criteria for the outcomes that the IO seeks to assess.<\/li>\n
  5. Accessible and equitable<\/strong>: IO assessments are marked according to the rubric, which should not usually include criteria on communication quality (unless this is a specific unit learning outcome). The educator’s role is to drive the authentic conversation<\/strong> and try to unpack what the student knows in a supportive way. These contribute to IOs being more accessible and equitable.<\/li>\n
  6. Professionally-focused<\/strong>: one of the cornerstones of the IO is the focus on professionally authentic conversations. The scenario helps students and educators place themselves into a role that allows access to the structure and language appropriate for the IO. For example, an education IO assessment might simulate a parent-teacher interaction, an architecture IO assessment might have students defending their work to a potential client, a law IO assessment might have students being asked to provide advice to a client, and an IO for language learning could have students being asked to act as guides to their city of choice.<\/li>\n<\/ol>\n

    Surely interactive orals can’t work for large cohorts?<\/h2>\n

    According to Dani and Popi, the IO scales exceptionally well. You can have tutors or coordinators taking on the assessment, much like you would distribute marking normally. Dani and Popi estimate that a 10-15 minute IO conversation effectively assesses the equivalent of a 3,000 word text-based submission. They recommend allowing for some buffer time, e.g. 20 minutes per student as you start out and then 15 per student as you get more comfortable. Even with a more generous allowance for a 5-10 minute buffer, the per-student time commitment is still significantly less than marking and providing individual feedback on a 3,000 word essay.\u00a0<\/span><\/p>\n

    Plus, educators who have given IO assessments say that they actually enjoy the marking<\/strong>, and are able to much more clearly understand student knowledge (and gaps).<\/span><\/p>\n

    You could consider running the IO during timetabled class sessions, or allowing markers to nominate a schedule and arrange for students to book in for available time slots. Some designs intentionally keep tutors and students matched while other designs encourage tutors to assess students they don\u2019t teach in their own classes.\u00a0<\/span><\/p>\n

    Marking is variable and it’s hard to keep it fair and well-calibrated<\/h2>\n

    The key is to remember that you are not just having a random conversation, nor are you peppering your students with set questions. The educator will use a rubric to <\/strong><\/span>prompt<\/strong> students<\/strong>, gently nudging them towards their potential to meet a higher criteria. For example, if your student has described an observation (if \u2018description\u2019 were aligned to a credit grade), the prompt might be \u201c<\/span>tell me more about what made you come to this conclusion<\/span><\/i>\u201d, opening the door to analysis (if \u2018analysis\u2019 were aligned to a distinction grade). The role of the assessor is to actively listen and use the rubric to prompt students to extend their response.\u00a0<\/span><\/p>\n

    Interactive oral assessments can’t be organised at scale<\/h2>\n

    Booking tools like <\/span>Microsoft Bookings<\/span><\/a> can be helpful here. The tool allows you to set up an online booking system where you determine the available time slots for each of your assessors, and students are able to select an assessment time that works with their schedule.\u00a0<\/span><\/p>\n

    To run the sessions, you may wish to run them online or in person. If you are considering running them online, we recommend using Zoom, creating meetings for your tutors or asking them to take responsibility for creating their own links. Don\u2019t forget Zoom has some limitations for how many concurrent sessions you can run so if as the coordinator you are setting up Zoom links for your tutors reach out to Educational Innovation for some help. Also, AI software exists now that can respond to online video-based interviews in a deepfake live video feed, so be aware that online assessments are not secure.<\/span><\/p>\n

    You’ll need a rubric to make the IO work properly.<\/strong> You can set up your rubric in Canvas or SRES based on what you currently use for your other assessments. If you do not currently use a rubric we recommend joining one of the upcoming sessions on <\/span>rubrics and Canvas<\/span><\/a>.\u00a0<\/span><\/p>\n

    These assessments disadvantage students<\/h2>\n

    Equity and access are key concerns that IOs address, according to Dani and Popi. While usual University adjustments and support remain available to students, which might allow extra time, pre-recording, a support person, etc, key aspects of IOs contribute to improved relevance, motivation and opportunity. The scaffolded nature of assessments and activities in the lead-up to an IO, and the provision of exemplars and demonstrations, enable students to build expertise and familiarity with the task<\/strong>. Further, because the task replicates workplace or \u2018real-world\u2019 scenarios, there is a heightened sense of relevance, which motivates student participation.<\/span><\/p>\n

    Ready to introduce interactive oral assessments into your unit?<\/h2>\n

    We would love to work with you and your unit as you progress with the design. We have academic developers and educational designers ready to help. If there is any support that you need, even if it’s just a conversation on how things are going, please reach out to us. Interactive oral assessments are fairly new to the University so we are keen to learn together and help share practice across the institution.\u00a0<\/span>Please get in touch witih\u00a0<\/span>eszter.kalman@sydney.edu.au<\/span><\/a>.<\/p>\n

    Tell me more!<\/h2>\n

    Check out Dani and Popi’s excellent resources on interactive oral assessments:<\/p>\n