In semester 2, 2021 we piloted an approach to find and support students who may have been struggling across multiple units of study in the Faculty of Science. Unique to the pilot was that each of the 23 units had different sets of indicators that were customised by each unit coordinator, informed by their deep understanding of unit and student contexts.
The program involved Peter Rutledge and Mike Wheatland from the Faculty of Science, and was supported by staff from the DVC (Education) Portfolio and Student Experience teams. The Student Relationship Engagement System (SRES) was used to gather data on metrics such as attendance, assessment completion and performance, and Canvas activity, which was then used to identify students who showed signs of early disengagement in one or more units of study. Peter was able to personally reach out to students at scale early enough in the semester, using metrics that were meaningful and providing just in time support.
Background and context of the pilot
Changes to the Higher Educational Loan Program (HELP) set to take effect in January 2022 mean that students who fail to maintain a “reasonable completion rate” after taking at least eight units of study now face losing Commonwealth financial assistance for their university fees. Therefore it is more important than ever that we are doing our best to find and support students who might be at risk of failing a unit of study and intervening in a timely and effective manner. Many coordinators across the university are already using SRES at the unit of study level; the aim of this pilot program was to see if it was possible to scale up the use of SRES to identify students with low levels of engagement across multiple units of study.
The units which were included in the pilot were chosen to maximize the number of students included in the pilot. Roughly 4000 Science students were enrolled in at least one of the 23 first-year units selected for the pilot, accounting for about 97% of all Science students enrolled in first-year units
What was done?
The Educational Innovation team worked with unit coordinators to decide on the most appropriate early indicators that could be used to gauge student engagement in their unit leading up to census date. Engagement is multi-dimensional so, where possible, we tried to use a combination of indicators, many of which could be sourced directly from Canvas and synchronised automatically into SRES.
Some example indicators included:
- time spent engaged with the unit’s Canvas site
- completion of early tasks
- attendance at lectures and tutorials
- performance in early assessment tasks
Once indicators had been identified, a column was set up in each unit’s SRES list with a ‘traffic light signal’ indicating each student’s level of engagement in that unit. For visual ease, a red light indicated little or no engagement within the unit. These traffic light columns were then aggregated into a single SRES list, fittingly dubbed the ‘Frankenlist’. In this list, we could see every student enrolled in one or more of our pilot units of study, and the traffic light indicator of engagement for each unit they were enrolled in.
Particularly important for our purposes, aggregating all the early engagement data into a single list meant that students with a red light in more than one unit of study could be quickly identified and supported. Key to the pilot was that unit coordinators had full control over the engagement metrics being used, and that these were unique to the needs of each unit.
Peter was then able to use this information to send out a friendly and personalised text message and email to any student with a red indicator in more than one unit of study, checking in and reminding them of the upcoming census date and offering support. If a student had a red indicator in only one unit of study they were not included in the program-level email but were instead contacted by the unit coordinator for that unit.
What was the outcome?
In total, 85 Science students with red indicators in more than one unit of study received an email from Peter (26 Engineering students also received an email). Of the Science students contacted, 60 students viewed the email at least once (roughly 70% of the students who were emailed). Peter received responses from 27 students. Many of these responses were indicative of the broad range of stresses and disruptions that students are currently dealing with. A few of the responses were also from students who hadn’t realised they were still enrolled in one or more units of study (either thinking they had withdrawn or not understanding the withdrawal procedure). Reaching out to these students prior to census date meant they still had the opportunity to withdraw from a unit of study without financial or academic penalty.
Towards the end of semester, we repeated the process with new indicators from all the units of study. Only 30 students were identified at this point as still at risk, whereas there were 16 students who had moved from the early-semester at-risk status to having improved engagement.
Moving forward
There is still work to be done reviewing the pilot program with a view to determining the next steps, but even at this early stage there are a few key takeaways:
- by using SRES and making use of information that coordinators are already collecting through it, it is possible to identify and reach out to students at risk across multiple units of study;
- unit coordinators have quite different conceptions of the measures that suggest students may be disengaged, and it is important to accommodate for these unit-by-unit contextual differences;
- this will make a big difference to those students we contact, either by re-engaging them or helping them avoid financial penalties.
