Last year, we introduced a Cogniti AI assistant to help educators provide more personalised and constructive feedback on first-year biology scientific reports. Cogniti allows educators to guide large language models like GPT-4 with specific instructions and resources, such as assignment information, rubrics, and examples of good feedback, to produce tailored outputs that support student learning.
Recognising its benefits, in 2024 we developed three different Cogniti AI assistants to support first-year students in various aspects of biology.
Designing AI agents to help students
“Patch the Biology Pup” assists students with general unit information and support, while “Dr. Patch the Academic” acts as a Socratic tutor, based on unit specific content, and generates multiple-choice questions (MCQs) for students to test their knowledge. With a large first year cohort of 1,500 students, Dr. Patch offers students a personal tutor that can respond instantly to lecture-based questions, eliminating the need to wait for the next class, email, or posting on discussion boards. Furthermore, Dr. Patch creates MCQs to help students prepare for quizzes and the final exam.
The third Cogniti agent, “Scientific Writing Assistant”, provides pre-submission feedback on scientific reports. First-year students are still developing their scientific writing skills, a core skill crucial for their future careers. Constructive feedback is essential, but with a diverse cohort of 1,500 students, providing personalised and timely feedback is challenging. To address this, we developed the “Scientific Writing Assistant” using Cogniti. This customised AI assistant enables students to submit drafts of their scientific reports and receive immediate feedback based on the assignment rubric, helping them improve their scientific writing skills before the final assessment.
Part of the system message used to guide the “Scientific writing assistant.”
Impact of AI agents on student learning
The introduction of the three Cogniti AI assistants into first year biology has significantly transformed the way we support student learning.
Over the semester, we saw remarkable engagement, with Patch the Biology Pup (1,205 conversations, 430 unique users), Dr Patch the Academic (4,183 conversations, 619 unique users) and the Scientific Writing Assistant (1,500 conversations, 720 unique users) representing more than half of the student cohort.
The pre-submission feedback tool has not only made it feasible to provide timely and constructive feedback on written drafts, but it has also offered a deeper insight into student needs through us being able to view students’ conversation histories. Through analysing these conversation histories in Cogniti, we identified several key areas where students were seeking more support, including academic writing, assessment preparation, application of scientific concepts and integration of research and evidence. These insights have proved invaluable in tailoring our teaching approaches to better meet student needs.
We have observed an overall improvement in student performance in scientific report writing compared to previous years. In addition, students who used the “Scientific writing assistant” for feedback on their written drafts scored higher in the scientific report assessment compared to students who did not use Cogniti.
An example of feedback based on the rubric after submitting a draft of a scientific report.
After submitting their scientific report assessments, students were encouraged to reflect on their engagement with Cogniti and share their thoughts. The reflective statements gathered revealed that students who utilised the tool found it particularly beneficial. One student said:
The tool was helpful in guiding me toward a more structured approach… interpret my results, suggesting that I focus less on restating findings and more on synthesising them into my hypothesis… innovative future research directions, …gave me valuable ideas on how to expand and provide clarity to my ideas.
Some students expressed reservations about using the “Scientific writing assistant,” stemming from concerns about accuracy of the feedback in relation to assessment criteria and data privacy. “I decided not to use this AI tool in my preparation, as I am aware that it is relatively new and may possess some limitations (whether that be in cohesion or scientific literacy) and as such was not comfortable using it for this assignment.”
To address these concerns, it is essential that we share more detailed information about how the “Scientific writing assistant” is developed. Transparency about the system message design, data handling processes and enhancing AI literacy will help to alleviate some of these concerns.
Feedback from educators has indicated a decrease in the number of postings on discussion boards and a reduction in enquiries via email and in-person regarding the structure of discussions, integration of literature, and the depth of knowledge required. This suggests that Cogniti has been effective in clarifying these aspects for students, potentially reducing the need for further clarification from teachers. One educator remarked:
I believe the Scientific Writing Assistant was extremely helpful in its ability to not only critique student’s discussions, but also at other stages of the writing process such as the initial structuring, planning, and developing. It is easy to use and helps the student learn by directing them to the answer as opposed to giving them the answer directly.
There are plans to continue using the three Cogniti AI assistants in future. For the feedback tool, we will further refine the AI’s design based on chat insights and include a more detailed introduction to the tool. It will be crucial to enhance communication with students about the “Scientific report assistant” design and data privacy to build AI literacy skills, trust and encourage wider use and engagement. This could involve more open dialogue and demonstrations of the AI tools at the beginning of the semester as well as developing skills in prompting, critical review of AI generated outputs and discussing limitations of the tool.
The chat history insights from Cogniti will help to inform future teaching practices related to unit content and scientific writing skills. The embedding of Cogniti has provided us with a way of addressing concerns related to the structure and content of the scientific report at an individual level. This has freed up our time to focus on other aspects of report writing such as literature searching, comprehension and citation.
Our teaching approach and use of Cogniti can be easily adapted and applied to any unit, assessment, or task. By simply updating the system message with unit content, topics, assignments, rubric, and examples of good feedback, it becomes highly adaptable and transferable. We have already begun implementing these types of Cogniti AI assistants in other units of study not only at the school level but across the whole University.
To find out more
- Read about the scientific report feedback expander.
- Read other Teaching@Sydney articles about generative AI and Cogniti.
- Register for a Cogniti workshop to learn more about the tool and start building your own AI agents.
