Our journey with generative artificial intelligence (AI) in higher education began in late 2023 with the release of Cogniti, a generative AI tool developed by educators at the University of Sydney. Cogniti allows users to guide large language models such as GPT-4 by providing specific instructions and resources. A group of us, an educational designer and four academics, came together to explore how this tool could be used to support students and educators.
The excerpts below are from an interview discussing our experiences with AI in education, emphasising the importance of collaboration. We discuss the tool’s evolution from its early use as a feedback expander for markers, to its adaptation for a second unit of study, and finally its application in providing pre-submission assessment feedback directly to students. We also discuss how we worked together within and across teams, and between central, faculty and school units to foster a collaborative environment. Our goal is to share insights on enabling strong partnerships. The excerpts have been edited for length and clarity; you can watch the full interview in the video below.
This presentation explores the challenges and opportunities we encountered in integrating AI into education. We discuss how we worked together within and across teams, between central and faculty units, to foster a collaborative environment.
The vision and objectives of implementing Cogniti
The idea for creating Cogniti AI assistants came from a mix of curiosity and apprehension about integrating AI into our teaching contexts and higher education more broadly. Initially, practical applications of AI were limited, and while we were eager to explore this new tool, we were also concerned about how best to incorporate it into our units of study. We were still learning about its capabilities and anticipating how educators and students might respond.
Recognising this as a perfect opportunity for collaboration, we aimed to demystify AI and Cogniti, creating a tool that educators could use confidently without needing any technical or programming expertise. By combining our expertise in educational design and subject matter knowledge, we developed a Cogniti agent to assist educators in enhancing and expanding feedback for students on scientific report assessments. Our focus was on practical applications to enhance teaching and learning experiences, ensuring the use of AI was authentic, accessible, and aligned with learning outcomes.
Overcoming challenges in the initial stages of development
During the initial design phase of the Cogniti AI assistants, we encountered several challenges due to the shortage of use cases. One of our primary goals was to clearly identify the teaching challenges that Cogniti would address. This required a deep understanding of the needs and expectations of both educators and students. As we progressed, the lack of use cases meant we had to rely heavily on extensive and thorough testing. This involved continuous feedback loops with users, both educators and students, who were crucial in determining Cogniti’s effectiveness. Their insights were vital in assessing how well the Cogniti AI assistant understood and responded to the teaching and learning challenges it was designed to solve.
Additionally, tweaking the system message to better suit our needs was a significant hurdle for some educators due to limited technical expertise. This challenge highlighted the importance of teamwork and the reliance on individuals more experienced with AI prompting to make the necessary adjustments. Through collaborative efforts, we iteratively refined the tool based on real-time feedback, which was essential for improving its functionality and ensuring it met the learning goals effectively.
User experiences and feedback
The feedback gathered from users of Cogniti has been very positive.
Markers in the first-year biology and statistics units have particularly been supportive of the Cogniti feedback expander. They valued its ability to generate detailed, kind, and constructive feedback efficiently, which not only saved them time but also enhanced the quality of feedback provided to students.
Data from the Cogniti chat history for the pre-submission feedback assistant has been crucial in refining the tool’s effectiveness and user-friendliness. This includes analytics showing how frequently students engaged with the AI assistant, the types of questions asked, and common misconceptions. Reflections written by students about their interactions with the AI assistant in preparation for assessments have provided deeper insights, helping to ensure the tool meets the needs of both educators and learners effectively.
Overall, the feedback and data collected suggest that Cogniti is a valuable educational tool, offering substantial benefits in terms of feedback quality and accessibility of educational insights, which can help inform future teaching practices. This continuous loop of feedback and subsequent adjustments ensures that the tool remains effective, relevant, and user centric.
The importance of collaboration
Reflecting on a year of integrating AI into our educational practices, a key lesson has emerged: the power of trust and collaboration. Initially, our roles were distinctly outlined; academics as subject matter experts, who needed to understand the teaching context and implement the tools in the classroom, working with an educational designer who managed the design and development of the tools. None of us were AI experts, which made our journey into using AI tools like Cogniti both challenging and rewarding. Our collaboration evolved as we navigated this new technology. Trust was crucial; we had to rely on each other’s expertise.
I really like the idea around trust because…that is a massive thing…I didn’t have the expertise that you had in the back end it’s like I have to trust that that part you can do and then you have to trust that like I can do the implementation part of it.
Open communication was critical, allowing us to give honest feedback without fear of offence, which was vital for adapting and refining our approaches. The partnership was not just about combining our respective expertise but also about learning from each other’s strengths and weaknesses. This collaboration led to a more enriched student and teacher experience, leveraging AI to enhance learning effectively. This experience has highlighted the importance of teamwork in innovation, especially when venturing into new technological tools in education.
Advice for educators looking to develop similar tools
For anyone keen to explore new AI tools to support your students, the best advice is to just give it a go. Surround yourself with colleagues who share your ambitions and dive in together. We all started as novices in AI, not getting everything right on our first attempt. Key to our progress was trust, starting small, keeping the design simple and user-friendly, and gradually building from there. We embraced a low stake beginning, which allowed us to iterate and refine our approach continuously.
We’re not experts here. We didn’t get it right the first time…an important part of this is starting small, trusting each other, building upon it. It may not be perfect but we can go back…we can iterate and develop the process further.
What really made a difference was our willingness to experiment. We weren’t sure of the outcomes, but the excitement of innovation drove us. It’s crucial to have a supportive team, as the success of such ventures hinges on collaborative efforts and shared expertise. Engaging with others who can join you on this journey is invaluable. Most importantly, listening to student needs and aligning the tools with learning outcomes is vital.
Future directions
In our journey of integrating AI tools like Cogniti into education, our next steps involve a deeper analysis of conversation histories to enhance how we tailor teaching and support for students. By examining common questions and misconceptions revealed in these interactions, we aim to personalise and improve the learning experience. Additionally, direct feedback from students who have engaged with these AI tools will be crucial. We plan to explore their perceptions and decision-making processes regarding the use of AI in their education, particularly how they receive, and trust AI expanded feedback.
A long-term goal is to ensure equitable AI literacy and experiences for all students, including those facing language barriers, by providing the necessary support. The integration of AI tools like Cogniti into educational practices at the University of Sydney represents a significant step forward in addressing the challenges of providing personalised and constructive feedback in large cohort units. As we continue to harness the potential of AI in education, the focus remains on enhancing and supporting the student learning experience.
To find out more
- Read about the scientific report feedback expander.
- Read Teaching@Sydney articles about generative AI and Cogniti.
- Register for a Cogniti workshop to learn more about the tool and start building your own AI assistants.
