by Dr Emma Heywood and Jesse Armstrong
.png.jpg)
The advent of Chat GPT in November 2022 left us all excited by its possibilities but also instantly aware of the problems it would bring in HE. Students, just like academics, would be eager to push the limits of this new tool. But how would this affect assessments? How could we know what the students had done and what AI had done? What could we allow and what would come under the yet-to-be-defined unfair means guidelines for this issue?
By engaging with students, rather than imposing an unworkable ban on AI use, we asked them to identify how they used AI in their assessments. We designed a template into which the students would write their copy and paste their essay assignment into a left hand column and annotate each paragraph as they wrote, in a right hand column, with comments on how they used AI, which AI, and which prompts they used, and whether AI was useful. Administratively, this was a straightforward and instant response which did not require formal approval as changes to module or assessment descriptors were not required.
The study targeted 2 cohorts of students, both new to HE in the UK: Level 1 UG students, the majority being home students in semester 1, and MA students, the majority being international students, in semester 2. There were just over 100 students on each module. Both assessments were essays (2500/3000 words). Our aim was to understand how students of different levels and backgrounds used Al in the academic year 2023-24, and to trial a tool to identify students' use of Al which could be adapted for future use.
We circulated an evaluation questionnaire to students after the submission deadline, but before the marks were released, about the new template. From their responses, we found they used AI for a) checking language content and for style b) information gathering and content, and c) structure and organisation. They revealed they used a broad range of tools including Google Translate, Deepl, Grammarly, ChatGPT, Perplexity AI, Claude, Bing, SciSpace, QuillBot, Bard and Docsium, but rarely the university-recommended Gemini.
Students found that while the various tools could be helpful in brainstorming ideas and constructing plans, the AI response often created more work for them. This was because they found that it fabricated information (often referred to as hallucinations), was generic, insufficiently detailed, inconsistent and as a result, that they had to revert to lecture notes and checking suggestions. Nonetheless, many did seem to use AI as a support to boost confidence and to overcome anxiety in a new context. Students were also using it instead of Google as the primary search tool, potentially hindering curiosity and their desire to research, triggering a lack of independence and self-reliance.
The template itself, as a tool, proved easy to use. It did not deter them from using AI but they were confused about which tools ‘counted’ as AI. Importantly, rather than simply checking a box on a generic form, the majority of the students concluded that this using template made them reflect on their AI use. We will continue using this template moving forwards, especially as AI tools are becoming increasingly integrated into word processing, translation and grammar assisting software.
The study fulfilled its two aims. First, students seem to be using AI as a complementary tool in their assessments, for initial research, language refinement, and structural organisation. They were aware of its limitations and the need to critically evaluate AI-generated content. However, the only information we can base this assumption on comes from the students themselves which is an obvious limitation. Second, by allowing students to reflect on their usage of AI, the use of our non-assessed tool enabled them to acknowledge they were using AI, which would always have been the case, and allowed them to feel they were being treated as adults and they responded accordingly.