The Open University Scholarship Exchange
Browse
Mike-Richards-Kevin-Waugh-Daniel-Gooch-Mark-Slaymaker.pptx (143.69 kB)

Poster: Does the use of Grammarly impact detection rates from Generative-AI detection tools?

Download (143.69 kB)
poster
posted on 2024-01-17, 09:37 authored by Mike Richards, Mark Slaymaker, Daniel GoochDaniel Gooch, Kevin Waugh

Generative AI tools can produce compelling text, even to University assessment questions. Our research has demonstrated that this poses a risk to the OU, as current versions of ChatGPT can produce material sufficient to pass undergraduate modules. We also tested two types of detection software including the standard OU TurnItIN AI detection service. This software produced mixed results https://oro.open.ac.uk/89325/.

This provisional work took a simplistic view, submitting material completely generated through ChatGPT. We recommend students use Grammarly – or other similar tools – to improve their writing skills. Grammarly now uses similar technology to ChatGPT to improve writing. It is currently unknown whether AI detection tools can distinguish between writing produced by generative AI and material which has been improved by the use of generative AI.

Our proposal is to run a study with the following steps:

  1. Obtain ethical and data protection sign-off.
  2. Download 10 random student scripts apiece from multiple C&C modules covering a range of study levels and assessment types from the 21J cohort (pre-release of ChatGPT, as used in our previous studies).
  3. Use Grammarly suggestions to progressively change the student submissions resulting in three levels of scripts – unchanged, minimal changes, significant changes.
  4. We will generate a separate set of scripts which manipulate the student submissions using the AI-based Microsoft Word editor.
  5. Upload the scripts to the MCT quality assurance area on TurnItIn ACQ have setup for our team as a testbed. We will record the resulting detection levels across the scripts.
  6. Analyse the data.
  7. Disseminate externally through a SIGCSE publication, and internally through seminars and the OU community of interest for generative-AI group on Teams.
  8. Based on the results, we will engage with the study skills team that provides tool advice for students, and the group working on AI policy for the OU.

Funding

eSTEeM

History

Sensitivity

  • Public document

Institutional priority category

  • Students Learning Experiences

Themes

  • Assessment
  • Innovative Teaching Approaches

Subject discipline

  • Computing and IT

Usage metrics

    eSTEeM

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC