When the Cheating Detector Cheats

Blog 10: When the Cheating Detector Cheats You Imagine writing an essay entirely on your own, submitting it, and then getting called into your teacher's office because a piece of software decided you probably did not write it. This is not a hypothetical. It is happening constantly, and it is happening most often to exactly the students who can least afford it.

Stanford researchers tested seven widely used GPT detectors on two sets of writing: essays by US eighth graders, and TOEFL essays written by non-native English speakers applying to study abroad. The detectors were nearly perfect at recognizing the eighth grade essays as human written. But they misclassified more than half of the non-native writers' essays as AI generated, an average false positive rate of 61.22 percent (Liang et al.). Some of those essays were unanimously flagged as AI by every single detector tested, meaning a student could get accused from multiple directions at once over writing that was completely their own.

Think about why this happens. These detectors work by measuring how predictable or repetitive a piece of writing is. Non-native speakers tend to rely more on common phrasing and simpler sentence structures while they are still building fluency in a second language, which is a completely normal part of learning English, not a sign of using a chatbot. The tool is not detecting AI. It is detecting a certain kind of writing style, and it is punishing students for not sounding like a native speaker.

As someone who has classmates from a dozen different language backgrounds, this is not an abstract statistic to me. A flag from one of these detectors is treated like hard proof in a lot of schools, and the burden usually falls on the student to somehow prove they did not cheat, which is a nearly impossible thing to prove for an essay you wrote in your own head. If a tool is this unreliable and this biased against a specific group of students, it should not be making disciplinary decisions on its own. At minimum, a flag should be treated as a reason to have a conversation, not as a verdict.

Works Cited Liang, Weixin, et al. "GPT Detectors Are Biased against Non-Native English Writers." Patterns, vol. 4, no. 7, 2023, article 100779.