Does Turnitin Detect ChatGPT? What the Data Shows

TL;DR

Turnitin added AI detection in 2023. Detection capability is limited — Scarfe et al. (2024) found 94% of AI submissions went undetected at the University of Reading. False positives and false negatives both occur. Detection improves slowly; assessment redesign is increasingly seen as more important.

How Turnitin's AI detection works

Statistical analysis approach

Turnitin's AI detection (added in 2023) analyses text features that distinguish AI-generated from human-written text:

• Word-distribution patterns
• Sentence-structure variation
• Idiomatic and stylistic characteristics
• Probability of word sequences

The tool produces a percentage estimate of likely AI-generated content in a submission.

Coverage

The detection is trained on output from major AI tools:

• ChatGPT (GPT-3.5, GPT-4, GPT-4 Turbo and successors)
• Claude (Anthropic)
• Gemini (Google)
• Other major large language models

Less-covered AI tools (smaller models, less-known services) may produce text that passes detection more readily.

The Scarfe 2024 study

The most cited independent assessment of AI detection capability:

Method

Scarfe, P., et al. (2024). "A real-world test of artificial intelligence infiltration of a university examinations system: A 'Turing Test' case study" [verify exact title].

The study:

• Submitted AI-generated coursework through normal channels at the University of Reading
• Used psychology undergraduate course assessments
• Tested combined human marker review plus Turnitin AI detection
• Tracked which submissions were identified as AI-generated

Result

94% of AI submissions went undetected by the combined detection system.

The 6% detection rate established the empirical benchmark for current AI detection capability. The finding has been widely cited as the most authoritative independent assessment.

Implications

If 94% of AI submissions evade detection in a controlled study at a research-active UK university with mature integrity infrastructure, detection alone cannot be relied upon as the primary defence against AI misconduct.

False positive problem

Non-native English speakers

Multiple studies have documented that text written by non-native English speakers can register as AI-generated:

• Less idiomatic English patterns can resemble AI output
• Statistical features may overlap with AI text characteristics
• This produces unjust accusations against students with English as a second language

Other patterns producing false positives

• Technical or formulaic writing styles
• Writing produced under time pressure
• Heavily edited human writing
• Translation from another language

The false positive rate is partly responsible for limited institutional confidence in AI detection.

False negative problem

Cases where AI submissions evade detection:

Light editing

Students editing AI output substantially reduce its statistical signatures. Even modest editing — replacing some words, restructuring some sentences — can move AI text below detection thresholds.

Iterative prompting

Students who use AI with extensive prompts and iterative refinement produce more human-like output. The more student input into the AI generation process, the harder detection becomes.

Paraphrasing through additional AI

Passing AI output through paraphrasing tools (or through additional AI passes with paraphrase prompts) produces text without clear AI signatures.

Hybrid drafts

Combining AI-generated content with human-written sections produces submissions that often pass AI detection — the mixed signal does not trigger detection thresholds.

What Turnitin says vs what works

Vendor claims

Turnitin publishes accuracy claims for its AI detection. Specific numbers vary as the product is updated. The claimed accuracy is typically higher than independent research finds.

Independent assessment

Scarfe 2024 and other independent studies have consistently found lower accuracy than vendor claims. The discrepancy is partly because:

• Vendor accuracy testing uses controlled inputs that may not reflect real student behaviour
• Real students adapt to detection capability
• Detection accuracy varies by text length, language, and edit-state

How students try to bypass detection

The cat-and-mouse dynamic is real. Common bypass methods:

Prompt engineering

Using extensive prompts that incorporate the student's own context, vocabulary, and style preferences. The AI output reflects student input rather than pure model output.

Editing AI output

Modifying AI-generated text:

• Replace some words with synonyms
• Rearrange sentences
• Add personal anecdotes or examples
• Insert intentional minor errors

Multiple passes

Running text through:

• Paraphrasing tools
• Additional AI with paraphrase instructions
• Translation to another language and back

Hybrid composition

Combining:

• Human-written introduction and conclusion
• AI-generated body sections
• Human-edited transitions
• Student-added citations and examples

Beyond Turnitin

Other AI detection tools

• GPTZero — early AI detector with similar limitations
• Originality.ai — commercial AI detector
• Copyleaks — combined plagiarism and AI detection
• Various academic tools — research-oriented detection approaches

All face similar fundamental limitations as Turnitin.

Language-specific detection

Detection capability is strongest for English-language AI text. Other languages have weaker detection:

• Spanish, French, German: reasonable but lower than English
• Less-resourced languages: very limited
• Translation plagiarism: typically undetected

Institutional alternatives

Some institutions deploy combinations of tools rather than relying on a single detector. Multiple-tool deployment reduces single-tool false positives but does not necessarily improve overall detection.

Why detection matters less than expected

The 94% miss rate

If 94% of AI submissions evade detection, detection is not the primary deterrent. The deterrent is whatever consequences students face when caught — which depends on the small fraction of cases identified.

Detection vs assessment redesign

Increasing institutional emphasis on assessment redesign:

• Oral examinations
• Live problem-solving
• Project work with iterative review
• Demonstrable understanding rather than text production

These approaches make AI use largely irrelevant rather than relying on detection to catch misuse.

Policy maturation

Many institutions are moving away from "detect and punish" toward "design assessment so AI use is irrelevant or required-with-disclosure." The Scarfe finding accelerated this shift.

What to expect going forward

Detection capability will improve

Turnitin and competitors continue to invest in detection. Iterative improvement is occurring. The 94% miss rate is a 2024 measurement; 2026 capability is somewhat better but still well below reliable thresholds.

AI capability will also improve

The AI tools students use continue to improve. Newer AI versions produce more human-like text. The cat-and-mouse dynamic is structural.

Assessment redesign will accelerate

The most durable response is assessment redesign. Universities investing in this are positioned better for the long-term than universities relying on detection alone.

Policy expectations will shift

Expectations around AI use will continue to mature:

• Permitted-with-disclosure is becoming dominant
• Detection-as-primary-defence is being abandoned by leading institutions
• Assessment design becomes the primary integrity tool

Bottom line

Does Turnitin detect ChatGPT? Partially, with substantial limitations. Detection is a useful signal but should not be relied upon as the primary defence against AI misconduct. The Scarfe 2024 finding (94% miss rate) is the empirical benchmark. Institutional responses are increasingly moving toward assessment redesign rather than detection.

Sources

• Scarfe, P., et al. (2024). University of Reading AI submission study
• Turnitin AI Writing Detection documentation
• Independent AI detection research (multiple)
• AMI v1.5 D2 dimension methodology

Full methodology | Download dataset