This article approaches AI text detection the way you would approach any binary classifier in production: what are the features, what is the decision boundary, where are the errors, and who absorbs them. The running example is one of the best-documented failure cases: non-native English writers.

The problem framing

Formally, a detector maps a document d to a score s(d) and flags it when s(d) > Ï„. The true label, whether a human or a model produced the words, is never observed by the detector. It only sees the surface text.

That makes every detector a proxy classifier. It does not test for authorship; it tests for properties that happened to differ between human and machine text in whatever data it was tuned on.

The proxy features

Statistical detectors typically draw on three families of features.

Token predictability. Score the document with a reference language model. If each next token is assigned high probability, the text has low perplexity. Model-generated text, decoded toward likely continuations, tends to land here.

Distributional uniformity. Compute something per sentence (length, or per-sentence predictability) and look at its spread. Human text tends to be "bursty"; generated text is often more even.

Lexical markers. Counts of stock transitions ("Moreover", "In conclusion"), formulaic list structures and generic phrasing.

Each of these is a reasonable feature. The engineering question is what else correlates with them.

The confound: proficiency

In ML terms, second-language writing is a confounder. A learner's prose shifts all three features toward the "machine" side for reasons that have nothing to do with machines:

A classifier trained to separate fluent native writing from model output has learned a boundary that L2 writing sits on the wrong side of. Two honest sentences make the point:

"Social media is important for young people. It has many advantages and disadvantages."

"My cousin runs her whole bakery from Instagram, which still surprises our grandmother."

The first is common words in a textbook pattern; it is easy to predict. The second is specific and personal. A predictability-based detector may well score the first as more "AI-like", though both are perfectly human.

Evidence of the distribution shift

Liang et al. (2023, Patterns) effectively ran the out-of-distribution test. Seven widely used AI-text detectors were applied to 91 human-written TOEFL essays and 88 US eighth-grade essays:

• US essays: classified accurately.
• TOEFL essays: an average false-positive rate of about 61%.
• TOEFL essays flagged by all seven detectors had significantly lower perplexity than the rest.

The neat part is the intervention. Simplifying the vocabulary of the native essays, pushing them toward the L2 distribution, substantially increased misclassification. Manipulate the confounder, and the label moves.

The authors flag the limits honestly: a pilot study, small samples, and detectors (early 2023) mostly using an older, smaller language model as their backbone. Newer systems differ in detail; whether they still lean on the same family of signals is worth checking for any tool you use.

Ensemble disagreement as an uncertainty signal

If you treat the seven detectors as an ensemble, their disagreement is informative. In the Stanford data, 97.8% of TOEFL essays were flagged by at least one detector, but only 19.8% by all seven. When independent models disagree that much, the inputs are near or across their decision boundaries: high uncertainty, presented to users as confident percentages.

Error costs are asymmetric

In most product classifiers you tune Ï„ against the relative cost of false positives and false negatives. In education, a false positive can mean a misconduct accusation against an honest student. Base rates make this worse at scale. In August 2023 Vanderbilt University disabled the AI-detection feature of its commercial plagiarism checker, noting that even the vendor's own claimed 1% false-positive rate could have meant around 750 wrongly labelled papers out of the 75,000 it submitted in 2022.

What responsible output looks like

If a detector's job is to inform a human decision, the interface should expose the evidence and the uncertainty:

• Localised signals. Highlight which sentences drove the score so a reader can see whether it is a formulaic intro or definition, a known false-positive zone.
• Human-in-the-loop by design. Pair the score with process evidence: drafts, version history, a conversation with the writer.

This is how we try to build the Coda One AI Detector. It is free with no daily limit and no signup. It highlights the specific sentences that read as machine-written instead of giving only a score. Our own framing is the one we would apply to any detector: detection is evidence, not proof.

Summary

Treat a detector score like any classifier output: a function of proxy features, a chosen threshold, and a training distribution. Second-language writing is a known confounder that shifts those features toward "AI". Build and read detectors accordingly, and keep the final decision with a person who has looked at more than one number.

Sources

• Liang, W., Yuksekgonul, M., Mao, Y., Wu, E. & Zou, J. (2023). Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779 (preprint: arXiv:2304.02819). The detectors tested are not named in this article.
• Vanderbilt University, blog post announcing that the university was disabling AI detection, August 2023. As reported in: Tom Carter, "Universities ditch AI detectors over fears students could be falsely accused of cheating", Business Insider, 22 September 2023. https://www.businessinsider.com/universities-ditch-ai-detectors-over-fears-students-falsely-accused-cheating-2023-9