ChatGPT creator OpenAI creates tool to recognize AI text
ChatGPT maker OpenAI has put a tool online that assesses whether a text is probably written by artificial intelligence or not. OpenAI has put the tool online as a ‘starting point in the discussion about AI literacy’.
OpenAI calls it AI Text Classifier and trained the model on 34 models from 5 different organizations, including ChatGPT. The tool returns a probability that a text was written by a human or artificial intelligence, but does not provide any certainty. Moreover, the model is often wrong. In a challenge set of texts on which the model was not trained, AI Text Classifier labeled 30 percent of human-written texts as “maybe” or “probably” AI-generated.
The tool is not available as an API, but only as a web interface. Moreover, there are the necessary restrictions. For example, the tool works particularly well in English and on adult texts, but children’s texts and in other languages work less well. OpenAI also takes into account that models can adapt to the tool, in order to make AI texts appear more human in order to pass the test. There are many concerns, including in education, about the use of ChatGPT. The chatbot could generate texts that people could then designate as their own.
| OpenAI Text Classifier | threshold | Proportion of human-written text* | Share of text written by AI* |
| Very unlikely to be AI generated | <0.1 | 5% | 2% |
| Unlikely to be AI generated | 0.1 – 0.45 | 15% | 10% |
| Unclear if it is AI written | 0.45 – 0.9 | 50% | 34% |
| Possibly AI generated | 0.9 – 0.98 | 21% | 28% |
| Likely AI generated | 0.98 – 1 | 9% | 26% |
* from the challenge set with which the model was tested
Got caught? OpenAI AI Text Classifier says this article ‘might be AI generated’ (it isn’t)