Facebook uses artificial intelligence to understand text in pictures
Facebook uses artificial intelligence to understand text in pictures. The system called Rosetta not only recognizes the text, but also understands the context. The system is used on Facebook and Instagram to find hate messages, among other things.
Facebook explains about Rosetta on its website. According to the social network, using traditional OCR techniques is not enough, because the way text appears in images is constantly changing and also because OCR cannot understand the context.
Rosetta uses a machine learning model that has viewed more than a billion images of text from Facebook and Instagram. These are public images. To recognize text, the artificial intelligence Faster uses R-CNN.
Initially, the model was trained with only short words, with a maximum of five letters. After the system had mastered all those words, longer and longer words were added. The artificial intelligence can handle different languages and characters, Rosetta currently recognizes English, Arabic and Hindi among others.
Rosetta is already widely used by several teams within Facebook and Instagram. The technology detects hate messages, among other things. Facebook says it is far from finished with the system. Many more languages need to be supported and recognizing text in video remains a challenge.
Researchers at Facebook have written a paper on Rosetta entitled: Large scale system for text detection and recognition in images.