Scientists working on photo editing through text input

Researchers at the American Brown University have developed a method for editing photos by means of text. They used machine learning and their own algorithm for the method.

For example, the method allows users to adjust the weather and time of certain landscapes. To do this, the researchers had volunteers analyze more than 8,000 photos with different weather types. They came from more than a hundred webcams worldwide, which had captured the landscape in different weather conditions and seasons.

The scientists had the input provided analyzed by their algorithm. This resulted in a total of forty attributes, which the researchers’ program has at its disposal to be able to edit photos. Those attributes consist of simple commands such as cloudy, sunny, snowy, rainy or foggy. In addition, there are also subjective commands that the user can choose from, including gloomy, bright, sentimental, mysterious and rustic.

The program then helps users manipulate photos by simply asking for text commands. For example, a command like ‘more rain’ makes the photo look rainier. To do this, the program divides the photo into different clusters of pixels. Then it colors it by looking at the properties of ‘rain’, which are described in the database with the eight thousand defined photos. The text commands should help people who are not as familiar with advanced photo editing software as Photoshop.

According to the researchers, their program offers more than just filters, as Instagram does, for example. “Changing the weather in a photo is more than changing the blue sky to gray sky. It requires subtle adjustments to color and contrast throughout the photo – adjustments normally only possible by a professional photo editor,” said the Brown University.

The program, which will be presented at the Siggraph conference next week, still has some limitations. For example, it cannot reproduce attributes that require new structures in a photo. “We cannot turn winter into summer, because this means that we have to show grass, for example, in places where there is currently snow,” the researchers said.