Researcher reconstructs screenshot of Toad by photographing shadow

Computer scientists at Boston University state that they have developed a method that makes it possible to see around a corner based on what is visible in a shadow of an object or subject projected onto a wall.

Vivek Goyal and other researchers at Boston University have created a new computer algorithm for analyzing photos for their method. In addition, the method requires a regular, relatively inexpensive camera. The system analyzes the light reflected from matte surfaces, such as walls. This makes it roughly possible to ‘look around the corner’, or at least to roughly reconstruct images that are not immediately visible.

This method is specifically concerned with analyzing the image information of the penumbra. This is also called the penumbra or penumbra and is the part of a shadow in which the light source behind can be partially seen. With the umbra or core shadow, the light source is completely blocked.

The researchers used a setup in a darkened room with no windows, where they had a monitor display several images, including a simple drawing of the Nintendo character Toad. A penumbra was created by placing a chair between the screen and the wall. Twenty photos were taken with a camera mounted on a tripod with a total exposure time of three seconds. This footage was then analyzed by the computer algorithm, after which a rough reconstruction of Toad’s drawing emerged after a few minutes.

From left to right: the images as they appeared on the monitor, their penumbras and the reconstructed images.

For this method it is important that part of the object to be reconstructed is blocked. No transparent object may be used for this. The shadows contain information for the computer algorithm that can analyze the distribution of the light and thus determine what the original scene looks like.

For now, the technology is in its infancy. The reconstructed images are still very pixelated and not exactly sharp. Moreover, the technology is not yet able to reconstruct much more detailed objects or subjects, such as human faces. Goyal says he hasn’t tried this method yet for reconstructing an image of a person around the corner, but he says there’s no reason why it shouldn’t be impossible. According to the researcher, the main limitation is the contrast between the penumbra and the surrounding environment. The method was tested in a darkened room. As the amount of light in the room was increased, the penumbra became harder to see and the system was also less able to make the reconstructions.

The researcher states that in the future a hybrid method may be used, whereby the system is able to locate the opaque objects in the foreground and include that information in the computer reconstruction of the scene. Goyal states that this research shows that a lot of information can be extracted from penumbras, and they can be found everywhere. For example, based on the orientation of the shadows, it could become possible in the evening to determine the locations of all street lamps. Reconstructing Toad is more difficult than mapping street lamps, Goyal tells Wired.

Gordon Wetzstein, an electrical engineer from Stanford University who was not involved in the study, is developing a similar technology that uses lasers. He tells Wired that Goyal’s research could help robots maneuver better and make self-driving cars safer. In the medical field, it can be used to detect tumors or to see around bones.

Darpa, an American defense institute that develops new technology for the military, has supported this research, among others. These include the Darpa program Reveal, which sponsors research to create images in very low light. Government surveillance or military applications could thus also benefit from Goyal’s research.

Goyal and the other researchers published their findings in an article in the scientific journal Nature, entitled Computational periscopy with an ordinary digital camera. This research builds on a 2017 MIT study that used shadows to record the direction people walked around a corner. In that older study, however, no clear pictures could be made and color did not yet play a role.