How memorable is your photo? A new tool will give you a score

This story was originally published by UChicago News. Read the article on their site here. 

UChicago psychologists tease apart what makes images stick in our brains

Why are some photographs remembered and recognized, while others are quickly forgotten? University of Chicago researchers are leveraging artificial intelligence and machine learning to search for an answer—and have developed a free tool that can predict how likely you are to remember a photo.

Called “ResMem,” the state-of-the-art tool is the most accurate and sophisticated software of its kind. Created by graduate student Coen Needell and Asst. Prof. Wilma Bainbridge in the Department of Psychology, their work builds on recent scientific advances toward identifying the factors that make some images more memorable than others. Their findings were described in a preprint paper released May 25.

ResMem allows anyone to upload photos and receive a “memorability score” between 0 and 1: A score of 0.85, for example, means that 85% of people viewing the image would remember having seen it. Because human memories tend to be fairly good, the average score is higher than 0.5—about 0.756.

“Memorability can be thought of as an intrinsic property to images,” said Bainbridge, an expert on memory. “It is actually something that a computational model can measure, predict and—eventually—maybe even manipulate.”

ResMem can computationally score the memorability of any photograph in a generalizable way. The tool leverages a model built on two datasets from earlier experiments, in which thousands of participants viewed about 70,000 total images and reported what they remembered.

The two datasets used to train the model collected real photographs (not cartoons or drawings) that sample a broad range of image types, from landscapes to photos of people and pets.

To identify factors—from lines to eye-like shapes—that contribute to the memorability of an image, Needell and Bainbridge used “residual neural networks,” a technique in artificial intelligence for teasing apart some of the components of memorability that might not be apparent to the human eye.

 

Built primarily by Needell, the model has yielded intriguing insights because of its ability to go beyond recognition of certain categorical objects—like a chair or a cat—and extract underlying factors and conceptual patterns that our brains pick up on as we focus on and begin to store an image in our brains.

The model then generates images of each neural “filter”—a pictographic representation of what the model thinks sticks best in our memories. Early filters tend to be abstract forms like lines, zig-zags or squiggle patterns. In deeper filters, these gradually give way to geometric forms that resemble honeycombs and shells. Finally, the model begins to trace and extract images that look recognizable to us: eyes and faces, for example.

“It’s cool to see the network picking up on categories of information that we know humans are sensitive to,” Bainbridge said. The patterns identified by these sorts of models map well onto how human brains actually process images in day-to-day life, according to the researchers.

“The reason researchers first made these neural networks was to try to mimic the visual processing system that had been observed in the human brain,” Needell said.

The authors recognize that the tool has vast potential. Bainbridge anticipates that educators might use it to figure out which figures students would be most likely to remember, and that clinicians could use it to create more memorable environments or other tools for assisting those with dementia and Alzheimer’s disease.
 

“We also envision people using this for more fun applications, like choosing a memorable LinkedIn profile picture or Instagram photo,” Bainbridge added.

Needell and Bainbridge plan to keep ResMem available to the public under an open-source license—but they have engineered it such that it cannot easily be used for profit-driven purposes like advertising. Individual users can only submit a few images at once, such that marketers, for example, would have difficulty sorting through hundreds of images to find the one most likely to leave an impression on consumers.

Among the most exciting uses of the tool, Bainbridge said, could be in further psychology research, which might yield more insights into how the brain processes and stores images.

“We could design really effective experiments where we could drive people’s memories,” she said, “hopefully helping us understand the underlying brain mechanisms behind how memories are formed.”