“Attention is known to modulate neural representations of perceived images,” says lead author of the study Ryohei Fukuma. “However, we didn’t know whether imagining a different image could also change these representations.”

To test this, the researchers developed a new technology by working with patients with epilepsy who already had electrodes implanted in their brains to record and display electrocorticogram readouts of images that they were imagining. The patients were shown an image of the real-time readout and instructed to mentally picture a different image representing a “landscape,” “human face,” or “word” (for example, thinking of a human face while looking at various types of images) to control the readout.

“The results clarified the relationship between brain activities when people look at images versus when they imagine them,” explains Takufumi Yanagisawa, senior author. “The electrocorticogram readouts of the imagined images were distinct from those provoked by the actual images viewed by the patients. They could also be modified to be even more distinct when the patients received real-time feedback.”

The time needed to generate a very clear distinction between the imagined image and the viewed image was different for imagining a “word” and a “landscape,” which could have something to do with the different parts of the brain involved in imagining these two concepts.

“Our findings suggest that a readout image controlled by the subject’s imagery can be inferred by an observer using this technology,” says Fukuma.

Given the accuracy with which this new technology displays images that exist within the subject’s mind, a similar approach could be used to develop a communication device for severely paralyzed patients, such as those with amyotrophic lateral sclerosis. Similar devices already used by some patients with this condition rely on motor control, which degenerates more quickly than visual cortical activity, so an imagery-based device could be highly valuable.

Fig. 1 Neural decoding and patient control of imagined image presentation

Electrocorticogram (ECoG) recordings were taken from 17 patients with epilepsy who had implanted subdural cortical electrodes related to visual perception. A decoder was trained to estimate the semantic meaning of the images that the patients were viewing from the intracranial ECoG recording using visual-semantic space. Based on the real-time inferred semantic information with the decoder, an image was displayed on a monitor placed in front of the patient. The patient then tried to display an image with instructed meaning by imagining it.

(CC BY, credit: 2022 Ryohei Fukuma et al. Voluntary control of semantic neural representations by imagery with conflicting visual stimulation. Communications Biology)

Fig.2 Closed-loop experiment

During the experiment, the subjects were instructed to imagine an image of target meaning (human face, landscape, or word). The subjects attempted to display the image of the target meaning on the monitor by visually imagining it.

Fig.3 Results of the closed-loop experiment

A total of 32 images, shown from left to right for one for every two images, were displayed for 250 ms each during each attempt to display the image of the target meaning. The images underlined in red are those that corresponded correctly to the target meaning.

(CC BY, credit: 2022 Ryohei Fukuma et al. Voluntary control of semantic neural representations by imagery with conflicting visual stimulation. Communications Biology)

The article, “Voluntary control of semantic neural representations by imagery with conflicting visual stimulation,” was published in Communications Biology at DOI: https://doi.org/10.1038/s42003-022-03137-x.