Researchers at California Institute of Technology studying the ability of the brain to intuitively interpret sound as form could prove beneficial to blinded veterans, enabling them to “see” basic shapes using commercially available technology.
Though the concept, and even the technology, enabling the blind to decode sound as images has been around for two decades, it has long been a given that it takes a great deal of training for someone to learn how to recognize what the sounds are showing them, according to researcher Noelle Stiles, who co-authored a paper on the development with principal investigator Professor Shinsuke Shimojo.
“It is difficult to interpret sound off of a sensory substitution device. It takes effort. But we found that in some cases it is intuitive in a surprising way,” Stiles told DefenseTech. “In everyone, there are connections across the senses. If we tap into that cross modality, it makes it easier to interpret some stimuli with this device.”
Stiles recently published a report on the research with colleague and co-author Prof. Shinsuke Shimojo. Their research is not funded by the VA, which didn’t respond to a request for comment. Tom Zampieri of the Blind Veterans Association said he is unaware of the department putting any funds into the work.
“I also sit on the [Defense Department’s] Telemedicine & Technology Research Center peer-reviewed DoD Vision Trauma Research Programs funding and know that of the 32 grants [we reviewed] last May, none included this kind of research,” Zampieri said. “So while I would hope both DoD and VA research would be investing in neurosensory vision research in technology advances, it seems to be mostly done outside of the system.”
The equipment used by Stiles is all off-the-shelf and includes a portable computer that runs Windows software, commercially available “spyglasses” with a miniature camera able to run a live feed to the computer, and headphones.
The live feed allows her to follow the process on screen — as the software converts pixels into sound, with brightness and location converted to pitch and volume.
The computer software — available for free download — translates images to sound. The sounds are sent back to the user through the headphones, she said.
The idea and even the technology to enable some sightless people to “see” using sound has been around for years, but an apparent breakthrough at CalTech indicates that to some extent the brain’s ability to interpret sound as form may be intuitive.
Peter Meijer developed the technology, called The vOICe — with OIC meaning “Oh, I see” — about 20 years ago to technically preserve mostly pictorial information that could be mentally decoded by the blind through sound.
“But this design goal does not necessarily yield intuitive results,” he said. “The main thing emerging from the recent work of Noelle Stiles and Shinsuke Shimojo is that at least for relatively simple visual patterns the mapping is intuitive, matching natural expectations of the brain that exist even before people start training with The vOICe.”
Stiles said there remains much research and work to do before the technology is at a stage and the mapping capabilities of the brain understood enough for a blind person to throw on a pair of glasses and headphones and see complicated images.
But in limited, controlled settings a person could — without a great deal of training — learn to recognize items such as key, door or other objects.
–Bryant Jordan can be reached at bryant.jordan@military.com. Follow him on Twitter at @bryantjordan.
