hidden thumbnail

Sonar - Web Auditory Map Generation

A sensory substitution interface using canny edge detection for BLV individuals' mouse usage

The experiment participant is looking around with the low-vision simulator on

Providing cues for mouse navigation

Sonar is a project aimed at assisting low-vision users' web navigation through sound. It provides a guidance on component location and size. With canny edge detection, Sonar traces out the contour of the web components. Followingly, it creates a sound map of the webpage by matching current cursor coordinates with the light intensity value of the contour.

The researcher is guiding the participant
The participant is testing the Sonar using the disability test kit

Therefore as the low-vision user slides the mouse over the item, Sonar conveys the presence and the size of the web components. Additionally, the user can grasp information about the relative distance between each element. The primary target of Sonar is a people with a low degree of visual impairment, capable of using the mouse. The system substitutes the blind spot within the navigation of the webpage, lowering the need for sequentially iterating over the web components [1].


Canny Edge Detection Pipeline

The process initiates with the canny edge detection deriving the intensity gradient given an image. Through thresholding, it efficiently traces out the border of the elements within the image at a low computing cost, making real-time processing possible. Also to enhance the usability, our team implemented a filtering process, smoothing the light intensity change to adjust the alert volume gradually.

An image depicting how the web component transforms into an auditory map using canny edge detection and a Gaussian blur

A Usability Test

Within the project, we conducted a usability test with 5 participants using a visual impairment simulator. The test was designed to comparatively analyze the task accomplish speed with five criteria. Each procedure was pre-specified with guidelines, including an introduction and guidance by the researcher. While the statistical significance was not measured we observed an increase in completion speed compared to the baseline condition.

Usability testing result for 5 participants. While most participants perceived the task as challenging, they also acknowledged that the interface was helping in conducting the task

The main technical challenge within the project was implementing real-time image processing of the user screen. Due to the Chrome security policy, capturing a video stream of the webpage in real time is inhibited by the design. Therefore, our team created an interval page capture to circumvent the guidelines.


Presentation at HCIK

Sonar was awarded the best paper at the HCI Korea Conference 2022.



Reference

  • [1] Borodin, Yevgen, et al. "More than meets the eye: a survey of screen-reader browsing strategies." Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). 2010.