NeighboAR: Efficient Object Retrieval using Proximity-and Gaze-based Object Grouping with an AR System

Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.

Aleksandar Slavuljica, Kenan Bektaş, Jannis Strecker, and Simon Mayer. 2024. NeighboAR: Efficient Object Retrieval using Proximity- and Gaze-based Object Grouping with an AR System. Proc. ACM Hum.-Comput. Interact. 8, ETRA, Article 225 (May 2024), 19 pages. https://doi.org/10.1145/3655599

@inproceedings{slavuljica2024,
author = {Slavuljica, Aleksandar and Bekta\c{s}, Kenan and Strecker, Jannis and Mayer, Simon},
title = {NeighboAR: Efficient Object Retrieval using Proximity-and Gaze-based Object Grouping with an AR System},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3655599},
doi = {10.1145/3655599},
abstract = {Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.},
booktitle = {Proc. ACM Hum.-Comput. Interact.},
volume = {8}
issue = {ETRA},
articleno = {225},
numpages = {19},
keywords = {augmented reality, cognitive offloading, eye tracking, object detection,human augmentation, mixed reality, working memory, visual search},
}