SOCRAR: Semantic OCR through Augmented Reality

To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.

Jannis Strecker, Kimberly García, Kenan Bektaş, Simon Mayer, and Ganesh Ramanathan. 2022. SOCRAR: Semantic OCR through Augmented Reality. In Proceedings of the 12th International Conference on the Internet of Things (IoT ’22), November 7–10, 2022, Delft, Netherlands. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3567445.3567453

@inproceedings{10.1145/3567445.3567453,
author = {Strecker, Jannis and Garc\\'{\\i}a, Kimberly and Bekta\\c{s}, Kenan and Mayer, Simon and Ramanathan, Ganesh},
title = {SOCRAR: Semantic OCR through Augmented Reality}, 
year = {2023}, 
isbn = {9781450396653}, 
publisher = {Association for Computing Machinery}, 
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3567445.3567453},
doi = {10.1145/3567445.3567453},
abstract = {To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.},
booktitle = {Proceedings of the 12th International Conference on the Internet of Things},
pages = {25–32},
numpages = {8},
keywords = {Augmented Reality, Knowledge Graph, Optical Character Recognition, Ubiquitous Computing, Web of Things},
location = {Delft, Netherlands},
series = {IoT '22}
}