EyeQ: Exploring Corneal Reflection for Mobile Context Sensing

arXiv preprint

Publication preview for EyeQ: Exploring Corneal Reflection for Mobile Context Sensing

Abstract

What if a single face-facing camera could sense your surroundings—through your eyes? We introduce EyeQ, which explores corneal reflection as a modality for context-aware mobile interaction. Traditional vision-based context sensing relies on world-facing cameras that demand explicit aiming or dual-camera setups; EyeQ instead shows that an off-the-shelf VLM can read scene context from corneal reflections captured with a smartphone camera. To recover usable detail from mobile corneal crops, EyeQ pairs on-device iris-guided capture with server-side AI enhancement, then leverages VLMs to extract scene types, names, and objects from warped, unfamiliar eye-image sequences. In an N=10 study, we characterize how capture conditions shape reflection fidelity, evaluate what VLMs detect, misclassify, and hallucinate, and demonstrate interaction opportunities, including detecting a steering wheel to distinguish drivers from passengers. This work reframes the eye in HCI—from a target for gaze estimation to a passive mirror of one’s surroundings—and examines the privacy tensions it raises.

Materials

  @inproceedings{lee2026eyeq,
    title = {EyeQ: Exploring Corneal Reflection for Mobile Context Sensing},
    author = {Lee, Jaewook and Amores, Judith and Shin, Donghoon and Choi, Bo-Eun and Kim, Jason and Kyi, Davin Win and Kim, Seok-Young and Froehlich, Jon E. and Wilson, Andrew D.},
    booktitle = {arXiv preprint},
    year = {2026}
  }