Abstract

While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually-defined environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, a mobile AI-assisted AR authoring system that aims to let anyone build anything, anywhere -- simply by speaking their imagination. ImaginateAR is powered by custom pipelines for offline scene understanding, fast 3D asset generation, and LLM-driven speech interaction. Users might say "a dragon enjoying a campfire" (P7) and iteratively refine the scene using both AI and manual tools. Our technical evaluation shows that ImaginateAR produces more accurate outdoor scene graphs and generates 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI in authoring, what and how users create in free-form use, and design implications for future AR authoring tools.

  @inproceedings{lee2025imaginatear,
    title = {ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality},
    author = {Lee, Jaewook and Aleotti, Filippo and Mazala, Diego and Garcia-Hernando, Guillermo and Vicente, Sara and Johnston, Oliver James and Kraus-Liang, Isabel and Powierza, Jakub and Shin, Donghoon and Froehlich, Jon E. and Brostow, Gabriel and Brummelen, Jessica Van},
    booktitle = {arXiv preprint, 2025},
    year = {2025}
  }