Reimagining How We Interact with Generative AI: How might Generative AI move beyond the screen into physical, spatial experience? At the Embodied Computations Lab (led by Dina El-Zanfaly at CMU), I design playful systems that blend sensors, projection, and real-time generation to invite people to co-create with AI through their bodies, voices, and gestures. Projects like Rapid Worldbuilding with GenAI and my Pose-to-Animation Prototype explore how embodied interaction can make emerging technologies more intuitive, performative, and human, ultimately transforming GenAI from a tool of text and image production into a medium for spatial storytelling and discovery.
Rapid Worldbuilding with GenAI combines physical blocks, voice prompts, and real-time image generation to create an intuitive prototyping tool for environmental design. Built with TouchDesigner and Stream Diffusion, the system translates tactile block arrangements into immersive, room-scale projected worlds that update instantly as designers manipulate the physical elements and refine their vision through speech. Inspired by Jesse Benjamin and Joseph Lindley's Shadowplay, this project explores embodied interaction with generative AI for spatial design.
Team: Hayden Lee, Dina El-Zanfaly, Kimberly Blacutt
Pose-to-Animation Prototype: This prototype explores how physical gestures can drive digital animation. It takes real-time body poses as input which captured through your webcam (if you allow it to) and processes via a lightweight pose detection model (Google’s Teachable Machine). The poses are mapped to distinct animated responses. I developed as an early experiment within Embodied Archives (my research with Dina) to investigate how embodied movement can activate archival or generative content. The prototype was also adapted into a Pepper’s Ghost holographic display to test how these motion-triggered animations might be experienced as physical, spatial artifacts.