Hold That Thought: Neuro-Symbolic Coupling for Shaping Ideas in Real Time

Chuanqi Sun
BA, Computer Science, Dartmouth College, 2015
[email protected]

Thesis proposal for the degree of
Master of Science in Media Arts and Sciences
at the
Massachusetts Institute of Technology

Advisor:
Hiroshi Ishii
Jerome B. Wiesner Professor of Media Arts and Sciences
Media Lab, Massachusetts Institute of Technology

Reader:
Zach Lieberman
Adjunct Associate Professor of Media Arts and Sciences
Media Lab, Massachusetts Institute of Technology

Reader:
Jean-Baptiste Labrune
Research Affiliate
Media Lab, Massachusetts Institute of Technology

Abstract

Historically, real-time interaction has led to breakthroughs in HCI, but its application has remained limited to data with direct spatial mappings or metaphors. Conversely, generative AI works with semantic data while lacking the structure and speed for real-time manipulation. Today, generative models are approaching interactive frame rates and sub-second response, yet human-AI interaction still relies on turn-based prompt-and-response loops. Hold That Thought addresses this gap through neuro-symbolic coupling of structured symbolic affordances with unstructured semantic data, materializing ideas as tangible handles that users can manipulate in real time. The work builds on foundational theories of direct manipulation, cybernetics, and tangible media, and on Tim Ingold's account of making as a continuous response to material, to extend the coupling of physical and digital bits to spatial and semantic media. Two completed systems, Dream Brush and Parametric Diffusion, provide the foundation. I will build Braindump, Put That There Again, Context Pucks, and Sable to cover getting ideas in, manipulating them, and holding them physically, and contribute interaction loops to Tangible Co-ideation, a multi-user, multi-AI collaboration platform at the Tangible Media Group. By evaluating the systems against conversational baselines using the Creativity Support Index (CSI) and complementary measures of agency, ownership, and joy, I test the hypothesis that neuro-symbolic coupling preserves human agency, enhances creativity, and sparks joy in human-AI collaboration.

Introduction

Nearly every major step in human-computer interaction has come with a new real-time interaction paradigm. Sutherland's Sketchpad let a person draw and constrain geometry with a light pen and see the result immediately [1]. Engelbart's 1968 demonstration showed people manipulating text, links, and views in a shared, live workspace [2]. Shneiderman named the pattern direct manipulation, built on "rapid, incremental, reversible operations whose impact on the object of interest is immediately visible" [3], and Hutchins, Hollan, and Norman explained why it works: it shrinks the distance between intent and action [4]. The Tangible Media Group carried the idea into the physical world with Tangible Bits and its successors [5].

But direct manipulation has been limited to data that has a direct spatial mapping (pixels, terrain, physical simulation) or a strong spatial metaphor (files, windows, sliders). It requires an object with a designed surrogate, and ideas, concepts, tone, and intent have none. AI took the other path. From ELIZA to ChatGPT it has accepted semantic input and produced semantic output in alternating turns. The costs of that shape are now documented: users face a gulf of envisioning between an intention and a prompt that anticipates what the model can do, and, unlike direct manipulation, model feedback is not immediate [6].

Generative models are now approaching real-time response in image, video, and speech [7], [8], so the turn-based shape is no longer forced by the technology. When latency falls below the threshold of perception, a series of requests becomes a coupled loop, as in the feedback between a camera and its own monitor [9]. This thesis assumes real-time generative AI will be available soon, and asks what to build for it.

Speed alone is not enough, because a faster chat is still chat. A thought expressed in language is not something a hand can hold. The second argument for real time comes from the theory of making. Ingold contrasts a hylomorphic model of creation, in which a maker imposes a preconceived form on passive matter, with the practice of the craftsperson, who works with the material, following its flows and responding to what it does as the work unfolds [10]. Turn-based prompting is hylomorphic: the user specifies the form up front and the model imposes it. Creativity in Ingold's sense needs a medium that answers back continuously, which is what real time provides and a request-response loop cannot.

My answer is to give thoughts handles. To hold a thought is to bind it to a discrete, inspectable object that a hand can manipulate. The handles are neuro-symbolic in a practical, interface-level sense: a neural model supplies fluid semantic content, and a symbolic layer (parameters, structures, programs, tokens) supplies the handle [11]. Handles make shaping possible, because an idea is fleshed out progressively by manipulating it instead of specifying it up front, and real time keeps the loop closed. This gives the thesis its central hypothesis:

Neuro-symbolic coupling of semantic and spatial control, with direct input and immediate output, will preserve human agency, enhance creativity, and spark joy in human-AI collaboration, compared with turn-based prompting.

Contribution

  1. Two completed systems. Dream Brush couples the semantics and spatial extent of calligraphic strokes to steer image generation [12]. Parametric Diffusion exposes symbolic structure as real-time handles over a diffusion model.
  2. Four proposed systems and one platform contribution, covering getting ideas in, manipulating them, holding them physically, and sharing them: Braindump (voice and gesture input), Put That There Again (spatial-semantic manipulation in AR), Context Pucks (physical handles with adaptive affordances), Sable (tangible direct manipulation of audio synthesis), and interaction loops for Tangible Co-ideation Across Time.
  3. A framework for neuro-symbolic coupling, extending the Tangible Media Group's coupling of physical and digital bits to spatial and semantic media, with an operational definition of real time.
  4. A catalogue of semantic manipulations ("verbs" such as steer, blend, scale, bind) that generalize across domains.
  5. Empirical evidence and design recommendations on how real-time, handle-based interaction affects creativity, agency, and joy relative to turn-based prompting.

Proposed Work

System Overview

The portfolio is organized around the idea lifecycle: getting an idea into the system, manipulating it, holding it physically, and sharing it.

System Role in the framework Medium Status
Dream Brush Spatial and semantic control coupled in one gesture Calligraphic canvas, diffusion Done (DIS '26 Companion [12])
Parametric Diffusion Symbolic structure as handles over a generative image model Real-time interface to diffusion Done
Braindump Input: getting ideas into the system Voice and gesture Started; ETA end of 2026
Put That There Again Manipulation: semantic operations on ideas CV and AR, tangible handles Prototyped; ETA Jan. 2027
Context Pucks Physical handles with adaptive affordances Tangible device Designing; ETA Mar. 2027
Sable Tangible direct manipulation of creative audio synthesis Wavetable synthesizer, tangible terrain Proposed; starts Jan. 2027
Tangible Co-ideation Across Time Social: multi-user, multi-AI ideation TMG platform In development (ongoing TMG project)

Completed Work

Dream Brush is a near-real-time canvas on which people steer image generation and editing with calligraphic marks. A vision model reads the written character, and the bounding box of the stroke selects the region to repaint. It couples semantics (the character) with spatial control (position, size, orientation) in one gesture [12]. Generation still takes seconds, and a "drying" animation masks the wait, so the system is also an honest measure of the gap between near-real-time and real-time.

Parametric Diffusion is a real-time interface for controlled image generation with direct manipulation of an explicit symbolic structure over a diffusion model, preserving user control and diversity in the output. It was built and shown as an art project, so no user feedback exists yet, and I make no claims about its effect on users here.

Research Questions

  1. What are the generalizable semantic manipulations of ideas? Which operations (steer, blend, scale, bind, pin, branch, merge) carry across domains, and how is each materialized as a handle?
  2. What does real time change? Compared with turn-based prompting, how does direct input with immediate output affect creativity, agency, ownership, and enjoyment, and how do these depend on latency?
  3. How should ideas get into the system? What multimodal input lets a person instantiate rich context quickly, so that input does not bottleneck manipulation?
  4. Which tangible handles best support semantic manipulation? What makes a puck a better handle than a slider or a mouse?
  5. What happens socially? How do communication and collaboration change when every participant is coupled to AI in real time?

Architecture

Each system shares a common loop: (1) instantiate, in which an idea enters as unstructured context (speech, gesture, sketch, text); (2) reify, in which a neural model proposes symbolic handles, typed parameters bound to semantic dimensions of the idea (for example, a scale from "academic" to "poetic"); (3) manipulate, in which the user grabs handles through spatial, tangible, or gestural input and drives a fast generative loop with immediate feedback; and (4) inspect and refine, in which handles can be rebound, merged, or discarded, and the agent can propose new ones. Only the fast loop (3) must be real time. The slower loops (1–2, 4) parallel the inner loop and tool-editing path in malleable software [35].

I define real time operationally as at least 24 frames per second of visible output with no more than 100 ms from input to perceptible response [16], treating the bound as a minimum given the stricter figures for direct touch [18]. Few current systems satisfy both conditions for every modality. Each system will therefore use the best available models, with a Wizard-of-Oz or simulated-latency layer to test conditions beyond today's capability, and latency is a manipulated variable in the evaluation.

Design Process

The process follows Ishii's vision-driven approach, with a long-term vision driving the work rather than current technology. Each system is built as a sequence of rapid prototypes tested early with small groups of creators, in the creative-coding spirit of daily, playful sketching. Joy is a design criterion as well as an outcome: Dream Brush unexpectedly became a social drawing game, and I will look for similar unplanned uses in the tangible systems.

Evaluation

Hypotheses. Relative to a turn-based prompting baseline, handle-based real-time interaction yields (H1) higher creativity support, (H2) a stronger sense of agency and ownership, and (H3) greater enjoyment. I also hypothesize (H4) that these differences shrink as latency exceeds the real-time bound.

Measures. The primary measure is the Creativity Support Index [54], which scores six factors (exploration, expressiveness, immersion, enjoyment, results worth effort, collaboration) out of 100. Complementary measures are the Sense of Agency Scale [55] as a baseline trait, with task-specific agency and ownership items that I will pilot and report as unvalidated; NASA-TLX for cognitive effort [56]; and interaction logs, think-aloud sessions, and interviews. The qualitative analysis will attend to how participants describe their relationship to the output, as the "manager of a machine" or the "author of a craft," in the sense of Ingold's maker [10].

Studies. All designs are proposals subject to IRB approval and committee feedback.

Limitations. Simulated latency is a weaker substitute for truly real-time systems. CSI was designed for single-tool studies, so cross-system comparison needs careful task matching. Order and novelty effects, particularly with tangible hardware, can inflate enjoyment. I will counterbalance, use well-engineered baselines, and look specifically for where the new interface performs worse.

Timeline

Task / Phase Nov '26 Dec Jan '27 Feb Mar Apr May
Braindump implementation (ETA end of 2026) X X
Put That There Again prototype to study-ready (ETA Jan. 2027) X X X
Context Pucks design and fabrication (ETA Mar. 2027) X X X X
Sable development (starts Jan. 2027) X X X
Tangible Co-ideation contributions (ongoing) X X X X X
IRB protocol and recruitment X X X
Study A X X
Study B X X X
Study C X
Analysis X X X
Thesis writing and publications X X X

Schedule risk. Studies A and B concentrate in March and April. To de-risk, Study B can begin in February with the screen-based and chat conditions, adding the puck condition when the hardware is ready.

Scope. Training new foundation models, large-scale deployment, actuated shape-changing hardware beyond the Context Puck, and networked remote collaboration are out of scope.

References