Deep Learning Vision Models Reveal Neural Dynamics of Digital Interface Interaction: A Design Neuroscience Approach

Abstract

Digital interface design profoundly impacts user experience, yet the underlying neural mechanisms governing how users perceive and interact with interfaces remain largely uncharted. Traditional user experience (UX) research relies heavily on behavioral metrics and subjective feedback, which often fail to capture the continuous and complex cognitive processes involved. To address this gap, we introduce a novel framework that integrates pre-trained deep learning vision models with simultaneous electroencephalography (EEG) and eye-tracking. We recorded high-density EEG and eye-tracking data from 32 participants as they performed both free-viewing and task-oriented interactions with a diverse set of 20 real-world web and mobile interfaces. By correlating neural activity with visual-semantic features extracted by the Contrastive Language-Image Pre-training (CLIP) model, we reveal a detailed neural map of interface processing. Our findings demonstrate that neural activity, particularly in the gamma frequency band, is significantly correlated with hierarchical features encoded by the CLIP model, reflecting the brain's processing of design elements from basic visual attributes to high-level semantic concepts. Furthermore, these neural patterns are dynamically modulated by the user's attentional focus, as measured by eye-tracking, and shift significantly during transitions between browsing and decision-making phases. These results provide the first direct neural evidence of how the human brain processes complex digital interfaces in naturalistic settings and establish a new, neuro-grounded paradigm for design evaluation. This approach offers a scalable and objective method to deconstruct the user experience, paving the way for neuro-adaptive interfaces and data-driven design optimization.