Cognitive abstractions for visual communication

Visual communication is a fundamental cornerstone of many human endeavors. The most ornate and staggering of buildings begin as strokes on paper shared between architects, Nobel prize winning scientific advances begin as graphs shared among lab members, and the most ineffable of emotions can be communicated with a single patch of color. These examples speak to the ability of the human mind to abstract relevant features from the naturalistic world in service of pithily creating new visual artifacts that convey complex ideas while raising the central question animating this dissertation - How does the mind abstract from prior experiences to produce and understand visual artifacts – sketches, charts, maps, and diagrams? Through a series of experiments organized around two domains — color semantics and drawing — the reported studies investigate the nature of the mental representations that support such visual abstraction. Evidence shows that these representations are best characterized by cross-modal models that capture the rich patterns of covariation not only in the visual world but between vision and natural language. We also show that the manner in which humans deploy these representations for effective visual communication are context sensitive, requiring computational cognitive models to ‘decode’ behavior from these representations. Chapter 1 presents a characterization of the conceptual space that underlies people’s color-concept associations and shows that it is distinct from both connotative and language-based conceptual spaces. Chapter 2 presents a theory of how people deploy their color concept associations using models of assignment inference to interpret the meanings of colors in visualizations and what the limits on the ability to do so are. Chapter 3 investigates how deep neural network models of vision can be leveraged in addition to cognitively circumscribed theories to explain how people represent similarities between abstract drawings of real-world objects. Lastly, chapter 4 describes how large-scale drawing production and recognition studies can both help unpack the contents of human visual semantic knowledge and also critically evaluate AI systems in their ability to perform visual abstraction.