Zhicheng Liu

Designing and Understanding Visual Sensemaking

Replies to a fictitious skeptic

Q: All these terms, mental models, schemas, coordination, etc. they are pretty fuzzy and not definite. I believe that to understand how InfoVis works we must understand how the brain works.

A: First, understanding a phenomenon often involves multiple levels of description and explanation (e.g. the famous AI researcher David Marr proposed three levels of analysis for information processing problems). I do not mean to completely dismiss the relevance of neuroscience for InfoVis (I seemed to do so in a previous post). Rather I just believe that a conceptual level description about the nature of representation and interaction in InfoVis can provide a more direct and relevant account. Ultimately internal representations have to be grounded at the neuro-physiological level, but neurological level explanations need to be constrained by conceptual and behavioral evidence too.

Second, what is meant by “how the brain works”? The undertone of this phrase is that the brain can be understood as a machine in isolation from the body and from the environment. This view is being increasingly questioned and criticized. Of course one can argue this is just a matter of opinion and assumption. I believe in a holistic approach of understanding InfoVis processes enabled by brains coupled with bodies embedded in a socio-cultural environment mediated by language, visual images and cultural practices.

Q: Theories of mental models, distributed cognition etc. have been around for a while, I saw no direct application or immediately usable results. Why do we care?

A: It might just be that these concepts or theories are not fully mature yet, or maybe these concepts can only be useful when combined with domain-specific investigations. We the InfoVis researchers can appreciate an incremental result in an interaction technique or a layout algorithm, but we do not have patience for the potentially slow development of theories. I believe we should play an active role in further developing existing theories that sound promising and relevant to InfoVis. We seem to be comfortable enough in considering InfoVis as an applied area and leaving theoretical questions to psychologists and cognitive science, but as Newell and Card nicely put, “nothing drives basic science better than a good applied problem”.

Q: What about perception? Isn’t it the most important aspect of InfoVis?

A: Perception is important, but perception is not a process where we take a snapshot of the visualization and then decode every bit of information inside. Empirical studies have shown how astonishingly little information we actually attend to and are aware of (e.g. the hidden gorilla). Perception, to me, is the active sampling and foraging of information, and hence must be understood in the context of dynamic, continuous loop of interaction.

There are of course qualitative aspects of visual sensation to be understood. Understanding how we perceive color difference or size difference, for example, is helpful, but when users are not certain if two colors are really that different, they often can just mouse over to get the exact values represented by the colors and do a comparison. Even in InfoVis, the foraging of information often is not purely achieved through non-textual visual means. Hence perception must also be understood in conjunction with action so that we know how they complement each other.

[Have better questions? Don't agree with the replies? Let me know!]

5 Responses to “Replies to a fictitious skeptic”

  1. Chris Parnin says:

    There are some neat things we understand about the brain that can help our thinking. But there are few outside of neuroscience that have sufficient understanding to even begin to apply it toward visualization. A common misunderstanding about neuroscience is that it would only help out with “bottom-up” processes, but there are many top-down aspects that are interesting:
    - Neuronal tuning during attention will sharply bias some signals and ignore others. Think of it like smart-sensors: for example, the ear can be tuned to focus in a particular frequency by stiffing some of the sensory hairs.
    - Semantic color perception. A picture of a gray leaf will often look green. This is a example of top-down processing that will override our bottom-up processing of information.

    What I want out of a theory of visualization is a kinda “frames-per-second” metric that we use in graphics, but instead a “information-per-second” that I can gain with a particular visualization/environment.

    • zcliu says:

      Thanks for the comment Chris. Very interesting. I still think the fact that ear can be tuned to focus on a particular frequency is the behavioral level analysis, which is grounded in neuronal tuning at the neurological level. Both behavioral level as well as neurological level analyses can account for both bottom-up and top-down processes. Is this fair? Also you care to elaborate on “information-per-second”?

      • Chris Parnin says:

        “Behavioral” is a weird word for me. Because in one sense it can describe my actions by taking into account my goals (which I’m aware of). But it can also be based on my learned associations or hard-wired processes (which I’m not aware of).

        I think something like information-per-second would be at the lowest level of any model. But if I really wanted to take advantage of more “brain-based” constraints, then I would want to know the sort of things imposed by perceptual, attentional constraints, etc that would cap my capacity to comprehend a scene/visualization. Though I would suspect other processes (memory,learned schema,etc) would start to kick in and become more important.

        • zcliu says:

          Doesn’t the same problem exist for the term “neurological”? :) How do we differentiate neuron activities that are results of deliberation and those that are “operant-conditioned”?

  2. [...] Leo is a Ph.D. student at GeorgiaTech and a very versatile researcher. On his blog, he covers topics like cognition, model-building in visualization, storytelling, etc. He does not write very much (the last posting as of this writing is from six months ago), but what he writes is very interesting. I also like the fact that he experiments with different posting formats, like in his Replies to a fictitious skeptic. [...]

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.

Powered by Wordpress
Copyright © 2006 - 2013 Zhicheng Liu