Zhicheng Liu

Designing and Understanding Visual Sensemaking
 

Syntactic Structure, linguistic theory and visualization theory

I always believed that the study of visualizations and the study of (natural) languages could mutually inform each other, so through examining the development of linguistic theory, we can learn something useful for pushing forward theoretical research on visualization.

Noam Chomsky’s Syntactic Structure is undoubtedly a classic piece of linguistic theory research. The philosophy it embodies was the Zeitgeist of its time: the 1960s was the golden age of the “cognitive revolution”, and a central belief was that cognition and intelligence were to be explained as the manipulation and transformation of symbols. Chomsky’s approach now
is characterized as “generative grammar” in differentiation with the previous dominant approach called “structuralism“. Before structuralism, as the Wikipedia tells us, the linguistic theory was all about how languages originate and change historically.

The development of linguistic theory in terms of paradigm shifts, to me, is not a continuous effort to refine existing theories. Instead it is about the changing notion of what theoretical questions are important, or simply, theories of what? Historical linguistics is undoubtedly concerned with the theoretical question of how do natural languages evolve? Structuralists such as Saussure seemed less concerned with these questions. He believed that language may be analyzed as a formal system of differential elements such as phonemes, morphemes, lexical categories, noun phrases, verb phrases, and sentence types.
Chomsky’s approach builds upon structuralism but he is more interested in a theory that explains the generative mechanisms of language. Structuralist approach, to him, is inadequate because the theory cannot produce the full range of diverse sentences in English. Chomsky’s theory, however, focuses exclusively on syntax and dismisses the problem of meaning. As a response, the emerging cognitive semantics becomes more interested in theoretical questions about semantics: what is meaning? how does meaning arise?

Visualization theory research is much less advanced than linguistic theory research, but I seem to notice some similar patterns in the development of visualization theory. The pinnacle of visualization theory research so far is represented in Bertin’s Semiology of Graphics, Mackinlay’s APT framework and Wilkinson’s Grammar of Graphics. Bertin’s work, to me, is inherently structuralist. The APT framework builds upon Bertin’s structuralist work and incorporates generative mechanisms. These two apporaches, however, do take into the considerations of the meaning and expressiveness of visualizations. Historical analysis and cognitive analysis in visualization theory have been lacking so far (and our lab’s work on Distributed Cognition and mental models may be characterized as cognitive).

I do not believe, however, that more recent theoretical paradigms will always be better than outdated paradigms – there is simply no such thing as “absolute good or better”. These paradigms are concerned with different theoretical questions, which may be deemed to be important or not important in different times. From a utilitarian perspective, the structuralist and generative paradigms in linguistics do seem to be the most useful approaches: they can be directly applied to build intelligent machines that can automatically produce syntactically correct sentences. Similarly, we have seen how the APT framework and the Grammar of Graphics lead to sophisticated visual analytics systems like Tableau.

2 Responses to “Syntactic Structure, linguistic theory and visualization theory”

  1. luc says:

    In Semiology of Graphics, Bertin doesn’t analyze the actual organisation of a visualization language, but try to define how to make visualization efficient.

    This is a big difference with language analyses. There is no good way to call a sheep. “sheep”, “mouton” etc. are neither better or worst. But using a size variation to mean a more important phenomena is better than using hue variation. The signifier is here dependant from the signified (a size variation can’t mean a quality difference between two objects).

    Bertin also use other kind of signifier in Semiology of Graphics. These are symbols. Symbols are closer to a language and don’t need to refer to the real world to mean something (as is language according to structuralist). Meaning can be reach through external cultural references (for example, a triangle is more used to mean an industry than a town in a map). Meaning can also be here reached through caption (so natural language references).

    Symbols and the visual elements of the first kind are not necessary opposed. An arrow is both a symbol and a natural way to lead the eye in the direction. It’s actually interesting to see the evolution of the arrows that Bertin draw. The first (that are not considered as efficient) look like the things launched from a bow (it has some kind of wings on it), and the last are both more abstract, symbolic and refer better to a natural eye movement.

    So as far as I understand these works, there are two structural theories in Bertin : One is about symbols and cultural references. This one can be independant from real world references (signifier is independant from signified), is closer to a real language. The other is what Bertin is trying to create : a structure of signifier dependendant from signified. This one really make Bertin work interesting and original.

    They are both structuralist in the way for both, meanings is reached from the variation between a limited number of elements inside a structure. They can be used in a complementary way in a visualization. Map especially use often both.

    Of course, we can discuss if the signifier of the first kind are absolutely better used with some kind of signified. It’s not obvious from people who create structures in visualization since they often make “semiology” mistakes when creating visualization. It’s neither not sure that in every civilisation, these relations between signifier and signified will be the same or as efficient.

  2. Hans-Georg Fill says:

    Thanks for the interesting blog entry. I had some similar thought in this direction when working on aspects of legal visualizations. However, these kinds of visualizations are rather more “knowledge” oriented than “data” oriented. maybe you want to take a look at these thoughts: http://homepage.dke.univie.ac.at/fill/papers/Fill_Polysyntactic_View.pdf

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.

 
 
Powered by Wordpress
Copyright © 2006 - 2013 Zhicheng Liu