The Applied Semiotics of Visual Modelling

In the digital age, every member of society needs to process, interrogate and understand
complex data. In addition, more and more of us take part in or depend on specialised fields in which
information has to be absolutely precise, such as the design and creation of
software. Graphical forms of communication are widely recognised as an effective way of
communicating certain types of information, whether through a simple data visualisation such as
a bar chart accompanying a newspaper article, or through a complex diagram which specifies a
computer system. Thus, there are many benefits inherent to understanding how these graphical
forms work, and what makes them more or less effective (relative to, say, the written word) for
certain tasks.

Semiotics is the perspective that analyses these graphical notations as so many arrangements of
signs. The field of semiotics began with the work of Charles Peirce, who provided a richly
thorough analysis of the types of signs, their components, the relations of the components to
one another and so on. Despite several initial approaches and numerous differing perspectives
on the same problem, no one has yet applied the semiotic method in depth to the analysis of
diagrammatic languages to make predictions of their cognitive potentials; that is what this
project sets out to do.

We will begin by analysing several widely used data visualisations, including those which are
used interactively, from a Peircean perspective. That is, we will use Peirce's classification
of types of sign and their correlation with different types of information to make predictions
about the potential effectiveness and suitability of the visualisations for particular
tasks. We will correlate these predictions with findings from cognitive science and HCI, where
such visualisations are studied from different points of view. This evaluation will result in
guidelines for the use of existing techniques and generalised criteria for the design of data
visualisations.

The next step in our project is to use the frame of reference explained above to analyse the
logic of concept diagrams. Concept diagrams were designed for knowledge representation and have
been found to be more effective than symbolic alternatives for some tasks. A diagrammatic logic
is more than data visualisation; it is a precisely defined graphical notation (e.g. Venn
diagrams or concept diagrams), together with a set of rules for transforming and possibly
combining diagrams in logically valid ways. Although they are not a panacea, when diagrammatic
logics are used in place of "traditional" symbolic alternatives, certain advantages can arise:
in some circumstances a diagram effectively reveals more information than was required for its
construction. This is a key source of the power of diagrammatic communication, whereby
information is revealed "by construction" that would need to be inferred as a result of some
chain of reasoning if the information were presented in a symbolic language. The means by which
this takes place are a central part of Peircean semiotics, and are also of great interest to
disciplines outside semiotics, such as human-computer interaction and cognitive science. One of
our contributions will be our attempt to reconcile and find agreements or inconsistencies among
these different accounts of the "natural" or "intuitive" power of diagrams and data
visualisation. In this way, we aim to enable people to make better use of existing notations
and design more effective notations for the future.

We will describe the intended impacts of the project in terms of the EPSRC indicators
Knowledge, Economy, Society and People. By producing a new
understanding of wherein lies the effectiveness of formal diagrammatic languages, the research
will deliver enhanced knowledge of the potential of diagrammatic representation and
reasoning in modelling processes such as ontology engineering and software modelling. This will
be available to the academic community as a short-term benefit (0-2 years), and to
beneficiaries such as ontology engineers and software modellers as a more medium (2-5 years)
and long-term benefit. In turn, this stands to benefit the economy by leading to
better understanding of the systems and data being modelled and to economic gains as a long
term benefit. The main way in which this research can benefit society is by having the
potential to enable practitioners to understand data visualisation in new ways, and thus to use
visualisations more effectively. In the 21st century the participation of
individuals in civic life increasingly depends on their ability to take in and understand large
amounts of information presented as data; our close analysis of graphical languages thus has
the potential to provide medium and long-term benefits to society and its people.