How to (re)represent it?

Lead Research Organisation: University of Cambridge
Department Name: Computer Science and Technology

Abstract

AI engines are ubiquitous in our lives: we talk to our mobile phones, ask directions from our sat-navs, and learn new facts and skills with our digital personal assistants. But most of the time, we need to learn how these systems work first: we have to adapt to them as they are not aware of our level of experience, expertise and preferences. AI engines are fast and can deal with a deluge of data much better than us, but they do so in machine-oriented ways, which are often inaccessible and unintelligible to humans.

The aim of this project is to identify and study how humans \textit{represent} information that they want to work with and from which they will obtain new knowledge. Humans have the capability to choose the representation that is just right for them to enable them to solve a new problem, and moreover, if the representation needs to be changed, they can spot this and change it. Unlike humans, machines in general have fixed representations and do not have the understanding of the user. For example, sat-nav systems will only give directions with elementary spatial commands or route planning functions, whereas humans give directions in many forms, for instance in terms of landmarks or other geographic features that are based on shared knowledge.

We want to model in computational systems this inherently human ability to choose or change appropriate representations, and make machines do the same. We want to find out what are the cognitive processes that humans use to select representations, what criteria they use to choose them, and how we can model this ability on machines. Our hypothesis is that when humans choose a representation of a problem, they use cognitive and formal properties of the problem and its representation to make their choice. In this project, we will test this hypothesis by achieving the following goals:

1. Collect a corpus of problems and candidate representations to study and categorise their cognitive and formal properties.

2. Devise coding schemes and conduct cognitive studies to identify cognitive and formal properties that people use in choosing representations. Develop cognitive theories based on these experiments.

3. Design and implement computational algorithms that allow users to choose alternative representations. Build a ranking and recommendation system based on the taxonomy from cognitive studies to suggest appropriate representation given a particular problem and user.

4. Evaluate the utility of the system and generalise the approach to other domains outside of mathematics. Investigate how to apply our cognitive and computational models in education in the form of AI tutors that are adaptable to users.

Our work is novel in that it will address the problem of appropriate representation choice. Moreover, we will build novel cognitive theories and computational models that will allow AI systems to operate in more human-like ways and adapt to the requirements of the problem and the needs of the user. Thus, the potential impact will span numerous domains where systems interact with humans to represent information and use it for extracting new knowledge.

Planned Impact

"How to (re)represent it?" is an ambitious project where we want to find out how humans choose and also change representation during problem solving. We want to develop engines that will give machines the same capability. Such machines will accrue many of the benefits that humans obtain from changing representation, including: more effective communication with a human by selecting a representation that is well-suited to their level of familiarity with the target topic; better understanding of a human by identifying and adopting the representation that the human favours; and greater flexibility in adapting to the needs of the user. As such, the project will have economic, societal and knowledge impact.

The impact of this work will be wide as AI systems are becoming ubiquitous. When interacting and exchanging information with humans, developers of such products need to represent information in human understandable ways. We will devise techniques that will help developers of AI systems to build products that choose representations appropriate for their users -- and thus have potential economic benefit to them.

Making machines more human accessible will also benefit the society at large, in particular, as it will potentially bring technical tools to those that are less technically versed.

We have already outlined how our work will contribute to the expansion of knowledge in Academic Beneficiaries. The techniques we will develop are novel and will provide a scientific advance in all areas (in academia and in industry) that must represent information for solving problems, reasoning, etc.

Whilst we exposed education as our initial concrete application target, our contributions will be general and could apply across different domains. They will lead to better understanding of representations and their relation to human expertise and preferences. As intelligent tools make part of our every day life, we will all benefit from having more human-like systems that adapt to us, rather than us adapting to them.

To achieve maximum impact, we will follow a comprehensive dissemination strategy. We will publish our results in all relevant communities (e.g., artificial intelligence, cognitive science, computer science, automated reasoning, diagrams, knowledge representation, human-computer interaction, information visualisation) to achieve wide dissemination. We will demonstrate our work by organising a workshop and preparing tutoring material aimed at academic as well as more specific industrial communities. This will also provide a community building opportunity for people interested in representations and more generally, human-like computing. We will continue to participate at outreach activities to raise the awareness of the importance of human-like computing. We will create a web repository to enable free and public access to our papers, corpus of problems and their solutions, software and tutorials. Finally, we have enlisted an advisory board of experts spanning all areas relevant to this project, and coming from academia and industry - their advice will help us stay focused on relevant problems, influence diverse communities that they lead, and transfer our technology widely.
 
Description • Established a taxonomy of formal properties of representational systems relating to their encoding of information.
• Established a taxonomy of cognitive properties of representational systems relating to the cognitive demands they impose on human users.
• Developed algorithms for rating the efficacy alternative representations for particular problems based on formal and cognitive properties.
• Evaluated the performance of algorithms compared to human expert ratings of alternate representations.
• Developed and empirically tested a prototype cognitive profile test focusing on quantity scales.
Exploitation Route When choosing a representation, they can apply our framework to assess what representation is appropriate for their task and their users.
Sectors Digital/Communication/Information Technologies (including Software)

URL http://www.cl.cam.ac.uk/research/rep2rep/
 
Description In this project, we addressed the following questions: • What are the cognitive processes that humans use to select a representation of a problem? • What are the criteria that we can use to assess the effectiveness of a representation? • How do we model this computationally? We distinguish between cognitive and formal properties of a representation, in an approach that radically, but systematically, reconfigures previously descriptive accounts of the nature of representations and notations. We use this to devise methods for measuring competency in alternative representation use, and also to engineer a system to automatically select representations. Cognitive properties characterise cognitive processes demanded of a particular representation (e.g., problem state space characteristics; applicable state space search methods; attention demands of recognition; inference operator complexity). Formal properties characterise the nature of the content of the representation domain (e.g., operation types like associative or commutative, symmetries, coordinate systems, quantity or measurement scales). This project has laid the foundations for understanding formal and cognitive properties that affect the choice of representation in problem solving. Our prototype implementations of algorithms that carry out this analysis show that it is possible to model these processes computationally. We now have the basic theoretical foundations to deploy this programme on a more general and larger scale with respect to problem solving domains.
First Year Of Impact 2018
Sector Education
Impact Types Societal

Policy & public services

 
Description Automating Representation Choice for AI Tools
Amount £755,608 (GBP)
Funding ID EP/T019603/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2020 
End 02/2023
 
Title A novel interaction for competence assessment using micro-behaviors: Extending CACHET to graphs and charts 
Description Data for paper published in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23) These files contain: The data from the Graph Familiarity questionnaire used in our study (competence assessment using micro-behaviours_Demographic data and questionnaire) The interactions produced by our participants on each stimulus (competence assessment using micro-behaviours _interactions on all stimuli) All the pauses produced by our participants on each stimulus (competence assessment using micro-behaviours _Pauses) Paper abstract Competence Assessment by Chunk Hierarchy Evaluation with Transcription-tasks (CACHET) was proposed by Cheng [14]. It analyses micro-behaviors captured during cycles of stimulus view- ing and copying in order to probe chunk structures in memory. This study extends CACHET by applying it to the domain of graphs and charts. Since drawing strategies are diverse, a new interactive stimulus presentation method is introduced: Transcription with In- cremental Presentation of the Stimulus (TIPS). TIPS aims to reduce strategy variations that mask the chunking signal by giving users manual element-by-element control over the display of the stimulus. The potential of TIPS, is shown by the analysis of six participants transcriptions of stimuli of different levels of familiarity and com- plexity that reveal clear signals of chunking. To understand how the chunk size and individual differences drive TIPS measurements, a CPM-GOMS model was constructed to formalize the cognitive process involved in stimulus comprehension and chunk creation. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact Paper published in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23) 
URL https://sussex.figshare.com/articles/dataset/A_novel_interaction_for_competence_assessment_using_mic...
 
Title RST editor 
Description Editor for representation system theory 
Type Of Technology Webtool/Application 
Year Produced 2020 
Open Source License? Yes  
Impact We can construct different representations in the language of Oruga. 
URL http://rep2rep.cl.cam.ac.uk