Recognition and Description of Diagrammatic Content in Images Using Neural Networks

This PhD project is aimed at diagrammatic content in images of all types, an area of image analysis and image description that has so far not seen the same degree of attention and success as depictions of real-world content (such as people, animals, inanimate objects), and textual content. Diagrammatic image content is defined to a first approximation as line drawings composed of geometrical shapes. The project will look at several subtasks of increasing complexity: (1) locating diagrammatic content in images; (2) breaking down diagrammatic content into component parts; (3) mapping to abstract diagram representations; and (4) generating natural language descriptions of diagrams. In addition to looking at each task separately, we will also look at integrated solutions for two or more of the tasks combined. Neural network methods and hybridised neural network methods will be central to all solutions. Determining what kind of description is appropriate in what kind of context (such as type of user, desired level of abstraction or size of diagram) will be very much part of the research. In summary, the main objectives of the PhD are:

1. to develop neural-network-based methods for generating abstract representations of diagram structure and natural language descriptions of diagrammatic content in images;

2. to create datasets of paired images containing diagrams, abstract representations and descriptions, for developing, training, and evaluating methods, and for public release as an important data resource for other researchers;

3. to create comprehensive evaluation strategies to facilitate thorough evaluation of the developed methods, both by automatic means and involving BPS participants.