# Semantic Information Pursuit for Multimodal Data Analysis

Lead Research Organisation:
University College London

Department Name: Computer Science

### Abstract

In 1948, Shannon published his famous paper "A Mathematical Theory of Communication" [88], which laid the foundations of information theory and led to a revolution in communication technologies. Shannon's fundamental contribution was to provide a precise way by which information could be represented,

quantified and transmitted. Critical to Shannon's ideas was the notion that the content of a message is irrelevant to its transmission, since any signal can be represented in terms of bits.

However, Shannon's theory has some limitations. In 1953, Weaver argued that there are three levels

of communication problems: the technical problem "How accurately can the symbols of

communication be transmitted?", the semantic problem "How precisely do the transmitted symbols

convey the desired meaning?", and the effectiveness problem "How effectively does the received

meaning affect conduct in the desired way?" Hence, a key limitation of Shannon's theory is that it

is limited to the technical problem.

This was also pointed out by Bar-Hillel and Carnap in 1953, who argued that "The Mathematical Theory of Communication, often referred to also as Theory (of Transmission) of Information, as practised nowadays, is not interested in the content of the symbols whose information it measures. The measures, as defined, for instance, by Shannon, have nothing to do with what these symbols symbolise, but only with the frequency of their occurrence." While Bar-Hillel and Carnap argued that "the fundamental concepts of the theory of semantic information can be defined in a straightforward way on the basis of the theory of inductive probability", their work was based primarily on logic rules that were applicable to a very restricted class of

signals (e.g. text). In the last 60 years there has been extraordinary progress in information theory,

signal, image and video processing, statistics, machine learning and optimization, which have led

to dramatic improvements in speech recognition, machine translation, and computer vision technologies.

However, the fundamental question of how to represent, quantify and transmit semantic is what this programme of research shall address.

quantified and transmitted. Critical to Shannon's ideas was the notion that the content of a message is irrelevant to its transmission, since any signal can be represented in terms of bits.

However, Shannon's theory has some limitations. In 1953, Weaver argued that there are three levels

of communication problems: the technical problem "How accurately can the symbols of

communication be transmitted?", the semantic problem "How precisely do the transmitted symbols

convey the desired meaning?", and the effectiveness problem "How effectively does the received

meaning affect conduct in the desired way?" Hence, a key limitation of Shannon's theory is that it

is limited to the technical problem.

This was also pointed out by Bar-Hillel and Carnap in 1953, who argued that "The Mathematical Theory of Communication, often referred to also as Theory (of Transmission) of Information, as practised nowadays, is not interested in the content of the symbols whose information it measures. The measures, as defined, for instance, by Shannon, have nothing to do with what these symbols symbolise, but only with the frequency of their occurrence." While Bar-Hillel and Carnap argued that "the fundamental concepts of the theory of semantic information can be defined in a straightforward way on the basis of the theory of inductive probability", their work was based primarily on logic rules that were applicable to a very restricted class of

signals (e.g. text). In the last 60 years there has been extraordinary progress in information theory,

signal, image and video processing, statistics, machine learning and optimization, which have led

to dramatic improvements in speech recognition, machine translation, and computer vision technologies.

However, the fundamental question of how to represent, quantify and transmit semantic is what this programme of research shall address.

### Publications

Bulathwela S
(2020)

*SUM'20: State-based User Modelling*
Donini M
(2019)

*Combining heterogeneous data sources for neuroimaging based diagnosis: re-weighting and selecting what is important.*in NeuroImage
Grant E
(2018)

*Hierarchical quantum classifiers*in npj Quantum Information
Guedj B
(2020)

*Kernel-Based Ensemble Learning in Python*in Information
Rocchetto A
(2018)

*Learning hard quantum distributions with variational autoencoders*in npj Quantum Information
Tse, L
(2018)

*Graph Cut Segmentation Methods Revisited with a Quantum Algorithm*in IJQIDescription | Key finding 1: Adversarial learning is one of the most successful approaches to modelling high-dimensional probability distributions from data. The quantum computing community has recently begun to generalise this idea and to look for potential applications. We derived an adversarial algorithm for the problem of approximating an unknown quantum state. Although this could be done on universal (large) quantum computers, the adversarial formulation enables us to execute the algorithm on near-term quantum computers. Key finding 2: Quantum mechanics fundamentally forbids deterministic discrimination of quantum states and processes. However, the ability to optimally distinguish various classes of quantum data is an important primitive in quantum information science. We trained quantum circuits to distinguish quantum states and provided an example of machine learning in the quantum setting for a task that has inherently no classical analogue. Key finding 3: Quantum circuits with hierarchical structure have been used to perform binary classification of classical data encoded in a quantum state. We demonstrated that more expressive circuits in the same family achieve better accuracy and can be used to classify highly correlated quantum states, for which there is no known efficient classical method. Key finding 4: We showed that DNF formulae can be quantum PAC-learned in polynomial time under product distributions using a quantum example oracle. The best classical algorithm (without access to membership queries) runs in superpolynomial time. Key finding 5: The number of parameters describing a quantum state is well known to grow exponentially with the number of particles. This scaling clearly limits our ability to do tomography to systems with no more than a few qubits and has been used to argue against the universal validity of quantum mechanics itself. However, from a computational learning theory perspective, it can be shown that, in a probabilistic setting, quantum states can be approximately learned using only a linear number of measurements. We experimentally demonstrate this linear scaling in optical systems with up to 6 qubits. |

Exploitation Route | Further research. |

Sectors | Digital/Communication/Information Technologies (including Software) |

Description | Graph Parameters and Physical Correlations: from Shannon to Connes, via LovĂˇsz and Tsirelson |

Form Of Engagement Activity | A magazine, newsletter or online publication |

Part Of Official Scheme? | No |

Geographic Reach | International |

Primary Audience | Industry/Business |

Results and Impact | Article in ERCIM news |

Year(s) Of Engagement Activity | 2018 |

URL | https://ercim-news.ercim.eu/en112/special/graph-parameters-and-physical-correlations-from-shannon-to... |

Description | ICML Tutorial |

Form Of Engagement Activity | A talk or presentation |

Part Of Official Scheme? | No |

Geographic Reach | International |

Primary Audience | Postgraduate students |

Results and Impact | Benjamin Guedj and I gave a tutorial on A Primer on PAC-Bayesian Learning at ICML 2019 one of the two premier conferences in machine learning and artificial intelligence. It was attended by approximately 750 people and was streamed live to a wider audience, see https://icml.cc/Conferences/2019/ScheduleMultitrack?event=4338. |

Year(s) Of Engagement Activity | 2019 |

URL | https://icml.cc/Conferences/2019/ScheduleMultitrack?event=4338 |

Description | NeurIPS Tutorial |

Form Of Engagement Activity | A talk or presentation |

Part Of Official Scheme? | No |

Geographic Reach | International |

Primary Audience | Other audiences |

Results and Impact | It was an invited tutorial at the premier machine learning conference, Neural Information Processing Systems held in December 2018 in Montreal. The audience was over 500 researchers and professional practitioners from industry and business. |

Year(s) Of Engagement Activity | 2018 |

URL | https://www.youtube.com/watch?v=m8PLzDmW-TY |