Semantic Information Pursuit for Multimodal Data Analysis
Lead Research Organisation:
University of Cambridge
Department Name: Engineering
Abstract
In 1948, Shannon published his famous paper "A Mathematical Theory of Communication" [88], which laid the foundations of information theory and led to a revolution in communication technologies. Shannon's fundamental contribution was to provide a precise way by which information could be represented,
quantified and transmitted. Critical to Shannon's ideas was the notion that the content of a message is irrelevant to its transmission, since any signal can be represented in terms of bits.
However, Shannon's theory has some limitations. In 1953, Weaver argued that there are three levels
of communication problems: the technical problem "How accurately can the symbols of
communication be transmitted?", the semantic problem "How precisely do the transmitted symbols
convey the desired meaning?", and the effectiveness problem "How effectively does the received
meaning affect conduct in the desired way?" Hence, a key limitation of Shannon's theory is that it
is limited to the technical problem.
This was also pointed out by Bar-Hillel and Carnap in 1953, who argued that "The Mathematical Theory of Communication, often referred to also as Theory (of Transmission) of Information, as practised nowadays, is not interested in the content of the symbols whose information it measures. The measures, as defined, for instance, by Shannon, have nothing to do with what these symbols symbolise, but only with the frequency of their occurrence." While Bar-Hillel and Carnap argued that "the fundamental concepts of the theory of semantic information can be defined in a straightforward way on the basis of the theory of inductive probability", their work was based primarily on logic rules that were applicable to a very restricted class of
signals (e.g. text). In the last 60 years there has been extraordinary progress in information theory,
signal, image and video processing, statistics, machine learning and optimization, which have led
to dramatic improvements in speech recognition, machine translation, and computer vision technologies.
However, the fundamental question of how to represent, quantify and transmit semantic is what this programme of research shall address.
quantified and transmitted. Critical to Shannon's ideas was the notion that the content of a message is irrelevant to its transmission, since any signal can be represented in terms of bits.
However, Shannon's theory has some limitations. In 1953, Weaver argued that there are three levels
of communication problems: the technical problem "How accurately can the symbols of
communication be transmitted?", the semantic problem "How precisely do the transmitted symbols
convey the desired meaning?", and the effectiveness problem "How effectively does the received
meaning affect conduct in the desired way?" Hence, a key limitation of Shannon's theory is that it
is limited to the technical problem.
This was also pointed out by Bar-Hillel and Carnap in 1953, who argued that "The Mathematical Theory of Communication, often referred to also as Theory (of Transmission) of Information, as practised nowadays, is not interested in the content of the symbols whose information it measures. The measures, as defined, for instance, by Shannon, have nothing to do with what these symbols symbolise, but only with the frequency of their occurrence." While Bar-Hillel and Carnap argued that "the fundamental concepts of the theory of semantic information can be defined in a straightforward way on the basis of the theory of inductive probability", their work was based primarily on logic rules that were applicable to a very restricted class of
signals (e.g. text). In the last 60 years there has been extraordinary progress in information theory,
signal, image and video processing, statistics, machine learning and optimization, which have led
to dramatic improvements in speech recognition, machine translation, and computer vision technologies.
However, the fundamental question of how to represent, quantify and transmit semantic is what this programme of research shall address.
Planned Impact
The proposed information-theoretic framework for characterizing information
content in multimodal data combines principles from information physics with probabilistic models that capture rich semantic and contextual relationships between data modalities and tasks. These information measures will be used to develop novel statistical methods for deriving minimal sufficient representations of multimodal data that are invariant to some nuisance factors as well as novel domain adaptation techniques that mitigate the impact of data transformations on information content by finding optimal data transformations.
The computation of such optimal representations and transformations for classification and perception tasks will require solving nonconvex optimization problems for which novel optimization algorithms with provable
guarantees of convergence and global optimality will be developed.
The uncertainty of such information representations derived from multimodal data will be characterized via novel statistical sampling methods that are broadly applicable to various representation learning problems.
The information representations obtained from multiple modalities will be integrated by using a novel information theoretic approach to multi-modal data analysis called information pursuit, which uses a Bayesian model of the scene to determine what evidence to acquire from multiple data modalities, scales and locations, and to coherently integrate this evidence.
The proposed methods will be evaluated in various complex multimodal datasets, including text,
images, video, cellphone data, and body-worn cameras.
content in multimodal data combines principles from information physics with probabilistic models that capture rich semantic and contextual relationships between data modalities and tasks. These information measures will be used to develop novel statistical methods for deriving minimal sufficient representations of multimodal data that are invariant to some nuisance factors as well as novel domain adaptation techniques that mitigate the impact of data transformations on information content by finding optimal data transformations.
The computation of such optimal representations and transformations for classification and perception tasks will require solving nonconvex optimization problems for which novel optimization algorithms with provable
guarantees of convergence and global optimality will be developed.
The uncertainty of such information representations derived from multimodal data will be characterized via novel statistical sampling methods that are broadly applicable to various representation learning problems.
The information representations obtained from multiple modalities will be integrated by using a novel information theoretic approach to multi-modal data analysis called information pursuit, which uses a Bayesian model of the scene to determine what evidence to acquire from multiple data modalities, scales and locations, and to coherently integrate this evidence.
The proposed methods will be evaluated in various complex multimodal datasets, including text,
images, video, cellphone data, and body-worn cameras.
Organisations
People |
ORCID iD |
| Mark Girolami (Principal Investigator) |
Publications
Akyildiz Ö
(2022)
Statistical Finite Elements via Langevin Dynamics
in SIAM/ASA Journal on Uncertainty Quantification
Boys B
(2021)
PeriPy - A high performance OpenCL peridynamics package
in Computer Methods in Applied Mechanics and Engineering
Briol F
(2019)
Probabilistic Integration: A Role in Statistical Computation?
in Statistical Science
Briol F
(2019)
Rejoinder: Probabilistic Integration: A Role in Statistical Computation?
in Statistical Science
Cockayne J
(2019)
Bayesian Probabilistic Numerical Methods
in SIAM Review
Duffin C
(2021)
Statistical finite elements for misspecified models.
in Proceedings of the National Academy of Sciences of the United States of America
Duffin C
(2022)
Low-rank statistical finite elements for scalable model-data synthesis
in Journal of Computational Physics
Gaskin T
(2023)
Neural parameter calibration for large-scale multiagent models.
in Proceedings of the National Academy of Sciences of the United States of America
Girolami M
(2021)
The statistical finite element method (statFEM) for coherent synthesis of observation data and model predictions
in Computer Methods in Applied Mechanics and Engineering
Glyn-Davies A
(2022)
Anomaly detection in streaming data with gaussian process based stochastic differential equations
in Pattern Recognition Letters
Gregory A
(2019)
The synthesis of data from instrumented structures and physics-based models via Gaussian processes
in Journal of Computational Physics
Kuok S
(2022)
Broad learning robust semi-active structural control: A nonparametric approach
in Mechanical Systems and Signal Processing
Manderson A
(2019)
Uncertainty Quantification of Density and Stratification Estimates with Implications for Predicting Ocean Dynamics
in Journal of Atmospheric and Oceanic Technology
Niederer S
(2021)
Scaling digital twins from the artisanal to the industrial
in Nature Computational Science
Oates C
(2019)
Convergence rates for a class of estimators based on Stein's method
in Bernoulli
Oates C
(2019)
Bayesian Probabilistic Numerical Methods in Time-Dependent State Estimation for Industrial Hydrocyclone Equipment
in Journal of the American Statistical Association
Povala J
(2020)
Burglary in London: Insights from Statistical Heterogeneous Spatial Point Processes
in Journal of the Royal Statistical Society Series C: Applied Statistics
Povala J
(2022)
Variational Bayesian approximation of inverse problems using sparse precision matrices
in Computer Methods in Applied Mechanics and Engineering
Ranasinghe G
(2019)
A Methodology for Prognostics Under the Conditions of Limited Failure Data Availability
in IEEE Access
Rappel H
(2022)
Intercorrelated random fields with bounds and the Bayesian identification of their parameters: Application to linear elastic struts and fibers
in International Journal for Numerical Methods in Engineering
Vadeboncoeur A
(2023)
Fully probabilistic deep models for forward and inverse problems in parametric PDEs
in Journal of Computational Physics
Vargas F
(2022)
Bayesian learning via neural Schrödinger-Föllmer flows
in Statistics and Computing
Virtanen S
(2021)
Spatio-Temporal Mixed Membership Models for Criminal Activity
in Journal of the Royal Statistical Society Series A: Statistics in Society