Twenty20Insight

Lead Research Organisation: University of Warwick
Department Name: Computer Science

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
 
Description Our key findings are summarised below:

(1) We have proposed a novel singular value transformation function to address the token uniformity problem of the widely-used Transformer architecture, where different tokens share a large proportion of similar information after going through stacked multiple layers in a transformer. We propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the token uniformity problem. Based on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space.

(2) We have developed a new explainable AI (XAI) approach for providing hierarchical interpretations for neural text classifiers. Most existing XAI approaches aim at identifying input features such as words or phrases important for model predictions. Neural models developed in NLP however often compose word semantics in a hierarchical manner. Interpretation by words or phrases only thus cannot faithfully explain model decisions. We have proposed a Hierarchical Interpretable Neural Text classifier, called Hint, which is able to identify the latent semantic factors and their compositions which contribute to the model's final decisions. This is often beyond what word-level interpretations could capture.

(3) We have developed an explainable recommender system by simultaneously considering both implicit user-item interactions and users' reviews on certain items. We can infer latent semantic factors from user-item reviews, which can be used for both recommendation and explanation generation. We have shown that our model significantly improves the interpretability of existing recommender systems built on variational autoencoder while achieving performance comparable to existing content-based recommender systems in terms of recommendation behaviours.
Exploitation Route The transformer architecture is widely used in pre-trained language models such as BERT, ALBERT, RoBERTa, DistilBERT, GPT, etc., and has been extensively employed in tackling various tasks in Natural Language Processing and computer vision. Our proposed singular value transformation function will thus have a great potential to address the token uniformity problem in models built on the transformer architecture.

Our developed XAI approach for neural text classification and interpretable recommender systems can be applied in a wide range of tasks such as sentiment analysis, topic classification, rumour veracity assessment, and produce recommendation.
Sectors Digital/Communication/Information Technologies (including Software),Education,Financial Services, and Management Consultancy,Healthcare,Pharmaceuticals and Medical Biotechnology
URL https://sites.google.com/view/yulanhe/model-interpretability
 
Title Addressing token uniformity in transformers using the singular value transformation function (SoftDecay) 
Description Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. We propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the token uniformity problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks 
Type Of Material Computer model/algorithm 
Year Produced 2022 
Provided To Others? Yes  
Impact The proposed approach is described in a paper published in UAI 2022. 
URL https://github.com/hanqi-qi/tokenUni
 
Title CHIME - Cross-passage hierarchical memory network for question answering 
Description CHIME is a cross-passage hierarchical memory network developed for question answering (QA) via text generation. It extends XLNet introducing an auxiliary memory module consisting of two components: the context memory collecting cross-passage evidences, and the answer memory working as a buffer continually refining the generated answers. 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact The efficacy of CHIME has been verified in the multi-passage generative QA. It outperforms the state-of-the-art baselines with better syntactically well-formed answers and increased precision in addressing the questions of the AmazonQA review dataset. 
URL https://github.com/LuJunru/CHIME
 
Title Hierarchical Interpretable Neural Text classifier (HINT) 
Description Recent years have witnessed increasing interest in developing interpretable models in Natural Language Processing (NLP). Most existing models aim at identifying input features such as words or phrases important for model predictions. Neural models developed in NLP, however, often compose word semantics in a hierarchical manner. As such, interpretation by words or phrases only cannot faithfully explain model decisions in text classification.We propose a novel Hierarchical Interpretable Neural Text classifier, called HINT, which can automatically generate explanations of model predictions in the form of label-associated topics in a hierarchical manner. Model interpretation is no longer at the word level, but built on topics as the basic semantic unit. Experimental results on both review datasets and news datasets show that our proposed approach achieves text classification results on par with existing state-of-the-art text classifiers, and generates interpretations more faithful to model predictions and better understood by humans than other interpretable neural text classifiers 
Type Of Material Computer model/algorithm 
Year Produced 2022 
Provided To Others? Yes  
Impact The approach is described in a paper published in the Computational Linguistics journal. 
URL https://github.com/hanqi-qi/HINT
 
Title Joint Aspect-Sentiment Extraction (JASE) model for understanding patient reviews 
Description The Joint Aspect-Sentiment Extraction (JASE) model is developed for simultaneous unsupervised aspect extraction at the sentence level and supervised sentiment classification at the document level. It can extract coherent aspects from patient reviews and can automatically infer the distribution of aspects under different polarities without requiring aspect-level annotations for model learning. 
Type Of Material Computer model/algorithm 
Year Produced 2021 
Provided To Others? Yes  
Impact The model achieves 98.2% sentiment classification accuracy when tested on the reviews about healthcare services collected from Yelp, outperforming several strong baselines. 
URL https://github.com/GuiLinNLP/PatientReviewUnderstanding
 
Title Knowledge-Aware Graph (KAG) Model for Emotion Cause Extraction 
Description Knowledge-Aware Graph (KAG) Model aims to identify the triggering factors for emotions expressed in text. It explicitly models the emotion triggering paths by leveraging the commonsense knowledge to enhance the semantic dependencies between a candidate clause and an emotion clause. 
Type Of Material Computer model/algorithm 
Year Produced 2021 
Provided To Others? Yes  
Impact The model has already extracted interests from industry such as WeChat AI from Tencent. 
URL https://github.com/hanqi-qi/Position-Bias-Mitigation-in-Emotion-Cause-Analysis
 
Title Neural Temporal Opinion Model (NTOM) 
Description Neural Temporal Opinion Model (NTOM) has been developed for predicting user opinions on Twitter. NTOM models users' tweet posting behaviour as a temporal point process to jointly predict the posting time and the stance label of the next tweet given a user's historical tweet sequence and tweets posted by their neighbours. A topic-driven attention mechanism is designed to capture the dynamic topic shifts in the neighbourhood context. 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact NTOM predicts both the posting time and the stance labels of future tweets more accurately compared to a number of competitive baselines including RNN, LSTM, GRU and approaches built on Graph Convolutional Network. 
URL https://github.com/somethingx01/TopicalAttentionBrexit
 
Description Featured in Futurum, an online magazine 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Schools
Results and Impact Yulan He was featured in Futurum Careers, an online magazine, discussing her work on teaching computers to understand human language and offering guidance to young people interested in AI and NLP. Futurum Careers is a free online resource and magazine aimed at introducing 14-19-year-olds worldwide to the world of work in science, tech, engineering, maths, medicine, social sciences, humanities and the arts for people and the economy.
Year(s) Of Engagement Activity 2022
URL https://futurumcareers.com/teaching-computers-to-understand-our-language
 
Description Interview by New Scientist 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact I have been interviewed by New Scientist to comment on the ChatGPT detector.
Year(s) Of Engagement Activity 2023
URL https://www.newscientist.com/article/2355035-chatgpt-detector-could-help-spot-cheaters-using-ai-to-w...
 
Description Invited talk at AI UK 2022 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presented my work on machine reasoning for natural language understanding in AI UK 2022. My talk led to a collaborative project with AQA and joint research proposals with a few UK universities.
Year(s) Of Engagement Activity 2022
URL https://www.turing.ac.uk/node/7396
 
Description Invited talk at the University of Cambridge 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Yulan He was invited to give a talk on "Hierarchical Interpretation of Neural Text Classification" in the Language Technology Lab (LTL) at the University of Cambridge, headed by Anna Korhonen and Nigel Collier. A follow-up discussion was held in March between Anna and Yulan to explore potential future collaborations.
Year(s) Of Engagement Activity 2022
URL http://131.111.150.181/talk/index/170564
 
Description Tutorial in the Oxford Machine Learning Summer School 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I delivered a tutorial on recent developments in sentiment analysis in the Oxford Machine Learning Summer School, targeting postgraduate students and researchers working in AI and machine learning.
Year(s) Of Engagement Activity 2022
URL https://www.oxfordml.school/oxml2022