Natural Language Generation for Low-resource Domains
Lead Research Organisation:
Edinburgh Napier University
Department Name: School of Computing
Abstract
It is expected that by 2021, Artificial Intelligence (AI) based dialogue systems such as Amazon's Alexa and Apple's Siri will exceed the earth's population [1]. Such interactive technology products have already become prevalent in many aspects of everyday life, offering support for decision making, education, and health as well as entertainment, by effectively communicating in natural language to answer questions, describe or summarise data, and assist in multiple areas. To develop such systems, however, AI requires access to vast amounts of examples of dialogues, which can (1) be hard to attain in many domains due to unavailability; and (2) pose privacy concerns, impacting user uptake [2]. Current response generation techniques are heavily based on pre-specified templates that limit language coverage. Generating naturally fluent responses is heavily dependant on example dialogues, that are scarcely available in many domains. To address these interlinked challenges, the project will firstly develop natural language generation techniques that are able to learn from limited resources by reusing the knowledge learnt in other data-rich domains, similar to the way the human brain learns new skills efficiently by building on prior knowledge. Secondly, we will develop novel privacy-preserving AI methods to address the second important challenge, and eliminate the risk for de-anonymisation of data.
Although recent advances in understanding natural language have made it possible to accurately predict the meaning of users' utterances and hence accurately inform the personal assistants' actions, responding in natural language remains a bottleneck for the current generation of dialogue systems and personal assistants. As more interactive systems generating natural language become available, the need for natural variability and novelty in the generated text becomes significant in order to increase end-user satisfaction and engagement. Therefore the project will also develop AI approaches that generate text that shows novelty and variability for enriching the word choice while keeping the semantics of the generated text unchanged. Finally, many real-world applications such as personal assistants (and also chatbots and social robots) that support health or education, will benefit from generated responses that show empathy and adapt to users' psychological state. This requires a deep understanding of emotions from text, therefore, this project will, for the first time, develop and integrate innovative, natural language 'concept' based approaches, to understand user emotions from underlying text, and inform novel text generation approaches. Practical case studies provided by our industrial partners will be used to validate our developed AI approaches, throughout this ambitious project.
References:
[1] https://ovum.informa.com/resources/product-content/virtual-digital-assistants-to-overtake-world-population-by-2021
[2] https://www.independent.co.uk/life-style/gadgets-and-tech/news/amazon-alexa-echo-listening-spy-security-a8865056.html
Although recent advances in understanding natural language have made it possible to accurately predict the meaning of users' utterances and hence accurately inform the personal assistants' actions, responding in natural language remains a bottleneck for the current generation of dialogue systems and personal assistants. As more interactive systems generating natural language become available, the need for natural variability and novelty in the generated text becomes significant in order to increase end-user satisfaction and engagement. Therefore the project will also develop AI approaches that generate text that shows novelty and variability for enriching the word choice while keeping the semantics of the generated text unchanged. Finally, many real-world applications such as personal assistants (and also chatbots and social robots) that support health or education, will benefit from generated responses that show empathy and adapt to users' psychological state. This requires a deep understanding of emotions from text, therefore, this project will, for the first time, develop and integrate innovative, natural language 'concept' based approaches, to understand user emotions from underlying text, and inform novel text generation approaches. Practical case studies provided by our industrial partners will be used to validate our developed AI approaches, throughout this ambitious project.
References:
[1] https://ovum.informa.com/resources/product-content/virtual-digital-assistants-to-overtake-world-population-by-2021
[2] https://www.independent.co.uk/life-style/gadgets-and-tech/news/amazon-alexa-echo-listening-spy-security-a8865056.html
Planned Impact
This multi-disciplinary project will have impacts beyond academia.
*Privacy-preserving personal assistants/NLG systems*
As the use of personal assistants grows globally, the need for privacy-preserving approaches to NLG increases. Most industries handle sensitive information such as personal and private data and although data scientists strive to anonymise data records, sophisticated de-anonymisation approaches can pose a threat not only to privacy but also current legislation. Therefore, the impact of innovative approaches that respect privacy and ethical constraints will be enormous.
*Increase productivity by automating the descriptions of products*
Additionally, industries that rely on online presence will benefit from approaches that automatically generate text summaries from structured data, such as descriptions of products and services, since these approaches can increase productivity by automating repetitive and laborious tasks. In addition, diversity-enriched NLG approaches can make the content of automatically generated summaries more interesting and less repetitive, and hence increase the overall user experience.
*Automatic narrative and report generation*
Narrative generation from data, such as automatic news story generation, will benefit from more natural NLG approaches that offer variability and empathy as stories can be enriched with emotion, and interesting non-repetitive text. Business intelligence and analytics reporting will also benefit from the approaches developed here, as privacy is integral when communicating data and insights. In addition, NLG has been shown to enhance decision making support [5].
*Support Health and Well-being*
AI-powered personal assistants, such as the Alli-chat developed by our project partner, have started to become prevalent in health support [1]. It can also be preferable for supporting specific groups such as younger people for stigma-attached conditions such as mental health. Younger people are particularly less likely to seek help when facing mental health challenges [4], therefore, it is of vital importance to create a safe space for younger people that empowers them to seek private advice and information regarding mental well-being which can be achieved through trusted, privacy-sensitive, empathy-enriched personal assistants. Promotion of mental well-being, management, and prevention of mental health illnesses has been indeed identified as a core priority of the World Health Organisation's mental health action plan 2013-2020 [2] as well as in NHS's "Five Years Forward View" for mental health [3].
References:
[1] https://www.healthcareitnews.com/news/special-report-ai-voice-assistants-making-impact-healthcare
[2] https://www.who.int/mental_health/publications/action_plan/en/
[3] https://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf
[4] Marcus, M. A., Westra, H. A., Eastwood, J. D., Barnes, K. L., & Mobilizing Minds Research Group (2012). What are young adults saying about mental health? An analysis of Internet blogs. Journal of medical Internet research, 14(1), e17. doi:10.2196/jmir.1868
[5] Gkatzia et al. (2017). Data-to-Text Generation Improves Decision-Making Under Uncertainty. IEEE Computational Intelligence Magazine, Special Issue on Natural Language Generation with Computational Intelligence.
*Privacy-preserving personal assistants/NLG systems*
As the use of personal assistants grows globally, the need for privacy-preserving approaches to NLG increases. Most industries handle sensitive information such as personal and private data and although data scientists strive to anonymise data records, sophisticated de-anonymisation approaches can pose a threat not only to privacy but also current legislation. Therefore, the impact of innovative approaches that respect privacy and ethical constraints will be enormous.
*Increase productivity by automating the descriptions of products*
Additionally, industries that rely on online presence will benefit from approaches that automatically generate text summaries from structured data, such as descriptions of products and services, since these approaches can increase productivity by automating repetitive and laborious tasks. In addition, diversity-enriched NLG approaches can make the content of automatically generated summaries more interesting and less repetitive, and hence increase the overall user experience.
*Automatic narrative and report generation*
Narrative generation from data, such as automatic news story generation, will benefit from more natural NLG approaches that offer variability and empathy as stories can be enriched with emotion, and interesting non-repetitive text. Business intelligence and analytics reporting will also benefit from the approaches developed here, as privacy is integral when communicating data and insights. In addition, NLG has been shown to enhance decision making support [5].
*Support Health and Well-being*
AI-powered personal assistants, such as the Alli-chat developed by our project partner, have started to become prevalent in health support [1]. It can also be preferable for supporting specific groups such as younger people for stigma-attached conditions such as mental health. Younger people are particularly less likely to seek help when facing mental health challenges [4], therefore, it is of vital importance to create a safe space for younger people that empowers them to seek private advice and information regarding mental well-being which can be achieved through trusted, privacy-sensitive, empathy-enriched personal assistants. Promotion of mental well-being, management, and prevention of mental health illnesses has been indeed identified as a core priority of the World Health Organisation's mental health action plan 2013-2020 [2] as well as in NHS's "Five Years Forward View" for mental health [3].
References:
[1] https://www.healthcareitnews.com/news/special-report-ai-voice-assistants-making-impact-healthcare
[2] https://www.who.int/mental_health/publications/action_plan/en/
[3] https://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf
[4] Marcus, M. A., Westra, H. A., Eastwood, J. D., Barnes, K. L., & Mobilizing Minds Research Group (2012). What are young adults saying about mental health? An analysis of Internet blogs. Journal of medical Internet research, 14(1), e17. doi:10.2196/jmir.1868
[5] Gkatzia et al. (2017). Data-to-Text Generation Improves Decision-Making Under Uncertainty. IEEE Computational Intelligence Magazine, Special Issue on Natural Language Generation with Computational Intelligence.
Organisations
- Edinburgh Napier University (Lead Research Organisation)
- Allen Institute for Artificial Intelligence (Collaboration)
- IBM (Collaboration)
- Charles University (Collaboration)
- University of Virginia (UVa) (Collaboration)
- University of Tilburg (Collaboration)
- Amazon.com (Collaboration)
- UNIVERSITY OF EDINBURGH (Collaboration)
- Google (Collaboration)
- Georgetown University (Collaboration)
- UNIVERSITY OF ABERDEEN (Collaboration)
- Carnegie Mellon University (Collaboration)
- Heriot-Watt University (Collaboration)
- Trivago NV (Collaboration)
- University of Helsinki (Collaboration)
- Trivago N.V. (Project Partner)
- Voxsio (Project Partner)
Publications
Farooq U
(2021)
Advances in machine translation for sign language: approaches, limitations, and challenges
in Neural Computing and Applications
Farouq MW
(2021)
A Novel Coupled Reaction-Diffusion System for Explainable Gene Expression Profiling.
in Sensors (Basel, Switzerland)
Alwaneen T
(2021)
Arabic question answering system: a survey
in Artificial Intelligence Review
Ieracitano C
(2021)
A novel automatic classification system based on hybrid unsupervised and supervised machine learning for electrospun nanofibers
in IEEE/CAA Journal of Automatica Sinica
El-Affendi M
(2021)
A Novel Deep Learning-Based Multilevel Parallel Attention Neural (MPAN) Model for Multidomain Arabic Sentiment Analysis
in IEEE Access
Aroyehun S
(2021)
Leveraging label hierarchy using transfer and multi-task learning: A case study on patent classification
in Neurocomputing
Al-Ghadir A
(2021)
A novel approach to stance detection in social media tweets by fusing ranked lists and sentiments
in Information Fusion
Zhang S
(2021)
Improving generative adversarial networks with simple latent distributions
in Neural Computing and Applications
Description | Enhancing Labour Market Intelligence using Machine Learning |
Amount | £60,000 (GBP) |
Organisation | Skills Development Scotland |
Sector | Public |
Country | United Kingdom |
Start | 08/2021 |
End | 10/2025 |
Description | Scottish Gaelic Generation for Exhibitions |
Amount | £5,000 (GBP) |
Organisation | Arts & Humanities Research Council (AHRC) |
Sector | Public |
Country | United Kingdom |
Start | 02/2022 |
End | 09/2022 |
Description | Sentinel: Security alert level automation |
Amount | £5,000 (GBP) |
Organisation | Government of Scotland |
Department | Scottish Funding Council |
Sector | Public |
Country | United Kingdom |
Start | 11/2021 |
End | 01/2022 |
Title | CEC - Commonsense Evaluation Card |
Description | The Commonsense Evaluation Card (CEC) aims to standardise human evaluation and reporting of commonsense-enhanced NLG systems, enabling researchers to compare models not only in terms of classic NLG quality criteria, but also by focusing on the core capabilities of such models. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | This tool has helped in better documenting experiments related to commonsense knowledge. |
URL | https://nlgknowledge.github.io/commonsense/ |
Title | Multi-modal Speech Enhancement Demonstrator Tool |
Description | We developed the world's first open web-based demonstrator tool that shows how recordings of speech in noisy environments can be multi-modally processed to remove background noise and make the speech easier to hear. The demonstrator tool works for sound only, as well as video recordings, and enables researchers to develop innovative multi-modal speech and natural language communication applications. Users can listen to sample recordings and upload their own personal (noisy) videos or audio files to hear the difference after audio-visual processing using a deep neural network model. No uploaded data is stored. User data is erased as soon as the web page is refreshed or closed. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | This innovative demonstrator tool was showcased at an international workshop organised as part of the 2022 IEEE Engineering in Medicine and Biology Society Conference (EMBC) in Glasgow, 11-15 July. Around 40 Workshop participants (including clinical, academic and industry researchers) were provided with an interactive hands-on demonstration of the audio-visual speech enhancement tool. The tool demonstrated, for the first time, the technical feasibility of developing audio-visual algorithms that can enhance speech quality and intelligibility, with the aid of video input and low-latency combination of audio and visual speech information. This served to educate participants and demonstrated the potential of such transformative tools to extract salient information from the pattern of the speaker's lip movements and to contextually employ this information as an additional input to speech enhancement algorithms, in future multi-modal communications and hearing assistive technology applications. |
URL | https://demo.cogmhear.org/ |
Title | Second COG-MHEAR International Audio-Visual Speech Enhancement Challenge (AVSEC-2): New baseline Deep Neural Network Model, Real-world Datasets and Audio-visual Intelligibility Testing Method |
Description | We organised the second COG-MHEAR international AVSEC-2 Challenge that featured as an approved Challenge at IEEE ASRU 2023, 16-20 Dec 2023. As part of AVSEC-2, we developed and made openly available a new benchmark pre-trained deep neural network model and real-world (TED video) datasets. We further refined our method for evaluating audio-visual speech enhancement systems at scale using a fully validated keyword-spotting design that was presented at ICASSP 2023 (https://ieeexplore.ieee.org/document/10096479). |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | The Challenge has provided excellent advocacy for COG-MHEAR, bringing opportunities to strengthen the audio-visual speech enhancement (AV SE) community and bring together interdisciplinary researchers to work towards improving future natural language speech and hearing technologies. AVSEC-2 received a higher number of submissions compared to the first challenge, and we were pleased to see high-profile companies such as Mitsubishi among the participants. Moreover, there was an increase in performance of the best performing systems for multimodal speech-in-noise processing. The data produced from the challenge has enabled us to gain further understanding about the role that features of the audio and visual components play in intelligibility; we also investigated how intelligibility is affected when audio-only and audio-visual stimuli are presented, leading to important findings about the efficacy of speech enhancement in in-person settings. Our teams developed a new baseline pre-trained deep neural network model and made this openly available to participants, along with raw and pre-processed audio-visual datasets - derived from real-world TED talk videos - for training and development of new audio-visual models to perform speech enhancement and speaker separation at signal to noise (SNR) levels that were significantly more challenging than typically used in audio-only scenarios. The Challenge evaluation utilised established objective measures (such as STOI and PESQ, for which scripts were provided to participants) as well as a new audio-visual intelligibility testing method developed by the COG-MHEAR teams for subjective evaluation with human subjects. The latter was further refined as part of AVSEC-2 for evaluating AV SE systems at scale using a fully validated keyword-spotting design that was presented at ICASSP 2023. The new baseline model, real-world datasets and subjective audio-visual intelligibility testing method are continuing to be exploited by researchers in speech and natural language communication and hearing assistive technology applications. |
URL | https://challenge.cogmhear.org/#/avsec2/ |
Title | World's first large-scale Audio-Visual Speech Enhancement Challenge (AVSEC): New baseline Deep Neural Network Model, Real-world Datasets and Audio-visual Intelligibility Testing Method |
Description | We developed and made openly available, a new benchmark pre-trained deep neural network model, real-world (TED video) datasets and a novel subjectve audio-visual intelligibility evaluation method as part of the world's first large-scale Audio-Visual Speech Enhancement Challenge. Details of the benchmark model, datasets and intelligibility testing method were published in peer-reviewed proceedings of the 2023 IEEE Spoken Language Technology (SLT) Workshop (https://ieeexplore.ieee.org/abstract/document/10023284). |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2022 |
Provided To Others? | Yes |
Impact | The new benchmark pre-trained model code and training and evaluation datasets were made openly available as part of the world's first large-scale Audio-Visual Speech Enhancement (AVSE) Challenge organised by our COG-MHEAR teams as part of the 2023 IEEE Spoken Language Technology (SLT) Workshop, Qatar, 9-12 January 2023. The Challenge brought together wider computer vision, hearing and speech research communities from academia and industry to explore novel approaches to multimodal speech-in-noise processing. Our teams developed a new baseline pre-trained deep neural network model and made this openly available to participants, along with raw and pre-processed audio-visual datasets - derived from real-world TED talk videos - for training and development of new audio-visual models to perform speech enhancement and speaker separation at signal to noise (SNR) levels that were significantly more challenging than typically used in audio-only scenarios. The Challenge evaluation utilised established objective measures (such as STOI and PESQ, for which scripts were provided to participants) as well as a new audio-visual intelligibility testing method developed by the COG-MHEAR teams for subjective evaluation with human subjects. The new baseline model, real-world datasets and subjective audio-visual intelligibility testing method are continuing to be exploited by researchers in speech and natural language communication and hearing assistive technology applications. |
URL | https://challenge.cogmhear.org/#/download |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | Allen Institute for Artificial Intelligence |
Country | United States |
Sector | Public |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | Amazon.com |
Country | United States |
Sector | Private |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | Carnegie Mellon University |
Country | United States |
Sector | Academic/University |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | Charles University |
Country | Czech Republic |
Sector | Academic/University |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | |
Department | Research at Google |
Country | United States |
Sector | Private |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Multi-lingual NLG benchmarking |
Organisation | IBM |
Department | IBM T. J. Watson Research Center, Yorktown Heights |
Country | United States |
Sector | Private |
PI Contribution | For this partnership, multiple groups around the world collaborated on creating benchmark datasets and models for various languages in NLG, so that to facilitate benchmarking and easier comparison between new models. Our group contributed a new Gaelic dataset and the formulation of a new dual dialogue and summarisation task. |
Collaborator Contribution | Several partners contributed models, datasets, and evaluation metrics. |
Impact | Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou. Gemv2: Multilingual nlg benchmarking in a single line of code. EMNLP 2022. |
Start Year | 2022 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | Charles University |
Country | Czech Republic |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | Georgetown University |
Country | United States |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | Heriot-Watt University |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | Trivago NV |
Country | Germany |
Sector | Private |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | University of Aberdeen |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | University of Helsinki |
Country | Finland |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | University of Tilburg |
Country | Netherlands |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Providing Recommendations of Error Analysis of NLG systems |
Organisation | University of Virginia (UVa) |
Country | United States |
Sector | Academic/University |
PI Contribution | This is a multi-partners collaboration between Edinburgh Napier, Heriot-Watt University, trivago, Charles University in Prague, and others. All partners worked together to analyse the state of error reporting of NLG systems and provide recommendations so that future NLG publications discuss both the benefits but also the errors made by the systems with the aim to focus on bettering these aspects. |
Collaborator Contribution | All partners worked together to analyse current trends in error reporting and provide recommendations on how error analysis in NLG systems should be performed with the aim to understand the limitations of current scientific advances. |
Impact | Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson and Luou Wen. (2021). Underreporting of errors in NLG output, and what to do about it. In INLG 2021. Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen. Barriers and enabling factors for error analysis in NLG research. In Northern European Journal of Language Technology. 2023 |
Start Year | 2021 |
Description | Multi-party collaboration on Scottish Gaelic Language Generation |
Organisation | University of Edinburgh |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | TBA |
Collaborator Contribution | TBA |
Impact | Funding from AHRC for a data collection |
Start Year | 2022 |
Description | Multi-party collaboration/study on Evaluation of Commonsense-enhanced NLG systems |
Organisation | Heriot-Watt University |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | TBA |
Collaborator Contribution | TBA |
Impact | Miruna-Adriana Clinciu, Dimitra Gkatzia, Saad Mahamood. 2021. It's Commonsense, isn't it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) at EACL 2021. |
Start Year | 2021 |
Description | Multi-party collaboration/study on Evaluation of Commonsense-enhanced NLG systems |
Organisation | Trivago NV |
Country | Germany |
Sector | Private |
PI Contribution | TBA |
Collaborator Contribution | TBA |
Impact | Miruna-Adriana Clinciu, Dimitra Gkatzia, Saad Mahamood. 2021. It's Commonsense, isn't it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) at EACL 2021. |
Start Year | 2021 |
Description | Blogpost at trivago.com website about our collaboration/joint work |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Our industry collaborator published a blogpost about our recent works in Natural Language Generation. The website is visited by a large number of people internationally. |
Year(s) Of Engagement Activity | 2022 |
URL | https://tech.trivago.com/post/2022-03-31-improving-evaluation-practices-in-natural-language-generati... |
Description | Invited seminar talk at the National Research Council of Canada. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | David Howcroft presented "Disentangling 20 years of confusion in NLG: toward standards for human evaluation" at the National Research Council of Canada's Natural Language Processing seminar, having been invited by Cyril Goutte. The discussion included useful similarities between evaluation for Natural Language Generation (NLG) and machine translation in particular, including gaps in terms of designing studies to measure the preferences of individual target groups as well as discussions of performing evaluation in low-resource settings. |
Year(s) Of Engagement Activity | 2021 |
Description | Invited talk at the Celtic Languages in the Digital Age (CLIDA 2024) workshop by D Howcroft |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | David's talk is about Scottish Gaelic NLP |
Year(s) Of Engagement Activity | 2024 |
URL | https://wp.lancs.ac.uk/celtic/ |
Description | Invited to participate at a panel on Explainable AI at the inaugural Scottish AI Summit |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | I was invited to join a panel on why AI is still black box and discusses limitations and opportunities of explainable AI. The event was attended by 300 people in person and over 500 online. Attendees included politicians, academics, industry and third sector such as Unicef. |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.scottishaisummit.com/ |
Description | NLP Seminar at Dublin City University by D Howcroft |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | TBA |
Year(s) Of Engagement Activity | 2022 |
Description | NLP seminar at Cardiff University by D Howcroft |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | David gave a presentation on our Scottish Gaelic data collection and a proposed shared task for data collection for low-resource languages. The invited talk gave the opportunity for an exchange of research ideas and led to ongoing discussions with Dawn Knight (Cardiff U.) and her collaborator Mo El-Haj (U. Lancaster), leading to another invited talk and plans to develop a grant proposal together. |
Year(s) Of Engagement Activity | 2023 |
Description | Professorial Talk at Edinburgh Napier University open day |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Undergraduate students |
Results and Impact | Around 160 students attended my professorial talk on "How close are we to achieving Human-like AI? From Eliza to Alexa and beyond", which described the current state of dialogue systems and natural language generation, discussed the limitations of current systems, and discussed the "misinformation" about AI as presented in media. The talk sparked a vivid discussion in the area. |
Year(s) Of Engagement Activity | 2021 |