Turing AI Fellowship:Neural Conversational Information Seeking Assistant
Lead Research Organisation:
University of Glasgow
Department Name: School of Computing Science
Abstract
There have been significant recent advances in Virtual Personal Assistants (VPAs) such as Google Assistant, Siri, and Alexa. However, development of these assistant systems is expensive and difficult (often requiring multiple skilled PhDs). And further, current systems are capable of limited "conversations", with most actions consisting of a single interaction in limited domains to perform simple tasks ("set a timer", "play music", etc...). The goal of this research is to develop research to enable a future conversational search systems that can help solve complex information tasks. Examples of these types of information tasks could be "Teach me about the causes of climate change." or "Help me write the literature survey for this paper." These require complex discussion and long-running modelling of the user and their information task. We propose building on recent advances in machine learning to adapt a general purpose information agent for specialized domains (like health, law, finance) by "machine reading" of text (such documents from a website) to learn a domain model and to discover information tasks automatically from existing interaction data such as search logs, existing conversations, or help tickets. The result of this work will be information agents that can effectively work with the user (including asking questions back and forth) and explain their reasoning more effectively than current information assistants.
Publications
Aliannejadi M
(2021)
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
Aliannejadi M.
(2021)
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
in EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings
Dalton
(2022)
CAsT 2021: The Conversational Assistance Track Overview
Dalton J
(2022)
Conversational Information Seeking
Dietz L
(2023)
Neuro-Symbolic Representations for Information Retrieval
Fionda V
(2023)
Tutorials at The Web Conference 2023
Fischer S
(2022)
VILT: Video Instructions Linking for Complex Tasks
Fischer S
(2022)
VILT
Related Projects
| Project Reference | Relationship | Related To | Start | End | Award Value |
|---|---|---|---|---|---|
| EP/V025708/1 | 01/01/2021 | 29/09/2023 | £1,623,810 | ||
| EP/V025708/2 | Transfer | EP/V025708/1 | 30/09/2023 | 31/12/2025 | £1,033,083 |
| Description | The result of this award includes the development of new algorithms and a new open-source system for virtual personal assistants, called the Open Assistant Toolkit (OAT). It was deployed on Alexa in the US in the Alexa Prize Taskbot challenge and won first and second prizes (2022, 2023) helping users accomplish complex real-world tasks like cooking and home repair. The award also enabled the creation of multiple new benchmark datasets and evaluation methods to measure progress on conversational search systems in partnership with the U.S. National Institute for Standards and Technology (NIST) in the Text Retrieval Conference (TREC). |
| Exploitation Route | Companies and researchers can use and extend the open-source tools and frameworks developed to build new virtual assistants for their target domains. They can leverage the benchmarks created to measure the progress of their search components and extend them to specialised sectors, such as financial research and medicine. |
| Sectors | Digital/Communication/Information Technologies (including Software) Financial Services and Management Consultancy Healthcare |
| Description | ChatGPT training |
| Geographic Reach | Local/Municipal/Regional |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Description | Amazon Alexa Prize Challenge |
| Amount | $250,000 (USD) |
| Organisation | Amazon.com |
| Sector | Private |
| Country | United States |
| Start | 05/2021 |
| End | 06/2025 |
| Description | Amazon Alexa Prize Challenge |
| Amount | $250,000 (USD) |
| Organisation | Amazon.com |
| Sector | Private |
| Country | United States |
| Start | 01/2022 |
| End | 09/2022 |
| Title | Open Assistant Toolkit |
| Description | OAT is a task-oriented open-source conversational agent platform for supporting multimodal complex tasks. OAT is modular and extensible, enabling complex stateful behavior for human-like long-form interaction. It supports multiple conversational voice platforms as well as multimodal output. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | It's been tested and evaluated by Amazon during multiple Taskbot challenges; winning first and second prize in the data challenges. |
| URL | https://github.com/grill-lab/OAT |
| Title | TaskMAD: Task Multimodal Agent Dialogue |
| Description | Task-oriented Multimodal Agent Dialogue (TaskMAD) is a new platform that supports the creation of interactive multimodal and task-centric datasets in a Wizard-of-Oz experimental setup. TaskMAD includes support for text and voice, federated retrieval from text and knowledge bases, and structured logging of interactions for offline labeling. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2022 |
| Provided To Others? | Yes |
| Impact | Multiple publications and demos at leading IR conferences. Used by other research labs (USC, Radboud) for conversational search research and data gathering. |
| URL | https://github.com/grill-lab/TaskMAD |
| Title | CODEC - COmplex Document and Entity Collection |
| Description | This is a new dataset for benchmarking systems for complex entity-based retrieval, such as financial analysts or social science researchers. It provides research topics, queries, judgments, etc... to use as training and evaluation data for information retrieval algorithms. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2022 |
| Provided To Others? | Yes |
| Impact | None significant, yet. It is currently under submission and was just released last month. |
| URL | https://github.com/grill-lab/CODEC |
| Title | Cooking With Conversation |
| Description | A dataset of wizard-of-oz conversations on the topic of cooking. It supports the paper: "Cooking with Conversation: Enhancing User Engagement and Learning with a Knowledge-Enhancing Assistant" |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | N/A - Just released. |
| URL | https://github.com/AlexFrummet/cooking-with-conversations |
| Title | Deep Learning HARD |
| Description | This is a new dataset focused on evaluation of deep learning-based retrieval algorithms. It extends previous benchmark datasets and introduces new methodology for identifying and incorporating challenging cases. It includes a rich set of data used to benchmark and evaluate deep-learning based web search algorithms. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | It has been used and cited by multiple research groups. It influenced the direction of the TREC Deep Learning track for future years, towards different types of problems. |
| URL | https://github.com/grill-lab/DL-Hard |
| Title | TREC Conversational Assiastance Track 2021 & 2022 |
| Description | This is a new benchmark dataset for conversational search developed in partnership with the US National Institute for Technology and Standards (NIST). It is used to measure progress and evaluate algorithms for conversational search. It was used by fifteen worldwide teams competing in the 2021 and 2022 TREC Conversational Assistance Tracks (CAsT). |
| Type Of Material | Database/Collection of data |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | It was the first to benchmark the use of new dense retrieval approaches for conversational search. |
| URL | https://trec.nist.gov/data/cast2021.html |
| Title | TREC Interactive Knowledge Assistant Track (iKAT) |
| Description | A benchmark for interactive conversational assistants with contextual personalised knowledge statements. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2023 |
| Provided To Others? | Yes |
| Impact | Used by leading research groups (dozens of participants) from leading research groups around the world. |
| URL | https://www.trecikat.com/ |
| Description | Amazon Alexa Prize |
| Organisation | Amazon.com |
| Country | United States |
| Sector | Private |
| PI Contribution | My group creates the GRILLBot conversational agent skill that is deployed to US Amazon Alexa users. This includes research, software development, and deployment. |
| Collaborator Contribution | The partners provide software support (Cobot), weekly demo and feedback sessions with software engineers and UI experts, AWS cloud compute resources, and more. |
| Impact | VILT: Video Instructions Linking for Complex Tasks GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representations GRILLBot: A flexible conversational agent for solving complex real-world tasks Open Assistant Toolkit (software) Open Assistant Toolkit -- version 2 (software) GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants GRILLBot-v2: Generative models for multi-modal task-oriented assistance |
| Start Year | 2021 |
| Description | BBC Scotland Beeb Virtual Assistant |
| Organisation | British Broadcasting Corporation (BBC) |
| Department | BBC Scotland |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | Myself and a research assistant worked closely with the BBC Scotland Voice and AI on a prototype system, UoG Bot. This was a full time position for an RA and approximately 20% of my time. We contributed and developed machine learning models for conversational exploration and conversational search. The work and partnership and impact from this project is ongoing as part of the Fellowship. |
| Collaborator Contribution | The partners provided substantial support including a repository of news and sound content, four engineers part time working on the project, as well as project managers and experience developers. They assisted by contributing user study. They are currently assisting with releasing data and software. |
| Impact | COMEX: A Multi-task Benchmark for Knowledge-grounded COnversational Media EXploration https://dl.acm.org/doi/abs/10.1145/3543829.3543830 |
| Start Year | 2020 |
| Description | Radboud-Glasgow Collaboration |
| Organisation | Radboud University Nijmegen |
| Country | Netherlands |
| Sector | Academic/University |
| PI Contribution | My team collaborations on creation of personal knowledge graphs from conversational data. This involves my postdoc Shubham Chaterjee and PhD Students Paul Owoicho working closely with Radboud students and staff to collect data and run experiments. We provide our open assistant toolkit (OAT) as platform for testing. |
| Collaborator Contribution | The partners provide students, staff, and data in the collaboration for running experiments. They also provide their CREL entity linking software for analyzing conversations and performing information extraction. |
| Impact | Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search |
| Start Year | 2022 |
| Title | CAsT Searcher |
| Description | CAsT Searcher is a simple tool developed to help with creating the evaluation topics for the 2021 edition of the Conversational Assistance Track. The tool allows topic developers to visually assess the behaviour of a retrieval system, ultimately making it easier to develop challenging, but interesting, topics for the Track. |
| Type Of Technology | Webtool/Application |
| Year Produced | 2021 |
| Open Source License? | Yes |
| Impact | Used in the development of the TREC CAsT evaluation benchmark dataset that the group released. |
| URL | https://github.com/grill-lab/CAsTSearcher |
| Title | DiffIR |
| Description | DiffIR is a tool for visually 'diffing' the difference between two sets of rankings. Given a pair of TREC runs containing rankings for multiple queries, DiffIR identifies contrasting queries that have "substantially" different results between the two systems and generates a visual side-by-side comparison illustrating how the key rankings differ. This was developed and appeared as a SIGIR 2021 demo. |
| Type Of Technology | Software |
| Year Produced | 2021 |
| Open Source License? | Yes |
| Impact | It's been used my multiple research groups, including being starred and watched by tens of groups. |
| URL | https://github.com/capreolus-ir/diffir |
| Title | Open Assistant Toolkit - OAT |
| Description | A conversational task agent, this is the open source version of the 2022 winning Alexa Prize Taskbot system, GRILLBot developed with the ability to allow conversational interactions with Alexa voice and screen devices. It is currently deployed and serving US customers in the Amazon Alexa Taskbot challenge 2022-2023. |
| Type Of Technology | Software |
| Year Produced | 2022 |
| Open Source License? | Yes |
| Impact | It's been used as the basis for a SIGIR 2022 tutorial, WWW 2023 tutorial, and workshops with leading research and industry groups, including Stanford University and Square (a US payments company). |
| Title | TaskMAD |
| Description | Task MAD is a new Wizard-of-Oz data collection platform for conversational systems. It supports the collection of conversational data for interactive tasks (like cooking or researching a product). It's novel in that it supports rich task context, structured system actions, integrates knowledge-based search, and include rich support for images and videos in a conversational interface. |
| Type Of Technology | Software |
| Year Produced | 2022 |
| Open Source License? | Yes |
| Impact | This software was just released. |
| URL | https://github.com/grill-lab/taskmad |
| Company Name | Malted AI Ltd |
| Description | |
| Year Established | 2023 |
| Impact | There are approximately 2 full-time scientific posts within the company. |
| Description | BBC about ChatGPT |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Appeared on BBC Tech Tent on December 9th https://www.bbc.co.uk/programmes/w3ct4khv This was talking about ChatGPT and the future of conversational models to the general public. This also aired on Click short on the BBC https://www.bbc.co.uk/iplayer/episode/m001gf3r/click-short-edition-17122022 This was again about the implications of ChatGPT for the general public and developments in conversational language models. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.bbc.co.uk/programmes/w3ct4khv |
| Description | CSaP Policy Leaders Fellowship Roundtable |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Spoke at a roundtable discussion to approximately a dozen leading civil servant and government workers on AI policy. It sparked significant discussion during the roundtable, the continued dinner discussion on related subject matters on the topic of: How should government respond to advances in AI technologies?. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.csap.cam.ac.uk/network/policy-fellowship/1344-ae0932d4b50d01d8a619058ddb59e5b2a394e71f/ |
| Description | European Chatbot Summit Keynote |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | A keynote talk at an industry and professional conference to primarily business leaders and practitioners in industry, given to approximately 60 people in person and 500-1000 virtually online. This sparked discussion on the future of voice technology and policy on privacy, etc... in related subject areas. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://theeuropeanchatbot.com/ |
| Description | Rework Conversational AI Summit |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | The goal of this was to communicate research advances in conversational AI to group of industry practitioners. There were approximately 100 people in the room as well as 2-300 attending online. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://london-conv-ai.re-work.co/ |
| Description | Sky News Podcast - Search Engine Wars |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | Interview on Sky News Podcast about BingGPT and the future of search https://news.sky.com/story/search-engine-wars-battle-of-the-chatbots-12807081 |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://news.sky.com/story/search-engine-wars-battle-of-the-chatbots-12807081 |