Scaling up Statistical Spoken Dialogue Systems for real user goals using automatic belief state compression

Spoken dialogue systems (SDS) are increasingly being deployed in avariety of commercial applications ranging from traditional CallCentre automation (e.g. travel information) to new ``troubleshooting''or customer self-service lines (e.g. help fixing broken internetconnections).SDS are notoriously fragile (especially to speech recognition errors),do not offer natural ease of use, and do not adapt to differentusers. One of the main problems for SDS is to maintain an accurateview of the user's goals in the conversation (e.g. find a good indianrestaurant nearby, or repair a broadband connection) underuncertainty, and thereby to compute the optimal next system dialogueaction (e.g. offer a restaurant, ask for clarification). Recentresearch in statistical spoken dialogue systems (SSDS) hassuccessfully addressed aspects of these problems but, we shall show,it is currently hamstrung by an impoverished representation of usergoals, which has been adopted to enable tractable learning withstandard techniques.In the field as a whole, currently only small and unrealistic dialogueproblems (usually less than 100 searchable entities) are tackled withstatistical learning methods, for reasons of computationaltractability.In addition, current user goal state approximations in SSDS make itimpossible to represent some plausible user goals, e.g. someone whowants to know about nearby cheap restaurants and high-quality onesfurther away. This renders dialogue management sub-optimal and makesit impossible to deal adequately with the following types of userutterance: ``I'm looking for french or italian food'' and ``NotItalian, unless it's expensive''. User utterances with negations anddisjunctions of various sorts are very natural, and exploit the fullpower of natural language input, but current SSDS are unable toprocess them adequately. Moreover, much work in dialogue systemevaluation shows that real user goals are generally sets of items withdifferent features, rather than a single item. People like to explorepossible trade offs between features of items.Our main proposal is therefore to: a) develop realistic large-scale SSDS with an accurate, extended representation of user goals, and b) to use new Automatic Belief Compression (ABC) techniques to plan over the large state spaces thus generated.Techniques such as Value-Directed Compression demonstrate thatcompressible structure can be found automatically in the SSDS domain(for example compressing a test problem of 433 states to 31 basisfunctions).These techniques have their roots in methods for handling the largestate spaces required for robust robot navigation in realenvironments, and may lead to breakthroughs in the development ofrobust, efficient, and natural human-computer dialogue systems, withthe potential to radically improve the state-of-the-art in dialoguemanagement.