Generation Challenges 2011: Towards a Surface Realisation Shared Task

Computers can now perform some writing tasks well (e.g. transcription and spell checking), but we still lack goodcomputational solutions for writing tasks which involve the creation of new text (as opposed to the typing orchecking of existing text). Natural language generation (NLG) is the branch of computer science that aims toaddress this lack, by developing methods and tools for the computational generation of spoken and writtenlanguage. NLG technology has a vast range of potential applications, including increasing the efficiency oftext-production processes (e.g. automated letter and report writing) and making information available in verbal formthat would otherwise be inaccessible (e.g. to the blind) or more time-consuming to process (e.g. converting weatherdata to a textual summary). However, NLG is only just beginning to fulfil this potential. Among the reasons is thefact that NLG did not until recently employ comparative forms of evaluation, as are essential for effectivecomparison of alternative approaches, consolidation and collective scientific progress.The NLG field's evaluation tradition lies in user-oriented and task-based evaluation of complete systems.This tradition is very different from the comparative evaluation paradigms that are predominant in otherareas of Natural Language Processing (NLP) where shared data resources, intrinsic, automatically computedmetrics, and human ratings of quality provide time-efficient and low-cost ways of comparing new systems andtechniques against existing approaches. In contrast, in NLG, until a few years ago, there simply was no comparative evaluation of independently developed alternative approaches. Yet without comparative evaluation there can be noconsolidation or collective progress in a field of research, and individual researchers and groups are left toprogress more or less separately. The GenChal initiative has firmly established comparative evaluation in NLG andproduced data sets and software tools to support it. Past shared tasks have addressed the subfield of referencegeneration and specific applications, and the time is now right to tackle a more ambitious challenge.With the Surface Realisation Task that forms the core of the present proposal, we are aiming for something trulygroundbreaking and of great potential use in practical applications: the development of a new generation of surfacerealisers that can be directly compared and, because they work from common input, can be substituted for eachother. Ultimately, this will mean that data-to-text generation, MT, summarisation and dialogue systems (amongother fields) will directly benefit from the availability of a range of reusable realisation components which systembuilders can test to determine which is best for their purpose, something that has not been possible before.

Computers can now perform certain writing tasks well (e.g. transcription and spell checking), but we still lack computational solutions for writing tasks which involve the creation of new text (as opposed to the typing or checking of given text). Natural language generation (NLG) is the branch of computer science that aims to address this lack, by developing methods and tools for the computational generation of spoken and written language. NLG can increase the efficiency of text-production processes (e.g. automated letter and report writing) and make information available in verbal form that would otherwise be inaccessible (e.g. to the blind) or more time-consuming to process (e.g. converting weather data to a textual summary). NLG technology has relevance to the EPSRC's Digital Economy and Assisted Living/Lifelong Health themes, and the number of potential applications is vast. However, NLG has so far not lived up to its great potential. Among the reasons for this is that until recently, NLG lacked comparative evaluation, hence consolidation of research results, and was isolated from the rest of NLP, therefore not directly benefiting from advances there. It was shrinking as a field and lacked the kind of funding and participation that natural language analysis (NLA) fields have attracted. In order to ultimately fulfill its great potential (including commercial applications), NLG needs to achieve substantial technological progress. For this to be possible, NLG needs to (i) increase available reusable and trainable NLG tools and data resources; (ii) establish comparative evaluation as standard in order to consolidate and incrementally improve research results; (iii) to increase the field's critical mass (in terms of numbers of researchers, projects and events); and (iv) bridge to neighbouring disciplines where language is generated (to benefit from technological advances and available resources there). These very considerable impacts are our overarching goals in the Generation Challenges (GenChal) initiative. With the Surface Realisation Task that forms the core of the present proposal, we are aiming for a truly groundbreaking impact that is of great potential use in practical applications: the development of a new generation of surface realisers that can be directly compared and, because they work from common input, can be substituted for each other. Ultimately, this will mean that data-to-text generation, MT, summarisation and dialogue systems (among other fields) will directly benefit from the availability of a range of reusable realisation components which system builders can test to determine which is best for their purpose, something that has not been possible before.