Generation Challenges 2009

Lead Research Organisation: University of Brighton
Department Name: Sch of Computing, Engineering & Maths


Natural Language Generation (NLG) is the subfield of Natural Language Processing (NLP) that is concerned with developing computational methods for automatically producing language. Possible applications of NLG technology include economising text-production processes and improving access to non-verbal information, as well as machine translation, text summarisation and human-computer dialogue. NLG is a field with vast but so far largely unrealised potential. In other NLP fields, shared data, shared core technology and organised shared-task competitions have been seen to galvanise research communities, and lead to rapid technological progress. Among NLG researchers, growing interest in comparative forms of evaluation led to a discussion which has now resulted in the first two shared-task evaluation competitions in NLG: in 2007, we organised a pilot NLG shared-task evaluation event, the Attribute Selection for Generating Referring Expressions (ASGRE'07) Challenge; this year we organised the Referring Expression Generation Challenge (REG'08) which saw participation double and was co-located with the leading international conference in NLG (INLG'08 in Columbus, Ohio). ASGRE'07 and REG'08 have met met with enthusiasm among NLG researchers, have resulted in the creation of new technology and data resources, and have drawn new researchers into the field. In order to continue and increase these beneficial effects, we are now planning a third NLG evaluation event, Generation Challenges, for 2009, in conjunction with the leading European NLG conference (ENLG'09). Generation Challenges 2009 will have more shared tasks and a larger variety of evaluation methodologies than the two previous events. It also includes a call for evaluation methods and, for the first time, a general call for task proposals, enabling the NLG research community to directly shape future shared-task events.Unlike leading evaluation initiatives in Machine Translation and Document Summarisation, which are funded and directed by US government agencies, ASGRE'07, REG'08, and now Generation Challenges 2009 arecommunity-based, UK-led evaluation initiatives. This proposal requests funding for umbrella organisation and administration activities, as well as data preparation and evaluation experiments for two of the shared tasks in Generation Challenges 2009, to enable us to carry out the full range of planned activities, and to keep this initiative community-based and UK-led.