SMART - Shaping Multilingual Access through Respeaking Technology
Lead Research Organisation:
University of Surrey
Department Name: Languages and Translation Studies
Abstract
Our day and age is characterised by a proliferation of live multimedia and multilingual content, such as news and TV programmes. Such content is not accessible to everyone. In many countries, live subtitles in the same language are required by law to enable deaf and hard-of-hearing audiences to enjoy access to information, culture and entertainment. In the UK and other countries (e.g. Canada, Spain, Switzerland), intralingual respeaking is the most well-established technique to produce these subtitles: it relies on human-machine interaction whereby a respeaker listens to the original sound of a live programme or event and simultaneously dictates it to a speech recognition software that turns the each sentence into subtitles displayed on screen.
Respeaking so far has been used to produce intralingual subtitles, i.e. in the same language. Given the current multilingual content boom, the SMART project aims to investigate whether respeaking can be used to produce interlingual subtitles, i.e. in a different language. This entails adding translation to an already challenging task which includes listening, speaking, adding oral punctuation, controlling prosody to minimise speech recognition errors, and accounting for space constraints to produce meaningful subtitles. Automatic translation alone is unable to provide live subtitles in a different language of sufficient quality. Interlingual respeaking (IRSP) represents therefore an innovative technique that builds on its monolingual predecessor to achieve this goal. This technique, however promising, has not been systematically tested yet. Therefore there is a need for empirical data to assess its viability. To this end, SMART sets out to achieve three main goals:
1. Investigate the feasibility of IRSP by analysing its challenges. Participants' performance during IRSP tasks will be studied in detail to inform optimal strategies to overcome these challenges. Two pilot projects on students have yielded promising results in this respect. We will need to explore these further in a sample of professionals;
2. Measure the quality of the live subtitles thus produced using an existing, recently-developed measurement tool (NTR model) which could represent a first step towards establishing a quality benchmark for IRSP;
3. Identify the key competencies needed by professionals already working in the language industry (e.g. interpreting, subtitling) to support timely and efficient acquisition of IRSP skills.
IRSP has the potential to provide interlingual subtitles of virtually any live events, such as lectures, conferences, theatre shows, business meetings, live TV interviews or programmes involving speakers of different languages. IRSP could also enhance traditional subtitling methods for pre-recorded programmes, improving productivity. By doing so, IRSP would make such events and programmes accessible to everyone, irrespective of any sensory or language barriers.
SMART research findings will inform the design of bespoke training courses to equip language professionals with optimal skills to ensure high-quality live interlingual subtitling. We will test these materials in a Summer School during the project; after its end, we will continue to offer training that will provide language professionals with a new career path and the possibility to diversify the services they offer. A public engagement event and informative video will raise awareness of IRSP among the wider public and any other stakeholders interested in providing or benefiting from this service.
The UK pioneered the development of intralingual respeaking (today used to subtitle 85% of live TV programmes). SMART will provide the data needed for the UK to become a world leader in IRSP. IRSP is a radical evolution in respeaking methodology and a smart move towards meeting current communication needs, whilst contributing to 'barrier-free' access to different domains of public and private life for all members of society.
Respeaking so far has been used to produce intralingual subtitles, i.e. in the same language. Given the current multilingual content boom, the SMART project aims to investigate whether respeaking can be used to produce interlingual subtitles, i.e. in a different language. This entails adding translation to an already challenging task which includes listening, speaking, adding oral punctuation, controlling prosody to minimise speech recognition errors, and accounting for space constraints to produce meaningful subtitles. Automatic translation alone is unable to provide live subtitles in a different language of sufficient quality. Interlingual respeaking (IRSP) represents therefore an innovative technique that builds on its monolingual predecessor to achieve this goal. This technique, however promising, has not been systematically tested yet. Therefore there is a need for empirical data to assess its viability. To this end, SMART sets out to achieve three main goals:
1. Investigate the feasibility of IRSP by analysing its challenges. Participants' performance during IRSP tasks will be studied in detail to inform optimal strategies to overcome these challenges. Two pilot projects on students have yielded promising results in this respect. We will need to explore these further in a sample of professionals;
2. Measure the quality of the live subtitles thus produced using an existing, recently-developed measurement tool (NTR model) which could represent a first step towards establishing a quality benchmark for IRSP;
3. Identify the key competencies needed by professionals already working in the language industry (e.g. interpreting, subtitling) to support timely and efficient acquisition of IRSP skills.
IRSP has the potential to provide interlingual subtitles of virtually any live events, such as lectures, conferences, theatre shows, business meetings, live TV interviews or programmes involving speakers of different languages. IRSP could also enhance traditional subtitling methods for pre-recorded programmes, improving productivity. By doing so, IRSP would make such events and programmes accessible to everyone, irrespective of any sensory or language barriers.
SMART research findings will inform the design of bespoke training courses to equip language professionals with optimal skills to ensure high-quality live interlingual subtitling. We will test these materials in a Summer School during the project; after its end, we will continue to offer training that will provide language professionals with a new career path and the possibility to diversify the services they offer. A public engagement event and informative video will raise awareness of IRSP among the wider public and any other stakeholders interested in providing or benefiting from this service.
The UK pioneered the development of intralingual respeaking (today used to subtitle 85% of live TV programmes). SMART will provide the data needed for the UK to become a world leader in IRSP. IRSP is a radical evolution in respeaking methodology and a smart move towards meeting current communication needs, whilst contributing to 'barrier-free' access to different domains of public and private life for all members of society.
Planned Impact
SMART is a project with a high socio-economic impact potential, namely by:
1. Demonstrating the viability of interlingual respeaking (IRSP) as a technique. This will allow access service providers to increase productivity while ensuring quality;
2. Demonstrating how IRSP can facilitate access to, and reuse of, multilingual content, thus fulfilling a social mission, broadening the concept of accessibility and contributing to a 'barrier-free' society;
3. Developing a sustainable and scalable model for IRSP skills acquisition that can contribute to capacity building for access service providers and career development for language professionals;
4. Recommending guidelines for effective and efficient IRSP use that account for different contexts and stakeholders' needs.
SMART aims to consolidate the UK's reputation as a pioneer in respeaking. SMART outcomes will benefit different stakeholder groups, including: access service providers, language industry professionals and professional bodies, members of the general public (especially hearing-impaired individuals and people who cannot access audiovisual content in its original language), third sector, governmental and international organisations. All these groups will contribute to and benefit from the project as follows.
Three access service providers will be involved from the outset as non-academic Advisory Board members and co-creators of knowledge. This will ensure that they are fully consulted throughout every project phase so that SMART goals are consistent with industry needs. This group will also actively contribute to the dissemination of findings and to debates about the scope and implementation of IRSP in different domains of public and private life, for instance through participation in the end-of-project symposium alongside other stakeholder groups (see below). In turn, they will benefit from insights into IRSP skills and how training can be optimised to ensure high quality live subtitling.
Language industry professionals will play an integral role in the project by participating in its experiments and impact activities, i.e. in the Summer School as main recipients of IRSP training, and in a public engagement event to provide a live IRSP demo. In turn, they will benefit from the potential of IRSP to open up a new career path. SMART can build on existing ties with professional bodies to reach this stakeholder group, and on support from access service providers, who are committed to identifying relevant profiles among their staff members (see Letters of support).
At a broader level, SMART will generate significant new insights that will inform and shape the debate about how IRSP can be integrated within current policies, services and processes concerned with the nature, cost and efficiency of accessibility to information, entertainment and culture. To this end, SMART will create opportunities for dialogue with professional bodies, third sector and governmental organisations, for instance through the end-of-project symposium. Direct contact with these groups will be possible through existing investigators' links based on previous collaboration (e.g. with the British Film Institute, charities about hearing loss and migrant rights, Ofcom, United Nations International Telecommunication Union). New links will be sought during the project life. Members of the general public will have the opportunity to experience IRSP-produced subtitles live during a public engagement event at a cultural venue in London and will provide feedback from their user perspective.
Other means for generating impact are included in a carefully planned set of media engagement activities, for instance a project website with integrated social media feed, radio interviews/podcasts on FRED Film Radio and presence on key online platforms (e.g. Media Accessibility Platform). For further details, please see the Pathways to Impact.
1. Demonstrating the viability of interlingual respeaking (IRSP) as a technique. This will allow access service providers to increase productivity while ensuring quality;
2. Demonstrating how IRSP can facilitate access to, and reuse of, multilingual content, thus fulfilling a social mission, broadening the concept of accessibility and contributing to a 'barrier-free' society;
3. Developing a sustainable and scalable model for IRSP skills acquisition that can contribute to capacity building for access service providers and career development for language professionals;
4. Recommending guidelines for effective and efficient IRSP use that account for different contexts and stakeholders' needs.
SMART aims to consolidate the UK's reputation as a pioneer in respeaking. SMART outcomes will benefit different stakeholder groups, including: access service providers, language industry professionals and professional bodies, members of the general public (especially hearing-impaired individuals and people who cannot access audiovisual content in its original language), third sector, governmental and international organisations. All these groups will contribute to and benefit from the project as follows.
Three access service providers will be involved from the outset as non-academic Advisory Board members and co-creators of knowledge. This will ensure that they are fully consulted throughout every project phase so that SMART goals are consistent with industry needs. This group will also actively contribute to the dissemination of findings and to debates about the scope and implementation of IRSP in different domains of public and private life, for instance through participation in the end-of-project symposium alongside other stakeholder groups (see below). In turn, they will benefit from insights into IRSP skills and how training can be optimised to ensure high quality live subtitling.
Language industry professionals will play an integral role in the project by participating in its experiments and impact activities, i.e. in the Summer School as main recipients of IRSP training, and in a public engagement event to provide a live IRSP demo. In turn, they will benefit from the potential of IRSP to open up a new career path. SMART can build on existing ties with professional bodies to reach this stakeholder group, and on support from access service providers, who are committed to identifying relevant profiles among their staff members (see Letters of support).
At a broader level, SMART will generate significant new insights that will inform and shape the debate about how IRSP can be integrated within current policies, services and processes concerned with the nature, cost and efficiency of accessibility to information, entertainment and culture. To this end, SMART will create opportunities for dialogue with professional bodies, third sector and governmental organisations, for instance through the end-of-project symposium. Direct contact with these groups will be possible through existing investigators' links based on previous collaboration (e.g. with the British Film Institute, charities about hearing loss and migrant rights, Ofcom, United Nations International Telecommunication Union). New links will be sought during the project life. Members of the general public will have the opportunity to experience IRSP-produced subtitles live during a public engagement event at a cultural venue in London and will provide feedback from their user perspective.
Other means for generating impact are included in a carefully planned set of media engagement activities, for instance a project website with integrated social media feed, radio interviews/podcasts on FRED Film Radio and presence on key online platforms (e.g. Media Accessibility Platform). For further details, please see the Pathways to Impact.
Publications
Davitti E
(2020)
Embracing the Complexity: A Pilot Study on Interlingual Respeaking
in Journal of Audiovisual Translation
Davitti Elena
(2023)
Shaping Multilingual Access Through Respeaking Technology - Project poster
Davitti, E.
(2021)
Automating the respeaking workflow: Hype or reality?
| Description | With AI-related technologies redesigning the landscape of language-related professions, SMART identified the human (procedural, cognitive, interpersonal) competences needed by the linguists at the core of a novel form of human-AI interaction for accessible multilingual communication, namely real-time interlingual speech-to-text via speech recognition, which challenges traditional ways of working while creating new upskilling opportunities. The study successfully achieved all its goals by (1) developing and implementing an innovative and exploratory methodological design to study this emerging practice; (2) providing new, empirical insights into its process and product and identifying key factors that influence its performance. The findings of the study also (3) carry significant implications for enhancing the upskilling of language professionals and promoting greater accessibility in multilingual communication; and (4) they serve as a benchmark for future studies that seek to compare the effectiveness and impact of semi-automated workflows with the human-centric approach examined in this research. SMART utilised an exploratory, mixed-methods approach integrating methods from translation and multimodal studies, cognitive psychology, linguistics and social science to collect quantitative and qualitative data on the competences required to perform this real-time practice and explore the impact of different variables (e.g. professional background, age, cognitive abilities, interpersonal traits). The study was conducted entirely online due to the pandemic and involved a multi-stage experiment with a purpose-made five-week upskilling-for-testing course (tot. 25h) at its core. The course allowed the collection of data on performance and perception during skills acquisition and testing, with an unprecedented battery of scales, tasks, and two surveys delivered before and after the course to collect data on participants' cognitive and interpersonal skills, profiling, and satisfaction. The study attracted over 250 people, with 51 participants ultimately selected for inclusion in the final study. The study yielded new insights into three key aspects of this emerging practice: its process, product, and upskilling. In relation to the process, the study explored the human factors underpinning the performance of language professionals. For instance, it analysed the impact of the upskilling course on cognitive abilities, with a particular focus on executive functions (EF), including working memory (WM), shifting skills, and sustained attention. The findings revealed that attending the upskilling course improved WM and shifting skills, highlighting their significance in this practice. However, sustained attention did not show significant enhancement, suggesting the need to explore alternative forms of attention. WM was also found to be a positive predictor of high performance, while conscientiousness and integrated regulation were negative predictors of accuracy. These robust findings were achieved via statistical methods and triangulated with qualitative ones, e.g. from Think-Aloud protocols and narratives based on participants' self-perception, which helped to corroborate the results. The accuracy of the respoken subtitles, which constitute the product of this speech-to-text practice, was measured through a specialised model that accounts for all shifts between the source text and target text. This was triangulated with in intelligibility scale for identifying high and low performers groups. The average score across all participants in all three testing scenarios (highlighting three key conditions: speed, planned/unplanned delivery and multiple speakers) was 95.37%, with higher scores achieved when respeaking into English than romance languages. Among the scenarios, speed posed the greatest challenge, and omissions were the strongest (negative) predictor of accuracy, followed by substitutions and misrecognitions. Effective editions, which involved changes between the source and target text without any loss of information, were significant positive predictors of accuracy across all scenarios. Through reflection points delivered during the remote upskilling course and an evaluation survey, the study identified key challenges that emerged during skills acquisition. Our modular approach proved successful, allowing us to monitor skills acquisition at a granular level. For instance, mental workload showed a statistically significant increase in interlingual compared to intralingual upskilling. Error correction and interaction with software were among the key challenges highlighted by participants. These findings are important for optimizing upskilling and tailoring it to participants' diverse backgrounds, possibly through a format that allows them to choose their own pathway. Although our upskilling course duration led to solid skills development, additional guidance on independent practice and personalized feedback could be helpful, as demonstrated in our in-person Summer School (September 2022, https://smartproject.surrey.ac.uk/2022/10/01/smart-summer-school-in-interlingual-respeaking-for-language-professionals/). For further insights into our findings, see: • SMART poster with key findings from the project (2023): DOI: 10.13140/RG.2.2.30610.12481 Available at: https://smartproject.surrey.ac.uk/2023/05/11/smart-project-poster-with-key-findings-from-the-project/ • SMART poster on key cognitive findings "Human-Computer Interaction (HCI) and Cognition in Multilingual Communication" (2022): DOI: 10.13140/RG.2.2.12526.08002 - Available at : https://www.researchgate.net/publication/363295814_Human-Computer_Interaction_HCI_and_Cognition_in_Multilingual_Communication • ReShare dataset: Davitti, E. & Wallinheimo, A.-S. (2024) Shaping Multilingual Access through Respeaking Technology, Project Data, 2021, In: Training in new forms of human-AI interaction improves complex working memory and switching skills of language professionals UK Data Service - DOI: 10.5255/ukda-sn-856687 Available at: https://reshare.ukdataservice.ac.uk/856687/ • Please see also related publications added to the relevant sections and updated over time. |
| Exploitation Route | The study has shown how language-related skills coupled with technology can enhance the inclusivity and accessibility of broadcast and live events and broadcasts beyond sensory, linguistic and cognitive barriers. In terms of academic research, the amount of data collected in SMART (beyond the initial scope) and the findings obtained have opened up new research avenues which are currently being pursued by the research team. A PhD project has recently (2023) been funded by Surrey to pursue the following study: "Exploring Human-AI Interaction for Real-Time Speech-to-Text: A Comparative Process- and Product-Oriented Analysis", which aims to analyse the intralingual dataset collected by SMART (beyond the scope of the initial research questions) and compare the findings with interlingual ones delivered by the project. This is a fundamental follow-up study which will shed even further light on matters related to benchmarking, output accuracy and adaptability of the human to this new form of HAII. Other important research questions opened up by SMART aim to comparing the practice analysed in the project with other semi- or fully-automated workflows. SMART laid the conceptual foundation for this by exploring the most human-centric form of live subtitling via SR and placing it on a continuum with other related hybrid practice (see short dissemination film on live interlingual subtitling workflows - https://www.youtube.com/watch?v=rxIRKLR2_7o). A related research project called SAPPHIRE (Semi-Automated Practices and Processes in Human-AI Interaction Research) is currently being submitted and under review with the ESRC. In relation to non-academic routes, broadcasters, access, and language service providers have shown interest in implementing this practice within their operations, with several project meetings and industry panels discussing the findings (see dissemination and engagement activities list). These stakeholders were primarily interested in understanding the achievable level of accuracy, the opportunities for upskilling professionals from diverse backgrounds, and the added value of human intervention in a technology-driven language sector. The success of the practice is evident from the interest shown by two industry members of the Advisory Board, who sent their staff to be trained in the Summer School held in September 2022 (https://smartproject.surrey.ac.uk/2022/10/01/smart-summer-school-in-interlingual-respeaking-for-language-professionals/). The modularised upskilling model developed is currently being refined and finetuned on the basis of research findings and stakeholders' input into bespoke continuous professional development (CPD) courses within the framework of a follow-up project called SMART-UP, funded by the ESRC Impact Acceleration Account (2023-2024, https://www.surrey.ac.uk/research-projects/smart-shaping-multilingual-access-through-respeaking-technology-upskilling). Its main goal is to turn the course prototype developed by SMART into a fully-fledged, adaptable and customisable CPD model catering to the diverse needs and professional backgrounds of professionals. Interest in hybrid practices for real-time interlingual speech-to-text has extended beyond the study, including professional associations such as ITI and CIOL and important industry outlets such as Slator Langauge Industry Intelligence. SMART team members have already delivered tasters and short workshops and have been invited to write pieces for professional magazines and bulletins (see relevant sections where these outputs are reported in detail). Given the significant interest in this practice, with over 250 people applying to the course in its first run, the next step is to tailor upskilling and promote it more formally among language professionals through these channels. By doing so, the potential of technology and language skills can be harnessed to increase the inclusivity and accessibility of various forms of content, from live events to entertainment, culture, information, arts, and digital media, and to break down the barriers that prevent some individuals from fully participating in these experiences. |
| Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Education Leisure Activities including Sports Recreation and Tourism Government Democracy and Justice Culture Heritage Museums and Collections |
| URL | https://smartproject.surrey.ac.uk/2023/05/11/smart-project-poster-with-key-findings-from-the-project/ |
| Description | The proliferation of live multimedia content from around the world enhances the demand for it to be enjoyed by a wide audience, across languages and settings. The SMART project has developed an innovative upskilling course and an extensive conceptual and methodological knowledge-base on interlingual respeaking, an emerging method real-time speech-to-text combining advanced human language-transfer skills with speech recognition (SR) to produce high-quality subtitles in real-time. The project also informs the debate on achieving responsible human-machine integration in multilingual communication. Our research has delivered all promised outputs, and beyond, thus meeting the objectives set in the Pathways to impact. These will be reported below according to the original headings. Co-Creation of knowledge with access service providers To ensure the relevance and significance of our roadmap to impact, we engaged with a range of stakeholders who provided essential information on the state of the industry and current needs to be addressed by research. Access service providers (Ai-Media, SUB-TI), broadcasters (Sky), and professional respeakers were all directly involved in the co-creation of the research. Despite the disruption caused by Covid-19, the SMART project conducted a series of in-depth virtual expert meetings with stakeholders during the first trimester of the project. These meetings were instrumental in collecting relevant information about service needs, operations, and hiring procedures, which informed the design of the experiment and information on what is involved in the career, what skills and competencies are most needed, and what training they would welcome, which strongly informed the design of the upskilling course. We also held project meetings at key milestones in the project (28 February 2022, 13 May 2022) to obtain feedback on interim results and steer our next steps. The same stakeholders took an active part in the project's Summer School for Language Professionals (September 2022) by sending staff to be upskilled during the course; and in the last engagement event - dedicated panel within Convergence Conference "Embracing the complexity: hybrid practices for interlingual communication in real time" (February 2023). This approach allowed for full engagement with stakeholders throughout the course of the project and ensured true co-creation of different outputs. Since then, access service provider SUB-TI, the broadcaster SKY and a new partner, langauge service provider ONCALL, have joined the SMART-UP project, funded by the ESRC Impact Acceleration Account to further refine and commercialised a flexible, modular and tailored CPD course for language professionals (2023-24). Professional Development and capacity building activities SMART has so far delivered five 25h tailored upskilling courses for linguists in two formats: a five-week online course via dedicated platform (run four times between April-September 2021, 250+ applications, 65 selected) and a face-to-face three-day intensive Summer School (23-25 September 2022, 40+ applications, 15 selected). Both covered six language combinations (from English into Spanish/French/Italian and vice-versa) and provided free licenses to industry-standard SR software. The former "upskilling -for-testing" delivered remotely was at the core of the experimental part of our work and launched on 5th April 2021. This course enabled us to collect data on the human variables that are required as this technique is learned and used professionally. The course was delivered as part of a broader experimental set-up delivered online due to the pandemic. While on the one hand this required additional time and technical expertise in setting up and integrating different platform for the remote delivery of the experiment (namely Moodle for the training-for-testing course, Pavlovia for the cognitive battery and Qualtrics for the survey), on the other hand, it broadened significantly the scope of the project, with participants potentially joining from anywhere in the world. The number of applications received (from UK, Spain, Italy, France, Austria, Germany, Belgium, Australia, Argentina, New Zealand, USA, Peru) is testament to the strong demand for training in this novel practice. Participants found the course "incredibly stimulating" and "exceeding their expectations", praised its "fantastic structure making it possible to improve along the way", "innovative tools" and "excellent resources". It provided a "truly unparalleled opportunity" "for linguists interested in expanding and consolidating their portfolios of services" to remain relevant in a rapidly evolving industry, while exploring new ways to fulfil a social mission. As their participation contributed to research, they also took an active role in shaping this practice. We will report on how they have used the skills acquired and how findings are helping up share SMART-UP CPD models in the next round. Media engagement activities The SMART project took a multi-pronged approach to raising awareness and engagement in interlingual respeaking. We have engaged in radio interviews and podcasts with the aim to raise awareness of this practice across various sectors of the language industry. Dr Davitti (PI on SMART) was invited to the Troublesome Terps podcast (https://www.troubleterps.com/), a popular podcast in the field of interpreting and translation, followed by many professionals, academics and industry stakeholders in the language and multilingual communication field. The episode about "New ways of working" (https://www.troubleterps.com/55) focused on the technique we are exploring in SMART and as a result of this initiative, we have shortlisted some participants for the experiment and sparked interest among different institutions in the course we are developing. In 2022, the PI Davitti was also interviewed for a piece on an important professional outlet in the language industry world - Slator Language Industry Intelligence (https://slator.com/what-is-respeaking-how-does-it-improve-multilingual-subtitles/). This was followed by two more requests for interviews by the same outlet, namely Slator Language Industry Intelligence, for the 2023 Slator Pro Guide: Subtitling and Captioning and 2024 Interpreting, Technology and AI report (with Korybski). Other media engagement activities involve Davitti and Sandrelli invited to high-profile outreach event INCinema - the first Italian Accessible Film Festival organised by industry partner SUB-TI (2023 - press coverage ranging from Hollywood Reporter to La Nazione and other national and international outlets). On that occasion, we screened the short film on hybrid practices developed within the framework of SMART to a wide audience. Additionally, the project website, which launched in March 2021 (https://smartproject.surrey.ac.uk/) is hosting information about the project, its outputs and a social media feed, and has served as a platform for participants recruitment and dissemination. It will continue to grow organically as a hub of information for interested stakeholders. It provides a Subscribe function to link to a mailing list, as well as a form for participant recruitment that proved very successful to attract language professionals with a suitable background to take part in our upskilling course, 'An Advanced Introduction to Interlingual Respeaking' (i.e. our upskilling-for-testing prototype). The experimental stage of SMART thus had a direct impact on language professionals, the work they do and their understanding of why different forms of access are needed and the ways in which they can be provided, all of this in addition to providing them with an opportunity to acquire new skills to diversify their portfolio of services. We are also in the process of finalising a short film on hybrid practices for speech-to-text in real time and across languages that will be used to raise awareness raising and dissemination purposes on the website. It was shown for the first time at the Convergence Conference in February 2023, attended by over 100 people from academia and industry alike. Overall, these efforts have had a direct impact on language professionals and their understanding of the importance of different forms of access and the ways in which they can be provided. Further indicators of uptakes will be reported in the next cycle. |
| First Year Of Impact | 2022 |
| Sector | Other |
| Impact Types | Cultural Societal Economic |
| Description | Upskilling for language professionals |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | Participants found the course "incredibly stimulating" and "exceeding their expectations", praised its "fantastic structure making it possible to improve along the way", "innovative tools" and "excellent resources". It provided a "truly unparalleled opportunity" "for linguists interested in expanding and consolidating their portfolios of services" to remain relevant in a rapidly evolving industry, while exploring new ways to fulfil a social mission. As their participation contributed to research, they also took an active role in shaping this practice. By partnering with key stakeholders including broadcasters and subtitling providers, we are currently building on the success of SMART with the new project SMART-UP (ESRC Impact Acceleration Account). It will build on the 'upskilling-for-testing' course prototype developed by the SMART team, with the aim of turning this into a fully-fledged, adaptable and customisable continuous professional development (CPD) model. The long term goal is to upskill language professionals to reduce the current skills barrier that prevents widespread adoption of live interlingual subtitling across the globe. |
| Description | Post-doctoral Enrichment Award |
| Amount | £2,000 (GBP) |
| Organisation | Alan Turing Institute |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 01/2022 |
| End | 10/2022 |
| Description | SMART-UP: Shaping Multilingual Access through Respeaking Technology - Upskilling |
| Amount | £20,000 (GBP) |
| Organisation | Economic and Social Research Council |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2023 |
| End | 11/2024 |
| Title | Shaping Multilingual Access through Respeaking Technology, Project Data, 2021 |
| Description | The recent global surge in audiovisual content has emphasized the importance of accessibility for wider audiences. The SMART project addressed this by exploring interlingual respeaking, a novel practice combining speech recognition technology with human interpreting and subtitling skills to produce real-time, high-quality speech-to-text services across languages. This method evolved from intralingual respeaking, which is widely used in broadcasting to create live subtitles for the deaf and hard-of-hearing. Interlingual respeaking, which involves translating live content into another language and subtitling it, could revolutionize subtitle production for foreign-language content, overcoming sensory and language barriers. Interlingual respeaking is defined as a type of simultaneous interpreting, producing text with minimal delay. It involves two shifts: interlingual (from one language to another) and intermodal (from spoken to written). This practice combines the challenges of simultaneous interpreting with the requirements of subtitling. Respeakers must accurately convey messages in another language to a speech recognition system, adding punctuation and making real-time edits for clarity and readability. This method leverages speech recognition technology and human translation skills to ensure efficient and high-quality translated subtitles. Interlingual respeaking offers immense potential for making multilingual content accessible to international and hearing-impaired audiences. It's particularly relevant for television, conferences, and live events. However, research into its feasibility, accuracy, and the skills required for language professionals is still in its early stages. The SMART project aimed to address these research gaps. It focused on the cognitive and interpersonal profiles needed for successful interlingual respeaking. The project extended a pilot study, including language professionals from interpreting, subtitling, translation, and intralingual respeaking, to explore how cognitive and interpersonal factors influence learning and performance in this field. The SMART project's main goals were to study interlingual respeaking's complexity, focusing on the acquisition and implementation of relevant skills, and the accuracy of the final subtitles. The research involved 23 postgraduate students with backgrounds in interpreting, subtitling, and intralingual respeaking. The research program examined three areas: process, product, and upskilling. It sought to understand the variables contributing to language professionals' performance, challenges faced during performance, and how performance can be sustained. Regarding the product, it aimed to identify factors affecting the accuracy of interlingual respeaking and the impact of various individual and content characteristics on accuracy. For upskilling, the focus was on the challenges and strengths of the training course. Key findings included the importance of working memory in predicting high performance and the enhancement of certain cognitive abilities through training. Interpersonal traits like conscientiousness and integrated regulation were also examined. In terms of product accuracy, the average was 95.37%, with omissions being the strongest negative predictor of accuracy. High performers outperformed low performers across all scenarios. The upskilling course was innovative, focusing on modular training and combining intralingual and interlingual practices. It addressed real-world challenges and was tailored to different professional backgrounds. The approach proved effective, with 82% of participants finding the course met their expectations and 86% acknowledging its challenging nature. The study confirmed the benefits of a modular and personalized training approach, highlighting the need for flexibility and adaptability to different skill levels and backgrounds. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | This is the first extensive and mixed-methods dataset ever shared for the specific domain explored. |
| URL | http://reshare.ukdataservice.ac.uk/id/eprint/856687 |
| Description | Collaboration with ILSA consortium (Interlingual Live Subtitling for Access, 2017-1-ES01-KA203-037948) |
| Organisation | University of Vigo |
| Country | Spain |
| Sector | Academic/University |
| PI Contribution | I was invited to serve as a member of the Advisory Board of this international project. In this capacity I was invited to provide critical evaluation of the project deliverables, reports and intellectual outputs (particularly IO5), and to join the closing panel of the final symposium and multiplier event as an invited speaker ('The Present and Future of Interlingual Live Subtitling:Research, Training and Practice', 6 July 2020 - see Engagement Activities). |
| Collaborator Contribution | Some of the members of the international ILSA consortium are contributing to the SMART project in various capacities: Dr Pablo Romero-Fresco as International Co-I, Prof Aline Remael and Prof Franz Pöchhacker as members of the Advisory Board. Their insights and expertise have contributed to developing the project conceptually and ensuring that it continues to address current research needs via participation in project meetings and review of (interim) deliverables. |
| Impact | Invited speaker at 'The Present and Future of Interlingual Live Subtitling: Research, Training and Practice', 6 July 2020 (see Engagement Activities) |
| Start Year | 2020 |
| Description | 'Exploring new workflows with respeaking for real-time interlingual communication: from human-centric to human-in-the-loop', Translating Europe Workshop - 'Artificial intelligence and translation technologies: what is the state of play?' |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The PI Davitti and PDR Korybski were invited speakers at the Translating Europe Workshop "Artificial intelligence and translation technologies: what is the state of play?" (15 January 2021), which is part of the Translating Europe project, created to bring together translation stakeholders in Europe. This event attracted interest from a variety of academic and non-academic stakeholders and increased the visibility of the project, as well as attracted language professionals to our training-for-testing course. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://ec.europa.eu/info/events/translatingeurope-workshops/artificial-intelligence-and-translation... |
| Description | 'Human-centric approaches to translation and interpreting research at CTS@Surrey', ITI (Institute of Translation and Interpreting) research network event 'How research impacts upon the Translation and Interpreting professions' |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The PI Davitti was invited to present SMART to the ITI (Institute of Translation and Interpreting) community, through a talk "How research impacts on the translation and interpreting profession" (Research Network Event, 26 June 2020). The event proved very successful to raise awareness of the hybrid human-centric practice of interlingual respeaking, which represents an opportunity for language professionals to upskill and remain relevant in the language industry. Engagement with one of the most important professional associations in our field has led to increased visibility of our research and we are aiming to secure their endorsement for the Summer School to be organised in the last year of the project. We will report on this further in the next cycle. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.iti.org.uk/resource/how-research-impacts-on-translation-interpreting-professions.html |
| Description | 'Human-centric workflows for live subtitling via speech recognition: interlingual respeaking and the SMART project', EMT Train the Trainer Summer School 'Technologies and practices in AVT and media accessibility training' |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The PI Davitti and International Co-I Sandrelli were invited to take part in a Translating Europe Workshop 2021 'From Translation to Accessibility: EMT Train the Trainer Summer school. Technologies and practices in AVT and media accessibility training' (Constantine the Philosopher University in Nitra, Slovakia, 4th June 2021). This was the first cross-borded workshop of a series and provided a framework to share expertise and good practice in audiovisual translation and media accessibility training across Europe, with a main focus on training strategies, resources and integration of technologies. The workshop aimed to reflect on practices in selected environments and provide an opportunity to trainers in academia and the public sector to enhance their current skills in training as well as encourage curriculum development and innovation. It sparked interest in this new practice and led to visibility and further involvement of language professionals in our research. |
| Year(s) Of Engagement Activity | 2021 |
| URL | http://www.ktr.ff.ukf.sk/index.php/en/events/summer-school-of-audiovisual-translation/from-translati... |
| Description | Embracing the complexity: hybrid practices for interlingual communication in real time |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | In 2023, we organised an industry-academia panel on hybrid practices for real-time interlingual communication within the Convergence Conference at the Centre for Translation Studies at Surrey (280+ registered participants, cited in Slator industry intelligence reports). The discussion was focused on speech-to-text practices, which are normally associated with full automation of spoken communication (i.e. technology-driven transformation of spoken input into written output, particularly in real-time). However, the rapid development of technologies driven by artificial intelligence, like automatic speech recognition and machine translation, has opened up a myriad of hybrid methods and possible configurations for real-time delivery of speech-to-text. These can be positioned at the crossroads of translation, interpreting, and subtitling and entail different degrees of human and machine input to provide what is ultimately the same product: live text in a different language. However, we still do not know much about the affordances and constraints brought about by different workflows, and how they can match different live communication scenarios. This panel discussed the current state of the art in terms of speech-to-text interlingual practices, attempting to provide a clear and comprehensive overview of what these practices are and where they can be placed on a continuum from human-centric to automated ones. This serves as a springboard to capture and systematise the complexity of the current landscape, in a way that emphasises the hybrid nature of these practices and the role that language professionals play in different processes. The panel brought together experts from a variety of backgrounds, attempting to converge on a topical and multifaceted issue that requires input from a variety of standpoints to be addressed in all its complexity. Invited panellists used their own unique perspectives (institutional, professional, academic, industry- and user-centred) to discuss how we can bring about and capitalise on the intelligent and responsible integration of human skills and technological capabilities for the benefit of all stakeholders involved. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.surrey.ac.uk/centre-translation-studies/convergence-2023/embracing-complexity |
| Description | High-profile outreach events - INCinema - the first Italian Accessible Film Festival (organised by industry partner SUB-TI) |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | Davitti and Sandrelli were invited to take part in a roundtable, with press and the general public, as a launch event for the "InCINEMA - Festival del cinema inclusivo" (InCINEMA - Inclusive cinema festival), the first Italian accessible cinema festival, an initiative by SUB-TI (one of the industry stakeholders in SMART) and MyMOVIES, in collaboration with Fondazione Sistema Toscana and FRED Film Radio. During this round table, Federico Spoletti (SUB-TI CEO), Annalisa Sandrelli (International Co-Investigator on SMART) and Elena discussed the concept of accessibility in relation to audiovisual content and provided an overview of research on this topic in Italy and Europe, before the accessible screening of a short film. (30 June 2023, Cinema La Compagnia, Florence). On this occasion, we also screened the SMART-produced brand new video clip explaining the different workflows for live interlingual subtitling via speech recognition, one of the outputs of the project (available at: https://www.youtube.com/watch?v=rxIRKLR2_7o). |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.hollywoodreporter.it/film/festival-e-premi/nasce-a-firenze-il-primo-festival-del-cinema-... |
| Description | Interview with Slator Language Industry intelligence for 'SLATOR 2024 Interpreting Technology and AI report' |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | Highly recognised Slator Language Industry Intelligence reports (slator.com) present the very latest from the language industry with a wealth of data, and the recent SLATOR 2024 Interpreting Technology and AI report features contributions from CTS' Elena Davitti and Tomek Korybski. Elena and Tomek were interviewed to provide insights based on their research projects, SMART (https://smartproject.surrey.ac.uk/) and MATRIC (https://smartproject.surrey.ac.uk/related-projects/). They contributed remarks on hybrid workflows for live interlingual communication including AI-powered real-time speech-to-text and the unique skills language professionals bring into such workflows, creating complex forms of Human-AI Interaction. After Elena's contribution to an earlier report on live subtitling (August 2023: https://slator.com/slator-pro-guide-subtitling-and-captioning/), this is the second time in less than 6 months that CTS staff has provided expertise for a major industry publication. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://slator.com/2024-interpreting-technology-and-ai-report/ |
| Description | Interview with Slator Language Industry intelligence for 'Slator Pro Guide: Subtitling and Captioning' |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | In 2023, Davitti was interviewed by Slator: Language Industry Intelligence, alongside other experts in the field of speech-to-text services to contribute to the 'Slator Pro Guide: Subtitling and Captioning'. This document also features a brand new video clip explaining the different workflows for live interlingual subtitling via speech recognition, which are placed on a continuum from more human-centric to semi and fully-automated. This video is aimed for the wider public and was created within the ESRC-funded SMART project (Shaping Multilingual Access through Respeaking Technologies, 2020-2023) with the support of the Surrey Institute for People-Centred Artificial Intelligence (https://www.youtube.com/watch?v=rxIRKLR2_7o). |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://slator.com/slator-pro-guide-subtitling-and-captioning/#:~:text=The%20Slator%20Pro%20Guide%3A... |
| Description | Invited contribution to the trends e-book for the Institute of Translation and Interpreting (ITI) |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The PI Davitti was invited to write an article in the trends e-book 2021 "Negotiating a new path. Trends in translation and interpreting 2021", published in February 2021. This paved the way to advertising the course developed as a CPD (Continuous Professional Development) activity and will hopefully ensure ITI endorsement for the Summer School to be delivered in the last year of the project. We will report further on this in the next cycle. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.iti.org.uk/resource/latest-iti-trends-e-book-published.html |
| Description | Invited panellist at Innovation in Interpreting Summit |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This online event involved professional practitioners and industry stakeholders from all over the world. The engaging discussion sparked questions and resulted in excellent feedback and requests for involvement in the upskilling course and Summer School we are designing as part of the project as well as invitation to organise further events to engage the professional community and raise awareness of new, hybrid practices for spoken, real-time multilingual communication. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.innovationininterpreting.com/ |
| Description | Real-time interlingual speech-to-text via speech recognition: Respeaking and beyond |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Davitti was invited to give a talk as part of the CIOL (Chartered Institute of Linguists) Languages and Accessibility on Audiovisual Translation and Tour Guiding - Focus session |
| Year(s) Of Engagement Activity | 2023 |
| Description | Research Webinar Series 'How technology is shaping traditional and emerging translation practices' - 'From Specialised Languages and Discourses to Accessible Translations', University of Palermo |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | The PI Davitti and PDR Korybski were invited to deliver a talk as part of these research seminars covering a wide range of topics within the fields of Specialised Discourses and Languages, and Audiovisual Translation. These include legal translation, migration, translation technology, audiovisual translation and media accessibility across a variety of genres (i.e., TV, cinema, museums, theatre, tourism, advertising). The seminars was attended by students attending the second-cycle postgraduate degrees, for third-year undergraduate students as well as scholars and experts/professionals working in international settings in the fields of specialised translation and audiovisual translation. The discussion sparked questions and interest and raised the visibility and profile of our project. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.unipa.it/dipartimenti/scienzeumanistiche/RESEARCH-WEBINAR-SERIES-From-Specialised-Langua... |
| Description | Roundtable discussion. Centre for Translation & Interpreting Studies in Scotland |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | This event was organised to celebrate the International Translation and interpreting day by addressing questions related to current developments in artificial intelligence, e.g. whether they are complementing or competing with the role performed by translators, interpreters, language brokers/mediators, how these developments are perceived by individuals and by agencies/organizations, how professionals can adapt to new workloads and roles and to the requirements of a shifting marketplace. The discussion sparked questions and raised awareness of new, hybrid practices and resulted in very positive feedback and requests for engagement in project deliverables. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.eventbrite.co.uk/e/celebration-of-international-translation-day-roundtable-discussion-ti... |
| Description | Troublesome Terps podcast interview |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The PI Davitti was invited to be interviewed in a podcast addressed to the language professionals community as described as follows: "This episode's topic seems to be of a particular relevance right now at a time, when interpreters around the world are diversifying more than ever. Elena Davitti joins the Troublesome Terps and details her research on interlingual respeaking, how there is now the option to shape a whole new language serivice and why and how interpreters and language professionals at large can get started in this new field. The episode led to feedback from language professionals who became interested in the practice and decided to get involved as reported understanding more about current developments and the fact that they represent an opportunity rather than a threat. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://uk-podcasts.co.uk/podcast/troublesome-terps/55-new-ways-of-working-w-elena-davitti |