Collaborative Cross Modal Interfaces

Lead Research Organisation: Queen Mary, University of London
Department Name: Sch of Electronic Eng & Computer Science


We live in an information society where multiple methods of communication are commonplace. The rapid advancements in Information Communication Technologies (ICT) over the last few years has led to new methods of communication; both socially and professionally. This technology has allowed collaboration to become a cornerstone of problem solving through enabling people to work together with ease, despite being geographically apart. The improvement in ICT has also increased the inclusion of people who have perceptual impairments, for example, text messaging allows a person with hearing impairments to communicate relatively easily on their mobile phone. There is still, however, much work needed in the area of technology that allows collaborative communication between people who have differing perceptual awareness. In this project we will examine this important issue by addressing the challenge: How do we design support for collaboration where participants have differing access to modalities?Specifically, we will look at collaboration between visually impaired users and sighted users in the context of editing diagrams in the software engineering workplace. The deliverable for this project is an open-source software tool that enables collaborative diagram editing between visually impaired and sighted users.To do this, we will draw on a pilot study undertaken by the authors which demonstrates that auditory access to diagrams created in a collaborative context, in conjunction with shared auditory awareness mechanisms, supports participants in non-visual collaborative diagram editing and construction. The project will enlist the assistance of two project partners who have experience in this area (Royal National Institute of Blind People and British Computer Association of the Blind). The partners will assist in the recruitment of the user groups and sit on the steering committee. The user group studies will be carried out 'in the wild' to ensure the improvements are relevant. The feedback from the user studies will allow us to add appropriate haptic and auditory components, producing a user centric software tool. This tool will be released as an open-source resource and we will make the findings of our studies available to both academic and practitioner communities. The results of this project will broaden inclusion in the workplace and give the UK an advantage in the area of assistive technology. As the ageing population increases, so does the need for assistive technology, the software produced from this project will be contributing towards both inclusion in work based collaboration as well as social and creative collaboration. This project will also lead to further work such as creating software plug-ins and addressing the area of assistive collaborative technology in the teaching environment.

Planned Impact

Software engineers with sensory impairments will feel immediate impact from this research as we will provide a tool to support collaborative design where participants have differing access to modalities. We will achieve this by distributing our tool as open-source, and providing online documentation of the tool. Participants in the studies involved in the project will get first hand experience of the tool, as will attendees at the events that we will exhibit at. We will also nurture a community of early adopters of the tool to help promote the tool after the end of the project. In the medium term, we hope that our work will be developed beyond the software engineering domain to support people with sensory impairments in a range of collaborative design domains. Our plan is that the community of early adopters, our public dissemination about our research, and our own follow on funding plans will support divergence into a wider set of domains. In the longer term, the people with sensory impairments in all walks of life will benefit from our research by being able to collaborate with others through cross-modal interaction. Publicising the research through media such as the BBC and the New Scientist will raise awareness of the possibilities of such tools with the general public, and may help to secure future funding to develop general purpose cross-modal tools. The project will create immediate commercial impact for companies with software engineering teams as it will increase the efficiency and inclusion of the workforce. Our open-source tool will be introduced to commercial companies through showcases and commercially focussed publicity. In the medium term, educational establishments will benefit from being able to teach in cross-modal environments, for example where teachers and students have differing access to modalities. We will support this by engaging with specialist educational establishments towards the end of the project to identify possible routes for our tool into the educational realm. In the longer term the project will have economic benefit to the UK as it will allow a wider range of people to engage in collaborative work, hence increasing opportunity and productivity. The project will also create impact by improving the UK's position in the assistive technology market, and making the UK an example of best practice in the field.


10 25 50
publication icon
Bryan-Kinns N (2013) Mutual engagement and collocation with shared representations in International Journal of Human-Computer Studies

publication icon
Oussama Metata (Author) (2012) The Effects of Using Headphones and Speakers on Collaboration in an Audio-only Workspace in Journal of the Audio Engineering Society

publication icon
Oussama Metatla (Author) (2010) Collaborative Cross-modal Interfaces

publication icon
Oussama Metatla (Author) (2011) Designing for Collaborative Cross-modal Interaction

Description The broad objective of the work is to increase workplace and social inclusion through the development of cross-modal collaborative tools. We have achieved the following: 1. Development and release of the open-source cross-modal diagram editing tool (the CCmI tool). 2. Real-world in-situ studies with visually impaired and sighted colleagues using the CCmI tool to edit workplace and educational diagrams. 3. Engagement with the RNIB, BCAB, and New College Worcester in developing and validating the cross-modal approach to diagram editing. 4. Engagement with visually impaired communities at QAC SightVillage. 5. Production of youtube tutorials for the general public, and academic papers.
Exploitation Route The software developed in this project can be used by general public, and is especially aimed at visually impaired people. We have successfully trialled the software in public exhibitions, schools, and local authority offices. We see exploitation routes through the open-source software developed in the project, and through identifying low-cost solutions to non-visual interaction with diagram. These routes would be through The British Computer Association of the Blind (BCAB) and Royal National Institute For The Blind (RNIB) who both served on our steering committee and directly contributed to the project through hands-on feedback on the design in workshops, identification of suitable places to raise awareness of our tool (e.g. mailing lists, newsletters, etc.), and identification of people to take part in workplace studies of the use of the tool.
Sectors Digital/Communication/Information Technologies (including Software)

Description The CCmI project has produced the first tool which supports groups of people editing diagrams using different modalities including combinations of audio, haptic, and visual modalities (referred to as cross-modal interaction). Furthermore, this project is the first to study the use and utility of cross-modal interaction in the wild - for real-world diagram editing activities in the workplace and educational settings with sighted and visually impaired users. The CCmI project contributes real-world validated knowledge on designing for cross-modal collaboration through publication in academic papers. The tool and approach developed in the CCmI project makes diagrams accessible to visually impaired people, and reduces barriers to inclusion in the work place as evidenced by our in-situ studies of people using the tool in both commercial and educational settings for real-world diagram editing activities. Our work direct impacts visually impaired people who we have been interacting with through communities fostered by the BCAB, RNIB, and the Association of Blind and Partially Sighted Teachers and Students (ABAPSTAS), for example, through newsletters, email lists, and word of mouth. We exhibited our tool at the QAC SightVillage public exhibition and engaged members of the public in creating and editing diagrams. We have also engaged staff and pupils at New College Worcester in trials of our system at their school. Our youtube videos are aimed at engaging the general public with our project and encouraging them to download the software for their own use and experimentation. Beneficiaries: Visually Impaired students, teachers, employees, employers.
First Year Of Impact 2011
Sector Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Security and Diplomacy
Impact Types Cultural,Societal,Economic

Description Tool use in Local Authority Context
Policy Influence Type Influenced training of practitioners or researchers
Impact Through our in-the-wild studies in local government offices we showed that cross-modal interaction is a feasible way to improve co-working between sighted and visually impaired employees. Target Audience: Government Department
Description EPSRC
Amount £1,009,614 (GBP)
Funding ID EP/J017205/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 10/2012 
End 10/2015
Description Collaboration with BCAB 
Organisation British Computer Association of the Blind (BCAB)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution The British Computer Association of the Blind (BCAB ) served on our steering committee and directly contributed to the project through hands-on feedback on the design in workshops , identification of suitable places to raise awareness of our tool (e.g. mailing lists, newsletters, etc.), and identification of people to take part in workplace studies of the use of the tool.
Start Year 2010
Description Partnership with RNIB 
Organisation Royal National Institute for Blind People
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Through the CCmI project we have developed and strengthened our partnerships with user advocates: RNIB, BCAB, and New College Worcester - in fact, RNIB and BCAB are supporting our new EPSRC funding proposal (DePIC). The RNIB and BCAB have contributed their time and knowledge to the CCmI project through the steering committee, and New College Worcester has contributed time and effort to the project through organising and taking part in studies of our tool with pupils at New College Worcester.
Start Year 2010
Title CCmI tool 
Description Our CCmI tool is available under open-source licence from the CCmI website, and the EPSRC SoundSoftware repository. 
Type Of Technology Software 
Year Produced 2011 
Open Source License? Yes  
Impact N/A 
Description Community Engagement Workshop 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Type Of Presentation workshop facilitator
Geographic Reach Regional
Primary Audience Study participants or study members
Results and Impact Exploratory workshop on 24th November 2010 with 8 visually impaired participants attending, including representatives from the BCAB, RNIB, RBS, and the Open University. This workshop provided a chance for participants to interact with early versions of the CCmI tool, to provide feedback, and to give details of current solutions they use to edit diagrams (typically, either using embossed paper or a human assistant).

Further engagement by community with the research project
Year(s) Of Engagement Activity 2010
Description Exhibition at QAC SightVillage 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact To reach out to our target audience we exhibited the CCmI tool at QAC SightVillage , Birmingham

12-14 July 2011. SightVillage is the "premier European events showcasing technology, support and services for people who are blind or visually impaired", and attendance is free, providing us with an opportunity to reach out to a wide range of visually impaired people. As well as hands-on demos for attendees we also gave two free seminars at SightVillage on using our tool to create, edit, and share di

Increased engagement with the research project
Year(s) Of Engagement Activity 2011
Description Listening to Diagrams 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact We have also contributed an article to Audio! and CS4FN to explain to the general public interested in technology how it is possible to create, edit, and share diagrams without seeing them.

Increased interest in the project
Year(s) Of Engagement Activity 2012
Description YouTube videos 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact In addition to producing academic papers we have produced youtube video tutorials promoting the project and the tool. These videos provide an explanation of the project aimed at the general public, and, importantly, are audio described making them accessible for visually impaired viewers.

see ccmi youtube channel:

Increased interest in project
Year(s) Of Engagement Activity 2012