Design Patterns for Inclusive Collaboration (DePIC)

Lead Research Organisation: Queen Mary University of London
Department Name: Sch of Electronic Eng & Computer Science

Abstract

Our interaction with the world around us relies on perception which exploits combinations of the senses we have available to us. Enabling people to use combinations of senses becomes critical in situations where people who have different senses available to them interact with each other. These differences can arise because of temporary or permanent sensory impairment, or due to the technology they are using. However, very little research has examined how people combine and map information from one sense to another, particularly for individuals with sensory impairments, and then used such mappings to inform the design of technology to make collaboration easier. The aim of this multi-disciplinary project is to develop new ways for people to interact with each other using different combinations of senses. This will reduce barriers to collaboration caused by sensory impairment, and improve social and workplace inclusion by optimising the use of available senses. We will combine empirical studies of mappings between senses with participatory design techniques to develop new ideas for inclusive design grounded in Cognitive Psychology. We will capture these design ideas and mappings in the form of Design Patterns and demonstrate their usefulness through the development of interactive systems to support assisted work, living, and leisure.

Planned Impact

Potential beneficiaries of this project outside the research community include anyone who could benefit from reducing barriers to collaboration due to sensory impairment. Examples from various sectors are given below.

Wider public:
* Visually impaired individuals, through reduced barriers to work, living, and leisure.
* Sighted individuals, through raising awareness of barriers to collaboration, ways to reduce these barriers, and increased opportunities for collaboration.
* Teachers in schools with visually impaired students, through increasing opportunities for collaborative learning.

Commercial private sector:
* Commercial companies in general, through increased inclusion of skilled workers.
* Accessible software developers, through examples of best practice in cross-modal design, and access to the open-source platform for development.
* Assistive hardware developers, through identification of new hardware requirements and new areas for assistive hardware development.
* Telehealth providers through examples of best practice in cross-modal collaboration design.
* HCI, Interaction Design, and Accessible Design practitioners, through Design Patterns and examples of best practice to inform their accessible design practice.
* New company development through possible spin-out consulting on cross-modal interaction and the open-source platform.

Policy-makers, charities, and governmental agencies:
* Charities such as the British Computer Association of the Blind, and Royal National Institute of Blind People, and policy-makers through examples of best practice in support for inclusive collaboration.
* Government agencies through the establishment of the UK as an international leader in the field of accessible systems.

Public sector, and others:
* Healthcare workers working with visually impaired patients through support for shared interaction with medical data.
* Visually impaired healthcare workers through being able to access and collaborate with medical data
* Museums and galleries, through examples of best practice in cross-modal design particularly for interactive exhibits and guides.

Researchers employed on the project:
* Improved skills in accessible software development and study methodologies, which may be transferable e.g. the commercial private sector on completion of the project.
* Improved public engagement skills through exhibitions and outreach work.

Others undertaking direct interaction with the project:
* Participants in Participatory Design, study participants, and steering committee, through exposure to new assistive interfaces, and contribution to design direction and refinement.

Publications

10 25 50
 
Description We have found new ways to combine sensory output (sound, sight, and touch) to allow people with different sensory abilities to use interactive systems. We developed design patterns and guidelines for how to combine different modalities, and developed new methods to engage people with different sensory abilities in design processes, and then tested these with real world users.

A key example of the use of the guidelines is our Accessible Digital Audio Workstation and Accessible Peak Meter which allows visually-impaired Audio Engineers to undertake complex work which was previously extremely difficult due to the visual nature of the user interfaces. We have also used the guidelines to develop an accessible game which can be played by sighted, blind, and deaf participants at the same time, and is available for public download.
Exploitation Route Design of more inclusive and accessible technologies across all domains of technology use.
Sectors Digital/Communication/Information Technologies (including Software)

Education

URL http://depic.eecs.qmul.ac.uk
 
Description We have presented our work in a number of non-academic venues to engage with general public and people living with visual impairments: A series of participatory design workshop with visually impaired musicians and audio production specialists AccessNIME workshop at NIME'2014 Sonification for Sports & Performances workshop at ICAD'2014 AccessNIME hackathon Demosntration of accessible technology at the Visually Impaired Musicians Lives conference Sensory Support Services Technology Day demonstrations of audio-haptic technology NESTA SoundLab Play Space Event demonstrations of ShapeTones Latitude Music Festival, Wellcome Trust Hub Bath Alumni Weekend "Ideas Hub" Science Museum "You have been upgraded" Festival General University Lecture Programme (in association with the University for the Third Age), University of Bath, United Kingdom, TEDx, Bath, United Kingdom Wiltshire Science Festival, United Kingdom Peak level meters, along with other components which rely on the sense of sight for their use, are inherently inaccessible to people living visual impairments. Our AccessiblePeakMeter is the first plug-in which makes these previously inaccessible meters completely accessible. It uses real-time sonification to deliver information to the user about audio levels and peaks in audio signals, and so supports core activities in audio production. The AccessiblePeakMeter comes as a VST, AU or AAX plug-in, two of the main industry standards for the deployment of digital audio effects into professional DAWs (e.g. Cakewalk Sonar, Cockos Reaper, Ableton Live). The plug-in can be run on both Windows (32/64) and Mac platforms and it is free for download and completely open source. AccessiblePeakMeter received the Best Solution by a Large Organization award in the AT&T and New York University Connect Ability Challenge, a global software competition for software technologies that improve the lives of people living with disabilities. The winner were announced on Sunday July 26 2015, the day of the 25th anniversary of the Americans with Disabilities Act. In addition, research on cross-cultural co-design supported by this project has led to further inclusive and tangible interaction design research with partners in China with the most recent publication on this topic in 2022.
First Year Of Impact 2014
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Cultural

Societal

 
Description Collaboration with BCAB 
Organisation British Computer Association of the Blind (BCAB)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution The British Computer Association of the Blind (BCAB ) served on our steering committee and directly contributed to the project through hands-on feedback on the design in workshops , identification of suitable places to raise awareness of our tool (e.g. mailing lists, newsletters, etc.), and identification of people to take part in workplace studies of the use of the tool.
Start Year 2010
 
Description Partnership with RNIB 
Organisation Royal National Institute for Blind People
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Through the CCmI project we have developed and strengthened our partnerships with user advocates: RNIB, BCAB, and New College Worcester - in fact, RNIB and BCAB are supporting our new EPSRC funding proposal (DePIC). The RNIB and BCAB have contributed their time and knowledge to the CCmI project through the steering committee, and New College Worcester has contributed time and effort to the project through organising and taking part in studies of our tool with pupils at New College Worcester.
Start Year 2010
 
Description Sensory Support Service Bristol 
Organisation Sensory Support Service
Country United Kingdom 
Sector Private 
PI Contribution The Sensory Support Service, part of the Bristol City Council, is a service providing support for children with sensory impairments in mainstream and specialised schools, covering four councils in the Southwest of England. Our contributions to this partnership is two folds: 1) We provide relevant staff (Qualified Teachers for Visual Impairments) with opportunities to be involved in the design and evaluation of novel technologies for visually impaired users; 2) We contribute activities (e.g. technology demonstrations, training workshops) to public events organised by the sensory support service
Collaborator Contribution They make contributions to the research project in two ways: 1) Provide access to their staff who are Qualified Teachers for Visual Impairment who contribute as study participants 2) They provide contacts with local mainstream schools who then contribute to the research project also as partners and provide access to teachers, teaching assistants and pupils who contribute as study participants
Impact - Staff from the Sensory Support Services have taken part in a scoping study - We are organising a joint event this summer to demonstrate accessibility technology for visually impaired children and their parents - We have identified two schools through this service, and established partnerships with them (where we are now conducting ethnographic field studies with staff, teachers, and pupils)
Start Year 2016
 
Title Accessible Peak Meter 
Description Peak level meters, along with other components which rely on the sense of sight for their use, are inherently inaccessible to people living visual impairments. Our AccessiblePeakMeter is the first plug-in which makes these previously inaccessible meters completely accessible. It uses real-time sonification to deliver information to the user about audio levels and peaks in audio signals, and so supports core activities in audio production. The AccessiblePeakMeter comes as a VST, AU or AAX plug-in, two of the main industry standards for the deployment of digital audio effects into professional DAWs (e.g. Cakewalk Sonar, Cockos Reaper, Ableton Live). The plug-in can be run on both Windows (32/64) and Mac platforms and it is free for download and completely open source! 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact AccessiblePeakMeter received the Best Solution by a Large Organization award in the AT&T and New York University Connect Ability Challenge, a global software competition for software technologies that improve the lives of people living with disabilities. The winner were announced on Sunday July 26 2015, the day of the 25th anniversary of the Americans with Disabilities Act. 
URL http://depic.eecs.qmul.ac.uk/apm/
 
Title Cross-Modal DAW Prototype 
Description A simple DAW prototype that can be operated with the usual graphical user interface or with its peculiar audio-haptic interface. The latter allows visually impaired users to interact with the program without having to look at the screen, by using haptics (through the Geomagic® Touchâ„¢ Haptic Device) and sound (speech as well as non-speech). It supports cut/copy/paste of audio clips and automation graphs of gain and panning. It is not really meant to be used for real tasks but rather to test out cross-modal approaches of interacting with DAW's. It features several sonifications and "haptifications" of different aspects of DAW's that are often inaccessible to visually impaired users. Waveform selection Waveform peaks Automation envelopes Its sonification of peaks underlies the sonification techniques used in the Accessible Peak Meter audio plug-in. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact First truly accessible Digital Audio Workstation prototype. 
URL http://depic.eecs.qmul.ac.uk/?q=dawprototype
 
Title ShapeTones 
Description ShapeTones is an audiovisual memory game for iOS (iPhone/iPad). A sequence of 3 shapes and tones ("ShapeTones") is played, and the player tries to reproduce it. Tapping different areas of the screen trigger different ShapeTones. The game starts with 3 ShapeTones. As the game evolves, more ShapeTones become available. When a new ShapeTone is added, a trial screen is shown to demonstrate where each ShapeTone is triggered. Some surprises happen along the way. As a one player game, the sequence is created automatically. As a two player game, one player creates the sequence, passes the device to the other player, who tries to repeat it. They then swap the roles. 
Type Of Technology Software 
Year Produced 2015 
Impact Public release on Apple iTunes 
URL http://depic.eecs.qmul.ac.uk/?q=shapetones
 
Title jHapticGUI 
Description The JHapticGUI library allows to embed force feedback devices in the development of Java GUI based programs Force feedback devices are widely used for a diverse range of application including 3D modeling, medical simulations, games and virtual sculpture. Normally these applications are based on a 3D vector graphics (e.g. OpenGL) scene which is "augmented" with sense of touch, namely the possibility to touch the virtual objects that are rendered on the screen. However, a relatively unexplored field is the use of such devices to provide a haptic modality of interaction with everyday programs featuring a 2D graphical user interface made of buttons, lists, menus etc. Such a modality, besides opening up new ways of interacting with graphical user interfaces, would have an impact on the access to common software by people with visual impairments. The JHapticGUI library helps building such software in Java. It provides an abstract and clean interface to the haptic device from the Java code allowing at the same time full freedom as to how to design the haptic scene. It takes care of the synchronization between the Java event dispatching thread and the thread that handles the device. Furthermore it provides a communication protocol based on general purpose messages for connecting haptic and graphic events. Details of communication implementation are hidden to the client code but the content of the message can be defined by the application, for maximum customization of the program logic 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact Provides a new way to make accessible interfaces 
URL http://depic.eecs.qmul.ac.uk/?q=jhapticgui
 
Title ofxEnveloper 
Description ofxEnveloper is an openFrameworks addon that converts gestures into an envelope type of graphical information, by recognising inflection points from the gesture. Thresholds for horizontal and vertical lines can also be defined. It is particularly aimed for animation and sound applications, and for use with drawing surfaces (such as Wacom tablets or touchscreen interfaces). Possible uses: audio effect or EQ envelopes; animation curves; creation/manipulation of audio and/or visual sequences; arpeggiators. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact Potential use in sound editing applications 
URL http://depic.eecs.qmul.ac.uk/?q=ofxenveloper
 
Description Haptic Wave at Visually-impaired Musicians' Lives 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We presented the Haptic Wave at the Visually-impaired musicians' lives conference at University College London
Year(s) Of Engagement Activity 2015
URL http://vimusicians.ioe.ac.uk/confer_train.html
 
Description Innovation in Haptics presentation at Royal Academy of Engineering 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact We were invited to present the Haptic Wave at the Royal Academy of Engineering, "Innovation in Haptics"
Year(s) Of Engagement Activity 2017
URL https://www.raeng.org.uk/events/events-programme/2017/july/innovation-in-haptics
 
Description NESTA SoundLab Play Space Event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact NESTA SoundLab Play Space Event demonstrations of ShapeTones which was developed in the DePIC project as an accessible game
Year(s) Of Engagement Activity 2015
 
Description Sensory Support Services Technology Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact Sensory Support Services Technology Day demonstrations of audio-haptic technology with school children with sensory impairments and their carers.
Year(s) Of Engagement Activity 2015
 
Description participatory design workshops 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Study participants or study members
Results and Impact A series of participatory design workshops with visually impaired musicians and audio production specialists
Year(s) Of Engagement Activity 2013,2014,2015