Design Patterns for Inclusive Collaboration (DePIC)
Lead Research Organisation:
Queen Mary University of London
Department Name: Sch of Electronic Eng & Computer Science
Abstract
Our interaction with the world around us relies on perception which exploits combinations of the senses we have available to us. Enabling people to use combinations of senses becomes critical in situations where people who have different senses available to them interact with each other. These differences can arise because of temporary or permanent sensory impairment, or due to the technology they are using. However, very little research has examined how people combine and map information from one sense to another, particularly for individuals with sensory impairments, and then used such mappings to inform the design of technology to make collaboration easier. The aim of this multi-disciplinary project is to develop new ways for people to interact with each other using different combinations of senses. This will reduce barriers to collaboration caused by sensory impairment, and improve social and workplace inclusion by optimising the use of available senses. We will combine empirical studies of mappings between senses with participatory design techniques to develop new ideas for inclusive design grounded in Cognitive Psychology. We will capture these design ideas and mappings in the form of Design Patterns and demonstrate their usefulness through the development of interactive systems to support assisted work, living, and leisure.
Planned Impact
Potential beneficiaries of this project outside the research community include anyone who could benefit from reducing barriers to collaboration due to sensory impairment. Examples from various sectors are given below.
Wider public:
* Visually impaired individuals, through reduced barriers to work, living, and leisure.
* Sighted individuals, through raising awareness of barriers to collaboration, ways to reduce these barriers, and increased opportunities for collaboration.
* Teachers in schools with visually impaired students, through increasing opportunities for collaborative learning.
Commercial private sector:
* Commercial companies in general, through increased inclusion of skilled workers.
* Accessible software developers, through examples of best practice in cross-modal design, and access to the open-source platform for development.
* Assistive hardware developers, through identification of new hardware requirements and new areas for assistive hardware development.
* Telehealth providers through examples of best practice in cross-modal collaboration design.
* HCI, Interaction Design, and Accessible Design practitioners, through Design Patterns and examples of best practice to inform their accessible design practice.
* New company development through possible spin-out consulting on cross-modal interaction and the open-source platform.
Policy-makers, charities, and governmental agencies:
* Charities such as the British Computer Association of the Blind, and Royal National Institute of Blind People, and policy-makers through examples of best practice in support for inclusive collaboration.
* Government agencies through the establishment of the UK as an international leader in the field of accessible systems.
Public sector, and others:
* Healthcare workers working with visually impaired patients through support for shared interaction with medical data.
* Visually impaired healthcare workers through being able to access and collaborate with medical data
* Museums and galleries, through examples of best practice in cross-modal design particularly for interactive exhibits and guides.
Researchers employed on the project:
* Improved skills in accessible software development and study methodologies, which may be transferable e.g. the commercial private sector on completion of the project.
* Improved public engagement skills through exhibitions and outreach work.
Others undertaking direct interaction with the project:
* Participants in Participatory Design, study participants, and steering committee, through exposure to new assistive interfaces, and contribution to design direction and refinement.
Wider public:
* Visually impaired individuals, through reduced barriers to work, living, and leisure.
* Sighted individuals, through raising awareness of barriers to collaboration, ways to reduce these barriers, and increased opportunities for collaboration.
* Teachers in schools with visually impaired students, through increasing opportunities for collaborative learning.
Commercial private sector:
* Commercial companies in general, through increased inclusion of skilled workers.
* Accessible software developers, through examples of best practice in cross-modal design, and access to the open-source platform for development.
* Assistive hardware developers, through identification of new hardware requirements and new areas for assistive hardware development.
* Telehealth providers through examples of best practice in cross-modal collaboration design.
* HCI, Interaction Design, and Accessible Design practitioners, through Design Patterns and examples of best practice to inform their accessible design practice.
* New company development through possible spin-out consulting on cross-modal interaction and the open-source platform.
Policy-makers, charities, and governmental agencies:
* Charities such as the British Computer Association of the Blind, and Royal National Institute of Blind People, and policy-makers through examples of best practice in support for inclusive collaboration.
* Government agencies through the establishment of the UK as an international leader in the field of accessible systems.
Public sector, and others:
* Healthcare workers working with visually impaired patients through support for shared interaction with medical data.
* Visually impaired healthcare workers through being able to access and collaborate with medical data
* Museums and galleries, through examples of best practice in cross-modal design particularly for interactive exhibits and guides.
Researchers employed on the project:
* Improved skills in accessible software development and study methodologies, which may be transferable e.g. the commercial private sector on completion of the project.
* Improved public engagement skills through exhibitions and outreach work.
Others undertaking direct interaction with the project:
* Participants in Participatory Design, study participants, and steering committee, through exposure to new assistive interfaces, and contribution to design direction and refinement.
Publications
Brown D
(2013)
Increased Signal Complexity Improves the Breadth of Generalization in Auditory Perceptual Learning
in Neural Plasticity
Brown D
(2016)
Audio-Vision Substitution for Blind Individuals: Addressing Human Information Processing Capacity Limitations
in IEEE Journal of Selected Topics in Signal Processing
Brown DJ
(2015)
Auditory scene analysis and sonified visual images. Does consonance negatively impact on object formation when using complex sonified stimuli?
in Frontiers in psychology
Brown DJ
(2014)
Visual objects in the auditory system in sensory substitution: how much information do we need?
in Multisensory research
Bryan-Kinns N
(2022)
Qi2He: A co-design framework inspired by eastern epistemology
in International Journal of Human-Computer Studies
Bryan-Kinns N
(2014)
Interactive Experience in the Digital Age - Evaluating New Art Practice
Bryan-Kinns N
(2018)
Exploring Interactivity and Co-Creation in Rural China
in Interacting with Computers
Dell'Erba S
(2018)
Synesthetic hallucinations induced by psychedelic drugs in a congenitally blind man.
in Consciousness and cognition
Haigh A
(2013)
How well do you see what you hear? The acuity of visual-to-auditory sensory substitution
in Frontiers in Psychology
Description | We have found new ways to combine sensory output (sound, sight, and touch) to allow people with different sensory abilities to use interactive systems. We developed design patterns and guidelines for how to combine different modalities, and developed new methods to engage people with different sensory abilities in design processes, and then tested these with real world users. A key example of the use of the guidelines is our Accessible Digital Audio Workstation and Accessible Peak Meter which allows visually-impaired Audio Engineers to undertake complex work which was previously extremely difficult due to the visual nature of the user interfaces. We have also used the guidelines to develop an accessible game which can be played by sighted, blind, and deaf participants at the same time, and is available for public download. |
Exploitation Route | Design of more inclusive and accessible technologies across all domains of technology use. |
Sectors | Digital/Communication/Information Technologies (including Software) Education |
URL | http://depic.eecs.qmul.ac.uk |
Description | We have presented our work in a number of non-academic venues to engage with general public and people living with visual impairments: A series of participatory design workshop with visually impaired musicians and audio production specialists AccessNIME workshop at NIME'2014 Sonification for Sports & Performances workshop at ICAD'2014 AccessNIME hackathon Demosntration of accessible technology at the Visually Impaired Musicians Lives conference Sensory Support Services Technology Day demonstrations of audio-haptic technology NESTA SoundLab Play Space Event demonstrations of ShapeTones Latitude Music Festival, Wellcome Trust Hub Bath Alumni Weekend "Ideas Hub" Science Museum "You have been upgraded" Festival General University Lecture Programme (in association with the University for the Third Age), University of Bath, United Kingdom, TEDx, Bath, United Kingdom Wiltshire Science Festival, United Kingdom Peak level meters, along with other components which rely on the sense of sight for their use, are inherently inaccessible to people living visual impairments. Our AccessiblePeakMeter is the first plug-in which makes these previously inaccessible meters completely accessible. It uses real-time sonification to deliver information to the user about audio levels and peaks in audio signals, and so supports core activities in audio production. The AccessiblePeakMeter comes as a VST, AU or AAX plug-in, two of the main industry standards for the deployment of digital audio effects into professional DAWs (e.g. Cakewalk Sonar, Cockos Reaper, Ableton Live). The plug-in can be run on both Windows (32/64) and Mac platforms and it is free for download and completely open source. AccessiblePeakMeter received the Best Solution by a Large Organization award in the AT&T and New York University Connect Ability Challenge, a global software competition for software technologies that improve the lives of people living with disabilities. The winner were announced on Sunday July 26 2015, the day of the 25th anniversary of the Americans with Disabilities Act. In addition, research on cross-cultural co-design supported by this project has led to further inclusive and tangible interaction design research with partners in China with the most recent publication on this topic in 2022. |
First Year Of Impact | 2014 |
Sector | Digital/Communication/Information Technologies (including Software) |
Impact Types | Cultural Societal |
Description | Collaboration with BCAB |
Organisation | British Computer Association of the Blind (BCAB) |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | The British Computer Association of the Blind (BCAB ) served on our steering committee and directly contributed to the project through hands-on feedback on the design in workshops , identification of suitable places to raise awareness of our tool (e.g. mailing lists, newsletters, etc.), and identification of people to take part in workplace studies of the use of the tool. |
Start Year | 2010 |
Description | Partnership with RNIB |
Organisation | Royal National Institute for Blind People |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | Through the CCmI project we have developed and strengthened our partnerships with user advocates: RNIB, BCAB, and New College Worcester - in fact, RNIB and BCAB are supporting our new EPSRC funding proposal (DePIC). The RNIB and BCAB have contributed their time and knowledge to the CCmI project through the steering committee, and New College Worcester has contributed time and effort to the project through organising and taking part in studies of our tool with pupils at New College Worcester. |
Start Year | 2010 |
Description | Sensory Support Service Bristol |
Organisation | Sensory Support Service |
Country | United Kingdom |
Sector | Private |
PI Contribution | The Sensory Support Service, part of the Bristol City Council, is a service providing support for children with sensory impairments in mainstream and specialised schools, covering four councils in the Southwest of England. Our contributions to this partnership is two folds: 1) We provide relevant staff (Qualified Teachers for Visual Impairments) with opportunities to be involved in the design and evaluation of novel technologies for visually impaired users; 2) We contribute activities (e.g. technology demonstrations, training workshops) to public events organised by the sensory support service |
Collaborator Contribution | They make contributions to the research project in two ways: 1) Provide access to their staff who are Qualified Teachers for Visual Impairment who contribute as study participants 2) They provide contacts with local mainstream schools who then contribute to the research project also as partners and provide access to teachers, teaching assistants and pupils who contribute as study participants |
Impact | - Staff from the Sensory Support Services have taken part in a scoping study - We are organising a joint event this summer to demonstrate accessibility technology for visually impaired children and their parents - We have identified two schools through this service, and established partnerships with them (where we are now conducting ethnographic field studies with staff, teachers, and pupils) |
Start Year | 2016 |
Title | Accessible Peak Meter |
Description | Peak level meters, along with other components which rely on the sense of sight for their use, are inherently inaccessible to people living visual impairments. Our AccessiblePeakMeter is the first plug-in which makes these previously inaccessible meters completely accessible. It uses real-time sonification to deliver information to the user about audio levels and peaks in audio signals, and so supports core activities in audio production. The AccessiblePeakMeter comes as a VST, AU or AAX plug-in, two of the main industry standards for the deployment of digital audio effects into professional DAWs (e.g. Cakewalk Sonar, Cockos Reaper, Ableton Live). The plug-in can be run on both Windows (32/64) and Mac platforms and it is free for download and completely open source! |
Type Of Technology | Software |
Year Produced | 2015 |
Open Source License? | Yes |
Impact | AccessiblePeakMeter received the Best Solution by a Large Organization award in the AT&T and New York University Connect Ability Challenge, a global software competition for software technologies that improve the lives of people living with disabilities. The winner were announced on Sunday July 26 2015, the day of the 25th anniversary of the Americans with Disabilities Act. |
URL | http://depic.eecs.qmul.ac.uk/apm/ |
Title | Cross-Modal DAW Prototype |
Description | A simple DAW prototype that can be operated with the usual graphical user interface or with its peculiar audio-haptic interface. The latter allows visually impaired users to interact with the program without having to look at the screen, by using haptics (through the Geomagic® Touchâ„¢ Haptic Device) and sound (speech as well as non-speech). It supports cut/copy/paste of audio clips and automation graphs of gain and panning. It is not really meant to be used for real tasks but rather to test out cross-modal approaches of interacting with DAW's. It features several sonifications and "haptifications" of different aspects of DAW's that are often inaccessible to visually impaired users. Waveform selection Waveform peaks Automation envelopes Its sonification of peaks underlies the sonification techniques used in the Accessible Peak Meter audio plug-in. |
Type Of Technology | Software |
Year Produced | 2015 |
Open Source License? | Yes |
Impact | First truly accessible Digital Audio Workstation prototype. |
URL | http://depic.eecs.qmul.ac.uk/?q=dawprototype |
Title | ShapeTones |
Description | ShapeTones is an audiovisual memory game for iOS (iPhone/iPad). A sequence of 3 shapes and tones ("ShapeTones") is played, and the player tries to reproduce it. Tapping different areas of the screen trigger different ShapeTones. The game starts with 3 ShapeTones. As the game evolves, more ShapeTones become available. When a new ShapeTone is added, a trial screen is shown to demonstrate where each ShapeTone is triggered. Some surprises happen along the way. As a one player game, the sequence is created automatically. As a two player game, one player creates the sequence, passes the device to the other player, who tries to repeat it. They then swap the roles. |
Type Of Technology | Software |
Year Produced | 2015 |
Impact | Public release on Apple iTunes |
URL | http://depic.eecs.qmul.ac.uk/?q=shapetones |
Title | jHapticGUI |
Description | The JHapticGUI library allows to embed force feedback devices in the development of Java GUI based programs Force feedback devices are widely used for a diverse range of application including 3D modeling, medical simulations, games and virtual sculpture. Normally these applications are based on a 3D vector graphics (e.g. OpenGL) scene which is "augmented" with sense of touch, namely the possibility to touch the virtual objects that are rendered on the screen. However, a relatively unexplored field is the use of such devices to provide a haptic modality of interaction with everyday programs featuring a 2D graphical user interface made of buttons, lists, menus etc. Such a modality, besides opening up new ways of interacting with graphical user interfaces, would have an impact on the access to common software by people with visual impairments. The JHapticGUI library helps building such software in Java. It provides an abstract and clean interface to the haptic device from the Java code allowing at the same time full freedom as to how to design the haptic scene. It takes care of the synchronization between the Java event dispatching thread and the thread that handles the device. Furthermore it provides a communication protocol based on general purpose messages for connecting haptic and graphic events. Details of communication implementation are hidden to the client code but the content of the message can be defined by the application, for maximum customization of the program logic |
Type Of Technology | Software |
Year Produced | 2015 |
Open Source License? | Yes |
Impact | Provides a new way to make accessible interfaces |
URL | http://depic.eecs.qmul.ac.uk/?q=jhapticgui |
Title | ofxEnveloper |
Description | ofxEnveloper is an openFrameworks addon that converts gestures into an envelope type of graphical information, by recognising inflection points from the gesture. Thresholds for horizontal and vertical lines can also be defined. It is particularly aimed for animation and sound applications, and for use with drawing surfaces (such as Wacom tablets or touchscreen interfaces). Possible uses: audio effect or EQ envelopes; animation curves; creation/manipulation of audio and/or visual sequences; arpeggiators. |
Type Of Technology | Software |
Year Produced | 2015 |
Open Source License? | Yes |
Impact | Potential use in sound editing applications |
URL | http://depic.eecs.qmul.ac.uk/?q=ofxenveloper |
Description | Haptic Wave at Visually-impaired Musicians' Lives |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | We presented the Haptic Wave at the Visually-impaired musicians' lives conference at University College London |
Year(s) Of Engagement Activity | 2015 |
URL | http://vimusicians.ioe.ac.uk/confer_train.html |
Description | Innovation in Haptics presentation at Royal Academy of Engineering |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | We were invited to present the Haptic Wave at the Royal Academy of Engineering, "Innovation in Haptics" |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.raeng.org.uk/events/events-programme/2017/july/innovation-in-haptics |
Description | NESTA SoundLab Play Space Event |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | NESTA SoundLab Play Space Event demonstrations of ShapeTones which was developed in the DePIC project as an accessible game |
Year(s) Of Engagement Activity | 2015 |
Description | Sensory Support Services Technology Day |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | Sensory Support Services Technology Day demonstrations of audio-haptic technology with school children with sensory impairments and their carers. |
Year(s) Of Engagement Activity | 2015 |
Description | participatory design workshops |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Study participants or study members |
Results and Impact | A series of participatory design workshops with visually impaired musicians and audio production specialists |
Year(s) Of Engagement Activity | 2013,2014,2015 |