Exploration of Ultrasound based haptic interaction on a multi-touch surface
Lead Research Organisation:
University of Bristol
Department Name: Computer Science
Abstract
Multi-touch tables, such as Microsoft Surface, are now widely available. Users can walk-up and use these systems in hotel lobbies and other public settings with very little instruction and with no need to wear or hold intrusive sensors with their hands or body. The ability to 'walk-up and use' with unadorned hands and fingers removes the barrier between human and technology, encouraging spontaneous use.
One of the primary disadvantages of current interactive surfaces is that users can touch objects, but they are unable to feel them. There are a plethora of applications where it is beneficial for the user to have their touches augmented with 'feel-based' haptic feedback. For example, medical applications, virtual training, and modelling applications require precise control from the user-haptic feedback can aid the user in effectively performing these tasks. These applications demonstrate the benefit of augmenting haptic feedback with visual feedback in an interactive application. Often, the visual space has been disconnected from the force-feedback, requiring a prolonged training period for the user to become accustomed to moving a digital object and watching the interactions a small distance away on a monitor.
In this proposal we will investigate the use of ultrasound transducers to provide 'feelable' feedback through air. The skin on a human hand can feel the ultrasonic pressure wave produced by a carefully calibrated series of transducers, in much the same manner that is apparent from loud sub-woofers on a stereo system. Ultrasound waves are outside the human's range of hearing and so provide silent, through-air haptic feedback. We will use this technology to provide multi-point haptic feedback on the surface of a multi-touch horizontal surface.
The team consists of Dr. Sriram Subramanian, Dr. Mark Marshall and Dr. Jason Alexander from the computer science departments and Prof. Bruce Drinkwater from the Mechanical Engineering department of the University of Bristol and Prof. Stephen Brewster from the Computer Science department of the University of Glasgow. The team is internationally recognised for its research in Human-Computer Interaction (HCI), novel integration of hardware for HCI, and Ultrasonic sensors. Microsoft research (Cambridge) and XMOS will serve as project partners and offer valuable resources and support for the project.
One of the primary disadvantages of current interactive surfaces is that users can touch objects, but they are unable to feel them. There are a plethora of applications where it is beneficial for the user to have their touches augmented with 'feel-based' haptic feedback. For example, medical applications, virtual training, and modelling applications require precise control from the user-haptic feedback can aid the user in effectively performing these tasks. These applications demonstrate the benefit of augmenting haptic feedback with visual feedback in an interactive application. Often, the visual space has been disconnected from the force-feedback, requiring a prolonged training period for the user to become accustomed to moving a digital object and watching the interactions a small distance away on a monitor.
In this proposal we will investigate the use of ultrasound transducers to provide 'feelable' feedback through air. The skin on a human hand can feel the ultrasonic pressure wave produced by a carefully calibrated series of transducers, in much the same manner that is apparent from loud sub-woofers on a stereo system. Ultrasound waves are outside the human's range of hearing and so provide silent, through-air haptic feedback. We will use this technology to provide multi-point haptic feedback on the surface of a multi-touch horizontal surface.
The team consists of Dr. Sriram Subramanian, Dr. Mark Marshall and Dr. Jason Alexander from the computer science departments and Prof. Bruce Drinkwater from the Mechanical Engineering department of the University of Bristol and Prof. Stephen Brewster from the Computer Science department of the University of Glasgow. The team is internationally recognised for its research in Human-Computer Interaction (HCI), novel integration of hardware for HCI, and Ultrasonic sensors. Microsoft research (Cambridge) and XMOS will serve as project partners and offer valuable resources and support for the project.
Planned Impact
Development of ultrasonic haptic feedback has the potential to not only revolutionise several current areas of research, but to also create opportunities for immersive new experiences with large commercial value. Haptics is currently a hot topic in industry (Next Generation Haptics: Market Analysis and Forecasts. ARC Chart. Feb 2011.www.arcchart.com) and our work will develop a new form of haptic technology and demonstrate its usefulness in a new range of areas. This will increase the potential opportunities for haptics beyond what is available now.
Our demonstrator systems (as listed in WP5) will allow Dynamic Holographic Assembler (DHA) operators to feel the forces required to move molecules on their multi-touch interfaces to these systems (currently force data cannot be used in any way as it is difficult to let users feel forces at multiple locations simultaneously). This will provide added value to current research in this area funded through EPSRC (EP/G037310/1 and EP/G012067/1). In turn, this will allow non-specialist researchers to also use a DHA with minimal training.
Once we have developed the technology and interaction techniques that support multi-point haptic feedback, there is a large number of potential domains for applying these techniques. Examples of these applications include mobile TV broadcasting and medical applications. Mobile TV broadcasts could be augmented with a haptic feedback channel to let users 'feel' the action. For example, multi-point ultrasonic haptic feedback could be applied to the back of the mobile device, with the user's hand sensing the forces exerted by the athletes. The gloves of amateur boxers already include sensors for registering the force of their punches-this data could be transmitted with the video stream to provide a more immersive user experience. A demonstrator for this application will be developed by a project technician to showcase this idea.
There are also several medical applications for this technology. Stem-cell researchers can also use multi-point haptic feedback as a method for interpreting, through mechanical property changes, structural changes in the cell even before chemical or other visible identification is possible.
Our demonstrator systems (as listed in WP5) will allow Dynamic Holographic Assembler (DHA) operators to feel the forces required to move molecules on their multi-touch interfaces to these systems (currently force data cannot be used in any way as it is difficult to let users feel forces at multiple locations simultaneously). This will provide added value to current research in this area funded through EPSRC (EP/G037310/1 and EP/G012067/1). In turn, this will allow non-specialist researchers to also use a DHA with minimal training.
Once we have developed the technology and interaction techniques that support multi-point haptic feedback, there is a large number of potential domains for applying these techniques. Examples of these applications include mobile TV broadcasting and medical applications. Mobile TV broadcasts could be augmented with a haptic feedback channel to let users 'feel' the action. For example, multi-point ultrasonic haptic feedback could be applied to the back of the mobile device, with the user's hand sensing the forces exerted by the athletes. The gloves of amateur boxers already include sensors for registering the force of their punches-this data could be transmitted with the video stream to provide a more immersive user experience. A demonstrator for this application will be developed by a project technician to showcase this idea.
There are also several medical applications for this technology. Stem-cell researchers can also use multi-point haptic feedback as a method for interpreting, through mechanical property changes, structural changes in the cell even before chemical or other visible identification is possible.
Organisations
Publications
Wilson, Graham
(2013)
Ultrasonic Haptic Feedback for Gestural Interfaces using a Moveable Hand-Mounted Array
in -
Obrist M
(2013)
Talking about tactile experiences
Seah SA
(2014)
Dexterous ultrasonic levitation of millimeter-sized objects in air.
in IEEE transactions on ultrasonics, ferroelectrics, and frequency control
Wilson G
(2014)
Perception of ultrasonic haptic feedback on the hand
Title | BBC Christmas Lecture |
Description | The output of the project was part of the BBC Christmas lectures from 2014. This was presented by Dr. Daniella George |
Type Of Art | Film/Video/Animation |
Year Produced | 2014 |
Impact | Publicity and enhanced awareness of our system and research. |
URL | http://www.bbc.co.uk/programmes/b04wgf6g |
Title | Part of Tate Modern IK prize 2015 |
Description | Ultrahaptics was featured in the artworks created by Flying Objects in 2015 as part of their winning entry to the Tate Moderns IK prize 2015. This was an installation of sensory stimulants which will allow users to smell, touch, taste and hear works of art. Ultrahaptics technology was used to create teh "touch" art work. |
Type Of Art | Artistic/Creative Exhibition |
Year Produced | 2015 |
Impact | The involvement of Ultrahaptics was through the University of Sussex (Dr. Marianna Obrist). The art piece was a huge success both within UK and beyond. It helped establish Ultrahaptics as a technology company that works with the creative industry. We got a lot of press coverage for this. Wall Street Journal - https://www.wsj.com/articles/please-touch-the-art-work-new-tate-exhibit-will-stimulate-all-five-senses-1439495457 The Independent - http://www.independent.co.uk/arts-entertainment/art/news/tate-sensorium-new-exhibition-at-tate-britain-invites-art-lovers-to-taste-smell-and-hear-art-10438783.html |
URL | http://www.tate.org.uk/about/projects/ik-prize/ik-prize-2015 |
Description | The project examined ways of creating tactile feedback on different interactive surfaces (like tablets). We examined the use of ultrasound transducers to create such tactile sensations and how this could be applied within the context of human-computer interaction. |
Exploitation Route | A spin-off company called ultrahaptics has been setup to commercialise the findings of this project. |
Sectors | Digital/Communication/Information Technologies (including Software) Electronics Transport |
Description | We are talking to many potential end-users ranging from automotive to consumer electronics and home appliances. All these avenues for further commercial exploitation is being explored through our startup Ultrahaptics. This project has also spawned further research interest in techniques for acoustic levitation. Acoustic Levitation has resulted in a new EPSRC proposal EP/N014197/1 Ultrahaptics is a Bristol based company that currently employs 25 people. More information can be found at www.ultrahaptics.com Updated on Jan 31 2019 Today Ultrahaptics is an international company with 100+ employees in the UK and offices in California and Singapore. The company is closely working with many international companies in automotive, VR, Consumer electronics and beyond. In Nov 2018, the company closed its Series C round taking on an additional £30m to grow its product offering and reach. Ultrahaptics won the Queens award for Enterprise in 2018 and there are innumerable press releases about the companies accomplishments and strategic partnerships. Updated feb 2020 Ultrahaptics has changed its name to Ultraleap following its successful acquisition of Leap Motion. This makes Ultraleap a global leader in provding spatial interaction solutions. |
First Year Of Impact | 2013 |
Sector | Creative Economy,Digital/Communication/Information Technologies (including Software),Electronics,Leisure Activities, including Sports, Recreation and Tourism,Transport |
Impact Types | Economic |
Company Name | Ultraleap Limited |
Description | Ultraleap develops hand tracking and haptics technology to enable natural interactions with digital worlds, making touchscreens, wearables, and controllers unnecessary. Their solutions are used in industries ranging from virtual reality to retail to interactive digital out-of-home. |
Year Established | 2013 |
Impact | none yet. |
Website | http://www.ultrahaptics.com |