Living With Interactive Decorative Patterns

Lead Research Organisation: University of Nottingham
Department Name: Horizon Digital Economy Research

Abstract

Decorative patterns are an ever-present feature of our everyday world. From motifs and borders, to swathes of colour and texture, almost every 'thing' of value is embellished with a pattern that has been carefully designed to enhance its aesthetic, meaning and function.

We aim to make such decorative patterns interactive so that all manner of everyday objects can become part of the Internet of Things simply by decorating their surfaces. Pointing a camera at a thing (or indeed pointing a thing at a camera) might then enable people to learn about what it is, how it was made, and how to use it; to access personal memories or review their history of use; or to trigger other contextually relevant services.

Our proposal builds on a technology called Aestheticodes that we have already developed and demonstrated through a feasibility project. This enables designers and craftspeople to draw beautiful decorative patterns that contain computer-recognisable visual codes embedded within them. So far, we have implemented the technology to run on smartphones, worked with ceramic designers to produce an initial portfolio of fifteen patterns, and worked with the Busaba Eathai restaurant chain to apply these designs to plates, placemats and menus. These were then deployed in one of their restaurants to demonstrate an enhanced dining experience during London Design Week.

This project will conduct an in the wild study of this technology actually being used in two complementary real world settings so as to learn how interactive decorative patterns can enhance everyday life, understand the challenges of interacting with patterns that contain embedded codes under testing real-world conditions, and establish ways of engaging designers and end-users in designing and reappropriating patterned objects and surfaces.

Our first setting is Busaba Eathai where we will build on our established collaboration to develop production tableware and a public app to be deployed and used by customers over a period of months. Our second setting is the home, where we will collaborate with Johnson Tiles to produce a range of domestic tiles that will be sold as products in retail outlets. These will come with an invitation to join our research project so that we can employ participatory design methods to create services for enhanced kitchens and bathrooms that users can live with over a period of six months (during which time we will study them) and potentially beyond.

These interventions will produce a series of papers reporting on the experiences and challenges of situating things that are decorated with interactive patterns in wild settings. These will also make broader contributions to discussions of interaction with invisible sensing systems and the design of new tangible materials within Human Computer Interaction, Ubiquitous Computing and related fields. Our project will also produce video documentation of the two case studies, public releases of the apps as well as an open source release of the design tools and supporting libraries.

Planned Impact

Short-term beneficiaries of this research will include our user groups: through the first trial, Busaba Eathai will obtain beautiful new place-settings and an publicly-released interactive smartphone application through which they can improve engagement with and experience of their customers. Through the second, Johnson Tiles will gain exciting new interactive designs opening up new product possibilities; and selected families will be able to decorate their homes with engaging and aesthetic tiles that will hopefully be a talking point for them for years to come.

Medium term beneficiaries will include researchers working in HCI and interaction design, and product designers more generally: they will have a better understanding of the means by which Internet of Things can be made more humanly acceptable through effective use of aesthetics in design to make currently passive objects interactive.

As a result we anticipate that, long term, we will all benefit from the results of this research. The Internet of Things promises increased interactivity and control over our virtual environments, accessed through physical interactions. Through this research we will enable this while retaining and encouraging a rich design aesthetic for these objects and environments which will enhance all our experiences of living with this future.

Publications

10 25 50
publication icon
Benford S (2017) Crafting Interactive Decoration in ACM Transactions on Computer-Human Interaction (ToCHI)

 
Title Interactive Coaster for Busaba Eathai 
Description Designs for coaster were created that both match the Busaba visual style and embed artcodes. The interactive coasters are designed to work with loyalty element of the existing Busaba app. Customers will be provided offers and rewards through scanning the coasters. 
Type Of Art Artefact (including digital) 
Year Produced 2015 
Impact The interactive coasters are enabling us to explore how Artcodes can be used as a service level interactive device in the dining context. 
URL http://www.artcodes.co.uk/rf-coaster/
 
Title Interactive illustrations 
Description Three large interactive illustrations created by pairs of professional writers and illustrators: "Stalkers", "Who am I?" and "A princess honeymoons in our town". Users can explore a rich digital narrative by scanning visual elements (artcodes) using their mobile phones (utilizing our novel spatial and layered interaction techniques). 
Type Of Art Artefact (including digital) 
Year Produced 2015 
Impact The interactive illustrations were exhibited at a special event at the Nottingham Writers' Studio and explored by about 20 attendees. One of the illustrations ( "A princess honeymoons in our town") will be permanently exhibited at a window facing the street so that the public can interact with it. The "Stalkers" illustration is currently being exhibited in the Horizon Digital Economy Research hub where visitors can interact with it. 
URL http://www.artcodes.co.uk/category/artefacts/
 
Title Interactive tiles 
Description Tiles decorated with aesthetically pleasing contemporary designs which incorporate artcodes, thus enabling interactivity by scanning the patterns with a mobile phone camera to reveal associated digital content. A panel of Artcode tiles demonstrates further interaction possibilities where uses can scan different paths or groups of codes to trigger different digital augmentations. 
Type Of Art Artefact (including digital) 
Year Produced 2015 
Impact The interactive tiles were exhibited at the British Ceramics Biennial in Stoke-on-Trent in 2015 and provoked discussion with visitors about how they might be used in the home. Further workshops are planed with home owners and interior designers to develop compelling use scenarios. 
URL http://www.artcodes.co.uk/rf-interactive-tiles/
 
Description Key findings:
- We have gained a deeper understanding of the process and value of crafting interactive decoration (artcodes) for everyday artefacts. (This involves adorning artefacts with decorative patterns that enhance their beauty while also triggering digital interactions when scanned with mobile cameras.) We have identified new visual design techniques to bridge between human and system interpretations of patterns and new material techniques to account for the materials, forms, structures and functions of artefacts. Our findings also highlight the need for transparent computer vision technologies. (Reported in "Crafting Interactive Decoration" by Benford et al, submitted to ACM ToCHI)
- We have developed and explored two new techniques that are intended to scale up interactions with visual markers on large surfaces (such as murals, tiled walls, posters, billboards, hoarding and street art) and extend the capabilities of graphic and media designers to create aesthetic designs that are associated with rich digital narratives. The first concerns spatial interactions in which users scan sequences or groups of visual markers that are embedded within the surface decoration, panning across the surface or stepping forwards and backwards to explore a narrative. The second addresses layered interactions in which different layers of visual markers appear to be overlaid on the same location on the surface, with users switching between them as they interact. (Reported in the paper "Exploring Interactive Public Illustrations" by Thorne at al, to be published at ACM DIS 2016)
- We have identified design and interaction issues for this new medium (of making static illustrations and decorations interactive through visual markers) to guide researchers and practitioners in future work. From the designer's perspective these include different strategies for incorporating using spatial and layered interaction, and the flow of the design process. From the interaction perspective these include guidelines around embedding instructions, interaction position, user collaboration, and designing for repeated experiences. (Reported in the paper "Exploring Interactive Public Illustrations" by Thorne at al, to be published at ACM DIS 2016)
- We have extended the capabilities of artcodes (as an example of a topological fiducial marker) by adding geometrical constraints to achieve greater scalability (larger code space), new interaction possibilities and reliability while at the same time retaining openness to designers and inspiring creativity. We also propose seven criterion for the evaluation of topological marker technologies - scalability, the impact on the number of available codes; visual aesthetic, impact on the appearance of the visual design; inspiration, how they inspire creativity in visual design; interaction, what new interactional possibilities they enable; comprehensibility, how easy is it for a designer to understand; reliability and process, how well does it fit into the design process of encoding a code. (Reported in paper "Extending Human Designed Topological Markers" by Preston et all, to be submitted to ACM UIST 2016)
Exploitation Route - Guides future research in the field of creating interactive physical objects through visual markers
- Guides practitioners (graphic and media designers) in how to adopt visual markers (and artcodes in particular) in order to create interactive pieces
- Demonstrates scenarios of use to the creative, services, leisure, and museum sectors to show the value of adorning physical objects with interactive decoration that links to digital augmentations
- Provides software (library and apps) which can be used by researchers and organizations to develop new artcode experiences
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

 
Description Professional designers and artists: through the Artcodes app, example artefacts, guidelines and events (such as London Tent at London Design Fair 2015 stand, British Ceramic Bianual 2015 exhibit, Nottingham Writers' Studio Storicodes exhibition, "All Makers Now?" conference presentation and demo, ArcInTex Network Workshop) designers have been supported in creating their own interactive objects and associated experiences. Artists Alice Angus and Simeon Featherstone have already independently used artcodes to create public experiences. (in an artwork inspired by the Lymington Saltmarshes and the Hidden Language Hidden Trails artwork in Dorchester respectively). Application domains: - Restaurant service setting - through our collaboration Busaba Eathai have explored scenarios of how Artcode decorated objects can be used to enhance the customer dining experience, new Artcode designs (matching the Busaba branding style) and examples of Artcode decorated objects and training of Busaba staff in designing Artcode patterns - Domestic setting (interior design and architecture) - through our collaboration Johnson Tiles have explored scenarios for using interactive tiles in the home and new Artcode designs for tiles (following Johnson Tiles latest trends). We have also engaged with interior designers and architects through events (British Ceramic Biannual 2015 exhibit and The Social: Artcode Workshop) to inspire the use of interactive decorative patterns in the built environment. - Cultural sites and museums: through the "Heritage and the Digital Hack Days" (Wollaton Hall) artcodes were promoted to practitioners and professionals who engage the public with heritage and example experiences were created. General public: members of the public have learned about how physical objects can be made interactive through artcode decorative patterns, including making their own at events we have organised or exhibited at (London Tent at London Design Fair 2015 stand, British Ceramic Biannual 2015 exhibit, Creative Class at Debbie Bryan Studio & Shop, May Fest 2015) and have been kept up to date with the #aestheticodes tag.
First Year Of Impact 2014
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal

 
Description Busaba Eathai (restaurant setting) 
Organisation Busaba Eathai
Country United Kingdom 
Sector Private 
PI Contribution - scenarios of how Artcode decorated objects can be used to enhance the customer dining experience - new Artcode designs (matching the Busaba branding style) and examples of Artcode decorated objects - training of Busaba staff in designing Artcode patterns (through workshops)
Collaborator Contribution - providing a real world setting enabling us to explore how Aestheticodes can enhance the customers' dining experiences in a managed service environment - hosting of workshops and recruitment of participants (both staff with different roles and customers)
Impact Outputs: - Tent London at London Design Week 2014 stand - Interactive Coasters for Busaba Eathai -"Crafting Interactive Decoration" paper Multidisciplinary: - HCI, design, vision, software development, restaurant services
Start Year 2010
 
Description Interactive Greeting Cards with Debbie Bryan 
Organisation Debbie Bryan Studio & Shop
Country United Kingdom 
Sector Private 
PI Contribution - training of Debbie Bryan and Jodie Paterson (creative professionals) in creating Artcodes - participation in planning of the creative class and facilitation of the artcode drawing session - documentation of the creative class - exploration of scenarios of how artcodes can be used in handcrafted products such as greeting cards
Collaborator Contribution - hosting of the Aestheticodes creative class at Debbie Bryan Studio & Shop (including providing craft materials) and recruiting participants - participation in meetings to design the creative class - preparation of sample materials (such as artcode templates and examples of pre-made cards)
Impact Outputs: - Aestheticodes creative class at Debbie Bryan Studio & Shop - Tent London at London Design Fair (September 2014) stand - "Crafting Interactive Decoration " paper
Start Year 2014
 
Description Johnson Tiles (domestic setting) 
Organisation Johnson's Tiles
Country United Kingdom 
Sector Private 
PI Contribution - New Artcode designs for tiles (following Johnson Tiles latest trends) - Scenarios for using interactive tiles in the home - Exploration of interior designers and home owners preferences (through participative design workshops)
Collaborator Contribution - fabrication of tiles (with Artcode decorative patterns) including materials and production time (machines and staff) - staff time for attending meetings and workshops
Impact Outcomes: - British Ceramics Biennial (BCB) 26 September-8 November 2015 exhibit - The Social: Artcode Workshop - Interactive Tiles - Artcodes app - Storicodes app - "Crafting Interactive Decoration" paper Multidisciplinary collaboration: - design, HCI, vision, software development, tile design and production
Start Year 2014
 
Description Nottingham Writers' Studio (interactive illustrations) 
Organisation Nottingham Writers' Studio
Country United Kingdom 
Sector Public 
PI Contribution - facilitated the process of creating large scale interactive illustrations by three teams of professional writers and illustrators - documented the exhibition at the Nottingham Writers' Studio and analysed findings - synthesized the opportunities and challenges of the medium (large scale illustrations embedded with artcodes) to guide practitioners (writers and illustrators) in creating future interactive works
Collaborator Contribution - hosting of the Interactive Illustrations exhibition at the Nottingham Writers' Studio, publicizing the event and inviting attendees - participation in planning meetings for the event
Impact - Nottingham Writers' Studio Storicodes exhibition - Interactive illustrations (artifacts) - Storicodes app - "Exploring Interactive Public Illustrations" paper
Start Year 2015
 
Title Artcodes library 
Description Library which allows Artcode scanning functionality to be integrated in apps. (The included URL if for the iOS version. An Android version is also available at: https://github.com/horizon-institute/artcodes-android) 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact It underpins the Artcodes and Storicodes mobile apps. The open source release of the library enables other developers to create bespoke apps that make use of Artcodes. The library is currently being used by undergraduate and PhD projects. 
URL https://github.com/horizon-institute/artcodes-ios/
 
Title Artcodes mobile app 
Description The Artcodes app allows users to scan artcodes (on objects or surfaces) and view associated digital content on their mobile phone. It also allows users to debug artcode patterns they are creating themselves by seeing what the app is currently detecting. The major extension added to the Artcodes app as a result of this project is the ability for users to create their own experiences on the phone (by providing an experience name, description, image and assigning URLs to codes). Created experiences can then be made public and shared with other users. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact The Artcodes app has been used in a number of workshops with external users (architects, local councils and museums) to teach them to create artcodes and associated experiences themselves. The app has already been used independently by artists to create public experiences. Most notably by Simeon Featherstone to create the Hidden Language Hidden Trails artwork in Dorchester and by Alice Angus in artwork inspired by the Lymington Saltmarshes. 
URL https://itunes.apple.com/gb/app/artcodes/id703429621?mt=8
 
Title Storicodes mobile app 
Description Enables users to interact with wall illustrations by scanning visual elements to explore associated digital narratives on their mobile phone Supports novel: - spatial interactions in which users scan sequences or groups of visual markers that are embedded within the illustration, panning across its surfaces or stepping forwards and backwards to explore a narrative - layered interactions in which different layers of visual markers appear to be overlaid on the same illustration (through colour filtering), with users switching between them as they interact 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact - has been used by 20 members of the public at an event at the Writers Studio Nottingham - will be used by members of the public during the planned long-term public deployment of the large scale interactive illustrations that have been created in the project - enables us to explore the research question of how we can enable interaction with large physical surfaces through embedding multiple visual codes 
URL https://itunes.apple.com/gb/app/storicodes/id1038128280?mt=8
 
Description "All Makers Now?" 2014 conference presentation and demonstration 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A presentation and demonstration of Artcodes was given to the conference delegates including teaching them how to draw their own.
This generated discussion about different materials that Artcodes could be applied to and led to some participants requesting to be considered for future collaborative projects involving Artcodes.
Year(s) Of Engagement Activity 2014
URL http://www.autonomatic.org.uk/allmakersnow/
 
Description Aestheticodes creative class at Debbie Bryan Studio & Shop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact 8 members of the public attended the creative class where they learned how to draw artcodes and then how to create them through paper-cutting and embroidery techniques. Each participant then created their own greeting card incorporating an artcode in the decoration so that it could be linked to a digital message.
Consequently we are exploring ideas about setting up a pop-up shop and incorporating artcodes in packaging. Debbie Brian has designed her own artcode stamp (linking to detail about her shop) to stamp wrapped gifts in her shop.
Year(s) Of Engagement Activity 2014
URL http://www.creativequarter.com/community/blog/583206/free-creative-class-aestheticodes
 
Description ArcInTex Network Workshop on "Narratives in An Internet of Soft Things" Nottingham Trent University (Aestheticodes & Colour Change Print) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact A 3 day workshop following a design centered approach to explore the application of Aestheticodes (Artcodes) for smart changeable textiles.
10 attendees learned how to create Artcode patterns and use photochromic inks to screen print their designs on textile surfaces. The workshop allowed us to explore the novel idea of designing dynamically changing (depending on light) codes and associated digital narrative.

The artist Alice Angus has consequently began to explore the use of photochromic and thermochromic inks to apply Artcodes on textiles (http://aliceeangus.net/Artcodes).
Year(s) Of Engagement Activity 2015
URL http://arcintex.hb.se/conferences-workshops/
 
Description British Ceramics Biennial (BCB) 26 September-8 November 2015 exhibit 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact A large interactive tile panel was exhibited (tiles fabricated by Johnson Tiles) prompting a wide range of visitors to consider how they might like to use interactive surfaces in their home. `

(listed on pp. 22-23 in the attached guide to BCB 2015)
Year(s) Of Engagement Activity 2015
URL http://britishceramicsbiennial.com/
 
Description Heritage and the Digital Hack Days (Wollaton Hall) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Third sector organisations
Results and Impact A series of 3 workshops with representatives from museums, cultural heritage sites and local council. There were about 15 attendees at each workshop where they were taught how to draw and use artcodes. Then they created artcodes for a specific site and an associated heritage experience. The events promoted artcodes to practitioners and professionals who engage the public with heritage.
Year(s) Of Engagement Activity 2015
 
Description Hong Kong University of Science and Technology talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invited talk given by Richard Mortier at Hong Kong University of Science and Technology on the Artcodes technology.
The students in the audience sketched their own codes and tried out the Artcodes app by scanning them.
Year(s) Of Engagement Activity 2015
URL http://www.cse.ust.hk/pg/seminars/S15/mortier.html
 
Description May Fest 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact At MayFest 2015 (an open day at the University of Nottingham for for the public) we showed visitors how to make their own Artcode (i.e. interactive) badges. Visitors learned how to draw Artcodes and create your own Artcode design which we then made into a button badge for the visitor to take away. Visitors also learned how to scan their button badge by downloading the Artcodes app onto their phones (which links to a special Mayfest webpage we created).
Year(s) Of Engagement Activity 2015
URL http://nottingham.ac.uk/mayfest/community/whatson/getcreative.aspx
 
Description Nottingham Writers Studio Storicodes exhibition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Graphic designers and writers were invited to a three-hour evening event where the interactive large scale illustrations were exhibited. Attendees were given an introduction to the project and then could freely explore the interactive illustrations using their phones.
Consequently one of the illustrations will be exhibited permanently at the Writers Studio at a window facing the street so that the public can interact with it.
Year(s) Of Engagement Activity 2015
URL http://www.creativequarter.com/community/blog/2416017/review-of-storicodes-at-nottingham-writers-stu...
 
Description Tent London at London Design Fair (September 2014) stand and interactive signage 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact 24,392 visitors in total to Tent London 2014
We had a large exhibition stall presenting Artcodes (Aestheticodes), teaching visitors to draw their own and running different craft activities. This provided us with the contact details of a range of businesses and professional designers who were interested in further information and making use of Artcodes in future projects.

We created a series of interactive Artcode signs/images placed across the event that could be scanned with the Artcode app by visitors to get more information about the event, different halls and exhibits. This serves as a demonstrator on now Artcodes can be used to guide visitors during large events and helped to promote our technology to the public (with many visitors consequently coming to our stall).
Year(s) Of Engagement Activity 2014
URL http://www.londondesignfair.co.uk/news/aestheticode-tent-app
 
Description The Social: Artcode Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A workshop was held with about 30 architects (in partnership with RIBA London) where they learned how to draw artcodes and how they can be used to access digital augmentations. The participants then discussed how artcodes can be used within the built environment to deliver content and services.
Year(s) Of Engagement Activity 2015
URL https://www.architecture.com/WhatsOn/Feb2015/TheSocialArtcodeworkshopwithAestheticodes.aspx
 
Description aestheticodes twitter feed 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Twitter account with 560 followers providing the public and professional designers with updates on artcode developments and events. Has lead to requests for more information and participation in workshops and other events.
Year(s) Of Engagement Activity 2014,2015,2016
URL https://twitter.com/aestheticodes