Evaluating Social Media to Identify and Leverage Engagement with Arts and Culture Experiences
Lead Research Organisation:
University of Warwick
Department Name: Sociology
Abstract
This project aims to (1) Build upon work on social media analytics from the Qualia project by taking that research in a new direction to develop deeper insights about mediated responses to arts and culture experiences, (2) expand on this prior research by including international expertise and cross disciplinary working- spanning arts technology, communication, sociology and computer science- to deliver new insights about social media analytics, (3) develop a preliminary model of the relationship between discourse on social media and authentic views held by social media users, based on researching discussions about arts and culture experiences occurring on social media, (4) establish an empirical basis for developing a new categorical sentiment analysis tool focusing on social media content using online ethnography and conversation analysis to identify the ways in which social media are used to communicate about arts and culture experiences and (5) Develop a new prototype open source sentiment analysis tool [SMILE] for arts and culture discourse to provide a practical test of the initial findings about automated social media analysis from the preceding Qualia project and (6) disseminate new sentiment analysis tool and associated research and practical recommendations through practitioner workshops and web-based communications.
The project will research the potential of making otherwise expensive digital tools and techniques available to arts and culture organisations, while pushing the frontiers of applied research using machine learning technology. Meanwhile, we will develop new knowledge about the 'medium effects' that characterise arts and culture discourse within social media. We propose to conduct an online ethnography focusing on a small stratified random sample of Twitter users discussing partner arts and culture organisations on Twitter, aiming to uncover the relationship between online and offline discourse. A realistic understanding of the limits of what social media discourse can reveal is essential at this time when such data is widely seen as an unproblematic source of audience insights.
Ultimately, the SMILE Project will deliver a Prototype Sentiment Analysis Tool for use on Twitter data ('SMILE tool') to provide a focal point for this research on the limits of social media analytics. The Sentiment Analysis tool will initially require developing a manually annotated corpus tuned and calibrated to arts and culture discourse. The SMILE sentiment analysis tool pulls tweets related to specified searches (i.e. for an arts organisation) from Twitter for processing. The SMILE tool extends beyond positive / negative, instead categorising responses according to 'quality of experience' categories, specific emotional responses and other relevant categories identified through the proposed online ethnography research and engagement with partner arts organisations.
The SMILE analytic tool may offer organisations a more robust understanding of their online audiences, while establishing a transferable tool for measuring cultural demand and interests. Meanwhile, an interdisciplinary dialogue amongst the partners representing arts, computer science and social science perspectives will run alongside the tool development process. This dialogue will yield clear findings about the technical, methodological and practical limitations of this approach to understanding publics. The work proposed here will use an online ethnography study to develop a preliminary theoretical model for Twitter interactions. While our immediate focus is arts and culture discourse, the project holds broad implications for big data analysis.
The project will research the potential of making otherwise expensive digital tools and techniques available to arts and culture organisations, while pushing the frontiers of applied research using machine learning technology. Meanwhile, we will develop new knowledge about the 'medium effects' that characterise arts and culture discourse within social media. We propose to conduct an online ethnography focusing on a small stratified random sample of Twitter users discussing partner arts and culture organisations on Twitter, aiming to uncover the relationship between online and offline discourse. A realistic understanding of the limits of what social media discourse can reveal is essential at this time when such data is widely seen as an unproblematic source of audience insights.
Ultimately, the SMILE Project will deliver a Prototype Sentiment Analysis Tool for use on Twitter data ('SMILE tool') to provide a focal point for this research on the limits of social media analytics. The Sentiment Analysis tool will initially require developing a manually annotated corpus tuned and calibrated to arts and culture discourse. The SMILE sentiment analysis tool pulls tweets related to specified searches (i.e. for an arts organisation) from Twitter for processing. The SMILE tool extends beyond positive / negative, instead categorising responses according to 'quality of experience' categories, specific emotional responses and other relevant categories identified through the proposed online ethnography research and engagement with partner arts organisations.
The SMILE analytic tool may offer organisations a more robust understanding of their online audiences, while establishing a transferable tool for measuring cultural demand and interests. Meanwhile, an interdisciplinary dialogue amongst the partners representing arts, computer science and social science perspectives will run alongside the tool development process. This dialogue will yield clear findings about the technical, methodological and practical limitations of this approach to understanding publics. The work proposed here will use an online ethnography study to develop a preliminary theoretical model for Twitter interactions. While our immediate focus is arts and culture discourse, the project holds broad implications for big data analysis.
Planned Impact
This proposal is born out of a drive to use technology to enhance the impacts of public engagement activities, initially within the arts and culture sector. The primary audiences for the project will be public-facing arts and culture organisations of many different shapes and sizes and academic researchers working in the field of social media analytics. The secondary beneficiary of the project will be current and future arts and culture audiences, who will be better served by arts organisations attuned to their interests, needs and experiences, as well as computer and data scientists, who may learn from this unique intervention into the big data arena.
If successful, the technology showcased through the prototype SMILE tool will offer arts and culture organisations a more robust understanding of audiences. The downstream technology could enable arts organisations to be better attuned to the needs and interests of current and potential audiences, enabling them to deliver more effective programming for a broader market. Moreover, the prototype tool we develop will open up audience analysis and market research to all arts and culture organisations, not just the select few that can afford expensive consultants or hire expert technical staff in house. The idea of a categorical sentiment analysis tool (suggested by the Qualia project) will be tested in practice through the SMILE sentiment analysis tool. As the development process will follow open source guidelines allowing for re-use of the technology and code, the long-term benefits of the project could emerge in completely unexpected ways.
While this project is focused on delivering a useful product for the arts and culture sector in the UK, the research, technical and organisational insights it develops will be of broad use across UK society and globally, providing a blueprint for the use of a refined categorical form of sentiment analysis to create economic and social impacts. The project findings and outputs will be used as a catalyst for sector change, informing digital cultural policy and best practice, and providing resources for staff development across the national arts and culture sector. We will also be contributing to development in technology and research that may have long-term impacts well beyond the immediate horizon of this project. This potential for unintended broader impacts is reinforced by the use of an open source approach.
In order to realise the important proximal and distal impacts, we have a multi-faceted dissemination plan including technical and practitioner-oriented project blogs, newsletters, briefings, dissemination workshops, website resources and academic publications. It is worth noting at the outset that the project team has excellent existing links and networks within this sector on which to draw upon to ensure uptake of the SMILE tool.
If successful, the technology showcased through the prototype SMILE tool will offer arts and culture organisations a more robust understanding of audiences. The downstream technology could enable arts organisations to be better attuned to the needs and interests of current and potential audiences, enabling them to deliver more effective programming for a broader market. Moreover, the prototype tool we develop will open up audience analysis and market research to all arts and culture organisations, not just the select few that can afford expensive consultants or hire expert technical staff in house. The idea of a categorical sentiment analysis tool (suggested by the Qualia project) will be tested in practice through the SMILE sentiment analysis tool. As the development process will follow open source guidelines allowing for re-use of the technology and code, the long-term benefits of the project could emerge in completely unexpected ways.
While this project is focused on delivering a useful product for the arts and culture sector in the UK, the research, technical and organisational insights it develops will be of broad use across UK society and globally, providing a blueprint for the use of a refined categorical form of sentiment analysis to create economic and social impacts. The project findings and outputs will be used as a catalyst for sector change, informing digital cultural policy and best practice, and providing resources for staff development across the national arts and culture sector. We will also be contributing to development in technology and research that may have long-term impacts well beyond the immediate horizon of this project. This potential for unintended broader impacts is reinforced by the use of an open source approach.
In order to realise the important proximal and distal impacts, we have a multi-faceted dissemination plan including technical and practitioner-oriented project blogs, newsletters, briefings, dissemination workshops, website resources and academic publications. It is worth noting at the outset that the project team has excellent existing links and networks within this sector on which to draw upon to ensure uptake of the SMILE tool.
People |
ORCID iD |
| Eric Jensen (Principal Investigator) | |
| Maria Liakata (Co-Investigator) |
| Description | We have noted some challenges in interpreting and labelling sentiments for analysis: There is a significant positive bias in the kinds of sentiments which users post about their museum experiences. (Associated with the self-selecting and public nature of this type of feedback) Museums should be made aware that users who have negative experiences are unlikely to post about them. Therefore a lack of negative feedback should not be interpreted as a lack of negative experiences. The bias means that sentiment analysis alone - even with a large sample of tweets - is unlikely to provide an accurate picture of museum visitors' overall experiences. A better means of judging the proportion of positive experiences would be to chart the number of tweets with positive labels such as Complimentary, Happy, Excited, Interested against the number of visitors in a given day. The variations over time would build a picture of the relative themes wihtin the bounds of generally positive feedback. Tweets with positive sentiment labels do often reflect authentic positive experiences. Sometimes these tweets indicate profound emotional attachment to the museum experience. Moreover, for all intents and purposes, Tweets with neutral labels also often reflect enjoyment of the museum, an implied positive sentiment, and are intended to publicise the museum to others. This should be taken account when analysing informative and descriptive tweets. Evidence of generational and gender differences in patterns of self-expression on Twitter (that could in future be taken into account in sentiment analysis). Of particular interest is the tendency for female users to avoid hostile speech to a greater degree than male users. Greater incidence of critical sarcasm among male users as opposed to female users, and a correspondingly greater proportion of "endorsement" tweets among female users. Examining patterns of self-expression not only within tweets about the museums, but also across individual users' accounts (who tweeted about the museums), could help to build a clearer picture of these differences. Based on our analysis of interview material, users view Twitter as a "performance platform" and their tweets as a "performance moment". Tweets are an opportunity to "promote" organisations such as museums while simultaneously building their online profiles. Due to self-selection, we should expect to see a high proportion of "professional" tweets - i.e. tweets from individuals in arts-related professions - in longer tweets about the museum organisations (containing more sentiment information). Shorter tweets with less information, which are often informational messages linking the tweet to a selfie or other photograph hosted on Flickr or Instagram, are not as much the domain of arts professionals. The main conclusion of this report is the importance of taking socio-cultural factors into account in sentiment analysis. The same social factors governing everyday speech also influence speech on Twitter, with the added complexity of new conventions particular to social media. Sentiments expressed on Twitter cannot be understood as raw, unmediated data about the inner workings of users' minds. If the factors described above are taken into consideration (which will be challenging technically and methodologically), Twitter provides a mine of valuable information about users' experiences. For the moment, live interviews with users remain an essential tool in analysing the significance of tweets. |
| Exploitation Route | Informal learning and public engagement institutions can use our findings to understand the limits of what they can glean from social media data, as well as the opportunities. The findings also contribute to general methodological knowledge about 'big data' approaches that use social media data. |
| Sectors | Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections |
| URL | http://culturesmile.org |
| Description | Our end of project workshops were well received, with attendees indicating they would use the insights from the project to benefit their practices (modifying how they use social media data to understand their audiences) |
| First Year Of Impact | 2016 |
| Sector | Culture, Heritage, Museums and Collections |
| Impact Types | Cultural |