Social Media Analysis for Social Geography
Lead Research Organisation:
Newcastle University
Department Name: Computing Sciences
Abstract
Tweet My Street (TMS) is a project within the SiDE Social Inclusion Through the Digital Economy Hub that has been exploring the extent to which data derived from Twitter can reveal more about spatial and temporal behaviours, and the meanings attached to these locally. This has been done with a longer-term view to supporting the coproduction and delivery of local services, effective complaint mechanisms and horizontal community support networks.
The project has exposed, and is tackling, 2 of the main barriers to realising the full potential for the use of Twitter (and related social media data) in the social sciences:
i) The lack of availability of tools that can allow social scientists to extract meaning from social media data. Capturing, storing, querying and visualising data derived from sources is very challenging given: a) the high rate at which it is being generated, b) the vast scale of historic data, c) the complexity involved in querying the data
ii) The lack of understanding of methodological issues in the analysis of social media data, for example ethics and the validity of any conclusions drawn from the skewed sample of the population that social media users represent
As these issues cannot be addressed by expertise in one field alone, the project is a multi-disciplinary collaboration involving social geographers, computer scientists and the computer industry (Red Hat Inc. who fund a PhD student working in the group, and take an active part in project meetings). The involvement of the latter represents the fact that the outputs of the research have relevance in many other areas, including marketing, e-commerce and policing.
To date the project has generated some promising early results, including:
- a scalable, cloud-based software infrastructure for the real-time and historic analysis of twitter data. We have demonstrated that this can scale to process data at the full rate that tweets are generated globally.
- a web-based tool to provide an easy-to-use way for social scientists to query social media data and visualise the results, for example on a map.
Just as importantly, it has established a successful virtuous circle for collaboration: the social scientists use the software tool to analyse twitter data in order to advance their case study; this leads to new requirements on the tool which the computer scientists attempt to address; the tool is enhanced and passed to the social scientists. The cycle then repeats.
This project will build on the work carried out in the SiDE Digital Economy Hub, leveraging its results and infrastructure to extend it in five key ways:
- to extend the software tool to include connection analysis (e.g. flows of tweets, changes in connection graphs following an event or intervention)
- to add to the collaboration a group of statisticians to explore the use of modern statistical techniques to extract understanding from twitter data, including social graph analysis, and modelling of network dynamics and meme propagation
- a new case study to explore the use of the tool to augment existing social science methods in connection analysis [Mike's work summarised here]
- a new case study to use the tool to augment existing social science methods, to understand the way third sector organisations are using social media and the barriers they have to overcome to do so
- to run an event to disseminate the results, the methodologies and give training on how to use the tooling
All the software generated by this project will be made freely available to the community as open-source software.
The project has exposed, and is tackling, 2 of the main barriers to realising the full potential for the use of Twitter (and related social media data) in the social sciences:
i) The lack of availability of tools that can allow social scientists to extract meaning from social media data. Capturing, storing, querying and visualising data derived from sources is very challenging given: a) the high rate at which it is being generated, b) the vast scale of historic data, c) the complexity involved in querying the data
ii) The lack of understanding of methodological issues in the analysis of social media data, for example ethics and the validity of any conclusions drawn from the skewed sample of the population that social media users represent
As these issues cannot be addressed by expertise in one field alone, the project is a multi-disciplinary collaboration involving social geographers, computer scientists and the computer industry (Red Hat Inc. who fund a PhD student working in the group, and take an active part in project meetings). The involvement of the latter represents the fact that the outputs of the research have relevance in many other areas, including marketing, e-commerce and policing.
To date the project has generated some promising early results, including:
- a scalable, cloud-based software infrastructure for the real-time and historic analysis of twitter data. We have demonstrated that this can scale to process data at the full rate that tweets are generated globally.
- a web-based tool to provide an easy-to-use way for social scientists to query social media data and visualise the results, for example on a map.
Just as importantly, it has established a successful virtuous circle for collaboration: the social scientists use the software tool to analyse twitter data in order to advance their case study; this leads to new requirements on the tool which the computer scientists attempt to address; the tool is enhanced and passed to the social scientists. The cycle then repeats.
This project will build on the work carried out in the SiDE Digital Economy Hub, leveraging its results and infrastructure to extend it in five key ways:
- to extend the software tool to include connection analysis (e.g. flows of tweets, changes in connection graphs following an event or intervention)
- to add to the collaboration a group of statisticians to explore the use of modern statistical techniques to extract understanding from twitter data, including social graph analysis, and modelling of network dynamics and meme propagation
- a new case study to explore the use of the tool to augment existing social science methods in connection analysis [Mike's work summarised here]
- a new case study to use the tool to augment existing social science methods, to understand the way third sector organisations are using social media and the barriers they have to overcome to do so
- to run an event to disseminate the results, the methodologies and give training on how to use the tooling
All the software generated by this project will be made freely available to the community as open-source software.
Planned Impact
The analysis of social media data has a range of potential beneficiaries:
Organisations in the public, private sector and 3rd sector who could improve their services and products though exploiting the information encapsulated in social media data. Early demos of the existing infrastructure at conferences and meetings have attracted interest from a range of organisations including the police, the home office, and a utility company. We are building relationships with these contacts , holding meetings to gain detailed feedback on the research outputs, and what would be needed for them to exploit it. Over time, we will also target companies in other sectors, for example marketing and advertising. We are making the infrastructure available as open-source in order to encourage take-up, and therefore impact.
Public Sector: Through SiDE and a TSB project we have excellent links to Newcastle City Council and have ongoing informal discussions with them on the potential value of using social media analysis to better understand it's citizens, and the impact of the services it delivers.
An aim is to look for opportunities to commercialise the outputs through licencing or a start-up company. Red Hat fund a PhD student working on the project, and their main contact gives regular input into the infrastructure design from a computer industry best-practise perspective.
Organisations in the public, private sector and 3rd sector who could improve their services and products though exploiting the information encapsulated in social media data. Early demos of the existing infrastructure at conferences and meetings have attracted interest from a range of organisations including the police, the home office, and a utility company. We are building relationships with these contacts , holding meetings to gain detailed feedback on the research outputs, and what would be needed for them to exploit it. Over time, we will also target companies in other sectors, for example marketing and advertising. We are making the infrastructure available as open-source in order to encourage take-up, and therefore impact.
Public Sector: Through SiDE and a TSB project we have excellent links to Newcastle City Council and have ongoing informal discussions with them on the potential value of using social media analysis to better understand it's citizens, and the impact of the services it delivers.
An aim is to look for opportunities to commercialise the outputs through licencing or a start-up company. Red Hat fund a PhD student working on the project, and their main contact gives regular input into the infrastructure design from a computer industry best-practise perspective.
Publications
Mearns G
(2015)
Enacting the internet and social media on the public sector's frontline
in New Technology, Work and Employment
Mearns G
(2014)
Tweet My Street: A Cross-Disciplinary Collaboration for the Analysis of Local Twitter Data
in Future Internet
Description | We have discovered new ways to understand how individuals and communities are affected by events through analysing social media data. This can be done in real time, or by looking back over a period of time. The tools we have designed and built allow users to home in on particular events, and visualise the results on a map. - design and evaluation of a novel, scalable architecture for the historic and real time analysis of social media data - new understandings of the effect of community interventions through the evaluation of social media data - new understandings of the evolution of social media networks |
Exploitation Route | We envisage that the tools we have created could be adopted by a wide range of sectors. The Digital Institute at Newcastle University (of which the PI is the Director) collaborates widely on multi-disciplinary projects and it is likely that other projects involving social media analytics will be undertaken in the coming months. For example, a (not yet formally announced) large new project about to begin in the Newcastle University medical school will utilise social media data analysis for predicting trends in new medical advances. The Digital Institute is a collaborator in the project and the results and experience from this SOMAG project will be built on in the new project. |
Sectors | Aerospace Defence and Marine Agriculture Food and Drink Chemicals Communities and Social Services/Policy Construction Creative Economy Digital/Communication/Information Technologies (including Software) Education Electronics Energy Environment Financial Services and Management Consultancy Healthcare Leisure Activities including Sports Recreation and Tourism Government Democracy and Justice Manufacturing including Industrial Biotechology Culture Heritage Museums and Collections Pharmaceu |
Description | The project's approach is to apply a combination of skills - in computing, social geography and statistics to social media analytics. We have explored it's potential impact outside academia in 2 ways: 1. Meetings with companies and organisations who see the potential to understand more about their product, services, customers and advertising. The knowledge gained in the project has allowed us to advise a range of companies on their social media analytics strategy, ranging from SMEs to a FTSE100 company. 2. Presentations at non-academic events such as special interest groups (e.g. the North East Fraud Forum) and commercial conferences. As a result of contacts made at these events, we have had several meetings in which we have informally advised public sector organisations on the potential (and pitfalls) of social media analytics in their areas of interest. These interactions have informed the design of tools created in the project, and highlighted opportunities for impact. We disseminated this knowledge, and the linkages we have made, in a set of events for external partners that were held towards the end of the project. Much of the work with industry and the public sector has been done through the DCMS-funded Cloud Innovation Centre. For example, on 10th May 2017, we conducted an all-day design review with the Department of Work and Pensions (DWP) at which we discussed and demoed the SOMAG system, before using the lessons we had learnt from the project to advise a group in DWP on how to address some problems they had faced in a new social media analysis project they had recently begun. In March 2016 we were awarded £30M to create the National Innovation Centre for Data (£15M from Treasury, £15M from Newcastle University). This will enable us to dramatically expand our work of helping companies increase their ability to extract value from their data through improving their skills, building their networks and carrying out joint projects with them. The knowledge we gained in "ES/M001636/1 - Social Media Analysis for Social Geography" will allow us to give better advice, not just in social media analysis, but also more generally in event processing, spatial data visualisation and textual analysis. One specific example is that the work on visualization has led to the provision of a new "Decision Theatre" facility in the National Innovation Centre for Data. Once commissioned (in April 2020) this will be used by tens of external organisations per year - in the private and public sector - to explore their data and use it to make decisions that benefit society and the economy. In parallel with this we continue to pursue research opportunities to build on the work of the project. Several projects have built on the outputs of this project. N.B. we were unable to publish the research data to a repository such as ReShare for 2 reasons: 1. The primary twitter data had to be purchased under a licence that prevents re-distribution. 2. The secondary data generated from it embodies a set of ethical issues that were explored by the social scientists in the project (please see the publications for a full explanation of the issues). For example, it can be used to identify individuals, so putting them at risk. |
First Year Of Impact | 2015 |
Sector | Government, Democracy and Justice,Security and Diplomacy,Other |
Impact Types | Economic |
Description | DCMS UKBD Go Digital Project |
Amount | £2,000,000 (GBP) |
Organisation | Department for Digital, Culture, Media & Sport |
Sector | Public |
Country | United Kingdom |
Start | 01/2014 |
End | 03/2015 |
Description | National Innovation Cenre for Data |
Amount | £15,000,000 (GBP) |
Funding ID | EP/R018855/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2016 |
End | 04/2020 |
Description | Red Hat |
Organisation | Red Hat UK |
Country | United Kingdom |
Sector | Private |
PI Contribution | Work with them on NoSQL databases and cloud platfroms. Designing and building prototypes |
Collaborator Contribution | Work with them on NoSQL databases. Joint Supervision and funding of PhD students |
Impact | They have created a Cloud Research Centre, co-located with us in Newcastle. It is one of only 2 Red Hat centres in the world. |
Start Year | 2008 |
Title | Social Media Analytics Software |
Description | The software allows the visualization and searching of social media data. It is used by researchers to explore subsets of twitter tweets that meet some criteria. |
Type Of Technology | Software |
Year Produced | 2016 |
Open Source License? | Yes |
Impact | This has given the sociologists new insights in the transmission of information through twitter. The data analytics skills developed by the computer scientist working on the project are now being applied by her in industry (Red Hat) |
Description | Conference Presentation |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Conference paper presented to an audience of approx. 70 geographers and other social scientists from cognate disciplines at the 7th Nordic Geographers Meeting. Paper entitled 'Visualising Inequalities. Engaging with (critical) visual methods in Geography'. |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.humangeo.su.se/english/ngm-2017 |
Description | Conference paper: "Big Data, the Internet of Things and innovation in the city: the case of transport". Paper presented at the Annual Conference of the Association of American Geographers (AAG), San Francisco, 29 March - 2 April, 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Richardson, R. and Robson, L. (2016) "Big Data, the Internet of Things and innovation in the city: the case of transport". Paper presented at the Annual Conference of the Association of American Geographers (AAG), San Francisco, 29 March - 2 April, 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Conference paper: 'Mapping transnational conversations about the International Day Against Homophobia and Transphobia using "big" geospatial data', Annual Meeting of the Association of American Geographers, April 21-24, Chicago, US. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Study participants or study members |
Results and Impact | Mearns, G. (2015) 'Mapping transnational conversations about the International Day Against Homophobia and Transphobia using "big" geospatial data', Annual Meeting of the Association of American Geographers, April 21-24, Chicago, US. |
Year(s) Of Engagement Activity | 2015 |
Description | Conference paper: 'Tweet my street: A geographical snapshot of digital culture in North-East England' International Geographical Union Conference, Krakow, Poland, August 18-22, 2014 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Study participants or study members |
Results and Impact | Mearns, G. and Simmonds, R. (2014) 'Tweet my street: A geographical snapshot of digital culture in North-East England'. International Geographical Union Conference, Krakow, Poland, August 18-22. |
Year(s) Of Engagement Activity | 2014 |
Description | Conference paper: Big Data and City Governance: Developing, managing and monitoring the (SMART?) city-region'. Paper presented at the Annual Conference of the Association of American Geographers, April 21-24, Chicago, US. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Study participants or study members |
Results and Impact | Robson, L. and Richardson, R. (2015) 'Big Data and City Governance: Developing, managing and monitoring the (SMART?) city-region'. Paper presented at the Annual Conference of the Association of American Geographers, April 21-24, Chicago, US. |
Year(s) Of Engagement Activity | 2015 |
Description | Conference paper:'Utilising "Tweet My Street" to understand geographical nuances in support and opposition to the 2014 International Day Against Homophobia and Transphobia', Congress on the Social Sciences and Humanities, 1-3 June, Ottawa, Canada. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Study participants or study members |
Results and Impact | Mearns, G. and Simmonds, R. (2015) 'Utilising "Tweet My Street" to understand geographical nuances in support and opposition to the 2014 International Day Against Homophobia and Transphobia', Congress on the Social Sciences and Humanities, 1-3 June, Ottawa, Canada. |
Year(s) Of Engagement Activity | 2015 |
Description | Workshop paper: "Introduction to research on sexuality and digital culture". Paper presented at Sexual Networking and the Geosocial Web Workshop, Newcastle University, 24 June 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Mearns, G. (2016a) "Introduction to research on sexuality and digital culture". Paper presented at Sexual Networking and the Geosocial Web Workshop, Newcastle University, 24 June 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Workshop paper: "Tweeting about the general election: who and where?" Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Coombes, M. (2016) "Tweeting about the general election: who and where?" Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Workshop paper: "Utilising the Antares data analytics tool to interpret online communications about the International Day Against Homophobia and Transphobia". Paper presented at Sexual Networking and the Geosocial Web Workshop, Newcastle University, 24 June 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Mearns, G. (2016b) "Utilising the Antares data analytics tool to interpret online communications about the International Day Against Homophobia and Transphobia". Paper presented at Sexual Networking and the Geosocial Web Workshop, Newcastle University, 24 June 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Workshop paper: 'Introducing Social Media Analysis and Social Geography'. Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Richardson, R. (2016) 'Introducing Social Media Analysis and Social Geography'. Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Workshop paper: 'Utilising the Antares Data Analytics Tool to Interpret Social Media Communications' Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Mearns, D. and Turner, M (2016) 'Utilising the Antares Data Analytics Tool to Interpret Social Media Communications'. Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Workshop paper: James, P. (2016) "Going to the Beach': analysing user patterns through Twitter". Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | James, P. (2016) "Going to the Beach': analysing user patterns through Twitter". Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Worskshop paper: Wilkinson, D. (2016) "Measuring social networks through Twitter" Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Wilkinson, D. (2016) "Measuring social networks through Twitter" Paper presented at Social Media Analysis Event, The Core, Newcastle University, 27 May 2016 |
Year(s) Of Engagement Activity | 2016 |