Development and validation of the Social Media Experience measure: Using objective assessment and adolescents' experience to inform its development
Lead Research Organisation:
University of Manchester
Department Name: Environment, Education and Development
Abstract
The increased use of social media among young people has attracted the attention of the public, the media and the government, and has led to growing concerns about its impact on young people's mental health, wellbeing and levels of loneliness. This concern stems from reported increase in mental health difficulties and increased social media use among this population. Research on this area is however relatively new and with mixed evidence. While some of the experiences with social media can be challenging, there is not sufficient evidence to support that social media is fundamentally bad. Indeed, recent evidence challenges this and suggests that the connection between social media and mental health might be a weak one, and its benefits, that have been largely overlooked, should also be considered.
This area of research has suffered the consequences of a rapidly changing field, resulting in quick, but methodologically flawed self-report measures of social media experience, that hinders progress. We have identified three potential problems in the self-report measurement of social media engagement and experience:
1) Most measures were developed without asking young people's experiences. This means that social media measures are being developed for young people without young people having any input. How can we be sure we are asking the right things if we do not consider their views?; 2) Many measures focus on "addictive social media", however this term is based largely on anecdotal evidence. In fact, the questions they use in these measures are based on nicotine dependence and gambling addiction criteria. Assuming these are the same can lead to misleading conclusions; 3) Many of the existing measures were not developed using rigorous and robust theoretical and statistical (psychometric) methods. Their validity is therefore questionable; 4) Even though the engagement with social media includes objective digital behaviours, this kind of information and data have not been considered during the development of measures. We cannot capture however the full picture of social media experience without assessing both, because they each offer unique information.
To address these challenges and limitations reported in the current literature we propose a 3-year project to co-develop, with young people, a comprehensive and freely available self-report social media experience measure that will be appropriate for young people. This will take into account existing research, objective social media data, and the views of social media experts, clinicians, parents/carers, teachers, and policy makers. Importantly, the development of the measure will be guided by the views and experiences of young people.
The proposed project will follow a novel method that combines traditional methods of scale development and a novel approach that triangulates objective (e.g. online social media comments) and subjective (e.g. self-report) assessment. The current project has a strong focus on the voices of young people and it will be based on a co-production model with young people. We will draw from different disciplines including digital behaviour and social media, mental health, loneliness, psychometrics, and computer science. The project has the potential to improve the way we measure, and therefore understand, young people's social media experience and how that influences their mental health, wellbeing, and loneliness. It will also provide a meaningful and productive engagement and partnership with young people to advance this area of work and share their views to key stakeholders.
This area of research has suffered the consequences of a rapidly changing field, resulting in quick, but methodologically flawed self-report measures of social media experience, that hinders progress. We have identified three potential problems in the self-report measurement of social media engagement and experience:
1) Most measures were developed without asking young people's experiences. This means that social media measures are being developed for young people without young people having any input. How can we be sure we are asking the right things if we do not consider their views?; 2) Many measures focus on "addictive social media", however this term is based largely on anecdotal evidence. In fact, the questions they use in these measures are based on nicotine dependence and gambling addiction criteria. Assuming these are the same can lead to misleading conclusions; 3) Many of the existing measures were not developed using rigorous and robust theoretical and statistical (psychometric) methods. Their validity is therefore questionable; 4) Even though the engagement with social media includes objective digital behaviours, this kind of information and data have not been considered during the development of measures. We cannot capture however the full picture of social media experience without assessing both, because they each offer unique information.
To address these challenges and limitations reported in the current literature we propose a 3-year project to co-develop, with young people, a comprehensive and freely available self-report social media experience measure that will be appropriate for young people. This will take into account existing research, objective social media data, and the views of social media experts, clinicians, parents/carers, teachers, and policy makers. Importantly, the development of the measure will be guided by the views and experiences of young people.
The proposed project will follow a novel method that combines traditional methods of scale development and a novel approach that triangulates objective (e.g. online social media comments) and subjective (e.g. self-report) assessment. The current project has a strong focus on the voices of young people and it will be based on a co-production model with young people. We will draw from different disciplines including digital behaviour and social media, mental health, loneliness, psychometrics, and computer science. The project has the potential to improve the way we measure, and therefore understand, young people's social media experience and how that influences their mental health, wellbeing, and loneliness. It will also provide a meaningful and productive engagement and partnership with young people to advance this area of work and share their views to key stakeholders.
Technical Summary
The proposed project aims to co-develop with young people a robust and comprehensive social media experience measure. It aims to address the methodological limitations of current social media measures, including small samples, absent focus groups with young people and experts, and inappropriate or limited theory and psychometric analyses. The proposed project will follow a comprehensive multiphase mixed method research consistent with scale development theory and practices. The item development will rely on existing evidence, focus groups/cognitive interviews with young people and a Delphi study with social media experts, clinicians, parents and teachers. Beyond that, we aim to apply a novel method that triangulates information from subjective (e.g. self-report data) and objective (e.g. online social media comments) social media experience to inform the development of the items. We will do so by developing a smartphone app that tracks the live time spent on social media and triggers an ecological momentary assessment study to assess how young people engage with social media, and how that relates to their mental health and related outcomes. This is a novel application within scale development practices and will help us develop valid items that are particularly useful for understanding mental health. The measure will be piloted and validated with two independent samples of young people aged 11-15. A range of psychometric analyses will be used, including item response theory, (network) factor analyses, measurement invariance, and reliability and validity analyses.
The measure, but also the app, will be freely available to use by national and international researchers and will be useful for those interested in assessing, and therefore understanding, young people's social media experience, but also how that relates to mental health and related outcomes. The triangulation of objective and subjective data will also be useful for scale development researchers overall.
The measure, but also the app, will be freely available to use by national and international researchers and will be useful for those interested in assessing, and therefore understanding, young people's social media experience, but also how that relates to mental health and related outcomes. The triangulation of objective and subjective data will also be useful for scale development researchers overall.
