Re-Thinking Duties of Care: Measuring Human Costs in the Creation of AI Systems for Detecting Online Harms

In a landmark 2020 lawsuit, Facebook paid a settlement of$52 million to its teams of moderators as compensation for mental health issues developed on the job such as PTSD, anxiety disorders, and social isolation (Newton,2020). As articulated in the UK government's recent Online Harms WhitePaper, internet platforms have a duty of care to protect their users from online harms such as hate speech, harassment, and abuse (Woods,2019). Due to the massive volume of content shared online, tech companies have turned to automation, AI systems, and machine learning (ML) algorithms to detect online harm (Gillespie,2020). Machines are not psychologically damaged by processing harmful items so these solutions have promise for detecting content at scale. However, these approaches are rarely 'fully-automated'; Instead, they usually require human moderators 'in-the-loop' to label existing data and review machine decisions (Ekbiaand Nardi,2014). The academic computer science literature on online harm detection is centered on the traditional paradigm of supervised learning on large, manually-annotated, and static datasets. Solutions developed under this paradigm are heavily reliant on human annotators - often in the form of 'click-workers' - for dataset labeling (Shmueliet al.,2021). The duty of care to users is operationalized and prioritized with traditional machine learning metrics like the accuracy of automated solutions to detect harmful content, but the duty of care to moderators and annotators is often ignored. This Ph.D. thesis focuses on understanding, measuring, and comparing the welfare burden and the human cost in traditional machine-learning approaches with new data-centric AI approaches in the context of harmful content detection. While computer scientists have sought to increase the performance of machine-learning solutions with larger models and datasets, this Ph.D. seeks to find more data-efficient strategies where highly-performing models can be developed with fewer but more carefully-curated training.