Mining the History of Medicine

This project, a cross-disciplinary collaboration between the National Centre for Text Mining (NaCTeM) and the Centre for the History of Science, Technology and Medicine (CHSTM) at the University of Manchester, seeks to demonstrate the potential of text mining in medical history. To do this, firstly an asset will be created out of two very large, long-running digital sources, the British Medical Journal (BMJ) (1840 - present) and the London-area Medical Officer of Health (MOH) reports (1848-1972), by applying text mining techniques to enrich these data with semantic annotations.

An important aspect of this work is to build tools to identify and record terminological variation and semantic shift over time, via construction of a temporal terminological inventory from the archives. Then, a semantic search system will be developed to help historians in broadening and deepening their work to ask 'big' questions that cover long periods, without losing sensitivity to changes in terminology and meaning.

The resulting asset and tools will be used and evaluated in two case studies, exploring the modern epidemiological transition and the creation of a medical surveillance culture, two massively important and interrelated changes in British health experience, where many questions remain unanswered. The methods and results of the case studies will serve as concrete examples of how such an asset and tools can be used.

The text mining tools and derived resources will be made available to the community via an interoperable text mining environment that will be contributed to the repositories of major digital humanities infrastructures (CLARIN, DARIAH, META-SHARE).

The project will be guided by an Expert User Advisory Group. Two workshops are planned to receive further guidance from and to inform the wider community.

The project plans to extend its impact to the following sectors: public health, public policy, publishing, media and libraries, with a view to ensuring sustainability and wider uptake of methods and technologies.

The direct impact of our project will be in the creation of an asset and technology that facilitate the work of humanities scholars from multiple disciplines, who seek new ways to reveal, explore, and discuss long-term, large-scale historical transformations related to medicine and health.

However, we have carefully designed our engagement strategy, supported by an Expert User Advisory Group, such that, indirectly, the project will impact on the following sectors: public health, public policy, publishing, media and libraries, thus ensuring not only high impact but also excellent potential for sustainability.

Evidence-based public health, particularly the National Institute for Health and Care Excellence (NICE), and public policy will benefit from the understanding of medical historical data at the large scale that our text mining methods will address. Text mining will uncover associations and hidden information from large-scale historical archives about disease and its impact on the environment and society, which can be used to guide and inform public health and policy. We offer a unique opportunity to advance our understanding of disease in the environment through historical data, which have hitherto been under-examined in all their complexity.

The text mining research methods using terms, semantic clustering, semantic metadata will impact on the publishing industry which is seeking to improve scholarly communications by enriching their archives using semantic methods but often lacks the expertise and technology.

Semantic analysis of historical data helps to inform the citizen: extending our links with the British Library, Wellcome Library, and the BBC, we shall also reach out to national archives (UK National Archives, National Archives of Scotland), in order to demonstrate how archives in the public domain can be enriched to allow the citizen to engage with big historical data semantically without being led astray by temporally-conditioned language.