Improving Experimentation and Measurement for Online Products and Services

The value of making data driven or data informed decisions has become increasingly clear in recent years. Key to making data driven decisions is the ability to accurately measure the impact of a given choice and to experiment with possible alternatives. We define Experimentation & Measurement (E&M) capabilities as the knowledge and tools necessary to run experiments (controlled or otherwise) with different digital or physical products, services, or experiences, and measure their impact. The capabilities may be in the form of an online controlled experiment framework, a team of econometrics analysts, or a system capable of performing machine learning-aided causal inference---the understanding of the cause and effect based on what one observed.

The research project aims to develop an array of statistical and machine learning methods, to boost the E&M capabilities of online businesses. So far, we have successfully (1) estimated the value of E&M capabilities themselves and (2) built an evaluation framework for controlled experiment designs, and seek to:

(3) Develop data-driven controlled experiment designs:
The ability to run many large-scale controlled experiments on the Web allows us to collect data generated by similar experiments in the past. Can we leverage these data to shorten the duration of an experiment if it yields results on a similar trend?

(4) Quantify the measurement uncertainty of observational studies a priori:
Observational studies (experiments w/o a control group) allow one to estimate both the direction and magnitude of the impact of an action/attribute using causal inference. Unlike its controlled counterpart, the uncertainty level around an estimate is often known only after running the analysis. Can we quantify the bounds on estimates produced from such analyses before running them, just like how we do so for controlled experiments?

(5) Combine insights generated from controlled and observational experiments:
Unlike in experimental science, where data are generally collected for a specific purpose, data on the Web are often used for purposes other than that originally intended. Can we do the same with experiments, i.e. supplement observational studies with data generated from controlled experiments that were designed for other purposes (or vice versa)?

The scope of the research project is ambitious. Many of the challenges (topics 1, 2, and 5) had received little consideration or serious attempts to the best of our knowledge due to the need to draw the latest results in multiple related fields. Other problems require building on the state-of-the-art in Bayesian Hypothesis Testing (topic 3) and Extreme Value Theory (topics 2 and 4). For the former we plan to combine Data-driven Priors, the specification on how a new experiment could possibly perform based on how previous experiments performed, and Non-local Priors, the specification on how an experiment could possibly perform that does not contradict the null hypothesis, to make an experiment more sensitive to any changes in a business metric. For the latter we seek to reduce the uncertainty level of the estimates produced in the face of heavy-tailed distributed responses, i.e. an extremely wide range of online behaviour that can be characterised using the 80/20 rule.

The project falls within the EPSRC Digital Economy Theme, touching on sub-themes including Behavioural research; Data, information and knowledge; and Value creation and capture. It is carried out in collaboration with, and part-funded by, ASOS.com, one of the largest UK-based online fashion retailers. The academic-industry collaboration enables the research to access the wealth of what would be propriety data that informs the development of measurement methods, and the business to benefit from insights and techniques developed immediately, generating new data along the way for iterative development.

The primary CDT impact will be training 75 PhD graduates as the next generation of leaders in statistics and statistical machine learning. These graduates will lead in industry, government, health care, and academic research. They will bridge the gap between academia and industry, resulting in significant knowledge transfer to both established and start-up companies. Because this cohort will also learn to mentor other researchers, the CDT will ultimately address a UK-wide skills gap. The students will also be crucial in keeping the UK at the forefront of methodological research in statistics and machine learning.
After graduating, students will act as multipliers, educating others in advanced methodology throughout their career. There are a range of further impacts:
- The CDT has a large number of high calibre external partners in government, health care, industry and science. These partnerships will catalyse immediate knowledge transfer, bringing cutting edge methodology to a large number of areas. Knowledge transfer will also be achieved through internships/placements of our students with users of statistics and machine learning.
- Our Women in Mathematics and Statistics summer programme is aimed at students who could go on to apply for a PhD. This programme will inspire the next generation of statisticians and also provide excellent leadership training for the CDT students.
- The students will develop new methodology and theory in the domains of statistics and statistical machine learning. It will be relevant research, addressing the key questions behind real world problems. The research will be published in the best possible statistics journals and machine learning conferences and will be made available online. To maximize reproducibility and replicability, source code and replication files will be made available as open source software or, when relevant to an industrial collaboration, held as a patent or software copyright.