SBE-UKRI:A Novel Theory of Ordered Judgment Processes
Lead Research Organisation:
University of Nottingham
Department Name: School of Computer Science
Abstract
Overview
There are three core elements to decision making: judgment, preference, and choice (Fischhoff & Broomell, 2020). Judgments represent how people come to understand the outcomes associated with choices along with their probabilities of occurrence. This proposal focuses on how people form judgments by aggregating multiple pieces of evidence gathered from different information sources. Much of the literature on judgment has relied on linear models as the prominent theoretical basis for understanding such judgments (e.g., Broomell & Budescu, 2009; Dawes, 1971; Karelaia & Hogarth, 2008). We propose a new theory of ordered judgment that fuses psychological theory with operational and theoretical advances from computer science in the areas of data aggregation and artificial intelligence.
Intellectual Merit
We propose to advance psychological theory beyond linear models by (1) leveraging computationally simple ordering processes with statistically desirable properties for information aggregation, (2) mimicking the relative nature of perception in judgment, and (3) seamlessly integrating this approach with linear theories previously used. Researchers have implicated ordering as a potential component of cognition but to date, lack the ability to empirically test for its presence. Our theory of ordered judgment will provide the first operational framework for empirical tests of the role of ordering in judgment and beyond. We will develop and test this novel theory through three research objectives. The first objective is to develop a predictive model of judgment based on preliminary work (Broomell & Wagner, 2023). Such a model will facilitate targeted experimentation to detect whether ordered judgment processes can account for human behavior. The second objective is to use lab experiments to understand the degree to which order-based processes naturally fit with judgment processes and can predict human behavior. These studies will reveal systematic and predictable behavior in how judgments react to momentary changes in context. The third objective is to develop methods for estimating the free parameters of the ordered judgment model from observed judgments. The development of such an estimation procedure will allow for a more detailed decomposition of judgments into stable and dynamic priorities that drive judgment. Additionally, such an estimation procedure would have implications for model fitting broadly in psychological and computer science work.
Broader Impacts
We anticipate that this theory integration will have impacts for both psychological and computer science research that go well beyond the intellectual merits. For psychology, our theory of ordered judgment has many implications for how to display data to facilitate accurate processing that will be useful for decision-support and human factors work in contexts ranging from graphical user interfaces to operating machinery. For computer science, we anticipate that this work will contribute to the crucial area of explainable artificial intelligence by articulating direct links between pervasive linear and non-linear aggregation processes and human reasoning. This can afford mechanisms to understand, evaluate and validate machine-learning driven decision-making approaches in critical applications such as security and defense, energy, and healthcare. Further, algorithmic implementations of the proposed theory hold the potential to offer efficient means of aggregating information in machine learning including neural networks, as alluded to in Kreinovich (2022). The work in this proposal will also serve to train doctoral students and postdoctoral reasearchers in interdisciplinary and internationally collaborative research leveraging mathematical, computational, and empirical methods. The results of this work will complement the PI and co-PI's teaching and instrunction at undergraduate and graduate levels in the US and the UK.
There are three core elements to decision making: judgment, preference, and choice (Fischhoff & Broomell, 2020). Judgments represent how people come to understand the outcomes associated with choices along with their probabilities of occurrence. This proposal focuses on how people form judgments by aggregating multiple pieces of evidence gathered from different information sources. Much of the literature on judgment has relied on linear models as the prominent theoretical basis for understanding such judgments (e.g., Broomell & Budescu, 2009; Dawes, 1971; Karelaia & Hogarth, 2008). We propose a new theory of ordered judgment that fuses psychological theory with operational and theoretical advances from computer science in the areas of data aggregation and artificial intelligence.
Intellectual Merit
We propose to advance psychological theory beyond linear models by (1) leveraging computationally simple ordering processes with statistically desirable properties for information aggregation, (2) mimicking the relative nature of perception in judgment, and (3) seamlessly integrating this approach with linear theories previously used. Researchers have implicated ordering as a potential component of cognition but to date, lack the ability to empirically test for its presence. Our theory of ordered judgment will provide the first operational framework for empirical tests of the role of ordering in judgment and beyond. We will develop and test this novel theory through three research objectives. The first objective is to develop a predictive model of judgment based on preliminary work (Broomell & Wagner, 2023). Such a model will facilitate targeted experimentation to detect whether ordered judgment processes can account for human behavior. The second objective is to use lab experiments to understand the degree to which order-based processes naturally fit with judgment processes and can predict human behavior. These studies will reveal systematic and predictable behavior in how judgments react to momentary changes in context. The third objective is to develop methods for estimating the free parameters of the ordered judgment model from observed judgments. The development of such an estimation procedure will allow for a more detailed decomposition of judgments into stable and dynamic priorities that drive judgment. Additionally, such an estimation procedure would have implications for model fitting broadly in psychological and computer science work.
Broader Impacts
We anticipate that this theory integration will have impacts for both psychological and computer science research that go well beyond the intellectual merits. For psychology, our theory of ordered judgment has many implications for how to display data to facilitate accurate processing that will be useful for decision-support and human factors work in contexts ranging from graphical user interfaces to operating machinery. For computer science, we anticipate that this work will contribute to the crucial area of explainable artificial intelligence by articulating direct links between pervasive linear and non-linear aggregation processes and human reasoning. This can afford mechanisms to understand, evaluate and validate machine-learning driven decision-making approaches in critical applications such as security and defense, energy, and healthcare. Further, algorithmic implementations of the proposed theory hold the potential to offer efficient means of aggregating information in machine learning including neural networks, as alluded to in Kreinovich (2022). The work in this proposal will also serve to train doctoral students and postdoctoral reasearchers in interdisciplinary and internationally collaborative research leveraging mathematical, computational, and empirical methods. The results of this work will complement the PI and co-PI's teaching and instrunction at undergraduate and graduate levels in the US and the UK.
