The role of familiarity and experience in the implementation of efficient visual search strategies

Lead Research Organisation: University of Aberdeen
Department Name: Psychology

Abstract

Imagine searching your office for your keys. You will likely start by scanning surfaces in your office such as your desk, table, and shelves. You may then check pockets, bags, and underneath papers, until you either find the keys or give up. How efficient was this search? How much time did you waste looking in places you already inspected, searching an area for too long, or looking in places that contained no useful information? In this proposal, we define search efficiency as the proportion of eye movements that are directed to locations that can be easily ascertained to provide new information. In the office example, some surfaces will be empty, and some cluttered with books and papers. If your keys were in the middle of an empty surface, you would already know where they were; no new information would be gained by looking directly at these locations. An efficient searcher would instead direct their eyes to the cluttered regions, where central vision is needed. Our recent studies using this metric to define efficiency have found a surprisingly large range of individual strategies, with some people being highly efficient, some random, and some highly inefficient. These differences suggest that rather than asking "is search optimal or random?" we should be asking for whom, and in what circumstances, search is optimal or random. This is the aim of the current proposal.

Much is already known about how visual information guides attention during search. Far less is known about search strategy, which contributes far more variance to performance measures. Our key hypothesis is that individual differences in strategy can be explained, at least in part, by differences in experience with the visual content and configuration, even though (in our experiments at least) these have no bearing on what the optimal eye movements are or the difficulty of implementing an efficient strategy. To assess this hypothesis, we systematically measure the effect on search efficiency of visual content, layout of the search array, individual motivation, learning, and prior expertise.

Understanding strategy is fundamental to building a complete model of visual search. The results have implications for understanding the role of experience in shaping strategy that could have relevance beyond the context of visual search. The results can also be useful in designing environments that promote more efficient search, and developing training programs that can lead to faster and more accurate detection of targets.

Planned Impact

The research is valuable to academics interested in modelling the visual, attentional, motor and decision processes and how these work together in visual search. Our results will account for a large source of variance and move this field forward substantially. The results can also be generalized to inform our understanding of human decision-making and strategic thinking, and thus will be of importance to researchers in this field as well. We will reach this audience through a wide range of conferences and symposia and publications, described in more detail in the section on academic beneficiaries.

Our key aim is to understand the visual and individual factors that contribute to search efficiency, and their interaction. Understanding how to improve search efficiency has obvious relevance to real-world situations in which fast and accurate visual search carries high stakes (security, search and rescue, and healthcare are some examples). The recruitment of airport security screeners to one of our experiments establishes a direct relationship with this industry, and we plan to host a knowledge exchange event where we invite these and other potential stakeholders to hear about the results of our work.

Our research in this area has already garnered a great deal of public interest. Evidently, spending too much time looking for lost objects is a situation that resonates for many people. Descriptions of our research on search efficiency have appeared in The Independent, New York Magazine, The New York Times, and The International Business Times, among many other newspapers and magazines. We will continue to engage with the media through journalists, but also by appearing at public engagement events, as described in the Impacts section.

The final impact of our project is educational. We have consulted closely with the postdoctoral researcher in devising both a programme of research, as well as flexible working schedule, that will allow her to thrive as a very promising early career researcher with substantial family responsibilities that might otherwise limit her career. We also provide research assistants with training, experience, and a multi-site network of support and management that will develop both their technical skills and their enthusiasm for basic research.

Publications

10 25 50
publication icon
Clarke A (2022) Visual search habits and the spatial structure of scenes in Attention, Perception, & Psychophysics

publication icon
Nowakowska A (2021) Search strategies improve with practice, but not with time pressure or financial incentives. in Journal of experimental psychology. Human perception and performance

publication icon
Nowakowska A (2023) Variable search for orientation, uniformly optimal search for identity. in Journal of Experimental Psychology: General

publication icon
Clarke AD (2022) Stable individual differences in strategies within, but not between, visual search tasks. in Quarterly journal of experimental psychology (2006)

 
Description Our research made substantial progress towards a more comprehensive understanding of search behaviour. Applying a consistent, adaptable measure of search efficiency across a range of experimental conditions allowed us to systematically isolate and test potentially important factors that could influence search. Using this approach, we identified conditions that promote consistently efficient search, and ruled out several that do not. We found that searching for a target with a particular orientation encourages the use of inefficient search routines, while searching for a target with a particular identity encourages efficient search based on visual analysis [Nowakowska et al, 2024]. We also found that areas of visual clutter that are distributed across a scene, by providing a better match to participants' default search routines, promote more efficient search than scenes split into more cluttered and less cluttered sides [Clarke et al., 2022]. In both cases, it is the simpler visual features and configurations that actually lead to less efficient search relative to the more complex versions. While counter-intuitive, it may be because default search strategies are better adapted to more complex circumstances.

Our second key question was for whom, and when, search becomes more optimal. A large (N=306) sample of participant completed a battery of tasks and measures alongside visual search. The results rule out many intuitively appealing explanations for inefficiency. Having completed a pre-registered report [https://osf.io/5aq4c/] evaluating specific predictions for what makes some individuals more efficient in search, we are now developing a more a descriptive, exploratory model of search behaviour based on this (now public) dataset.

We also implemented various interventions to encourage participants to search more efficiently. We found slow and steady improvements over time that were unaffected by performance-related rewards or deadlines [Nowakowska et al., 2021]. Our efforts here inspired us to refine how we model effects of time in behavioural research, so we also have been developing this approach as a general-purpose tool with many potential uses [https://osf.io/preprints/psyarxiv/fkbza/].

During covid-19, we pivoted to visual foraging, which shares many features with visual search because it involves selecting a series of spatial locations (the difference is that in foraging, there are many potential targets to "collect", typically by mouse-clicking, so experiments can be run over the internet). Through this work we developed a computational model of foraging behaviour that estimates a given individual's spatial and strategic biases [Clarke et al., 2022a; 2022b]. The model is generative, able to predict based on the parameters it learns from a person's data what object in a spatial array they are likely to select next. It is also modular; parameters can easily be added or removed to suit the conditions. This model forms a strong foundation on which to build a generalised theory of serial spatial selection.
Exploitation Route We have shown that the way in which the search target is defined has large effects on how consistently efficient the general population can be in findings targets, even though these surface-level details have no bearing on what the optimal strategy is or how easy it is to implement. We have also found that distributing the information-rich regions widely across a search array facilitates better strategies than clumping them together. In addition to the implications for theories of visual search, these two findings have obvious applications in user interface design and training in visually complex environments. They are also important for academic researchers to be aware of, as the typical laboratory experiment involves simplified and abstracted environments, and conclusions may or may not generalize to more complex and concrete environments.

Our results also suggest a slow and steady increase in search efficiency with experience; without changing the environment, we have so far failed to find any "shortcut" to more efficient search through incentives or emphasis on speed. This could have training implications in industries that require efficient information-sampling (e.g. aviation, security, healthcare). Two new collaborations fostered by the grant-funded research projects facilitate the application of our research findings in helicopter aviation and defense industries.

Finally, we have developed and presented a tool for the field to use in analysing data from visual foraging tasks. Our tool is more sensitive and accurate than existing techniques, is easily adapted and expanded to different contexts. We are already finding it useful in continuing our own research questions and hope others will too. The paper is published in an open-access journal and the code is publicly available for download here. https://osf.io/7yuaz/
Sectors Aerospace

Defence and Marine

Digital/Communication/Information Technologies (including Software)

Healthcare

Security and Diplomacy

Transport

 
Description Our research in visual search and information gathering has kindled a relationship with the applied psychology and human factors (APHF) group at the University of Aberdeen and with local helicopter transport and training companies CHC, Bristow, and NHV. We had virtual discussions and presentations over the course of the pandemic and in May 2022, ARH was awarded a Knowledge Exchange and Commercialisation award to fund a three-month PhD student interns and undergraduate research assistant to conduct structured interviews with a population of helicopter pilots on their information gathering strategies and to write an industrial report. The student was co-supervised by Dr Amy Irwin, who heads the APHF group. The project idea was based largely on evidence produced through the current ESRC-funded research, and the demonstrated utility of our paradigm. Dr Irwin, the PI (AH) and the PhD student intern on the above project, Nejc Sedlar, who is now a lecturer at Robert Gordon University, submitted an ESRC standard grant (in September 2023) with the aim of understanding the efficiency of information gathering and sharing between helicopter pilots and rear crew. In July 2021, the PI (ARH) gave a "Little Lecture" for the general public on the grant research and findings to date, sponsored and organized by the University of Aberdeen media team. Because of covid, the lecture was presented on line but live, to an audience of about 30. https://www.abdn.ac.uk/study/student-life/little-lectures.php A series of meetings with researchers from the US Army Research Laboratory has led to an industrial collaboration. In October 2021, ARH was invited to present to an audience of about 60 army researchers, with the co-I and postdoctoral fellow also attending. Several follow-up discussions with members of the Army Research Labs focused on eye movements in visual search in high-stress situations and when working in teams. This developed into a successful bid by the co-I ADFC and his colleague Anna Hughes, for a collaborative PhD studentship from the ESRC-funded SENSS network, which commenced in October 2023.
First Year Of Impact 2023
Sector Aerospace, Defence and Marine
Impact Types Societal

 
Title A Bayesian statistical model for estimating biases in target selection 
Description We present a way to model foraging behaviour as a generative sampling without replacement procedure, implemented in a Bayesian multilevel model. This allows us to break down behaviour into a number of independent biases that influence target selection, including the proximity of targets, a bias for selecting targets in runs and a bias for a particular target type, in a way that is not dependent on the number of targets present. The usefulness, sensitivity and flexibility of the model is demonstrated with a wide range of existing datasets, and the source code is freely available to download and use. 
Type Of Material Model of mechanisms or symptoms - human 
Year Produced 2022 
Provided To Others? Yes  
Impact The fully developed and validated tool resulted in a publication in PLoS Computational Biology. We are currently using the tool to explore subtle biases in foraging that would not have been detectable using the standard approach that our tool replaces, and extending it to be usable in a broader range of contexts. 
URL https://osf.io/7yuaz/
 
Title Dataset from paper published in Journal of Experimental Psychology: General 
Description The paper presents a series of main experiments, pilot data, and follow-up control experiments that together comprise more than 100 participants performing visual search tasks with a range of different stimulus types, with the common feature that they have been split into hard and easy sides to measure and compare how participants distribute their eye movements over the two sides across different stimulus and target types. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact The paper has only just been published, but our presentation of this work has been interesting to many researchers and has been an important aspect of our own theory development on the contribution of strategy to search outcomes and on how visual context contributes to strategy, and on the limitations of generalizing from one context to another. 
URL https://osf.io/9edx6/
 
Title Individual differences in visual search 
Description This repository contains data, analysis scripts and the written reports from a project that was pre-registered with the Royal Society Open Science. The pre-registration was peer-reviewed and given "in principle" acceptance on January 29th, 2020. The submitted final report of the results is in the document entitled "RSOSRegReportWithResults.pdf". This paper was accepted in December of 2023. A total of 306 participants completed a visual search task alongside a battery of other tasks and measures and the fixation data and results of all the other measures are included in this public database. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact Clarke, A.D.F., Sauerberger, K., Nowakowska, A.M., Rosenbaum, D.A., Zentall, T.R. & Hunt, A.R. (in press). Does precrastination explain why some observers are sub-optimal in a visual search task? A pre-registered report. Royal Society Open Science. 
URL https://osf.io/5aq4c/
 
Description Does pre-crastination explain why some observers are sub-optimal in a visual search task? 
Organisation University of California, Riverside
Country United States 
Sector Academic/University 
PI Contribution In this collaborative project, we conducted an initial pilot study (N=30) in Aberdeen to verify the methods were appropriate, to estimate the required sample size, and to develop the analysis approach. Participants were recruited and run in Aberdeen and Essex (N=306). The PDRF supervised data collection by research assistants paid by the grants. The Co-I, PDRF and PI on this ESRC grant took the lead in writing a pre-registered report and submitting it, and in responding to editor and reviewer comments on the submission. Financial compensation for participants has been provided by the grant funds.
Collaborator Contribution Collaboration partners (David Rosenbaum and Kyle Sauerberger in California, and Tom Zentall in Kentucky) consulted on the design of the tasks, and made comments on the pre-registered report and responses to editor and reviewers comments. They hosted the software collecting questionnaire and demographic data, compiled those results and uploaded them to a shared github repository.
Impact 1. A publication based on the results will appear as a pre-registered report in the journal Royal Society Open Science. The title of the report is: "Does pre-crastination explain why some observers are sub-optimal in a visual search task?" and the authors are Alasdair Clarke, Anna Nowakowska, Kyle Sauerberger, David A. Rosenbaum, Tom Zentall, and Amelia Hunt. The stage 1 report was accepted in January 2020. Data collection was delayed by covid-19 but the final report with results was accepted in December 2023 and is now in press. It is not listed as an output because the publisher's version has not yet appeared in the journal, but will be added in our next report. 2. The results were presented at the Vision Sciences Society conference (Florida, USA, May 2020) entitled: Optimal Visual Search, Individual Differences and Pre-Crastination. The authors are the same as those listed above. Other aspects of the data were presented at the Scottish Vision Group meeting (Dundee, April 2023) and the Applied Vision Association meeting (December 2023). 3. There is a large dataset associated with this project available at https://osf.io/5aq4c/
Start Year 2019
 
Description Does pre-crastination explain why some observers are sub-optimal in a visual search task? 
Organisation University of Kentucky
Country United States 
Sector Academic/University 
PI Contribution In this collaborative project, we conducted an initial pilot study (N=30) in Aberdeen to verify the methods were appropriate, to estimate the required sample size, and to develop the analysis approach. Participants were recruited and run in Aberdeen and Essex (N=306). The PDRF supervised data collection by research assistants paid by the grants. The Co-I, PDRF and PI on this ESRC grant took the lead in writing a pre-registered report and submitting it, and in responding to editor and reviewer comments on the submission. Financial compensation for participants has been provided by the grant funds.
Collaborator Contribution Collaboration partners (David Rosenbaum and Kyle Sauerberger in California, and Tom Zentall in Kentucky) consulted on the design of the tasks, and made comments on the pre-registered report and responses to editor and reviewers comments. They hosted the software collecting questionnaire and demographic data, compiled those results and uploaded them to a shared github repository.
Impact 1. A publication based on the results will appear as a pre-registered report in the journal Royal Society Open Science. The title of the report is: "Does pre-crastination explain why some observers are sub-optimal in a visual search task?" and the authors are Alasdair Clarke, Anna Nowakowska, Kyle Sauerberger, David A. Rosenbaum, Tom Zentall, and Amelia Hunt. The stage 1 report was accepted in January 2020. Data collection was delayed by covid-19 but the final report with results was accepted in December 2023 and is now in press. It is not listed as an output because the publisher's version has not yet appeared in the journal, but will be added in our next report. 2. The results were presented at the Vision Sciences Society conference (Florida, USA, May 2020) entitled: Optimal Visual Search, Individual Differences and Pre-Crastination. The authors are the same as those listed above. Other aspects of the data were presented at the Scottish Vision Group meeting (Dundee, April 2023) and the Applied Vision Association meeting (December 2023). 3. There is a large dataset associated with this project available at https://osf.io/5aq4c/
Start Year 2019
 
Description Individual differences across search tasks 
Organisation Ohio State University
Country United States 
Sector Academic/University 
PI Contribution Expertise in visual search and research methods
Collaborator Contribution Expertise in attentional control settings and research methods
Impact Clarke, A.D.F., Irons, J.L. James, W.R.G., Leber, A.B. & Hunt, A.R. (2022). Stable individual differences in strategies within, but not between, visual search tasks. Quarterly Journal of Experimental Psychology, 75, 289-296.
Start Year 2019
 
Description Real world visual search 
Organisation University of Essex
Country United Kingdom 
Sector Academic/University 
PI Contribution Expertise in visual search and research methods
Collaborator Contribution Expertise in camouflage and research methods
Impact 10.3390/vision6040066 10.1371/journal.pcbi.1009813 British Academy / Leverhulme Small Grant (to Hughes & Clarke): British Academy/Leverhulme (2020). £8,971. Co-Investigator. Developing a collaborative framework for naturalistic visual search.
Start Year 2020
 
Description Spatial exploration and collaboration/competition 
Organisation US Army Research Lab
Country United States 
Sector Public 
PI Contribution Computational modeling expertise and theoretical background in visual search/eye movements
Collaborator Contribution Virtual reality expertise, data collection, and special population
Impact Hughes, A., Cohen, R. & Clarke, A.D.F. (May, 2023). Computational modeling of 3D team foraging to understand human behaviour and cognition. Accepted for presentation at the Vision Sciences Society. St Pete's Beach, USA.
Start Year 2021
 
Description Visual search in helicopter pilots 
Organisation University of Aberdeen
Country United Kingdom 
Sector Academic/University 
PI Contribution Expertise in attention and visual search theory and eye tracking methods, laboratory equipment and space
Collaborator Contribution Expertise in qualitative/applied research and cognitive task analysis from the Applied Psychology and Human Factors group at the University of Aberdeen (headed by Dr Amy Irwin), strong collaborative links with industrial partners
Impact Industrial Report provided to local stakeholders (NHV, CHC and Bristow) Two conference presentations by Nejc Sedlar (PhD intern) at the European Association of Work and Organizational Psychology Congress, May 2023 and the International Symposium on Aviation Psychology, May 2023 An ESRC standard grant application (outcome not yet known) on mental model development in helicopter pilots and rear crew with A. Irwin as PI and Amelia hunt as Co-PI, submitted in September 2023. An successful application for a collaborative PhD studentship co-funded by the Office for Nuclear Regulation and the SGSSS (the Scottish Graduate School for Social Sciences, an ESRC-funded DTP) on peer-checking to commence October 2024
Start Year 2021
 
Description A poster presented by the postdoc at the Vision Sciences Society conference in St Pete's Beach, Florida 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The conference and poster session provided an opportunity for the postdoc to network with interested researchers and to find out about current research in the field.
Year(s) Of Engagement Activity 2022
 
Description A set of three talks presented at the European Conference on Eye Movements (ECEM), Leicester, UK. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We presented a set of three talks to an audience of academic researchers at this international conference. The conference presented an opportunity to reach an international audience with our findings. The talks led directly to a series of further meetings and a collaborative project (still in process with no formal outputs, so it does not yet appear on the list of collaborations) with Monica Castelhano at Queen's University, Canada, on the role of scene understanding in search strategies.
Year(s) Of Engagement Activity 2022
 
Description A set of three talks presented at the Scottish Vision Group (SVG) meeting, Dundee, UK. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The PI, Co-I and postdoctoral fellow were able to attend this conference together and presented different aspects of the grant's outcomes to a specialised audience. These sparked questions and later discussions about the research that have been useful as we prepared these for submission, with the expectation that this will become published grant outputs.
Year(s) Of Engagement Activity 2023
 
Description Applied Psychology and Human Factors Seminar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Presentation jointly to collaborators and industrial stakeholders in helicopter aviation delivered by the PhD intern (Nejc Sedlar)
Year(s) Of Engagement Activity 2022
 
Description Talk presented at the Vision Sciences Society (VSS) meeting, St Pete's Beach Florida, USA. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Conference presentation made by the co-I (ADFC) at a large international conference attended by a large and specialised audience. Sparked conversation and follow-up discussions via email. One audience member in particular requested, read and provided detailed feedback on the article we then submitted for publication.
Year(s) Of Engagement Activity 2023
 
Description Talk presented to the US Army Research laboratory. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact By invitation, delivered an on-line presentation to researchers working on problems of visual search in high-stress contexts and in teams in a military context (October 29th, 2021). They were interested our observations of individual differences in visual search strategies and how to align team members better when engaging in team-driven search and foraging. The PI presented the grant research for 30 minutes, followed by an hour of further discussion and questions, with the PI and postdoc also present. Since then, we have had several smaller meetings to discuss the implications of our findings for the problems they are trying to solve.
Year(s) Of Engagement Activity 2021
 
Description Two talks delivered at the European Conference on Visual Perception in Nijmegen, the Netherlands 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Two talks were delivered by the co-I and postdoc on outcomes from the funded research. It was also an opportunity to network with other researchers and gather information about related findings and research from international laboratories.
Year(s) Of Engagement Activity 2022
 
Description University of Aberdeen Public Lecture: "Why is it so hard to find my keys?" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The event was part of a series organized by the media team at the University of Aberdeen. An audience of around 30 tuned in for an on-line live presentation that presented the research funded by the grant, described in a way that was accessible, and ending with some "practical tips" for how to be better at looking for lost objects. The talk was about 30 minutes and was followed by about 30 minutes of audience questions. The questions were lively and sparked interesting discussion. Following the talk, one of the attendees requested a visit to the laboratory to see the experiments and equipment in person.
Year(s) Of Engagement Activity 2021
URL https://www.abdn.ac.uk/study/student-life/little-lectures.php