The International Cognitive Ability Resource

Lead Research Organisation: University of Cambridge
Department Name: Psychology

Abstract

After more than a century of active research on cognitive abilities, there now exist dozens of assessment tools of tremendous variety. Many of these are no longer widely used on account of being outdated or poorly maintained. Most of the remainder - those tests which are actively used and well maintained - can generally be organized into two broad types: (i) proprietary measures owned and administered by profitable commercial entities or (ii) idiosyncratic measures which are usually developed and used by isolated research groups for narrow purposes. This circumstance is problematic for the international community of cognitive ability researchers because it inhibits the generalization of findings across sub-fields and, more generally, slows the pace at which knowledge can advance. The tangible consequences of this circumstance on the societal level are manifested through limited utility of cognitive ability assessment in clinical settings (as used for diagnoses of cognitive impairment) and selection contexts (as used for personnel recruitment or academic admissions).

These consequences will be addressed by the development of the International Cognitive Ability Resource (ICAR), a public-domain and open-source tool which will incorporate modern item generation and adaptive testing techniques. At its essence, ICAR is a framework for international collaboration. This framework will facilitate the development of a large and dynamic bank of cognitive ability measures for use in a wide variety of applications. By encouraging the use, revision, and ongoing development of these measures among qualified research groups around the world, the ICAR will further understanding about the structure of cognitive abilities as well as the nature of associations between cognitive ability constructs and the life outcomes they predict.

This framework is not without precedent; a similar project has been implemented for use in psychology. The International Personality Item Pool (IPIP) is a large collection of publicly available personality items which was introduced to resolve similar problems with measurement standards. This project has met with considerable success; data collected with IPIP measures have contributed to more than 500 journal articles and book chapters over the last decade. The proposed ICAR is not entirely analogous, mainly because cognitive ability items are fundamentally different from typical, self-report personality items. While the latter are measures of typical behaviour, most intelligence items are intended to assess maximal performance. This distinction makes the development of ICAR considerably more challenging, though it has already been demonstrated that such development is feasible.

The development project proposed herein will result in a resource which contains many types of "item generators". These will be distributed as functions in psychometrically informed, easily implemented, open-source software. Research
groups will be able to use these item generators by downloading the software through the widely-used, open-source statistical software platform, R. As needed, the revision of existing item generators and the development of additional types will be encouraged among qualified international researchers. These item generators will be well suited for large scale, international data collection as might be conducted over the internet.

Planned Impact

The primary aim of this research is to integrate recent innovations in test construction, large-scale data collection, and statistical methodology for the purpose of improving the measurement of cognitive ability. This presents a formidable challenge, though one which is well-matched with the collective expertise of the collaborators. The most obvious outcome, of course, will be the creation of an open-source, public-domain tool which will be iteratively improved upon for many years to come. Additional benefits to society-at-large include:
1. better understanding of the ways in which cognitive abilities relate to one another (their internal structure);
2. broader inclusion of cognitive ability assessment in social sciences research generally, including several sub-domains of psychology where ability assessment is often omitted due to the practical limitations posed by extant proprietary
measures;
3. following from the above benefit, more empirical data collection which objectively relates cognitive abilities to a wide variety of life-outcomes and societal goals. Examples include:
(a) broader STEM (science, technology, engineering, and maths) involvement among females and historically under-represented minorities based on the earlier identification of spatial abilities and more simple assessments of the efficacy of STEM interventions;
(b) more tailored treatment of physical and mental health issues based on cognitive ability assessments designed to gauge patients' capacity for self-care (recent research suggests that cognitive abilities/impairments are strong predictors of health outcomes for a variety of chronic diseases and unhealthy behaviours);
(c) better understanding of the ways in which cognitive abilities change over the lifespan, including study of the factors which exacerbate or protect against cognitive ageing;
(d) enhanced well-being resulting from knowledge about the ways in which an individual can be matched to academic and occupational pursuits that are likely to bring personal fulfilment;
4. Most importantly, this research program will lead to the development of an international "collaboratory" platform without precedent in cognitive ability research. The reach of this platform will extend well beyond the collaborators named in this proposal as many research teams have already expressed a strong interest in using the item types and/or developing their own. Given this breadth, it is likely that the most meaningful outcomes will stem from the international cross-fertilization of ideas, the more efficient use of resources across universities and nations, and ultimately the more rapid advancement of knowledge in this field.

Publications

10 25 50
 
Description Since the ICAR project started on April 1st, 2014, we have successfully created and maintained the infrastructure for the dissemination of the International Cognitive Ability Resource. A secure server has been set up, on which independent Concerto testing platforms have been installed, which accommodate a wide range of cognitive ability measures. The official website, www.icar-project.com, making use of the redmine open-source software, serves as a useful tool for both collaborators and users to communicate and contribute to the development of item types. Till February, 2019 there have been around 1500 registrations, among whom more than 300 qualified users have been granted access to the ICAR measures. They benefit greatly from being able to freely employ the open-source cognitive ability measures in their own research.

We are actively collaborating with international scholars on the development of cognitive ability measures. So far the ICAR project has shared 14 item types among the ICAR community. The Cambridge team has led the development of automatic item generators for matrix reasoning items, number series items, linear syllogism items, perceptual maze items, 2d and 3d rotation items, spatial planning items as well as basic arithmetic items. We have also investigated the stimulus features that contribute to the item difficulty using various IRT models, and applied such knowledge in the creation of R packages. The findings have been presented at international conferences and reported in academic journals. We have also published three newsletters, where we showcased research outcomes about the ICAR project.

To summarise, for the past four years, we have generated new knowledge for cognitive ability assessment, developed new methods for automatic item generation, established useful collaborations and partnership, created and more importantly shared substantial research resources with the academic community, and significantly increased research capacity for social sciences, biological sciences and other areas where cognitive ability assessment is involved.
Exploitation Route The research is mainly disseminated through the project website, from which users gain access to the cognitive ability measures developed by the ICAR core team and international collaborators. Qualified users are free to implement these outputs in their research. Moreover, scholars are also welcome to contribute to the project by providing data or developing new item types, so that more impact can be achieved. Now we have secured collaborations with researchers from research institutes based in Europe, North America, China, Singapore, Brazil and so on. Additionally, research findings have been presented at various academic conferences, and published on academic journals. More manuscripts are under preparation for submission.

Upon completion of the project in Cambridge in 2018, the responsibility of maintaining the website has been transferred to the German collaborator, who will continue the project and make sure that users can still access the ICAR measures. We consider it a major legacy for the entire research community.
Sectors Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Education,Financial Services, and Management Consultancy,Healthcare,Other

URL http://icar-project.com/
 
Description One of the major research outcomes derived from the ICAR project is a broad range of cognitive ability measures, which have been made available on the ICAR website and shared with the academic community. Currently there are around 1500 registered users, among whom more than 350 scholars and research groups have been granted free access to these ICAR measures. Unlike other grants that contribute new knowledge and skills to the academia, the ICAR project allows its outcomes to be directly used by other researchers, leading to great economic impact. A huge amount of research costs associated with the cognitive ability measures would have incurred without the free access to the ICAR measures. Additionally, with careful verification and close monitoring, the ICAR measures have been made available to organisations in public sectors, for instance, NPOs, charities, institutes affiliated with the government and the army, hospitals and clinical practices, etc. With free access to the high quality cognitive measures, their services are improved and result in higher efficiency. Based on the newly developed item types, sample tests have been constructed and implemented on multiple testing websites (e.g. discovermyprofile.com and sapa-project.org). These tests have been visited by tens of thousands of users, who are interested in learning more about themselves from the feedback provided by the tests. Not only can they gain useful insights into their cognitive potentials, but the general public, by visiting the websites, can widen their knowledge of the recent development in the psychometric research as well. The ICAR project has paid special attention to the technique of automatic item generation and developed several R packages that are capable of generating a large number of cognitive ability items. Individuals and organisations from various sectors are able to create their own item sets for cognitive ability assessment. This is a step change in the field of psychometrics, significantly encouraging the use of cognitive ability measures in a wide variety of applications. With better assessment, social justice can be reinforced. Research findings have been effectively disseminated via conference presentations, journal publications, websites and public engagement activities. Different user groups have deepened their understanding of cognitive ability measures and benefited from the project. The cognitive ability measures derived from the ICAR project have been employed in private sectors and greatly facilitated research and practice of educational, occupational and clinical assessment. For instance, the reasoning items have been frequently used as a screening test for recruitment and in education settings. They are also important indicators of cognitive functions in the health context. Through diverse pathways of collaborating with individuals and organisations beyond academia, economic and societal impacts of the project are maximised and the quality of life can be enhanced.
First Year Of Impact 2015
Sector Digital/Communication/Information Technologies (including Software),Education,Financial Services, and Management Consultancy,Healthcare,Government, Democracy and Justice,Other
Impact Types Societal,Economic,Policy & public services

 
Title Item types 
Description A number of item types have been made available on the ICAR website, including validated item sets and item generators. 
Type Of Material Improvements to research infrastructure 
Year Produced 2014 
Provided To Others? Yes  
Impact Access to the ICAR measures has greatly benefit the research projects of more than 200 qualified research groups. 
URL https://icar-project.com/projects/icar-project/wiki/Item_types
 
Title Sample tests 
Description A variety of sample tests are provided on the ICAR website. 
Type Of Material Improvements to research infrastructure 
Year Produced 2014 
Provided To Others? Yes  
Impact The sample tests can be freely used for research and facilitate neuropsychological assessment. 
URL https://icar-project.com/projects/icar-sample-tests
 
Title Database 
Description Test data is collected via the Concerto testing platform in order to validate the cognitive measures. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? No  
Impact Item sets and item generators are validated based on the test data collected. Then these cognitive measures can be shared on the ICAR website and used freely for research purposes. 
 
Description Dr Bateman from Oliver Zangwill Centre for Neuropsychological Rehabilitation 
Organisation Oliver Zangwill Centre
Country United Kingdom 
Sector Hospitals 
PI Contribution The ICAR team in Cambridge is collaborating with Dr Bateman on the analysis of the Ekman 60 Faces test. Advanced statistical analysis is performed and adaptive testing procedure simulated.
Collaborator Contribution Dr Bateman provides test data of brain injuries patients.
Impact Research findings have been presented at various conferences.
Start Year 2014
 
Description Prof Doebler from Technical University of Dortmund 
Organisation Technical University of Dortmund
Department Faculty of Statistics
Country Germany 
Sector Academic/University 
PI Contribution The ICAR team in Cambridge is collaborating with Prof Doebler on projects to develop automatic item generators for figural analogies test, number series test, syllogism reasoning test, and propositional reasoning test. Tests are implemented on the Concerto platform and made available via the testing website hosted by the Psychometrics Centre.
Collaborator Contribution Prof Doebler provides valuable suggestions on experiment design and consults on statistical analysis. His team also created the automatic generators for figural analogies test and propositional reasoning test.
Impact All item generators have been created, and two manuscripts have been submitted for publication. Both validated item sets and item generators will be made available on the ICAR website. Two generators have also been translated into multiple languages, awaiting further validation.
Start Year 2014
 
Description Prof Luo from Beijing Normal University 
Organisation Beijing Normal University
Department School of Psychology
Country China 
Sector Academic/University 
PI Contribution The ICAR team in Cambridge is collaborating with Prof Luo from Beijing Normal University on projects to develop automatic item generators for number series reasoning test, English comprehension test, and matrices reasoning test.
Collaborator Contribution Prof Luo helps with data collection and provides valuable suggestions on statistical analysis.
Impact Two manuscripts are currently under preparation for submission. Cognitive measures under study will be made available via the ICAR website.
Start Year 2015
 
Description Prof Revelle and Dr Condon from Northwestern University 
Organisation Northwestern University
Department Department of Psychology
Country United States 
Sector Academic/University 
PI Contribution The ICAR team in Cambridge is collaborating with Prof Revelle and Dr Condon on projects to validate cognitive measures and link psychological databases. Tests are implemented on the Concerto platform and made available via the testing website hosted by the Psychometrics Centre. ICAR measures are also cross-validated via the SAPA project.
Collaborator Contribution Prof Revelle and Dr Condon provide original test items and help create tests on the SAPA website. Prof Revelle provides valuable suggestions on manuscripts of intelligence research.
Impact Data is being collected, based on which conference presentations and journal publications are expected. Validated item sets will be shared on the ICAR website.
Start Year 2014
 
Description Prof Revelle and Dr Condon from Northwestern University 
Organisation Northwestern University
Country United States 
Sector Academic/University 
PI Contribution The ICAR team in Cambridge is collaborating with Prof Revelle and Dr Condon on projects to validate cognitive measures and link psychological databases. Tests are implemented on the Concerto platform and made available via the testing website hosted by the Psychometrics Centre. ICAR measures are also cross-validated via the SAPA project.
Collaborator Contribution Prof Revelle and Dr Condon provide original test items and help create tests on the SAPA website. Prof Revelle provides valuable suggestions on manuscripts of intelligence research.
Impact Data is being collected, based on which conference presentations and journal publications are expected. Validated item sets will be shared on the ICAR website.
Start Year 2014
 
Title AIG 
Description AIG: Automatic Item Generator. R package version 0.1.8. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Anyone can make use of the R package to generate arithmetic items, number series reasoning items, syllogism reasoning items, 2D and 3D rotation items. 
 
Title mazeGen 
Description mazeGen: Elithorn Maze Generator. R package version 0.1.3. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Anyone can make use of the R package to generate maze items. 
 
Title networkGen 
Description networkGen: Network Maze Generator. R package version 0.1.1. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Anyone can make use of the R package to generate network items. 
 
Title numGen 
Description numGen: Number Series Generator. R package version 0.1.0. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Anyone can make use of the R package to generate number series reasoning items/ 
 
Description Big Data 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The Psychometrics Centre organised an international conference on Educational Data Mining and Applications in China together with Beijing Normal University, where David Stillwell, Fiona Chan and Aiden Loe presented their research on advanced assessment methods.
Year(s) Of Engagement Activity 2017
 
Description Festival of Ideas 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Aiden Loe organised an event at the Festival of Ideas, where he showcased the adaptive cognitive ability tests.
Year(s) Of Engagement Activity 2017
 
Description Forum 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Aiden Loe attended The Psychometrics Forum, where he presented his research on 3D rotation item generation.
Year(s) Of Engagement Activity 2017
 
Description IACAT 2014 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Luning Sun and Fiona Chan attended the 2014 summit of IACAT in USA, and presented their research on facial expression recognition and matrices reasoning.
Year(s) Of Engagement Activity 2014
 
Description IACAT 2015 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Luning Sun and Roeland Verhallen attended the 2015 summit of IACAT and presented the newly developed adaptive face detection test.
Year(s) Of Engagement Activity 2015
 
Description ICAR newsletter 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact ICAR project produces newsletters to introduce recent progress in research, particularly new item generator developments. The newsletter reach out to the whole ICAR community, including both core members, collaborators as well as more than 800 registered users on the website.
Year(s) Of Engagement Activity 2015,2016,2017
 
Description ICAR website 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact A website has been constructed for the ICAR project, where validated item sets and item generators are made available to qualified research groups around the world. There have been more than 900 registered users, and more than 250 of them have been granted access to the ICAR measures.
Year(s) Of Engagement Activity 2014,2015,2016,2017,2018
URL https://icar-project.com/
 
Description IMPS 2015 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The ICAR team in Cambridge attended the 2015 IMPS in Beijing, and presented their research.
Year(s) Of Engagement Activity 2015
 
Description IMPS 2017 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Luning Sun attended the 2017 IMPS in Zurich, met ICAR collaborators and presented his research.
Year(s) Of Engagement Activity 2017
 
Description ISIR 2016 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Luning Sun and Aiden Loe attended the ISIR conference in Russia in 2016, and presented their research on intelligence measurement.
Year(s) Of Engagement Activity 2016