A cognitive diagnostic investigation of inequality in mathematics mastery in England

Many education studies use subject-level measurements for achievement, such as scores from PISA or national-level assessments. This approach implies disadvantage affects all areas of a subject equally. However, fine-grained studies have found achievement gaps below the subject level. For example, in mathematics, studies have found girls performed better on calculation questions, while boys performed better on algebra and geometry. As a result, policies or interventions using subject-level score as the sole indicator of mathematics ability would overlook differences between genders on specific topics.
Skills rather than topics
Most modern testing is based on Item Response Theory (IRT). Traditional IRT statistical models assume that a test is unidimensional (i.e. the test measures a single, overall ability) and that questions should function the same for all sociodemographic student groups.
Cognitive diagnostic models (CDM) bypass these assumptions of IRT since they allow multidimensionality. Simply put, CDM are designed by identifying specific skills required to successfully solve each test question. The model estimates each student's mastery of the skills and the difficulty of each skill. The information can then be used to identify groups with similar mastery characteristics.
Studies show that CDM can produce better model-fit compared with IRT (Ma et al., 2020; Yamaguchi & Okada, 2018). Also, true to the goal of cognitive diagnosis, CDM have been used to identify curriculum weaknesses, such as finding that students in the USA need extra support for reading data from tables and graphs (Lee et al., 2011).
Focus
Mathematics is an especially pertinent topic in England considering the government is considering making maths compulsory in post-16 education. Important information for England can be found by investigating groups' mathematics mastery with CDM; for example, identifying whether particular sociodemographic groups struggle with a specific area of the curriculum.
The following questions will guide the research:
1. Can English pupils with similar mathematics mastery profiles be meaningfully described as different sociodemographic groups?
2. Are groups consistent across different mathematics tests for England?
3. Are groups different in different countries?
4. Are groups consistent across IRT models and CDM?
5. Do CDM or IRT models provide better model-fit?
Methodology
International and national mathematics tests can provide the response-level data that is required for CDM. At national level, data from KS2 SATs and GCSEs may be accessed, and for international tests, PISA and TIMSS will be used.
A variety of CDM approaches exist, based on different assumptions. While the DINA model is most common, Yamaguchi & Okada (2018) found that main effects models gave better model fit for TIMSS data (i.e. R-RUM, A-CDM, LLM). Furthermore, CDM have been developed to account for partial mastery of attributes, multilevel data, and hierarchical attributes (i.e. mastery of one skill requires mastery of lower skills). These variations will be considered when designing the model.
Implications
Special attention will be given to policy and practice recommendations. Large-scale assessments have been criticised for providing little actionable information, and the fined-grained analysis of CDM could be a partial solution to this.
The findings can have important curriculum implications. If students who are not choosing to continue studying mathematics post-16 have different mastery profiles than those who are, then the curriculum may be developed to provide more support for the relative weaknesses.
Teaching strategies and interventions will be reviewed for the maths skills where outcomes differ. Furthermore, some claim that assessment research might often ignore learning theory. Reviewing developments in learning sciences could provide insight into the causes of groups' different mathematics mastery.