The Research Design Basis for Undergraduate Quantitative Methods (QM) Teaching

Lead Research Organisation: Durham University
Department Name: Education

Abstract

This project will show QM trainers how to encourage social science undergraduates (UGs) in many subject areas to engage with, use and re-use numeric evidence both wisely and appropriately. The project has assembled a team of international design experts to work with a large number of UG social science methods trainers in seven major UK universities. Together they will all prepare template course materials that link research designs to methods of quantitative data collection and analysis, and adapt these templates for use in their own subject areas. In this way, research design can be embedded in the teaching of undergraduate quantitative methods, through provision of training for current and future providers. The renewed emphasis on design at the outset of a study, currently rather neglected in UK researcher development, will have a number of immediate and longer-term benefits. It discourages schismic thinking among UGs since designs are largely independent of specific methods of data collection. It encourages true prediction and forethought in collecting data for new studies. It encourages the correct choice of methods of analysis for data that have already been collected. Above all, it holds the promise of simplifying analysis to such an extent that undergraduates may not even be aware that they are, in fact, working with 'quantitative' methods.

The applicants and their team of experts and hosts are internationally-renowned methods experts who specialise in QM design, as well as being professionally-recognised and lauded adult educators. This combined expertise, together with the close engagement of trainers in QM in undergraduate teaching, will fill a major gap in UK social science QM teaching. International experts will prepare a wide range of resources and materials to show how design naturally and easily underpins the teaching of QM, accessible to undergraduates across all areas of social science. Although the focus will be on undergraduate teaching, we will build into our programme of events a progression from introductory through intermediate and to advanced training in order to up-skill QM teachers in the embedding of design at all levels. This will enable a step-change in teaching at undergraduate level and even beyond. The work programme of up-skilling of research methods trainers includes the creation of support and curricular materials tailored to all relevant areas, sustained collaborative development of materials and approaches, and delivery of intensive clinics UK-wide. THe materials will be trialled in practice in social science UG teaching among the seven HEIs involved, and ideally more widely by the end of the project. The project is led by the College of Social Sciences at the University of Birmingham. It will be delivered in co-operation with the Universities of Cardiff, Leicester, Manchester, Nottingham, Warwick and York, and will involve experts from US Northwestern University.

Planned Impact

To a large extent the immediate impact of this project, as part of the QM intitiative and RDI, will be on academic beneficaries (as described elsewhere). Those academics who are or will be responsible for delivery of lectures on social science research and methods should gain something from participation in or use of materials from this project. It is, as explained elsewhere, using the approach of design as a kind of Trojan horse to get more trainers to engage with QM teaching, and to help more students find enthusiasm for, and the relevance of, QM in their studies. The applicants are trained and experienced teachers at UG and adult edcuation level, lauded in official inspections, and with dozens of successful methods teaching texts and textbooks to their name, and whose work forms the entire basis of some methods courses in other HEIs across the world. The latter is largely due to their ability to make the complex seem simple and relevant to all UGs. The impact on trainers will be estimated via attendance, repeat 'business', website hits and downloads, evaluation forms and by a small-scale observed test of the new teaching approaches in action in a rela-life setting with UGs.

An indirect outcome of the proejct would be an increase in the supply of graduates, taught by the trainers who are the key focus of this project, for roles beyond the academic. It will not be possible to evaluate such longer term outcomes within the life of the project. We therefore rely on the opinion of those working in industry, policy or practice who are looking to recruit numerate analyticl graduates in the social sciences, such as the two nomiated user reviewers of this project.

Impact through World Class Research: The initiative will deliver a cohort of trainers across many areas of social sciencee, able to better assist budding researchers and analysts to achieve their potential. Design is a major gap in existing methods training, and essential for the contemporary and global policy environment.

Impact through Skilled People: The initiative addresses a significant cross-sectoral skill gap through a variety of mechanisms. It creates collaborative partnerships between researchers/academics at different stages of their careers. The community emerging from these collaborations will improve the prospects for skill transferability, employability and inter-sectoral exchanges of skilled people, thus improving flexibility and adaptability.

Impact through Infrastructure: The initiative will significantly enhance the pool of skills among research mmethods trainers in relation to the integration of methods in an overall research design of strong scientific credibility, and assist the use of large scale data sets now becoming available through the radical transparency agenda.

Impact through International Leadership: The involvement of leading international scholars is a major component of the initiative's profile. Engaging these contributors consolidates existing relationships and opens the way for long-term future research collaborations on the theory and application of rigorous evaluations.

Impact through Partnerships: Intrinsic to the operation of the initiative will be the creation of partnerships between eight US and UK universities from the outset, and it is hoped and expected that this number will grow as part of the planned cascade. These networks and partnerships will be in place for future research, and for the career development of up-and-coming methods experts.

Publications

10 25 50
 
Description In Phase I of the project, we registered 204 teachers (and upcoming teachers) of 'quantitative' methods from a wide range of social sciences and the humanities. These were in addition to the members of the project team and their consultant experts. All expressed interest in the premise of the project, and hoped both to contribute and learn more. The project had a focus on working with the teachers, rather than their students directly.

We held seven recruitment events, with 146 distinct attendees. The events took place in seven university centres, across England and Wales, and were open to all QM teachers. Each recruitment event consisted of a workshop led by the local host, on a range of themes including developing research questions, the elements of research design, problems of attrition, and more generally what is 'wrong' with QM teaching. The last two such events had already begun to work on possible solutions. Each workshop was followed by small group discussion of the challenges and barriers to teaching social science topics and analyses using numbers. The results were collected by a scribe in each group and also via a written confidential feedback form completed by all attendees. The key points of a meta-synthesis of feedback were as follows.

There is considerable variability in the experience and prior qualifications of QM-relevant teachers, with some just thrown in as the most recent arrivals with no pedagogic training. Many teachers mentioned fear both for themselves and their students. Senior staff who use QM regularly for research are seldom required to teach these skills to UGs. The teachers themselves often confused methods of data collection with research design. What they had been taught was often poorly explained and plain incorrect - especially in relation to significance testing and related issues. These errors are then transmitted to new generations of students. Numeric data are seldom used naturally in substantive courses, and those lecturers prepared to use numbers are forced to become QM 'experts' for methods courses seen as irrelevant by many students. Fuller reports are available at:
https://www.dur.ac.uk/resources/education/research/ESRC-Interim-report-July-2013-.pdf)
https://www.dur.ac.uk/resources/education/research/ug-quantitative-methods-phaseone-completion.pdf.
The users for this RDI project are other academics such as QM teachers, and less directly their UG charges. The project forms part of the move towards a 'new statistics', embracing judgement of substantive findings about extent rather than apparent but mistaken precision concerning binary decisions. Such an approach is safer, encourages researcher and research integrity, and allows findings to be inspected and judged by a wider audience. As part of that move, this project may, via impact on staff and students, help improve the quality of work with numbers in social science, its wider use, and its easier interpretation by potential research users.

For example, we used the same approach as with QM teachers on four occasions when training groups of school leaders and teachers how to plan, conduct and analyse a rigorous evaluation of an innovation in practice. This was part of two aggregated randomised controlled trials led by schools, and funded by the Educational Endowment Foundation. The focus on effect sizes, for pre-specified primary outcomes, was simple and very enthusiastically received by practitioners. They proved able to conduct these analyses correctly and accurately themselves. There will be other areas of public policy such as housing, health and criminal justice where these approaches would be useful.

One of the many barriers to impact is the sheer number of intiatives (reducing attendance at each event perhaps), and the lack of quality control in the kinds of QM resources now publicly available. The latter can lead trainers and students to regard what are matters of fact as though they were matters of opinion.




Some of the issues raised were specific to individuals and their HE contexts. Some were general but also beyond the scope of this project - low levels of numeracy among potential UGs, for example. Phase II of the project focussed on working together to simplify approaches, correct errors in teaching, provide realistic resources for teacher development, and overcome barriers to the natural integration of numbers in social science curricula. As part of this, we held four training and resource development events in three universities in England, with 70 invited attendees selected from those in Phase I, plus consultant experts. Around 40 academics continued to be involved, and 15 of these were primary authors of resources. We held two public showcase events at successive national confences, one on the value of considering reseach as an argument, and the second illustrating how design aids analysis, each attended by around 60 academics and doctoral researchers across the social sciences and public policy. We organised a session at the second conference in which Larry Hedges urged the importance of effect sizes, and presented a typology with their relative merits.

We developed a number of resources mostly for the website, but also in print, and this process is ongoing as part of the sustainability of the project. So far, these include books, chapters, papers, brief teaching guides, videos, and other resources (see below). In Phase III of the project, these were tried out in a variety of settings, most notably the UG QM teaching by Patrick White for all social scientists at the University of Leicester. Feedback has been excellent, and the trial at Lecicester formed part of a successful application for a Teaching Fellowship (just awarded). We have also presented the work of the project around the country in at least 25 invited seminars and workshops (see below).
The resources from Phase II are being collected on the project website which will be sustained indefinitely. The resources are for the development of QM trainers themselves and for UGs. They need organising further into a narrative for each audience so that they can be used more easily in modules, and in an appropriate sequence. They also need 'advertising' and in some cases academic justification. This is being achieved by a combination of peer-reviewed publication, presentation, lecture tours, videos, social media, and email list discussions. Considerable discussion has taken place on SAGE Methodspace, where queries by researchers worldwide have been addressed, project resources have been directly uploaded, and a Case Study produced for the first round to be published by SAGE.

So far, the project has produced one methods book, 10 peer-reviewed methods papers, 4 book chapters in methods texts, one case study, 29 invited presentations wholly or partly about this project, and the resources themselves. These include sections on:

Starting out
Research Design
Identifying the cases
The behaviour of numbers
Presenting one value at a time
Presenting a comparison of two or more values
Random sampling theory and its serious limitations
The meaning of numeric results
Working with existing data

Perhaps most notable are the series of videos for UGs, showing how to use SPSS. These are intended for stand-alone use or as part of a taught module in a blended learning format. Although in some ways SPSS is not ideal, since there is not yet a button to simply turn off the default emphasis on significance testing and asterisk-based presentation without consideration of the nature of the cases involved, the videos are innovative in the way that they ignore this. Instead they focus on the ability to explore, display and model datasets, and they do so in a way that is non-threatening to most social scientists. Feedback from UGs has been universally excellent.
Exploitation Route -To be used as a teaching resource
-To be applied for data analysis
-To be used by researchers in all the field of sciences
Sectors Aerospace, Defence and Marine,Agriculture, Food and Drink,Chemicals,Communities and Social Services/Policy,Construction,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Electronics,Energy,Environment,Financial Services, and Management Consultancy,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Government, Democracy and Justice,Manufacturing, including Industrial Biotechology,Culture, Heritage, Museums and Collections,Pharmaceu

URL https://www.dur.ac.uk/education/research/current_research/quantitative-methods/
 
Description Warranting research claims 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Presentation given at the University of Leicester's staff development day.

Requests for further details and publications
Year(s) Of Engagement Activity 2010