The Research Design Basis for Undergraduate Quantitative Methods (QM) Teaching

Lead Research Organisation: University of Birmingham
Department Name: Education

Abstract

This project will show QM trainers how to encourage social science undergraduates (UGs) in many subject areas to engage with, use and re-use numeric evidence both wisely and appropriately. The project has assembled a team of international design experts to work with a large number of UG social science methods trainers in seven major UK universities. Together they will all prepare template course materials that link research designs to methods of quantitative data collection and analysis, and adapt these templates for use in their own subject areas. In this way, research design can be embedded in the teaching of undergraduate quantitative methods, through provision of training for current and future providers. The renewed emphasis on design at the outset of a study, currently rather neglected in UK researcher development, will have a number of immediate and longer-term benefits. It discourages schismic thinking among UGs since designs are largely independent of specific methods of data collection. It encourages true prediction and forethought in collecting data for new studies. It encourages the correct choice of methods of analysis for data that have already been collected. Above all, it holds the promise of simplifying analysis to such an extent that undergraduates may not even be aware that they are, in fact, working with 'quantitative' methods.

The applicants and their team of experts and hosts are internationally-renowned methods experts who specialise in QM design, as well as being professionally-recognised and lauded adult educators. This combined expertise, together with the close engagement of trainers in QM in undergraduate teaching, will fill a major gap in UK social science QM teaching. International experts will prepare a wide range of resources and materials to show how design naturally and easily underpins the teaching of QM, accessible to undergraduates across all areas of social science. Although the focus will be on undergraduate teaching, we will build into our programme of events a progression from introductory through intermediate and to advanced training in order to up-skill QM teachers in the embedding of design at all levels. This will enable a step-change in teaching at undergraduate level and even beyond. The work programme of up-skilling of research methods trainers includes the creation of support and curricular materials tailored to all relevant areas, sustained collaborative development of materials and approaches, and delivery of intensive clinics UK-wide. THe materials will be trialled in practice in social science UG teaching among the seven HEIs involved, and ideally more widely by the end of the project. The project is led by the College of Social Sciences at the University of Birmingham. It will be delivered in co-operation with the Universities of Cardiff, Leicester, Manchester, Nottingham, Warwick and York, and will involve experts from US Northwestern University.

Planned Impact

To a large extent the immediate impact of this project, as part of the QM intitiative and RDI, will be on academic beneficaries (as described elsewhere). Those academics who are or will be responsible for delivery of lectures on social science research and methods should gain something from participation in or use of materials from this project. It is, as explained elsewhere, using the approach of design as a kind of Trojan horse to get more trainers to engage with QM teaching, and to help more students find enthusiasm for, and the relevance of, QM in their studies. The applicants are trained and experienced teachers at UG and adult edcuation level, lauded in official inspections, and with dozens of successful methods teaching texts and textbooks to their name, and whose work forms the entire basis of some methods courses in other HEIs across the world. The latter is largely due to their ability to make the complex seem simple and relevant to all UGs. The impact on trainers will be estimated via attendance, repeat 'business', website hits and downloads, evaluation forms and by a small-scale observed test of the new teaching approaches in action in a rela-life setting with UGs.

An indirect outcome of the proejct would be an increase in the supply of graduates, taught by the trainers who are the key focus of this project, for roles beyond the academic. It will not be possible to evaluate such longer term outcomes within the life of the project. We therefore rely on the opinion of those working in industry, policy or practice who are looking to recruit numerate analyticl graduates in the social sciences, such as the two nomiated user reviewers of this project.

Impact through World Class Research: The initiative will deliver a cohort of trainers across many areas of social sciencee, able to better assist budding researchers and analysts to achieve their potential. Design is a major gap in existing methods training, and essential for the contemporary and global policy environment.

Impact through Skilled People: The initiative addresses a significant cross-sectoral skill gap through a variety of mechanisms. It creates collaborative partnerships between researchers/academics at different stages of their careers. The community emerging from these collaborations will improve the prospects for skill transferability, employability and inter-sectoral exchanges of skilled people, thus improving flexibility and adaptability.

Impact through Infrastructure: The initiative will significantly enhance the pool of skills among research mmethods trainers in relation to the integration of methods in an overall research design of strong scientific credibility, and assist the use of large scale data sets now becoming available through the radical transparency agenda.

Impact through International Leadership: The involvement of leading international scholars is a major component of the initiative's profile. Engaging these contributors consolidates existing relationships and opens the way for long-term future research collaborations on the theory and application of rigorous evaluations.

Impact through Partnerships: Intrinsic to the operation of the initiative will be the creation of partnerships between eight US and UK universities from the outset, and it is hoped and expected that this number will grow as part of the planned cascade. These networks and partnerships will be in place for future research, and for the career development of up-and-coming methods experts.

Publications

10 25 50
 
Description Research design in social sciences is somewhat under-represented in UG methods development at present. This project proposes to use design as a kind of Trojan Horse to 'smuggle' QM approaches into areas where UGs have traditionally found the use of statistics difficult and unappealing. Our initial pilot work at the University of Birmingham suggested that students can more easily appreciate complex aspects of design (with a subsequent comparison of means for example) than a complex post hoc analysis, and that the former leads to greater willingness to tackle the latter. The project has aimed to develop template teaching materials, using design as a basis; explain the proposed approach to a wide circle of methods trainers and student representatives; enthuse them, and together convert the templates into useable teaching resources for many specific areas of social science; trial and adapt them, and then make them widely and sustainably available for QM teachers across the UK.
July - October 2012: Meta-synthesis
A meta-synthesis of the focused discussion has been developed as an outcome of the Phase I. This document includes the barriers in the learning and teaching of research design and QM as
4
reported by the teachers. The development of these points is based on the group work by the participant teachers. The task was to discuss and present their main difficulties in learning and teaching of research design and QM.
In each of the workshop the groups were divided among experienced teachers/ researchers, new teachers/ researchers and PhD students. These three groups brought their varied experiences and through discussions amongst each other they were able to address the barriers in successful teaching of research design and QM.
The meta-synthesis given below illustrates the reported barriers and difficulties according the frequency of issues mentioned. This highlights the aims of Phase I of the project in the end of which we have achieved a substantial amount of information to develop an action plan that includes trials of simple teaching methods and assessment of teaching/learning techniques and resources.
1. Teachers/trainers: Variability, diversity in teachers/trainers: confidence, expertise and experience in teaching/training/supervising; decisions about: mode of delivery of teaching in quantitative methods ('off the peg' or bespoke); problem-based learning or not; specifics (e.g., NHST or effect sizes); 'faulty' teaching, particularly in relation to statistics; - randomness, population data, sample sizes, research designs all are taught with varied ideas and often wrong information; FEAR - inexperience, lack of knowledge, lack of support from colleagues; unexciting teaching; teachers confuse data collection methods and research design; the examples used for teaching QM are often irrelevant to group of targeted students or from an insufficient range of designs; presentation of quantitative data is often complex and inaccessible to understand; real life examples are not included (enquiry and interpretation - use of 'real' examples and 'real' analysis - makes it exciting); presentation of quantitative data is often complex and inaccessible to understand; generalist versus specialist statistics [20]
2. Curricular/course/module pressures: Teaching and practice of QM is not sufficient according to the available time: too much to cover; small amount of time to teach a great deal; courses too full; curriculum design issues; courses isolated and disjointed; titles of modules and courses should have appeal for novice researchers or beginners; quantitative 'methods' (design, statistics) teaching divorced from content teaching: teachers/trainers; structure; modules; assignments; time for teaching QM is very limited and often not repeated in the complete syllabus; ancient inherited teaching materials and quality control across modules [9]
3. Research questions and design: Research questions not the starting point of research; the discussions about research topics instantly skip the development of research questions. The common mistake is to pick random methods with less thought on the nature of research questions; teachers confuse data collection methods and research design; teachers' perceptions about methods are passed on to students; supervisors become barriers to students learning experiences; supervisors - relationship with research methods trainers; lack of integration of 'stats' with 'design'; 'quantitative'/'qualitative' divide: methods bias (institutions, teachers/trainers; students; disciplines; traditions; teaching/training); Stats 'in a box; how to focus on high level concepts (design) rather than on 'stats'; Focus on research design - not separate 'qual'/'quant' [8]
4. Students: Variability, diversity in students: interests; learning styles; prior knowledge (particularly in statistics); perceptions about statistics and QM.; expectations and goals; motivations; engagement (lack of early engagement); FEAR - of numbers - unfamiliarity, irrelevance, technical barriers, lack of motivation, prior perceptions and knowledge [7]
5. Resources: Journals and journal articles: Lack of knowledge and practice to read research articles; better resources (text books; journals) needed; access to accurate and simplified knowledge.
about
5
QM is not widely available; readability of journal articles; journal articles - 'quant' methods not transparent, not accessible [6]
6. Terminology and language: Students' difficulties in understanding; language barriers; language/terminology daunting and complicated [3]
Cardiff (Cardiff Workshop addressed the barriers and desired solutions of the problems)
7. Standards: There are no set standards of Research Design modules across all social science disciplines. There is an essential need to validate Research Design courses by higher educational councils that must ensure the main elements of research design techniques.
8. Dichotomies: Method presented, taught and 'evangelised' to students as divided between qualitative and quantitative. These adverse dichotomies of QM should be discouraged through writing journals and training of researchers/ teachers.
9. Have the students design their own study in Year 1. They would develop a research question and then a design which best answers that question, choosing from among available methodologies without labelling those approaches as either quantitative or qualitative. This allow the question to lead the design without the student making choices based on value judgements of qualitative or quantitative being intrinsically better or worse than each other, but simply better fitted for gathering and analysing different types of data.
10. Encouraging critical thinking by introducing a consideration of research methods in other content areas. For example, when talking about evidence in a social psychology class, including an evaluation of the methods would highlight both good and bad practice for students and model an understanding of the strengths and limitations of different methods and analyses.
Universities
University staff are in a position to both model and teach good research practice, and so there are several actions which could improve both practice and the teaching of undergraduates.
1. Staff should receive on-going training to enable them to integrate better practice into their teaching and curriculum(for example, points 1 and 2 above).
2. Departments could scrutinise course materials and text books to ensure they adhere to a high standard in their explanation of research methods. They could also identify gaps in publishers' lists relating to areas such as research design to encourage more attention and writing in these areas.
3. Highlight and reward positive examples of good practice to encourage a higher degree of academic honesty and integrity.
4. Increase the value and profile of the teaching of research methods by making it part of funding conditions, REF and validation, to make sure cutting edge researchers are involved in teaching undergraduate research methods so that senior, experienced leaders are passing on their expertise to future researchers.
5. Reduce the artificial divide between qualitative and quantitative methods in the minds university staff and researchers through renaming courses, training, and departmental discussion. Done properly, both methods should be of equal value and rigour.
The culture and practices of the wider research community also need to be challenged if better practice is to become a reality. Journals demand particular statistical analyses, and the need to be published and the demands of editors in turn affect the attitudes and practices of researchers, who develop their study designs and analyses with publication in mind. Changes in this culture.
could be achieved by:
1. Letter-writing campaigns could target both poor practice and the divide between different methods, such as qualitative and quantitative, and be directed to both professional and government bodies as well as the editors of journals who publish studies with faulty statistics and design.
6
2. Publishing papers or other media (such as blogs and social media) which highlight good practice and the consequences of basing public policy on problematic research, in the way that authors such as Ben Goldacre do for medical science.
Exploitation Route -To be used by teachers and researchers
-To be used by HE policy makers
Sectors Communities and Social Services/Policy,Education

URL https://www.dur.ac.uk/resources/education/research/ESRC-Interim-report-July-2013-.pdf
 
Description Most recently the findings have been the subject of considerable discussion in social media - prompted by the latest open access papers.
First Year Of Impact 2014
Sector Education
Impact Types Cultural,Policy & public services

 
Description Warranting research claims 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Presentation given at the University of Leicester's staff development day.

Requests for further details and publications
Year(s) Of Engagement Activity 2010