example of quantitative research paper about education

• Subject List
• Take a Tour
• For Authors
• Subscriber Services
• Publications
• African American Studies
• African Studies
• American Literature
• Anthropology
• Architecture Planning and Preservation
• Art History
• Atlantic History
• Biblical Studies
• British and Irish Literature
• Childhood Studies
• Chinese Studies
• Cinema and Media Studies
• Communication
• Criminology
• Environmental Science
• Evolutionary Biology
• International Law
• International Relations
• Islamic Studies
• Jewish Studies
• Latin American Studies
• Latino Studies
• Linguistics
• Literary and Critical Theory
• Medieval Studies
• Military History
• Political Science
• Public Health
• Renaissance and Reformation
• Social Work
• Urban Studies
• Victorian Literature
• Browse All Subjects

How to Subscribe

• Free Trials

In This Article Expand or collapse the "in this article" section Quantitative Research Designs in Educational Research

Introduction, general overviews.

• Survey Research Designs
• Correlational Designs
• Other Nonexperimental Designs
• Randomized Experimental Designs
• Quasi-Experimental Designs
• Single-Case Designs
• Single-Case Analyses

Related Articles Expand or collapse the "related articles" section about

About related articles close popup.

Lorem Ipsum Sit Dolor Amet

Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Aliquam ligula odio, euismod ut aliquam et, vestibulum nec risus. Nulla viverra, arcu et iaculis consequat, justo diam ornare tellus, semper ultrices tellus nunc eu tellus.

• Educational Research Approaches: A Comparison
• Methodologies for Conducting Education Research
• Mixed Methods Research
• Multivariate Research Methodology
• Qualitative Data Analysis Techniques
• Qualitative, Quantitative, and Mixed Methods Research Sampling Strategies
• Researcher Development and Skills Training within the Context of Postgraduate Programs
• Single-Subject Research Design
• Social Network Analysis
• Statistical Assumptions

Other Subject Areas

Forthcoming articles expand or collapse the "forthcoming articles" section.

• Cyber Safety in Schools
• Girls' Education in the Developing World
• History of Education in Europe
• Find more forthcoming articles...
• Export Citations
• Share This Facebook LinkedIn Twitter

Quantitative Research Designs in Educational Research by James H. McMillan , Richard S. Mohn , Micol V. Hammack LAST REVIEWED: 24 July 2013 LAST MODIFIED: 24 July 2013 DOI: 10.1093/obo/9780199756810-0113

The field of education has embraced quantitative research designs since early in the 20th century. The foundation for these designs was based primarily in the psychological literature, and psychology and the social sciences more generally continued to have a strong influence on quantitative designs until the assimilation of qualitative designs in the 1970s and 1980s. More recently, a renewed emphasis on quasi-experimental and nonexperimental quantitative designs to infer causal conclusions has resulted in many newer sources specifically targeting these approaches to the field of education. This bibliography begins with a discussion of general introductions to all quantitative designs in the educational literature. The sources in this section tend to be textbooks or well-known sources written many years ago, though still very relevant and helpful. It should be noted that there are many other sources in the social sciences more generally that contain principles of quantitative designs that are applicable to education. This article then classifies quantitative designs primarily as either nonexperimental or experimental but also emphasizes the use of nonexperimental designs for making causal inferences. Among experimental designs the article distinguishes between those that include random assignment of subjects, those that are quasi-experimental (with no random assignment), and those that are single-case (single-subject) designs. Quasi-experimental and nonexperimental designs used for making causal inferences are becoming more popular in education given the practical difficulties and expense in conducting well-controlled experiments, particularly with the use of structural equation modeling (SEM). There have also been recent developments in statistical analyses that allow stronger causal inferences. Historically, quantitative designs have been tied closely to sampling, measurement, and statistics. In this bibliography there are important sources for newer statistical procedures that are needed for particular designs, especially single-case designs, but relatively little attention to sampling or measurement. The literature on quantitative designs in education is not well focused or comprehensively addressed in very many sources, except in general overview textbooks. Those sources that do include the range of designs are introductory in nature; more advanced designs and statistical analyses tend to be found in journal articles and other individual documents, with a couple exceptions. Another new trend in educational research designs is the use of mixed-method designs (both quantitative and qualitative), though this article does not emphasize these designs.

For many years there have been textbooks that present the range of quantitative research designs, both in education and the social sciences more broadly. Indeed, most of the quantitative design research principles are much the same for education, psychology, and other social sciences. These sources provide an introduction to basic designs that are used within the broader context of other educational research methodologies such as qualitative and mixed-method. Examples of these textbooks written specifically for education include Johnson and Christensen 2012 ; Mertens 2010 ; Arthur, et al. 2012 ; and Creswell 2012 . An example of a similar text written for the social sciences, including education that is dedicated only to quantitative research, is Gliner, et al. 2009 . In these texts separate chapters are devoted to different types of quantitative designs. For example, Creswell 2012 contains three quantitative design chapters—experimental, which includes both randomized and quasi-experimental designs; correlational (nonexperimental); and survey (also nonexperimental). Johnson and Christensen 2012 also includes three quantitative design chapters, with greater emphasis on quasi-experimental and single-subject research. Mertens 2010 includes a chapter on causal-comparative designs (nonexperimental). Often survey research is addressed as a distinct type of quantitative research with an emphasis on sampling and measurement (how to design surveys). Green, et al. 2006 also presents introductory chapters on different types of quantitative designs, but each of the chapters has different authors. In this book chapters extend basic designs by examining in greater detail nonexperimental methodologies structured for causal inferences and scaled-up experiments. Two additional sources are noted because they represent the types of publications for the social sciences more broadly that discuss many of the same principles of quantitative design among other types of designs. Bickman and Rog 2009 uses different chapter authors to cover topics such as statistical power for designs, sampling, randomized controlled trials, and quasi-experiments, and educational researchers will find this information helpful in designing their studies. Little 2012 provides a comprehensive coverage of topics related to quantitative methods in the social, behavioral, and education fields.

Arthur, James, Michael Waring, Robert Coe, and Larry V. Hedges, eds. 2012. Research methods & methodologies in education . Thousand Oaks, CA: SAGE.

Readers will find this book more of a handbook than a textbook. Different individuals author each of the chapters, representing quantitative, qualitative, and mixed-method designs. The quantitative chapters are on the treatment of advanced statistical applications, including analysis of variance, regression, and multilevel analysis.

Bickman, Leonard, and Debra J. Rog, eds. 2009. The SAGE handbook of applied social research methods . 2d ed. Thousand Oaks, CA: SAGE.

This handbook includes quantitative design chapters that are written for the social sciences broadly. There are relatively advanced treatments of statistical power, randomized controlled trials, and sampling in quantitative designs, though the coverage of additional topics is not as complete as other sources in this section.

Creswell, John W. 2012. Educational research: Planning, conducting, and evaluating quantitative and qualitative research . 4th ed. Boston: Pearson.

Creswell presents an introduction to all major types of research designs. Three chapters cover quantitative designs—experimental, correlational, and survey research. Both the correlational and survey research chapters focus on nonexperimental designs. Overall the introductions are complete and helpful to those beginning their study of quantitative research designs.

Gliner, Jeffrey A., George A. Morgan, and Nancy L. Leech. 2009. Research methods in applied settings: An integrated approach to design and analysis . 2d ed. New York: Routledge.

This text, unlike others in this section, is devoted solely to quantitative research. As such, all aspects of quantitative designs are covered. There are separate chapters on experimental, nonexperimental, and single-subject designs and on internal validity, sampling, and data-collection techniques for quantitative studies. The content of the book is somewhat more advanced than others listed in this section and is unique in its quantitative focus.

Green, Judith L., Gregory Camilli, and Patricia B. Elmore, eds. 2006. Handbook of complementary methods in education research . Mahwah, NJ: Lawrence Erlbaum.

Green, Camilli, and Elmore edited forty-six chapters that represent many contemporary issues and topics related to quantitative designs. Written by noted researchers, the chapters cover design experiments, quasi-experimentation, randomized experiments, and survey methods. Other chapters include statistical topics that have relevance for quantitative designs.

Johnson, Burke, and Larry B. Christensen. 2012. Educational research: Quantitative, qualitative, and mixed approaches . 4th ed. Thousand Oaks, CA: SAGE.

This comprehensive textbook of educational research methods includes extensive coverage of qualitative and mixed-method designs along with quantitative designs. Three of twenty chapters focus on quantitative designs (experimental, quasi-experimental, and single-case) and nonexperimental, including longitudinal and retrospective, designs. The level of material is relatively high, and there are introductory chapters on sampling and quantitative analyses.

Little, Todd D., ed. 2012. The Oxford handbook of quantitative methods . Vol. 1, Foundations . New York: Oxford Univ. Press.

This handbook is a relatively advanced treatment of quantitative design and statistical analyses. Multiple authors are used to address strengths and weaknesses of many different issues and methods, including advanced statistical tools.

Mertens, Donna M. 2010. Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods . 3d ed. Thousand Oaks, CA: SAGE.

This textbook is an introduction to all types of educational designs and includes four chapters devoted to quantitative research—experimental and quasi-experimental, causal comparative and correlational, survey, and single-case research. The author’s treatment of some topics is somewhat more advanced than texts such as Creswell 2012 , with extensive attention to threats to internal validity for some of the designs.

back to top

Users without a subscription are not able to see the full content on this page. Please subscribe or login .

Oxford Bibliographies Online is available by subscription and perpetual access to institutions. For more information or to contact an Oxford Sales Representative click here .

• About Education »
• Meet the Editorial Board »
• Academic Achievement
• Academic Audit for Universities
• Academic Freedom and Tenure in the United States
• Action Research in Education
• Adjuncts in Higher Education in the United States
• Administrator Preparation
• Adolescence
• Advanced Placement and International Baccalaureate Courses
• Advocacy and Activism in Early Childhood
• African American Racial Identity and Learning
• Alaska Native Education
• Alternative Certification Programs for Educators
• Alternative Schools
• American Indian Education
• Animals in Environmental Education
• Art Education
• Artificial Intelligence and Learning
• Assessing School Leader Effectiveness
• Assessment, Behavioral
• Assessment, Educational
• Assessment in Early Childhood Education
• Assistive Technology
• Augmented Reality in Education
• Beginning-Teacher Induction
• Bilingual Education and Bilingualism
• Black Undergraduate Women: Critical Race and Gender Perspe...
• Black Women in Academia
• Blended Learning
• Case Study in Education Research
• Changing Professional and Academic Identities
• Character Education
• Children’s and Young Adult Literature
• Children's Beliefs about Intelligence
• Children's Rights in Early Childhood Education
• Citizenship Education
• Civic and Social Engagement of Higher Education
• Classroom Learning Environments: Assessing and Investigati...
• Classroom Management
• Coherent Instructional Systems at the School and School Sy...
• College Admissions in the United States
• College Athletics in the United States
• Community Relations
• Comparative Education
• Computer-Assisted Language Learning
• Computer-Based Testing
• Conceptualizing, Measuring, and Evaluating Improvement Net...
• Continuous Improvement and "High Leverage" Educational Pro...
• Counseling in Schools
• Critical Approaches to Gender in Higher Education
• Critical Perspectives on Educational Innovation and Improv...
• Critical Race Theory
• Crossborder and Transnational Higher Education
• Cross-National Research on Continuous Improvement
• Cross-Sector Research on Continuous Learning and Improveme...
• Cultural Diversity in Early Childhood Education
• Culturally Responsive Leadership
• Culturally Responsive Pedagogies
• Culturally Responsive Teacher Education in the United Stat...
• Curriculum Design
• Data Collection in Educational Research
• Data-driven Decision Making in the United States
• Deaf Education
• Desegregation and Integration
• Design Thinking and the Learning Sciences: Theoretical, Pr...
• Development, Moral
• Dialogic Pedagogy
• Digital Age Teacher, The
• Digital Citizenship
• Digital Divides
• Disabilities
• Distance Learning
• Distributed Leadership
• Doctoral Education and Training
• Early Childhood Education and Care (ECEC) in Denmark
• Early Childhood Education and Development in Mexico
• Early Childhood Education in Aotearoa New Zealand
• Early Childhood Education in Australia
• Early Childhood Education in China
• Early Childhood Education in Europe
• Early Childhood Education in Sub-Saharan Africa
• Early Childhood Education in Sweden
• Early Childhood Education Pedagogy
• Early Childhood Education Policy
• Early Childhood Education, The Arts in
• Early Childhood Mathematics
• Early Childhood Science
• Early Childhood Teacher Education
• Early Childhood Teachers in Aotearoa New Zealand
• Early Years Professionalism and Professionalization Polici...
• Economics of Education
• Education For Children with Autism
• Education for Sustainable Development
• Education Leadership, Empirical Perspectives in
• Education of Native Hawaiian Students
• Education Reform and School Change
• Educational Statistics for Longitudinal Research
• Educator Partnerships with Parents and Families with a Foc...
• Emotional and Affective Issues in Environmental and Sustai...
• Emotional and Behavioral Disorders
• English as an International Language for Academic Publishi...
• Environmental and Science Education: Overlaps and Issues
• Environmental Education
• Environmental Education in Brazil
• Epistemic Beliefs
• Equity and Improvement: Engaging Communities in Educationa...
• Equity, Ethnicity, Diversity, and Excellence in Education
• Ethical Research with Young Children
• Ethics and Education
• Ethics of Teaching
• Ethnic Studies
• Evidence-Based Communication Assessment and Intervention
• Family and Community Partnerships in Education
• Family Day Care
• Federal Government Programs and Issues
• Feminization of Labor in Academia
• Finance, Education
• Financial Aid
• Formative Assessment
• Future-Focused Education
• Gender and Achievement
• Gender and Alternative Education
• Gender, Power and Politics in the Academy
• Gender-Based Violence on University Campuses
• Gifted Education
• Global Mindedness and Global Citizenship Education
• Global University Rankings
• Governance, Education
• Grounded Theory
• Growth of Effective Mental Health Services in Schools in t...
• Higher Education and Globalization
• Higher Education and the Developing World
• Higher Education Faculty Characteristics and Trends in the...
• Higher Education Finance
• Higher Education Governance
• Higher Education Graduate Outcomes and Destinations
• Higher Education in Africa
• Higher Education in China
• Higher Education in Latin America
• Higher Education in the United States, Historical Evolutio...
• Higher Education, International Issues in
• Higher Education Management
• Higher Education Policy
• Higher Education Research
• Higher Education Student Assessment
• High-stakes Testing
• History of Early Childhood Education in the United States
• History of Education in the United States
• History of Technology Integration in Education
• Homeschooling
• Inclusion in Early Childhood: Difference, Disability, and ...
• Inclusive Education
• Indigenous Education in a Global Context
• Indigenous Learning Environments
• Indigenous Students in Higher Education in the United Stat...
• Infant and Toddler Pedagogy
• Inservice Teacher Education
• Integrating Art across the Curriculum
• Intelligence
• Intensive Interventions for Children and Adolescents with ...
• International Perspectives on Academic Freedom
• Intersectionality and Education
• Knowledge Development in Early Childhood
• Leadership Development, Coaching and Feedback for
• Leadership in Early Childhood Education
• Leadership Training with an Emphasis on the United States
• Learning Analytics in Higher Education
• Learning Difficulties
• Learning, Lifelong
• Learning, Multimedia
• Learning Strategies
• Legal Matters and Education Law
• LGBT Youth in Schools
• Linguistic Diversity
• Linguistically Inclusive Pedagogy
• Literacy Development and Language Acquisition
• Literature Reviews
• Mathematics Identity
• Mathematics Instruction and Interventions for Students wit...
• Mathematics Teacher Education
• Measurement for Improvement in Education
• Measurement in Education in the United States
• Meta-Analysis and Research Synthesis in Education
• Methodological Approaches for Impact Evaluation in Educati...
• Mindfulness, Learning, and Education
• Motherscholars
• Multiliteracies in Early Childhood Education
• Multiple Documents Literacy: Theory, Research, and Applica...
• Museums, Education, and Curriculum
• Music Education
• Narrative Research in Education
• Native American Studies
• Nonformal and Informal Environmental Education
• Note-Taking
• Numeracy Education
• One-to-One Technology in the K-12 Classroom
• Online Education
• Open Education
• Organizing for Continuous Improvement in Education
• Organizing Schools for the Inclusion of Students with Disa...
• Outdoor Play and Learning
• Outdoor Play and Learning in Early Childhood Education
• Pedagogical Leadership
• Pedagogy of Teacher Education, A
• Performance Objectives and Measurement
• Performance-based Research Assessment in Higher Education
• Performance-based Research Funding
• Phenomenology in Educational Research
• Philosophy of Education
• Physical Education
• Podcasts in Education
• Policy Context of United States Educational Innovation and...
• Politics of Education
• Portable Technology Use in Special Education Programs and ...
• Post-humanism and Environmental Education
• Pre-Service Teacher Education
• Problem Solving
• Productivity and Higher Education
• Professional Development
• Professional Learning Communities
• Program Evaluation
• Programs and Services for Students with Emotional or Behav...
• Psychology Learning and Teaching
• Psychometric Issues in the Assessment of English Language ...
• Qualitative, Quantitative, and Mixed Methods Research Samp...
• Qualitative Research Design
• Quantitative Research Designs in Educational Research
• Queering the English Language Arts (ELA) Writing Classroom
• Race and Affirmative Action in Higher Education
• Reading Education
• Refugee and New Immigrant Learners
• Relational and Developmental Trauma and Schools
• Relational Pedagogies in Early Childhood Education
• Reliability in Educational Assessments
• Religion in Elementary and Secondary Education in the Unit...
• Researcher Development and Skills Training within the Cont...
• Research-Practice Partnerships in Education within the Uni...
• Response to Intervention
• Restorative Practices
• Risky Play in Early Childhood Education
• Role of Gender Equity Work on University Campuses through ...
• Scale and Sustainability of Education Innovation and Impro...
• Scaling Up Research-based Educational Practices
• School Accreditation
• School Choice
• School Culture
• School District Budgeting and Financial Management in the ...
• School Improvement through Inclusive Education
• School Reform
• Schools, Private and Independent
• School-Wide Positive Behavior Support
• Science Education
• Secondary to Postsecondary Transition Issues
• Self-Regulated Learning
• Self-Study of Teacher Education Practices
• Service-Learning
• Severe Disabilities
• Single Salary Schedule
• Single-sex Education
• Social Context of Education
• Social Justice
• Social Pedagogy
• Social Science and Education Research
• Social Studies Education
• Sociology of Education
• Standards-Based Education
• Student Access, Equity, and Diversity in Higher Education
• Student Assignment Policy
• Student Engagement in Tertiary Education
• Student Learning, Development, Engagement, and Motivation ...
• Student Participation
• Student Voice in Teacher Development
• Sustainability Education in Early Childhood Education
• Sustainability in Early Childhood Education
• Sustainability in Higher Education
• Teacher Beliefs and Epistemologies
• Teacher Collaboration in School Improvement
• Teacher Evaluation and Teacher Effectiveness
• Teacher Preparation
• Teacher Training and Development
• Teacher Unions and Associations
• Teacher-Student Relationships
• Teaching Critical Thinking
• Technologies, Teaching, and Learning in Higher Education
• Technology Education in Early Childhood
• Technology, Educational
• Technology-based Assessment
• The Bologna Process
• The Regulation of Standards in Higher Education
• Theories of Educational Leadership
• Three Conceptions of Literacy: Media, Narrative, and Gamin...
• Tracking and Detracking
• Traditions of Quality Improvement in Education
• Transformative Learning
• Transitions in Early Childhood Education
• Tribally Controlled Colleges and Universities in the Unite...
• Understanding the Psycho-Social Dimensions of Schools and ...
• University Faculty Roles and Responsibilities in the Unite...
• Using Ethnography in Educational Research
• Value of Higher Education for Students and Other Stakehold...
• Virtual Learning Environments
• Vocational and Technical Education
• Wellness and Well-Being in Education
• Women's and Gender Studies
• Young Children and Spirituality
• Young Children's Learning Dispositions
• Young Children's Working Theories
• Privacy Policy
• Cookie Policy
• Legal Notice
• Accessibility

Powered by:

• [185.80.151.41]
• 185.80.151.41

Quantitative Research in Education

In Asrifan, A. & Isumarni, N. (Ed.), Interdisciplinary Research: Collaborative Insights (Vol. 2, pp. 30-53). India: Island Publishers

Posted: 13 Mar 2023

Supaprawat Siripipatthanakul

Bangkok Thonburi University, Thailand; Manipal GlobalNxt University, Malaysia (MGNU)

Muthmainnah Muthmainnah

Universitas al asyariah mandar, andi asrifan.

Universitas Muhammadiyah Sidenreng Rappang

Sutithep Siripipattanakul

Faculty of Education, Kasetsart University, Thailand

Pichart Kaewpuang

Phranakhon Rajabhat University

Patcharavadee Sriboonruang

Kasetsart University

Pongsakorn Limna

Independent Researcher; Rangsit University; Unitar International University

Parichat Jaipong

Manipal GlobalNxt University

Tamonwan Sitthipon

University of Geomatika

Date Written: March 1, 2023

In the past few decades, educational practices have changed drastically, particularly regarding how information and learning are delivered and processed. Education research frequently employs quantitative methods. Quantitative education research provides numerical data that can prove or disprove a theory, and administrators can easily share the quantitative findings with other academics and districts. While the study may be based on relative sample size, educators and researchers can extrapolate the results from quantitative data to predict outcomes for larger student populations and groups. Educational research has a long history of utilising measurement and statistical methods. Commonly quantitative methods encompass a variety of statistical tests and instruments. Educators and students could transition to the digital era and research-based knowledge, including quantitative research in advanced higher education, as the technology has advanced. The quantitative research methods in education emphasise basic group designs for research and evaluation, analytic methods for exploring relationships between categorical and continuous measures, and statistical analysis procedures for group design data. The essential is to evaluate quantitative analysis and provide the research process, sampling techniques, the advantages and disadvantages of quantitative research in the article.

Keywords: Quantitative Research, Education, Learning, Technology, Statistical Analysis

Suggested Citation: Suggested Citation

Bangkok Thonburi University, Thailand ( email )

F16/10 Leabklongtaweewatana Rd., Khet Taweewatana Bangkok 10170 Thailand

Manipal GlobalNxt University, Malaysia (MGNU)

Jalan BBN 1/7, 71800 Nilai, Negeri Sembilan Malaysia

Jl. Budi Utomo No. 2 Manding Kecamatan Polewali Ka Polewali Mandar, West Sulawesi 91315 Indonesia +62-428-21038 (Phone)

Universitas Muhammadiyah Sidenreng Rappang ( email )

Jl. Angkatan 45 No. 1A. Lt. Salo Rappang, Sidenren Rappang, Sulawesi Selatan 91651 Indonesia

Faculty of Education, Kasetsart University, Thailand ( email )

50 Ngam Wong Wan Rd, Lat Yao Chatuchak Bangkok 10900 Thailand

Phranakhon Rajabhat University ( email )

Bangkok Thailand

Kasetsart University ( email )

Kasetsart University Kamphaeng Saen, Nakhon Pathom 73140 Thailand

Pongsakorn Limna (Contact Author)

Independent researcher ( email ).

Krabi Thailand

Rangsit University ( email )

52/347 Colonel Thaksin Lak Hok Pathum Thani Thailand

Unitar International University ( email )

Petaling Jaya Malaysia

Manipal GlobalNxt University ( email )

University of geomatika ( email ), do you have a job opening that you would like to promote on ssrn, paper statistics, related ejournals, engineering & applied sciences education ejournal.

Subscribe to this fee journal for more curated articles on this topic

Quantitative research in education : Background information

• Background information
• SAGE researchmethods SAGE Research Methods is a tool created to help researchers, faculty and students with their research projects. Users can explore methods concepts to help them design research projects, understand particular methods or identify a new method, conduct their research, and write up their findings. Since SAGE Research Methods focuses on methodology rather than disciplines, it can be used across the social sciences, health sciences, and other areas of research.

• The American freshman, national norms for ... From the Higher Education Research Institute, University of California, Los Angeles
• Education at a glance : OECD indicators
• Global education digest From UNESCO
• Next: Recent e-books >>
• Recent e-books
• Recent print books
• Connect to Stanford e-resources
• Related guides

• Last Updated: Aug 29, 2024 2:37 PM
• URL: https://guides.library.stanford.edu/quantitative_research_in_ed

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Doing Quantitative Research in Education

about quantitative research

Related Papers

Interdisciplinary Research: Collaborative Insights

Pongsakorn Limna , Dr. Sutithep Siripipattanakul , Tamonwan Sitthipon

In the past few decades, educational practices have changed drastically, particularly regarding how information and learning are delivered and processed. Education research frequently employs quantitative methods. Quantitative education research provides numerical data that can prove or disprove a theory, and administrators can easily share the quantitative findings with other academics and districts. While the study may be based on relative sample size, educators and researchers can extrapolate the results from quantitative data to predict outcomes for larger student populations and groups. Educational research has a long history of utilising measurement and statistical methods. Commonly quantitative methods encompass a variety of statistical tests and instruments. Educators and students could transition to the digital era and research-based knowledge, including quantitative research in advanced higher education, as the technology has advanced. The quantitative research methods in education emphasise basic group designs for research and evaluation, analytic methods for exploring relationships between categorical and continuous measures, and statistical analysis procedures for group design data. The essential is to evaluate quantitative analysis and provide the research process, sampling techniques, the advantages and disadvantages of quantitative research in the article.

DR FREDRICK ONASANYA

Quantitative research is a more consistent, coherent and data-resulted means of arriving in measuring of what people think from a statistical point of view. Quantitative research can gather prominent amount of data can easily be arranged and controlled into reports for analysis. In quantitative research, numerical data are gathered and mathematical based methods are used for analysis Quantitative research is basically about gathering numerical data to explicate a development especially that needs prompt answers using quantitative methods. It is used to measure mental outlook, beliefs, demeanors, and other defined variables that can be used to generalized results from prominent sample populations Quantitative research pertains to taxonomical of practical thorough check of social developments by the way of statistical, mathematical or computational proficiencies. Quantitative research data are gathered by surveys, audits, purchase points and so on. Quantitative research is quantifiable data to develop the truth or facts and reveal practices in research

Sandra Mathison

Trevor Male , David Needham

This book will help you to plan, design and conduct quality research within the specific context of education and educational studies. An impressive cast of contributors discuss the reality of conducting research in different educational settings and provide practical advice for both undergraduate and postgraduate students and early career researchers doing research in education. The book discusses key philosophical issues such as understanding research paradigms, ethics and selecting appropriate methodologies but remains grounded in the practical experience of the researcher. It has comprehensive coverage of the whole research process from start to finish, is easy to navigate and helps develop key skills such as: •Time management •Creating good research questions and hypotheses •Constructing the literature review •Structuring a project •Writing a proposal •Managing data •Analysing data •Writing for specific audiences Packed full of learning features and showcasing a wide range of voices and opinions this book is an ideal guide for anyone conducting research in education or educational studies.

andrea azures

• Identify the interest • Study the interest • Identify possible interesting dimension or problem in the area of interest • Formulate the initial research question in response to the identified dimension or problem in the area of interest • Make an initial research to gain further understanding of the initial research question • Incorporate new findings to the initial research question • Assess whether the research question remains relevant to the area of interest by studying the current state of literature All research inquiry starts from a simple identification of an area on which someone is interested in. A student researcher could be interested in the current state of information and communication technology, educational system, career tracks, use of social media, effects of personal issues on academic performance and impact of social background on education opportunities to name a few. Interests are either developed through life path, innate (such as inclination to music and other performing arts) or ideologically acquired. The identification of interest as the first stage of research inquiry is only reasonable since demands of the process requires undivided commitment and human nature has it that we struggle doing things when it is outside our area of interest.

Oroiyo K Peter

Rachel Irish Kozicki

Research Journal

Dr. Odera C O N Z A L K Amos Ouma

This study explores the possible suitability and efficacy of utilizing a quantitative research approach within the realm of educational psychology. While quantitative research methods have gained recognition across various disciplines, their appropriateness in the intricate and ever-evolving context of educational psychology necessitates thorough examination. This paper delves into the fundamental characteristics of quantitative research, underscoring its ability to furnish empirical evidence and conduct statistical analyses. Furthermore, the research probes into the advantages and challenges associated with the application of quantitative methods in educational psychology research, taking into account considerations such as sample size, generalizability, and the intricate nature of the psychological phenomena under scrutiny. Through a critical review of existing literature and empirical studies, the aim of this research is to provide insights into the feasibility of employing quantitative approaches, addressing reservations linked to the intricate and multifaceted nature of psychological phenomena within educational settings. The outcomes contribute to the ongoing conversation surrounding research methodologies in educational psychology, presenting scholars and practitioners with a nuanced viewpoint on the potential merits and constraints of embracing a quantitative research approach in this dynamic field.

SObia Mumtaz

Sobia Ahmed

It is about "educational research' and its types

mark vince agacite

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Munyaradzi Moyo

ASHRAFALSADAT HAKIM

Flanny Alamparambil

Samson Chukwuedo

Darrell M Hull , Robin Henson , Cynthia S. Williams

Ruddy Lesmana

Ridwan Osman

IJECA (International Journal of Education and Curriculum Application)

erita rahmaniar

Katrin Niglas

A.P.H. PUBLISHING CORPORATION

BinVa Chang

Nguyen Tran

Monika Jakubicz

amelya herda losari

Research on Humanities and Social Sciences

Brian Mumba

Adesoji Oni

International Journal of Research

Tahani Bsharat

zanariah hamid

Proceedings of INTED2015 Conference

Antonio Marzano , Rosa Vegliante , Marta De Angelis

Research, critical thinkers, science philosophy

H. Afandi, Afandi

Kezang sherab

BRIJENDRA GAUTAM

Abla BENBELLAL

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Studies in Engineering Education

• Download PDF (English) XML (English)
• Alt. Display

New Epistemological Perspectives on Quantitative Methods: An Example Using Topological Data Analysis

• Allison Godwin
• Brianna Benedict
• Jacqueline Rohde
• Aaron Thielmeyer
• Heather Perkins
• Justin Major
• Herman Clements
• Zhihui Chen

Background: Education researchers use quantitative methodologies to examine generalizable correlational trends or causal mechanisms in phenomena or behaviors. These methodologies stem from (post)positivist epistemologies and often rely on statistical methods that use the means of groups or categories to determine significant results. The results can often essentialize findings to all members of a group as truth knowable within some quantifiable error. Additionally, the attitudes and beliefs of the majority (i.e., in engineering, White cis men) often dominate conclusions drawn and underemphasizes responses from minoritized individuals. In recent years, engineering education research has pursued more epistemologically and methodologically diverse perspectives. However, quantitative methodologies remain relatively fixed in their fundamental epistemological framings, goals, and practices.

Purpose: In this theory paper, we discuss the epistemic groundings of traditional quantitative methods and describe an opportunity for new quantitative methods that expand the possible ways of framing and conducting quantitative research—person-centered analyses. This article invites readers to re-examine quantitative research methods.

Scope: This article discusses the challenges and opportunities of novel quantitative methods in engineering education, particularly in the limited epistemic framings associated with traditional statistical methods. The affordances of person-centered analyses within different epistemological paradigms and research methods are considered. Then, we provide an example of a person-centered method, topological data analysis (TDA), to illustrate the unique insights that can be gained from person-centered analyses. TDA is a statistical method that maps the underlying structure of highly dimensional data.

Discussion/Conclusions: This article advances the discussion of quantitative methodologies and methods in engineering education research to offer new epistemological approaches. Considering the foundational epistemic framings of quantitative research can expand the kinds of questions that can be asked and answered. These new approaches offer ways to conduct more interpretive and inclusive quantitative research.

• Epistemology
• Quantitative Methods
• Topological Data Analysis
• Person-Centered Analysis
• Latent Diversity

Introduction

Broadly, the purpose of research is to develop new knowledge or insight regarding a specific topic. As such, researchers and research communities must reflect on how they theorize and frame knowledge (i.e., their epistemologies and methodologies) and their processes to build that knowledge (i.e., their methods). This reflection not only facilitates alignment between research questions, theory, methodology, and methods but also can identify new opportunities for expanding the kinds of questions that can be asked and approaches to conducting research. In this theory paper, we explore emerging epistemic possibilities for quantitative research in the context of engineering education, particularly regarding person-centered analyses. These possibilities may offer ways to conduct more interpretive and inclusive quantitative research.

Engineering education research is practiced within a community that is shaped by the very engineering education systems being studied ( Kant & Kerr, 2019 ). Two major discourses in engineering education research methodologies have emerged from this history: rigor and methodological diversity (Beddoes, 2014). Rigor discourse historically focused on legitimating engineering education as an emerging research field. This discourse has resulted in a history of engineering education research that has emphasized objective and generalizable research methods ( Jesiek et al., 2009 ; Riley, 2017 ). Similarly, this discourse has been critiqued as enforcing limited epistemic framings of what counts as high-quality engineering knowledge and perpetuating inequity (Beddoes, 2014; Riley, 2017 ). More recently, methodological diversity discourse has created calls for and value of varied research approaches, particularly in qualitative research methodologies ( Douglas et al., 2010 ). Researchers have faced challenges with qualitative methods in their inculcation into engineering education research due to boundary spanning between engineering and social science ( Douglas et al., 2010 ). However, in recent years, engineering education has seen a surge in published qualitative papers with methodological diversity ( Walther et al., 2017 ). There have been dedicated conversations to clarifying methodological rigor ( Streveler et al., 2006 ), epistemic foundations (Baille & Douglas, 2010; Douglas, et al., 2010 ), and a holistic framework for qualitative inquiry in engineering education ( Walther et al., 2013 , 2015 , 2017 ). However, there has been little reflection on the epistemic norms of quantitative research. Targeting this reflection towards quantitative studies can situate current scholarship in engineering education as well as identify new possibilities that move beyond research methods aligned with a postpositivist epistemology (i.e., truth is knowable within some margin of error) that may be currently overlooked due to norms in the field ( Baillie & Douglas, 2014 ; Koro-Ljungberg & Douglas, 2008 ).

The purpose of this paper is to outline a discussion that invites readers to re-examine quantitative research methods and provides reflections on how an emerging set of quantitative methods—person-centered analyses (PCA)—can expand how we frame research in engineering education. Approaches that employ PCA treat the individual as a unique, holistic entity and work to maintain their whole response in the analysis, as opposed to traditional variable-centered approaches. We also provide an example of a person-centered analysis in engineering education to illustrate the possibilities of this approach. This paper does not provide an exhaustive review of all possible ways that quantitative research can be reconsidered beyond the epistemic norms of (post)positivism. 1 We use a research example to support the arguments made rather than present this example as a set of research findings or specific implications. Instead, we outline a gap in current methodological approaches to quantitative research and invite dialogue around embedded assumptions and norms within quantitative research.

Epistemologies in Social Science and Educational Research

Epistemology refers to beliefs about knowledge and how knowledge is constructed. It is one part of the philosophical assumptions that influences which methodologies and methods researchers consider appropriate ( Crotty, 1998 ; Lather, 2006 ). All aspects of the research process are informed by one’s epistemology, from embedded assumptions about what is known to the development of theories, research questions, and study designs ( Pallas, 2001 ; Collins, 1990 ). Upon the dissemination of findings, epistemologies also influence how research is interpreted and understood within a research community ( Pallas, 2001 ). In social science research, common terms have been developed to describe general categories of epistemologies. We describe three of these categories in this paper: (post)positivism, constructivism, and critical theory. We do not present these categories to continue the “Paradigm Wars” between quantitative and qualitative research as incompatible research approaches (see Bryman, 2008 ). Instead, we present the categories to provide context to the proposed discussion of quantitative methods and non-positivist approaches.

Postpositivism refers to a set of beliefs characterized by the assumption that reality can be known within some degree of certainty. Historically, postpositivism emerged as a response to positivism, an epistemology that was popular in early social science work ( Reed, 2010 ). Positivism takes a narrow view on knowledge production, focusing only on what can be measured and observed, with a strict focus on causality and the separation between knowledge and observer. Postpositivism allows for the role of human perspective and error, but still maintains a commitment to objective measurement and observation. Researchers leveraging a postpositivist perspective are often concerned with determining averages and trends in the dataset, attempting to minimize or control variation from these trends, and generalizing results to a larger population. Quality or validity is traditionally focused on measurement, generalization, and controlling variables to reduce bias ( Hammersley, 2008 ). While quantitative research is not a monolith, few studies have taken epistemological framings different from positivism or postpositivism ( Bryman, 2008 ).

In contrast, constructivism is often concerned with how an individual develops a socially negotiated and personal understanding of reality ( Baillie & Douglas, 2014 ). This understanding is varied for each individual, leading the researcher to study complexity and shared reality. Research leveraging constructivism recognizes individuals’ perspectives and the constellation of factors that may shape their lived experiences. It also acknowledges that research is a co-production between the researcher and participant(s). Thus, constructivism focuses on the subjective experience and its value for knowledge production.

Similarly, critical approaches emphasize the subjective reality of lived experiences to reveal power and oppression within social contexts with aims for social transformation (i.e., move away from (re)producing knowledge laden with inequity). Critical paradigms include feminist scholarship, Critical Race Theory, and disability studies or Crip Theory, among many others ( Lather, 2006 ). Critical epistemologies acknowledge that conceptions of knowledge are not value-neutral and that marginalized forms of knowledge must be valued and studied. This epistemological approach opposes how postpositivism imposes structural laws and theories that do not fit marginalized individuals or groups and posits that constructivism does not adequately address needed action against oppressive social structures.

Even though epistemologies are not tied to specific research methods, the affordances and foci of these common epistemological paradigms have resulted in historically bifurcated research approaches, where quantitative methods are typically associated with (post)positivism and qualitative methods are typically associated with constructivist, critical, or other non-positivist epistemologies ( Tuli, 2010 ). For instance, education researchers often use quantitative methodologies to study generalizable correlational trends or causal mechanisms. They typically rely on traditional statistics that use the means of groups (e.g., engineers versus non-engineers or women versus men) to determine statistically significant differences between groups or average effects of a variable on an outcome (i.e., variable-centered approaches). Research findings typically report means, line or bar graphs, p -levels, or Bayes factors. These methodologies often result in essentializing results of analyses to all members of a group as truth (a [post]positivist approach) and perpetuate a problematic dichotomy of identity.

As an alternative to such essentializing approaches, this theory paper focuses on the links between novel quantitative research methods in person-centered analyses and non-positivist epistemologies. However, we acknowledge that epistemology informs other components of the research process besides methodology, such as theory and dissemination. Douglas, Koro-Ljungberg, and Borrego ( 2010 ) argued against approaching theory, method, and epistemology separately or decontextualizing the framing of research (p. 255). Thus, despite a focus on methods of analysis, this work also demonstrates the potential need for alternatives to traditional conceptions of quantitative research that are reformulated from the epistemic foundations.

Epistemic Standpoint of Research Team

We are a team of researchers engaged in mixed-methods research focused on identity and diversity in engineering education. Some of us specialize more deeply in quantitative or qualitative paradigms, but together we recognize the value in each paradigm to answer particular kinds of questions, and an added richness in combining research approaches. As such, we approach our research and this discussion from a pragmatic epistemology. Pragmatism emerged in the late 19th century ( Maxcy, 2003 ), and is a set of philosophical tools rather than solely a philosophical standpoint ( Biesta, 2010 ), which focus on research choices that result in anticipated or desired outcomes ( Tashakkori & Teddlie, 2008 ). Pragmatism holds that knowledge is individual and socially constructed; nevertheless, it also posits that much of this knowledge is socially shared and research can begin to examine these shared realities ( Morgan, 2014 ). Pragmatism has been used recently in social science as the epistemology guiding mixed and multiple methods ( Creswell & Clark, 2011 ; Johnson & Onwegbuzie, 2004 ) as it “rejects traditional philosophical dualism of objectivity and subjectivity” ( Kaushik & Walsh, 2019, p. 4 ). With a focus on meaningful research that has utility for action for making purposeful difference in practice, pragmatism is also consistent with action for social justice ( Morgan, 2014 ).

One of the challenges in mixed methods research is synthesizing research findings from qualitative or quantitative paradigms. In this process, we have begun to engage in newer quantitative methods that provide additional nuance and the ability to preserve individuals’ responses within the data. We have found these practices both demanding and rewarding. From this standpoint, we open discussion of considering research questions and approaches in the quantitative paradigm from non-positivist epistemologies.

Traditional Methodological Approaches in Quantitative Research

Stemming out of (post)positivism, most quantitative methodologies emphasize objectivity, replicability, and causality. Most quantitative studies in social science research were designed to address research questions using variable-centric methods. Variable-centered approaches (i.e., correlations, regressions, factor analysis, and structural equation models) are appropriate for addressing inquiries concerned with “how variables, observed or latent, relate to each other” ( Wang et al., 2013, p. 350 ) and generate outcomes based on an averaged set of parameters. In engineering education, the study population is often cisgender, White, male, upper-middle-class, able-bodied, continuing generation, and heterosexual ( Pawley, 2017 ). Historically, this population has been accepted as the default in engineering education research, resulting in findings and implications for practice that are often decontextualized from the social reality of individuals’ backgrounds and experiences. By conducting research with demographic homogeneity, the understanding of phenomena for individuals who are not the default is limited and warrants a need for researchers to justify their rationale for generating theory based on individuals with a dominant presence in engineering ( Slaton & Pawley, 2018 ; Pawley, 2017 ). For our research, particularly in focusing on diversity in engineering education, traditional quantitative methods have provided useful answers to important questions; however, they also present challenges in adequately representing all students.

To illustrate these challenges and highlight how variable-centric statistical methods can reinforce dominant norms, we provide an example related to research on gender segregation in science, technology, engineering, and math (STEM) professions. This example, drawing on common, and well-known phenomena, illustrates the ability of variable-centered approaches to ask nuanced questions while still essentializing the findings of an individual to a group. Thus, even as this approach provides valuable and important research findings, it also shows the ways in which even carefully constructed quantitative studies that meet standards of quality still align with (post)positivism.

The phenomenon in question emerges from studies comparing the future goals and outcome expectations of men and women that find women are more interested in person-oriented or altruistic roles. Engineering, as a male-dominated and thing-oriented field, is not consistent with this characterization (e.g., Ngambeki et al., 2011; Su & Rounds, 2015 ). Therefore, studies conclude that misaligned orientations are a key reason for women’s lack of representation in engineering ( Bairaktarova & Pilotte, 2020 ; Cejka & Eagly, 1999 ; Miller, Eagly, & Linn, 2015 ; National Academy of Engineering, 2008 ; Su & Rounds, 2015 ). These studies give some important general characterization of how engineering culture is gendered, and their findings are consistent across repeated studies and cultural contexts.

However, the limits of this variable-centered approach emerge when we explore the question from an alternate direction. For example, a study of women in engineering disciplines with above-average (i.e., biomedical, industrial, etc.) and below-average female enrollment (i.e., mechanical, electrical, etc.) indicate different patterns, with women in the below-average female enrollment group having less interest in stereotypically feminine outcome expectations ( Verdín et al., 2018 ). This study points to the reality that not all women follow general findings about interests and goals. Thus, even with careful explanation by researchers that quantitative results are true for most women, the nuance of individual differences is not captured by these approaches. Indeed, most social science studies focus on variation between groups and make conclusions based on statistically significantly different average effects ( Fanelli, 2010 ). However, differences between groups, even with so-called large effect sizes, can occur even when two groups are much more similar than different ( Hanel et al., 2019 ). Additionally, the attitudes and beliefs of the majority (i.e., in engineering, White men) dominate conclusions drawn and underemphasizes responses from minoritized individuals.

Slaton and Pawley ( 2018 ) argued that it is not sufficient for scholars to justify the exclusion of individuals based on traditional quantitative norms of sampling and large-n studies. Instead, engineering education must create and learn new methods that empower researchers to learn from small numbers. The number of participants or lack thereof in a study is not an excuse to generate theory based on homogenous populations and perpetuate limited standards of representation ( Pawley, 2018 ; Slaton & Pawley, 2018 ). There is a need for epistemic shifts to advance our understanding and challenge what counts as adequately representative in engineering education research ( Slaton & Pawley, 2018 ). Otherwise, engineering education researchers reinforce systemic inequities through our logic and methods, unconsciously or otherwise.

Pawley and colleagues have offered small-n qualitative studies as a valuable solution to large quantitative studies’ important criticisms. The purpose of these studies is to capture and highlight the experiences of individuals often minoritized in engineering and sometimes (but not always) identify patterns across participants ( Merriam & Tisdell, 2016 ). These studies also can leverage the complexity and power of intersectionality studies to reveal inequities in engineering education. Through the thick description of individuals’ experiences, these qualitative studies lead to a richer and more nuanced understanding of phenomena otherwise left ignored or masked in studies that prioritize large-n studies. However, the level of detail often precludes the breadth of participants seen in quantitative studies. While this focus is a feature of qualitative research rather than a problem, it does constrain the kinds of questions that qualitative research can and cannot answer. There is still a need to conduct quantitative studies that are generalizable, are inclusive, and do not essentialize results to a single average or group.

As a result, in addition to qualitative studies that provide valuable insight into individual lived experiences, new quantitative methodological approaches have emerged in the social sciences that also begin to address the critiques raised about (post)positivist quantitative paradigms. These new approaches can introduce epistemologically novel ways to approach quantitative research questions that fill a gap not addressed by qualitative, mixed methods, or traditional quantitative research alone. New quantitative approaches do not need to replace traditional methods, but instead offer additional ways of understanding and querying a phenomenon. We describe some of these approaches below before focusing on person-centered analyses.

New Methodological Approaches in Quantitative Research

Multi-modal approaches.

Emerging scholarship in engineering education has begun to re-examine quantitative methods, particularly in using multi-modal approaches to understand cognition and emotion in authentic contexts. We provide a few but not exhaustive examples of these approaches. Villanueva, Di Stefano, Gelles, Vicioso Osoria, and Benson ( 2019 ) conducted a study with multi-modal approaches to data collection, including interviews and electrodermal activity sensor data, from 12 womxn students to study psychophysiological responses to academic mentoring. This approach treated inequity issues as core to participants’ experiences rather than moderating quantitative analysis variables. The quantitative data were analyzed using MANOVA and representative response profiles before synthesizing the findings with qualitative data. This approach allowed for both conscious (interview responses) and unconscious (electrodermal activity sensor data) to be examined simultaneously. This multi-modal approach has also been applied to an experimental study of students’ emotional experiences during testing with electrodermal activity sensor data saliva testing during a practice exam ( Villanueva et al., 2019 ).

Other researchers have used similar multi-modal protocols to study design thinking. Gero and Milanovic ( 2020 ) proposed a framework for design thinking that involves design cognition, design physiology, and design neurocognition. Gero and Milanovic ( 2020 ) provided a detailed description of prior studies and various measurement methods for these dimensions (i.e., brain imaging, electrodermal activity, eye movements, protocol analysis, surveys, interviews, etc.). These measurements are combined to inform a larger understanding of these processes in contexts that are often studied separately (i.e., affect and emotion or cognition). These data are examined using traditional statistical techniques but also using novel approaches like linkography to examine relationships between design moves ( Goldschmidt, 2014 ), Markov modeling to examine probable transitions in design reasoning or processes ( Gero & Peng 2009 ; Kan & Gero 2010 ), and correspondence analysis to describe the degree and extent of relationships between categories ( Greenacre & Hastie, 1987 ).

These multi-modal approaches offer new ways to examine complex phenomena and provide ways to integrate the strengths of quantitative and qualitative data. Two of the biggest challenges of multi-modal approaches are the effort (i.e., time, cost, etc.) associated with data collection and synthesis of heterogeneous data. As such, these studies are often conducted with small sample sizes and most studies rely on traditional statistical methods such as the correlation of quantitative results (where qualitative data streams are coded into quantitative frequencies or patterns; Gero & Milanovic, 2020 ). These approaches have strength in examining the underlying mechanisms in rich and nuanced ways.

The novelty of these methods is predominantly in data collection tools and integration of results of these tools to generate new insights and questions in educational research. Fewer studies have deeply examined the epistemic and statistical methods of solely quantitative research for the same goal. We believe that person-centered statistical analyses offer ways to reimagine quantitative educational research using more common numeric data collection approaches such as surveys and observations. This approach re-imagines how student responses are characterized and understood in context through statistical methods.

Person-Centered Approaches

Person-centered approaches sit in contrast to traditional variable-centric approaches and assume that the population under study is heterogeneous. The results of such studies focus on preserving the variation in individual’s responses resulting in authentic groupings of individuals, as opposed to imposing superficial characterizations of groups ( Laursen & Hoff, 2006 ; Morin et al., 2018 ). In a variable-centered approach, individual differences are treated as outliers from a mean value, or even erased, due to low sample size, a decision that disproportionately impacts minoritized individuals. While these approaches are not a panacea for all challenges with quantitative methods, especially concerning measurement and fairness ( Douglas & Purzer, 2015 ), they do open new avenues for quantitative inquiry beyond (post)positivist epistemologies. In doing so, they provide new avenues of research and potentially more equitable approaches to quantitative methodologies.

Person-centered analyses are a relatively young methodological approach arising alongside the increased availability of computing resources ( Laursen & Hoff, 2006 ). As with all innovations, they occupy an ill-defined space with concepts that both overlap and differ in key ways. Consequently, a call for increased use of person-centered analyses requires some discussion for readers to navigate this confusing morass of shared terminology. A central area of overlap and potential confusion that new researchers will likely encounter is between the terms person-centered analysis and data-driven approach . For instance, discussions of specific techniques (e.g., cluster analysis or mixture modeling) occur in both spheres, and both approaches rely on modern computational power and sprawling datasets (also called Big Data; Lazer et al., 2009 ; Gillborn, Warmington, & Demack, 2018 ).

A data-driven approach rejects traditional formulations of the scientific method that begin and end with theory developments. Instead, it lets the data “tell their own story,” independent of researchers’ assumptions and preconceptions, and then reconcile findings and theories once the analysis is complete ( Qiu et al., 2018 ). Data-driven approaches thus utilize bottom-up frameworks centered on relationships instead of top-down frameworks driven by explanations and causality ( Qiu et al., 2018 ). It is not surprising that data-driven approaches have increased in popularity as more and more data is created as part of our daily lives ( Gero & Milanovic, 2020 ; Villanueva, Di Stefano, et al., 2019 ), which also lessens the need for experiments that control for confounds and the influence of covariates. Instead, data-driven approaches accommodate for the lack of control in data generation and collection through sheer numbers and advanced computational power ( Lazer et al., 2009 ).

Person-centered analyses, in contrast, challenge assumptions about group homogeneity, variable effects, and the generalizability of conventional inferential analyses (e.g., linear regression; Eye & Wiedermann, 2015 ). The mean of a dataset is not always the best way to describe or represent a population—not only can it be distorted by a small number of outliers (e.g., the average net worth in the United States where wealth is concentrated among a relatively small group of individuals), but it may also represent an impossible or otherwise inaccurate value (e.g., the average of 2.5 children per American household; Eye & Wiedermann, 2015 ). Similarly, variable-centered analyses estimate the effects of individual variables by controlling for, or removing the effects of, other variables in the model, although this separation cannot occur in real life (e.g., attempting to attribute an outcome to racism or socioeconomic inequality when these experiences exist in a state of mutual or spiraling causality; McCall, 2002 ). Thus, person-centered analyses utilize the identification of underlying groups (i.e., latent profile/class analysis; Jack et al., 2018 ), hidden clusters or structures (i.e., cluster analysis, Topological Data Analysis, Principal Component Analysis, Self-Organizing Maps, and Multidimensional Scaling; Chazal & Michel, 2017 ; Everitt et al., 2011 ), or mixture components (i.e., mixture modeling; Jack et al., 2018 ) when examining the relationships of individual response patterns within the data. This approach preserves heterogeneity instead of masking or minimizing it. In other words, person-centered analyses adopt a data-driven approach and use this approach to identify subpopulations not readily visible to the naked eye and use these subpopulations to improve the clarity and accuracy of predictions and explanations. Although person-centered analyses incorporate data-driven approaches, not all data-driven approaches are person-centered; many other exploratory and Big Data techniques, including Classification and Regression Trees (CART; Breiman et al., 1984 ), still foster variable-centered approaches that aim to reconcile variables with predefined (and thus potentially biased or inaccurate) categories. We provide a description, but not an exhaustive list, of these different analyses in Table 1 .

Examples of person-centered and data-driven analyses.

Person-centered analyses are not necessarily associated with a particular epistemological paradigm. The techniques associated with person-centered analysis may be used to make (post)positivist claims, such as clustering engineering students based on learning orientations and study strategies, then evaluating the study success of each cluster (e.g., GPA; Tynjälä et al., 2005 ). However, a benefit of person-centered analyses is that it disrupts some of the assumptions typically associated with (post)positive, variable-centered approaches. Below, we provide an example of one kind of person-centered analysis that takes a non-positivist viewpoint.

An Example of Person-Centered Analysis from Engineering Education

We use a research project that employed Topological Data Analysis (TDA) to demonstrate the kinds of knowledge afforded by a specific type of person-centered analysis. This empirical example was a part of a study titled, CAREER: Actualizing Latent Diversity: Building Innovation through Engineering Students’ Identity Development (NSF Grant No. 1554057), focused on understanding first-year engineering students’ latent diversity through a national survey and longitudinal narrative interviews. Latent diversity refers to students’ underlying attitudes, mindsets, and beliefs that are not readily visible in engineering classrooms yet have the potential to contribute to innovation in engineering solutions ( Godwin, 2017 ). This latent diversity is often undervalued or unacknowledged in engineering education with an emphasis on particular ways of being, thinking, and knowing aligned with rigid norms and expectations centered in engineering’s historic lack of diversity ( Benedict et al., 2018 ; Danielak et al., 2014 ; Foor et al., 2007 ). We hypothesized that these cultural norms force students to conform to these expectations, thus reducing capacity for innovation and creating identity conflict that results in a lack of belonging and, ultimately, attrition. The goal of this project was to characterize latent diversity in incoming students to understand different subpopulations in engineering and how their experiences within the dominant culture of engineering affected their development as engineers to provide more inclusive ways of educating engineering students. The Purdue University Internal Review Board approved this study under protocol number 1508016383.

This study was executed in three consecutive phases: 1) instrument development; 2) characterization of latent diversity from a nationally representative sample; 3) longitudinal narrative interviews. For more details about the survey development, see Godwin et al. ( 2018 ). We used TDA to identify six data progressions among engineering students’ attitudinal profiles. These groups were later used to identify and recruit students to participate in bi-annual longitudinal narrative interviews designed to capture student identity trajectories. Our example focuses on the second phase of research focused on characterizing latent diversity. It demonstrates the type of person-centered characterizations that can be conducted in engineering education research.

Data Sources

We recruited U.S. institutions to participate based on a stratified sample of small (7,750 or fewer), medium (7,751 to 23,050), and large (23,051 or more) institutions in the United States ( Godwin et al., 2018 ). We chose this sampling approach to ensure there was equal representation among the institution types (i.e., small, medium, and large), instead of an overrepresentation of large, public engineering institutions. The survey instruments were administered in common first-year engineering courses via paper-and-pencil format at 32 ABET-accredited institutions during the Fall 2017 semester. This timing captured students’ incoming latent diversity before being influenced by the process and culture of engineering education and captured students interested in a wide range of engineering disciplines. The data were digitized and cleaned by removing indiscriminate responses resulting in 3,711 valid responses.

Study Participants

Students indicated their self-reported demographics at the end of the survey instrument. These measures were designed to include a wide range of identities and included a multi-select question ( Fernandez et al., 2016 ). The majority of participants identified as men ( n = 2150), with other students identifying as a woman ( n = 720), transgender ( n = 70), agender ( n = 17), or genderqueer ( n =14). Some students used the self-identify write-in option to indicate a gender not listed ( n =75), and some did not respond ( n = 782). The majority of the students identified as White ( n = 2089). The remaining students identified as Asian ( n = 380), Latino/a or Hispanic ( n = 347), African American/Black ( n = 209), Middle Eastern or Native African ( n = 65), Pacific Islander or Native Hawaiian ( n = 34), Native American or Alaska Native ( n = 49), used the self-identify write-in option to indicate another race/ethnicity not listed ( n = 72), or did not respond ( n = 793). We note that a large portion of students did not report demographics; often, students do not complete surveys due to fatigue, lack of time, or loss of interest. The survey was extensive, and some students dropped off in responding at the end of the survey. These reasons may account for students who did not report a gender identity or race/ethnicity, which were asked at the end of the survey. Students were allowed to select all that applied regarding their gender and race/ethnicity with which they identified. For example, out of the 2,089 (56%) students who identified as White, 291 (14%) of them also identified with another race/ethnicity. Additionally, students were asked to report their home ZIP code. These ZIP codes were plotted on the U.S. map to provide a geographic distribution of the overall first-year engineering student sample in the dataset, Figure 1 .

The map represents students’ self-reported home Zip Codes from a national survey. Each dot may represent more than one student. This image was generated in R ( R Core Team, 2018 ) using the ggplot2 package ( Wickham, 2009 ).

An Overview of Topological Data Analysis

Generally, the field of topology refers to an area of mathematics, persistent homology, that relies on the study of shapes and structures to make sense of the world. However, more recently, topological data analysis (TDA) has emerged as a person-centered analysis that allows quantitative researchers to take an exploratory approach to draw insights from complex high-dimensional datasets (see Wasserman, 2018 for a detailed review). These shapes or structures allow the researcher to identify subgroups that may not have been considered when using traditional pairwise comparative methods that rely on researchers’ predetermination of groups ( Lum et al., 2013 ). TDA differs from other person-centered approaches (i.e., Principal Component Analysis, multidimensional scaling, and clustering methods) based on its capabilities to capture geometric patterns that may have been ignored by other statistical methods ( Lum et al., 2013 ). Instead, TDA provides a mapping of the data into a two-dimensional representation while maintaining the complex structure of the data. The resulting map is constructed from the shape and proximity of the data to itself, rather than a reference or seed point. As such, the mapping is not influenced by the measurement scale or random generation of multiple possible models. Topological methods are capable of handling the data by compressing the infinite data points into a finite, manageable network of nodes ( Lum et al., 2013 ).

TDA has proven useful for wide-ranging applications in fields such as natural science, social science, and other computational fields. Studies have identified subgroups within breast cancer patients for targeted therapy ( Lum et al., 2013 ), real-time air detection of bacterial agents ( McGuirl et al., 2020 ), stratification of basketball positions above the traditional five characterizations of players ( Lum et al., 2013 ), and player and team performance of football data ( Perdomo Meza, 2015 ). Despite such broad and useful applications, TDA has been underutilized among engineering education and social science research except for two studies. Of the two studies, the first focused on distinguishing between normative and non-normative attitudinal profiles among incoming engineering students at four institutions ( n = 2,916; Benson et al., 2017 ). In that study, TDA was useful for identifying groupings of students based on latent constructs rather than demographic variables. This study also provided evidence that some students’ attitudes differ from the normative group, especially in terms of feeling recognized as an engineer ( Benson et al., 2017 ). The second study is the example used below. The specific results from this study have been published previously (see Godwin et al., 2019 for more detailed discussions of the specific study and TDA analysis); here, we focus on highlighting the ways in which the study illustrates the contributions afforded by person-centered approaches.

Analysis Steps in Topological Data Analysis

The process for conducting TDA for the example provided, including the sensitivity of these parameters is discussed in detail in our previous work ( Godwin et al., 2019 ), but we highlight key details here for context. Before conducting TDA, several considerations must be made to minimize error and bias. First, methods to estimate missing data must be used to address potential errors when computing distance between points within the metric space ( Lum et al., 2013 ; Godwin et al., 2019 ). This specific consideration is especially important in social science research, where missing data are common. Next, if using latent variable measures, a typical practice in engineering education survey methods, a valid factor space must be created. This step involves verifying the study measurements through confirmatory factor analysis and generating factor scores based on the results of this factor analysis. Finally, the TDA algorithm parameters must be tuned to detect the underlying structure of the data. These parameters include the filtering method, clustering method, number of filter slices (n), amount of overlap of individuals, and cut height.

Interpreting TDA Maps

TDA generates a rich graphical representation of the data structure that consists of nodes and edges. The nodes represent multiple students, and the edges represent the overlap of student membership with other nodes. The size of the node indicates the number of students present in that area of the map. The color indicates the density of student responses within the node. Density indicates how similar student response patterns are across all dimensions. The resulting map is descriptive rather than inferential in group determination and differences between groups. It is particularly important to emphasize how TDA results are not a defined group but a representation of the structure of interconnectedness and difference within the data ( Laubenbacher, 2019 ). This approach contrasts with other statistical methods that rely on specifying a probability at which a group is considered different or forcing data into deterministic groups (as in clustering and latent profile analysis. This approach allows for more nuanced relationships and patterns to be identified between groups and individuals while also preserving the individual’s response within the study. The resulting map shows data progressions, which are groupings of students and their relation to one another—the groupings were determined visually by the researchers from this descriptive method rather than from the method’s results.

We created a 17-dimensional factor space based on the items used to measure students’ attitudes, mindsets, and beliefs concerning their STEM role identities (physics, mathematics, and engineering), motivation beliefs (control and autonomous regulation), epistemic beliefs, sense of belonging (engineering and engineering classroom), and two personality dimensions (neuroticism and conscientiousness). The results of TDA indicate six data progressions (i.e., A–F) for the characterization of latent diversity (Figure 2 ).

TDA map generated from the analyses, including groupings based on the distribution of the network of nodes. The colors shown in the map above represent the density of the map. The blue nodes denote a population of approximately 200 students, while the red nodes denote a smaller population of approximately three to five students. Our final parameters included a k-nearest neighbors filtering method, a single-linkage hierarchical agglomerative clustering method, 35 filter slices (n), a 50% overlap in data, and a 4.0 cut height (ɛ).

The resulting data progressions show descriptive differences across various factors, as shown in Figure 3 . We provide these descriptive differences to illustrate the utility of this approach in producing data progressions that indicate unique student groupings and relationships within the dataset. We avoid conducting traditional variable-centered comparisons that reduce these data progressions to finite groups or clusters to avoid the knowledge claims we have critiqued in this paper. The discussion that follows provides the description of these data progressions as evidence for pragmatic validation or the utility of this method to reveal structure in complex, noisy data while still maintaining individual student responses ( Walther et al., 2013 ).

Spider plot of average student responses on factors within TDA. Measures include disciplinary role identity constructs: Math_Int = mathematics interest; Math_PC = mathematics performance/competence beliefs; Math_Rec = mathematics recognition; Phys_Int = physics interest; Phys_PC = physics performance/competence beliefs; Phys_Rec = physics recognition; Eng_Int = engineering interest; Eng_PC = engineering performance/competence beliefs; Eng_Rec = engineering recognition. Two factors from the Big Five Personality measure were used: Ocean_NC = conscientiousness and Ocean_Neu = neuroticism. Belonging was measured in two contexts: Bel_Fac1 = in the engineering classroom and Bel_Fac2 = in engineering as a field. Students’ motivation was captured by Motiv_CR1 = controlled regulation for engaging in courses; Motiv_CR2 = controlled regulation for completing course requirements; and Motiv_AR2 = autonomous regulation for completing course requirements. Students’ epistemic beliefs (Epis_Fac4) captured the certainty of engineering knowledge (i.e., absolute to emergent).

First-year engineering students’ incoming attitudes and beliefs vary across the dimensions, but students also share similarities between the groups. Group A has the largest number of students ( n = 952) with moderately strong STEM role identities, motivation beliefs, epistemic beliefs, and a sense of belonging. In contrast, students in Group E ( n = 144.5, average partial membership because edges in Figure 2 are shared membership) shared moderately low beliefs about their STEM role identities and indicated low emotional stability. These qualities of Group E were similar to students identified in groups A, B ( n = 517), C ( n = 21), and D ( n = 27). Interestingly, students in Group F ( n = 51.5) had high emotional stability, STEM role identities, and a sense of belonging, but indicated low motivation beliefs (i.e., Controlled Regulation).

While additional similarities and differences can be drawn about each progression, such discussion is outside the scope of this paper. Rather, this paper focuses on the utility of person-centered approaches and how the results assert the assumptions of person-centered analysis. Thus, through our example, we wish to highlight how multiple subpopulations can exist among a sample and to explicitly draw attention to the power of taking an exploratory approach to data analysis, as opposed to methods that require defined hypotheses. By relying on the shape of the data, we were able to draw meaningful insights about the landscape of students’ attitudes, beliefs, and mindsets rather than binning students into groups based on demographic variables. Some data progressions show strong common patterns with small sample sizes (for example, Groups C and D). Many statistical techniques would ignore these groups in inferential testing because of this limitation. TDA allows these patterns to be detected and placed within the large dataset structure.

Implications of TDA Example

The TDA map (Figure 2 ) illustrates a wide variation among students’ attitudes, beliefs, and mindsets in engineering education. Students’ incoming latent diversity in U.S. engineering programs is not homogeneous. Additionally, results from this work often reveal small groups of student attitudes that would not emerge using variable-centered methods. This approach also allows new ways of framing research questions to understand general positions of students’ multidimensional attitudes, beliefs, and mindsets in relation to one another rather than forcing students into rigidly defined groupings based on probability. Importantly, this approach highlights how a one-size-fits-all approach to engineering education cannot adequately support the variation of students entering engineering programs with differing ways of seeing themselves in STEM. This variation includes students’ motivation to engage in courses and assignments, personalities, and beliefs about knowledge. Teaching all students in the same way or portraying a stereotype of the kind of person that becomes an engineer can communicate dominant norms that push students out of engineering ( Benedict et al., 2018 ; Cech, 2015 ). This finding indicates how non-positivist epistemologies help frame research questions aimed at understanding how students build their understanding and knowledge of the world. In answering these questions, engineering educators can create experiences and reflection opportunities that support the diversity of students in the classroom.

Comparison to Traditional Methods

To further illustrate the contributions of TDA specifically and person-centered analyses generally, we compared the TDA results to more traditional statistical methods. For example, we examined the demographic representation of students within each data progression by gender identity and race/ethnicity individually and, where possible based on sample sizes, at the intersection of race and gender (i.e., White women, Black women, Asian women, Latinas, White men, Black men, Asian men, and Latinos). We did not find any differences in representation across data progressions using a chi-square test with a Holm-Bonferroni correction for gender, race/ethnicity, and intersectional groups of gender and race/ethnicity at the alpha value of 0.1. In this comparison, we emphasize that these tests rely on traditional statistical tests and do not consider individual responses with small numbers, particularly non-binary students across racial/ethnic categories and Native Hawaiian, Alaska Native, Native American, or other Pacific Islander students within the dataset.

However, when examining the data by traditional demographic groups using a Kruskal-Wallis test with a follow-up Dunn’s test, we did find statistically significant differences across the majority of the 17 factors. For example, we found that students’ controlled regulation motivation for engaging in engineering courses (Mov_CR1) showed significant differences by intersectional gender and race/ethnicity (H(7) = 93.787, p < 001) with a small effect size (η 2 = 0.023; Cohen, 1988 ) as shown in Figure 4 . A post hoc Dunn’s test indicated that Black men and Latinos reported statistically significantly lower controlled regulation motivation ( p < 0.01) than all other groups and that Black women and Latinas reported statistically significantly higher scores than all-male groups ( p < 0.001).

Differences in controlled regulation for classroom engagement by intersectional gender and race/ethnicity groups. Groups with large enough samples for comparisons include: WW = White women, AW = Asian women, BW = Black women, LW = Latinas, WM = White men, AM = Asian men, BM = Black men, and LM = Latinos.

From these results, one might conclude that Black and Latinx groups show average differences (i.e., lower motivation from external sources) by gender and race/ethnicity. However, a focus on demographics as explanations for student outcomes treats minoritized groups as homogeneous and often implicitly suggests race or gender as a causal variable for differences rather than other structural issues ( Holland, 2008 ). Other analyses focused on investigating differences in latent constructs by demographic characteristics often bin together groups of minoritized students to satisfy sample size requirements (i.e., all underrepresented racial and ethnic groups in engineering). This practice assumes that the experiences of minoritized students are a monolith and ignores the context as to why certain norms and inequities exist in engineering education.

Our TDA results, in contrast, indicate that these conclusions, based on a traditional approach to understanding gender and racial/ethnic diversity within our sample, oversimplify students’ responses within the data. Black and Latinx men and women have a wide range of attitudes and are equally represented in the data progressions within our results. This person-centered analysis allows for individual student differences to exist in complex large datasets. Additionally, the person-centered analysis allows for students who do not meet the sample size requirements for traditional statistical comparisons to be included within data analysis. Even with a large social science sample greater than 3,000 responses, many intersectional groups with small numbers were excluded from the demographic analyses presented. A person-centered analysis allows for inclusive representation where data analysis and conclusion include all responses rather than only those with dominant group status. Finally, this approach allows the structure and connections within the data to be uncovered.

Our example illustrates how engineering education researchers might reframe research questions and approaches from non-positivist epistemologies. Engineering culture and structures have been constructed as raced, classed, and gendered, and negatively affect all students. Engineering culture emphasizes and perpetuates demographic normativity of Whiteness, masculinity, competition, and emphasis on technical solutions ( Akpanudo et al., 2017 ; Secules et al., 2018 ; Slaton, 2015 ; Uhlar & Secules, 2018 ).

Challenges and Opportunities for Person-Centered Analysis

Person-centered analysis can provide ways to ask research questions outside of the “to what extent” research questions or hypotheses often tested with quantitative research in (post)positivist paradigms. In our example, we examined the data structure with no a priori hypotheses about how gender, race/ethnicity, or other demographic factors might influence students’ incoming underlying attitudes, beliefs, and mindsets in engineering. TDA allowed us to find the emergent structure of relationships among student responses within the dataset and make generalized and descriptive conclusions about our results. This statistical approach provided ways to re-think the types of questions we asked of our data and the assumptions we brought to our analysis.

Additionally, these methods do not replace the need for qualitative, mixed methods, and multi-modal studies that have different purposes for generating knowledge. However, research methods focused on retaining the integrity of the individual within the dataset do provide opportunities to ask more complex and potentially novel research questions than the ones traditional quantitative methods can address. Person-centered analyses can help reveal relationships and patterns between large amounts of information by allowing discovery to be emergent. This approach aligns more closely with constructivist or even critical epistemologies. As discussed previously, many of our approaches to knowledge are implicitly biased, influenced by an epistemological racism and discrimination woven into the fabric of our social history ( Scheurich & Young, 1997 ). While it is necessary to address these biases and acknowledge the reality of research, traditional variable-centric methods are often framed as “objective” and researchers often do not interrogate the assumptions of statistical tests, prohibiting them from making these types of considerations. Person-centered analysis alleviates some of the systemic discrimination within our research paradigms by challenging or eliminating a priori knowledge necessary for quantitative research methods. More importantly, these new approaches provide new insight and knowledge to bolster our current understanding.

Critical Alternatives to Person-Centered Approaches

While person-centered analyses can address many systemic issues embedded within traditional quantitative research methods, there remain related problems that person-centered analyses still cannot solve. As an option for other research approaches, we discuss critical methodologies, which are approaches that do not distinguish between the methodologies/methods and epistemologies used. Instead, these approaches frame methods and epistemologies in critical studies as inextricably linked. These approaches often used person-centered analysis in conjunction with qualitative data and have specific tenants and framings that make them unique from general person-centered methods.

Critical quantitative methodological approaches are quantitative methodological approaches consistent with critical epistemologies. There are numerous books and excellent studies that give a complete discussion of these approaches (see McCall, 2002 ; Oakley, 1998 ; Sprague & Zimmerman, 1989 ; Sprague, 2005 ; and a special issue by Gillborn, 2018 ). Nevertheless, we still include basic descriptions of these methodologies to illustrate other methodological framings of quantitative inquiry that directly challenge, refute, or build upon (post)positivist approaches to research. There are many bodies of critical quantitative research; here, we focus on just two that are consistent with Feminist and Critical Race Theory: FemQuant and QuantCrit. These two bodies formed separately with FemQuant forming and developing much earlier than the other. Both bodies have similar underlying tenets that provide ways to frame and conduct quantitative research critically.

Feminist-specific or not, critical quantitative approaches build upon general ideas of the feminist paradigm or feminist ethics, assuming systemic power relations beyond gender rule all aspects of social life through the organization of institutions, structures, and practices ( Jagger, 2014 ). This organization of resources results in an unequal system of advantages and disadvantages ( Acker, 1990 ; Ray, 2019 ). The feminist paradigm requires that research and praxis be positioned to promote a more just and equitable society ( Collins & Bilge, 2016 ). In this approach, all methodologies—created and used by researchers who are also social participants—influence and can be influenced by the hierarchical social system in which research is situated ( Oakley, 1998 ). This framing contrasts (post)positivist epistemology, which situates context (including the positionality and influence of the researcher if this context is even acknowledged) as a weakness to the supposed objectivity of quantitative research ( Hundleby, 2012 ; Sprague & Zimmerman, 1989 ). Harding ( 2016 ) wrote that reflexive incorporation actually makes quantitative research more objective or strong. She and others emphasized that the doing of research is messy, unpure, and laden with power relations, and the acknowledgment of these dynamics is essential ( Harding, 2016 ; Hesse-Biber & Piatelli, 2012 ). Quantitative researchers need to explore, and make explicit, how their methodological use is complicit in that larger system of hierarchical power relations.

FemQuant and QuantCrit are based in these same basic epistemological framings but also advance their individual ethical positions to focus on race and racism (QuantCrit) and gender and sexism (FemQuant). Both approaches acknowledge the intersectional nature of multiple identities and different power relations associated with them. Still, each has developed from different historical and theoretical roots. QuantCrit maintains primary adherence to the first tenet of Critical Race Theory, that racism is a normal and ordinary component of daily life ( Delgado & Stefancic, 2012 ), and that other power relations such as gender and class are used to support a larger racist project ( Gillborn et al., 2018 ). FemQuant centers Feminist Theory with the incorporation of post-modern and post-feminist Intersectionality Theory ( Codiroli Mcmaster & Cook, 2019 ), a partnership that highlights the many ways in which gender inequality exists and is enacted through the unique interactions of inequality due to gender, race, class, sexuality, disability, and more ( Bowleg, 2008 ). While FemQuant and QuantCrit’s moral commitments and directions are different, their underlying reflexive methods and feminist philosophy are the same.

We present a very brief summary of these complex ideas here. In addition, we provide multiple brief engineering education-specific examples to situate our summary. Generally, the methodological and epistemological commitments of approaches can be summarized in six tenets ( Major, Godwin, & Kirn, 2021 ) adapted from prior work ( Bowleg, 2008 ; Gillborn et al., 2018 ; Hesse-Biber & Piatelli, 2012 ; Oakley, 1998 ; Sigle-Rushton, 2014 ; Sprague & Zimmerman, 1989 ):

• Naturality – Domination is a central component of society that is not natural but rather is socially constructed and supported through multiple dimensions of difference or categories that quantitative research cannot be absent from. For example, accepted government categories of race and ethnicity that are typically recognized and used in quantitative research, such as in engineering education, have changed over time according to changing U.S. and broader global political motivations, not for natural reasons ( Omi & Winant, 2014 ). Such motivations directly impact the ways in which racially diverse populations in engineering education are represented numerically.
• Neutrality – Numbers cannot be neutral , but are rather numerically constructed representations of domination based on locally or globally rectified meanings relating to differences in human bodies. As such, neutrality often parallels naturality in that what is deemed natural is often connected to political ideology ( Oakley, 1998 ). In a similar example to that of naturality, the gender identity of students, such as those in engineering education, is often assumed according to physical traits such as the existence of sexual organs, or according to social performances of gender that relate to name, hair length and color, and even symbolic expressions of femininity or masculinity ( Connell, 2009 ; Akpanudo et al., 2017 ). These considerations conflate sex and gender. Thus, like race/ethnicity, numerical representations of gender, and their relation to ones’ ability to be an engineer or participate in engineering education, are tied to non-neutral local or global beliefs about gender identity and gender performance.
• Intersectionality – Inequality exists beyond one’s social position. In addition, inequality is multiplicative for persons experiencing multiple inequalities, and that multiplicative effect is not representable by simple variable positions, or identities. Rather, Intersectionality must be acknowledged and quantified as the unique experience it is, including its implications in engineering education, specifically. As one identity-specific example, one may want to consider the unique gendered-raced experiences of Black women as a combined numerical category rather than consider the additive or interactional effects that one who is Black or a woman might experience. In another more inequality-specific example, one instead may want to consider measures of the causes and implications of socioeconomic inequality itself rather than income itself ( Major & Godwin, 2019 ).
• Humanity – Data cannot speak for itself or act anthropomorphically in any other way. Rather, data is interpreted by researchers through their scientific understandings and global enculturation. There are thus implications to ones’ interpretations. For example, if researchers have results in which a control for race/ethnicity or gender is significant, they must consider the social processes associated with the tenets of naturality and neutrality. The data may suggest that race/ethnicity or gender creates statistical difference, but these are not casual variables. Instead, the researcher should identify and discuss the systems of hierarchy and oppression that benefits White and male identified individuals ( Holland, 2008 ; Gillborn, Warmington, & Demack, 2018 ).
• Counter-Majority – Quantification unduly supports assumptions that there is an average , or dominant, group from which marginalized and minoritized individuals simply differ, and that quantification must also seek out counter-stories (quantitative or qualitative) which concurrently challenge those assumptions. Results of person-oriented methodologies, such as those we discuss in this work, may identify narratives that are counter to what may be extracted from traditional variable-oriented engineering education work. Similarly, small-n qualitative accounts of student experience may also identify quantitative components which have gone unaccounted or wrongly accounted (such as identity rather than inequality) in traditional accounts ( Sigle-Rushton, 2014 ).
• Reflexivity – Research is inherently political, biased, and essentialized, as shown through prior tenets. As such, disseminated research containing and striving for the equitable participation of diverse people, such as in engineering education, must be vocal about its association with a socially just political direction. It must also articulate how its data, methods, or results might otherwise support an oppositional direction. For example, one may want to openly disseminate details regarding their political directionality and positionality more broadly, and more specifically as it relates to methods of quantifying experience.

These tenets provide additional epistemic guidance for how quantitative research should be conducted from a critical epistemology. In this paper, we have focused on person-centered analyses as a novel quantitative method that could be used across non-positive paradigms. In conducting work aligned with critical epistemology and theory, person-centered methods may be used but must be grounded in these tenants and supplemented with other research methods.

Conclusions

In writing this paper, our goal is not to replace research traditions in qualitative methodologies with quantitative ones nor to indicate that all quantitative analyses must be person-centered. While methodologies and methods such as TDA, FemQuant, QuantCrit, and others provide more robust and nuanced understandings of relationships, groupings, experiences, and qualities within a dataset, ultimately, there are still individuals who can be misrepresented or unnoticed. As person-centered analyses are used to search for generalizable patterns among large, sprawling information, there remains space for over-generalizations or lack of representation in research findings. Even though the results from person-centered analyses are not restricted to a small number of dimensions or rigid relationships, an individual still may only partially fit within a pattern. Thus, results can give insight into a portion of their experience but may not fully capture the lived experiences of individuals.

We offer this discussion as a way to ask the engineering education research community to evaluate what we can ask and conclude from research aligned with non-positivist epistemologies. We hope that this discussion can expand the conceptualizations and operationalizations of new quantitative methods aligned with non-positivist epistemologies within engineering education research and open new frontiers within the field to serve students better and more inclusively.

In this article we use (post)positivism to refer to the family of epistemologies related to positivism. For concision, we use the term non-positivist to refer to epistemologies outside of this family.  

Acknowledgements

We would like to thank the editors and anonymous reviewers for the input on this work that strengthened the focus and argumentation. We would also like to thank the anonymous participants for their time in engaging with this research. This work was supported in part by the National Science Foundation under Grant No. 1554057, and through two Graduate Research Fellowships (DGE-1333468). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We would also like to thank the STRIDE (Shaping Transformative Research on Identity and Diversity in Engineering) research group for their assistance in data collection and review of findings for this project. Specifically, the authors would like to thank Dr. Jacqueline Doyle for her work in developing the Mapper algorithm ( Doyle, 2017 ) used to conduct the TDA analysis and her consultation in data analysis. We would also like to thank Dr. Adam Kirn for his conversations about person-centered analyses and Dr. Elliot Douglas for his discussion of epistemic framings in research with the first author.

Competing Interests

The authors have no competing interests to declare.

Authors Contributions

Regarding this manuscript, AG conceptualized the idea for research, supervised all aspects of the research, conducted post-TDA analyses, wrote portions of each of the sections, and edited the document for flow and consistency. AG also wrote the sections describing the TDA analyses and results. JR wrote the introduction and epistemology section, as well as contributed throughout to link person-centered analysis to particular epistemological framings. In the example project described in this article, AT led and AG and JR assisted with data analysis and interpretation. BB contributed to the sections focused on new methodological approaches in quantitative research and the example of TDA used in engineering education. BB also contributed to the data collection and interpretation of the national survey data, as well as the data collection and analysis of the longitudinal narrative interviews. HP wrote sections on person-centered analyses. JM wrote sections on critical quantitative methodologies. RC contributed to the challenges and opportunities associated with person-centered analysis. RC also contributed to the data collection and analysis of the longitudinal narrative interviews. SC edited the document, found references for claims made in the paper, and properly cited all references used.

Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A., & Arshad, H. (2018). State-of-the-art in artificial neural network applications: A survey. Heliyon , 4(11), e00938. DOI: https://doi.org/10.1016/j.heliyon.2018.e00938  

Acker, J. (1990). Hierarchies, jobs, bodies: A theory of gendered organizations. Gender & Society , 4(2), 139–158. DOI: https://doi.org/10.1177/089124390004002002  

Akpanudo, U. M., Huff, J. L., Williams, J. K., & Godwin, A. (2017, October). Hidden in plain sight: Masculine social norms in engineering education. In IEEE Frontiers in Education Conference. DOI: https://doi.org/10.1109/FIE.2017.8190515  

Baillie, C., & Douglas, E. P. (2014). Confusions and conventions: Qualitative research in engineering education. Journal of Engineering Education , 103(1), 1–7. DOI: https://doi.org/10.1002/jee.20031  

Bairaktarova & Pilotte. (2020). Person or thing oriented: A comparative study of individual differences of first-year engineering students and practitioners. Journal of Engineering Education , 109(2), 230–242. DOI: https://doi.org/10.1002/jee.20309  

Benedict, B., Baker, R. A., Godwin, A., & Milton, T. (2018). Uncovering latent diversity: Steps towards understanding ‘what counts’ and ‘who belongs’ in engineering culture. In ASEE Annual Conference & Exposition, Salt Lake City, UT. DOI: https://doi.org/10.18260/1-2-31164  

Benson, L., Potvin, G., Kirn, A., Godwin, A., Doyle, J., Rohde, J. A., Verdín, D., & Boone, H. (2017). Characterizing student identities in engineering: Attitudinal profiles of engineering majors. In ASEE Annual Conference & Exposition, Columbus, OH. DOI: https://doi.org/10.18260/1-2--27950  

Biesta, G. (2010). Pragmatism and the philosophical foundations of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social and Behavioral Research (pp. 95–118), SAGE. DOI: https://doi.org/10.4135/9781506335193.n4  

Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression Trees . New York, NY: Routledge. DOI: https://doi.org/10.1201/9781315139470  

Bowleg, L. (2008). When Black+ lesbian+ woman≠ Black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles , 59(5–6), 312–325. DOI: https://doi.org/10.1007/s11199-008-9400-z  

Bryman, A. (2008). The end of the paradigm wars? In Alasuutari, P., Bickman, L. and Brannen, J. (Eds.), The SAGE Handbook of Social Research Methods (pp. 13–25), London, UK: SAGE. DOI: https://doi.org/10.4135/9781446212165  

Cech, E. (2015). Engineers and engineeresses? Self-conceptions and the development of gendered professional identities. Sociological Perspectives , 58(1), 56–77. DOI: https://doi.org/10.1177/0731121414556543  

Cejka, M. A., & Eagly, A. H. (1999). Gender-stereotypic images of occupations correspond to the sex segregation of employment. Personality and Social Psychology Bulletin , 25(4), 413–423. DOI: https://doi.org/10.1177/0146167299025004002  

Chazal, F., & Michel, B. (2017). An introduction to Topological Data Analysis: Fundamental and practical aspects for data scientists. Retrieved from http://arxiv.org/abs/1710.04019  

Codiroli Mcmaster, N., & Cook, R. (2019). The contribution of intersectionality to quantitative research into educational inequalities. Review of Education , 7(2), 271–292. DOI: https://doi.org/10.1002/rev3.3116  

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates.  

Collins, P. H. (1990). Black feminist thought: Knowledge, consciousness, and the politics of empowerment . Unwin Hyman.  

Collins, P. H., & Bilge, S. (2016). Intersectionality . Cambridge, UK: Polity Press.  

Connell, R. W. (2009). Gender: Short introductions (2nd ed.). Cambridge, UK: Polity Press.  

Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd Ed.). SAGE.  

Crotty, M. (1998). The foundations of social research: Meaning and perspective in the research process . SAGE.  

Danielak, B. A., Gupta, A., & Elby, A. (2014). Marginalized identities of sense-makers: Reframing engineering student retention. Journal of Engineering Education , 103(1), 8–44. DOI: https://doi.org/10.1002/jee.20035  

Delgado, R., & Stefancic, J. (2012). Critical race theory: An introduction (2 nd ed.). New York, NY: New York University Press. https://ssrn.com/abstract=1640643  

Douglas, E. P., Koro-Ljungberg, M., & Borrego, M. (2010). Challenges and promises of overcoming epistemological and methodological partiality: Advancing engineering education through acceptance of diverse ways of knowing. European Journal of Engineering Education , 35(3), 247–257. DOI: https://doi.org/10.1080/03043791003703177  

Douglas, K. A., & Purzer, Ş. (2015). Validity: Meaning and relevancy in assessment for engineering education research. Journal of Engineering Education , 104(2), 108–118. DOI: https://doi.org/10.1002/jee.20070  

Doyle, J. (2017). Describing and mapping the interactions between student affective factors related to persistence in science, physics, and engineering (Publication No. 10747700). [Doctoral dissertation, Florida International University]. ProQuest Dissertations & Theses Global.  

Everitt, B. S., Landau, S., Leese, M., & Stahl, D. (2011). Cluster analysis (5 th ed.). John Wiley & Sons, Inc. DOI: https://doi.org/10.1002/9780470977811  

Eye, A., & Wiedermann, W. (2015). Person-Centered Analysis. In Emerging Trends in the Social and Behavioral Sciences (pp. 1–18). John Wiley & Sons, Inc. DOI: https://doi.org/10.1002/9781118900772.etrds0251  

Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PloS one , 5(4), e10068. DOI: https://doi.org/10.1371/journal.pone.0010068  

Fernandez, T., & Godwin, A., & Doyle, J., & Verdín, D., & Boone, H., & Kirn, A., & Benson, L., & Potvin, G. (2016). More comprehensive and inclusive approaches to demographic data collection. In ASEE Annual Conference & Exposition, New Orleans, LA. DOI: https://doi.org/10.18260/p.25751  

Foor, C. E., Walden, S. E., & Trytten, D. A. (2007). “I wish that I belonged more in this whole engineering group”: Achieving individual diversity. Journal of Engineering Education , 96(2), 103–115. DOI: https://doi.org/10.1002/j.2168-9830.2007.tb00921.x  

Garcia-Dias, R., Vieira, S., Pinaya, W. H. L., & Mechelli, A. (2020). Clustering analysis. In Machine Learning (pp. 227–247). Academic Press. DOI: https://doi.org/10.1016/B978-0-12-815739-8.00013-4  

Gero, J., & Milovanovic, J. (2020). A framework for studying design thinking through measuring designers’ minds, bodies and brains. Design Science , 6, E19. DOI: https://doi.org/10.1017/dsj.2020.15  

Gero, J. S., & Peng, W. (2009). Understanding behaviors of a constructive memory agent: A Markov chain analysis. Knowledge-Based Systems , 22(8), 610–621. DOI: https://doi.org/10.1016/j.knosys.2009.05.006  

Gillborn, D. (2018). QuantCrit: Rectifying quantitative methods through Critical Race Theory [Special Issue]. Race Ethnicity and Education , 21(2), 149–273. DOI: https://doi.org/10.1080/13613324.2017.1377675  

Gillborn, D., Warmington, P., & Demack, S. (2018). QuantCrit: education, policy, ‘Big Data’ and principles for a critical race theory of statistics. Race Ethnicity and Education , 21(2), 158–179. DOI: https://doi.org/10.1080/13613324.2017.1377417  

Godwin, A. (2017). Unpacking latent diversity. In ASEE Annual Conference & Exposition, Columbus, OH. DOI: https://doi.org/10.18260/1-2--29062  

Godwin, A., Benedict, B. S., Verdín, D., Thielmeyer, A. R. H., Baker, R. A., & Rohde, J. A. (2018). Board 12: CAREER: Characterizing latent diversity among a national sample of first-year engineering students. In ASEE Annual Conference & Exposition, Tampa, FL. https://peer.asee.org/32207  

Godwin, A., Thielmeyer, A. R. H., Rohde, J. A., Verdín, D., Benedict, B. S., Baker, R. A., Doyle, J. (2019). Using topological data analysis in social science research: Unpacking decisions and opportunities for a new method. In ASEE Annual Conference and Exposition, Tampa, FL. https://peer.asee.org/33522  

Goldschmidt, G. (2014). Linkography: unfolding the design process . MIT Press. DOI: https://doi.org/10.7551/mitpress/9455.001.0001  

Greenacre, M., & Hastie, T. (1987). The geometric interpretation of correspondence analysis. Journal of the American Statistical Association , 82(398), 437–447. DOI: https://doi.org/10.1080/01621459.1987.10478446  

Hammersley, M. (2008). Assessing validity in social research. In P. Alasuutari, L. Bickman, & J. Brannen (Eds.), The SAGE Handbook of Social Research Methods (pp. 42–53), SAGE. DOI: https://doi.org/10.4135/9781446212165.n4  

Hanel, P. H., Maio, G. R., & Manstead, A. S. (2019). A new way to look at the data: Similarities between groups of people are large and important. Journal of Personality and Social Psychology , 116(4), 541–562. DOI: https://doi.org/10.1037/pspi0000154  

Harding, S. (2016). Whose science? Whose knowledge? Thinking from women’s lives . Cornell University Press. DOI: https://doi.org/10.7591/9781501712951  

Hesse-Biber, S. N., & Piatelli, D. (2012). The feminist practice of holisitic reflexivity. In S. N. Hesse-Biber (Ed.), Handbook of Feminist Research Theory and Praxis (2nd ed., pp. 557–582). SAGE. DOI: https://doi.org/10.4135/9781483384740.n27  

Holland, P. W. (2008). Causation and race. In T. Zuberi & E. Bonilla-Silva (Eds.), White logic, white methods: Racism and methodology . Rowman & Littlefield.  

Hout, M. C., Papesh, M. H., & Goldinger, S. D. (2013). Multidimensional scaling. Wiley Interdisciplinary Reviews: Cognitive Science , 4(1), 93–103. DOI: https://doi.org/10.1002/wcs.1203  

Hundleby, C. E. (2012). Feminist empiricism. In S. N. Hesse-Biber (Ed.), Handbook of Feminist Research: Theory and Praxis (2nd ed., pp. 28–45). SAGE. DOI: https://doi.org/10.4135/9781483384740.n2  

Jack, R. E., Crivelli, C., & Wheatley, T. (2018). Data-Driven Methods to Diversify Knowledge of Human Psychology. Trends in Cognitive Sciences, 22(1), 1–5. DOI: https://doi.org/10.1016/j.tics.2017.10.002  

Jagger, A. M. (2014). Introduction: The project of feminist methodology. In A. M. Jagger (Ed.), Just Methods: An Interdisciplinary Feminist Reader (2nd ed., pp. vii–xiii). Paradigm Publishers. DOI: https://doi.org/10.4324/9781315636344  

Jesiek, B. K., Newswander, L. K., & Borrego, M. (2009). Engineering education research: Discipline, community, or field? Journal of Engineering Education , 98(1), 39–52. DOI: https://doi.org/10.1002/j.2168-9830.2009.tb01004.x  

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher , 33(7), 14–26. DOI: https://doi.org/10.3102/0013189X033007014  

Kan, J. W., & Gero, J. S. (2010). Exploring quantitative methods to study design behavior in collaborative virtual workspaces. In New Frontiers, Proceedings of the 15th International Conference on CAADRIA (pp. 273–282).  

Kant, V., & Kerr, E. (2019). Taking stock of engineering epistemology: Multidisciplinary perspectives. Philosophy & Technology , 32(4), 685–726. DOI: https://doi.org/10.1007/s13347-018-0331-5  

Kaushik, V., & Walsh, C. A. (2019). Pragmatism as a research paradigm and its implications for social work research. Social Sciences , 8(255), 1–17. DOI: https://doi.org/10.3390/socsci8090255  

Kherif, F., & Latypova, A. (2020). Principal component analysis. In Machine Learning (pp. 209–225). Academic Press. DOI: https://doi.org/10.1016/B978-0-12-815739-8.00012-2  

Koro-Ljungberg, M., & Douglas, E. P. (2008). State of qualitative research in engineering education: Meta-analysis of JEE articles, 2005–2006. Journal of Engineering Education , 97(2), 163–175. DOI: https://doi.org/10.1002/j.2168-9830.2008.tb00965.x  

Lather, P. (2006). Paradigm proliferation as a good thing to think with: Teaching research in education as a wild profusion. International Journal of Qualitative Studies in Education , 19(1), 35–57. DOI: https://doi.org/10.1080/09518390500450144  

Laubenbacher, R., and Hastings, A., (2019). Topological Data Analysis. Bulletin of Mathematical Biology . 81(7), 2051. DOI: https://doi.org/10.1007/s11538-019-00610-3  

Laursen, B., & Hoff, E. (2006). Person-centered and variable-centered approaches to longitudinal data. Merrill-Palmer Quarterly , 52(3), 377–389. DOI: https://doi.org/10.1353/mpq.2006.0029  

Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabasi, A. L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., Jebara, T., King, G., Macy, M., Roy, D., & Van Alstyne, M. (2009). Computational social science. Science , 323(5915), 721–723. DOI: https://doi.org/10.1126/science.1167742  

Lum, P. Y., Singh, G., Lehman, A., Ishkanov, T., Vejdemo-Johansson, M., Alagappan, M., Carlsson, J. & Carlsson, G. (2013). Extracting insights from the shape of complex data using topology. Scientific Reports , 3, 1236. DOI: https://doi.org/10.1038/srep01236  

Major, J., Godwin, A., & Kirn, A. (2021). Working to achieve equitable access to engineering by redefining disciplinary standards for the use and dissemination of quantitative study demographics. In Collaborative Network for Engineering and Computing Diversity Conference, Washington, DC. https://peer.asee.org/36147  

Major, J. C., & Godwin, A. (2019). An intersectional conceptual framework for understanding how to measure socioeconomic inequality in engineering education. In ASEE Annual Conference & Exposition, Tampa, FL. DOI: https://doi.org/10.18260/1-2--33594  

Maxcy, S. J. (2003). Pragmatic threads in mixed methods research in the social sciences: The search for multiple modes of inquiry and the end of the philosophy of formalism. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social and Behavioral Research (pp. 51–89), SAGE.  

McCall, L. (2002). Complex inequality: Gender, class, and race in the new economy . Routledge. DOI: https://doi.org/10.4324/9780203902455  

McGuirl, M. R., Volkening, A., & Sandstede, B. (2020). Topological data analysis of zebrafish patterns. Proceedings of the National Academy of Sciences , 117(10), 5113–5124. DOI: https://doi.org/10.1073/pnas.1917763117  

McNicholas, P. D. (2010). Model-based classification using latent Gaussian mixture models. Journal of Statistical Planning and Inference , 140(5), 1175–1181. DOI: https://doi.org/10.1016/j.jspi.2009.11.006  

Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation (4th ed.). John Wiley & Sons.  

Miller, D. I., Eagly, A. H., & Linn, M. C. (2015). Women’s representation in science predicts national gender-science stereotypes: Evidence from 66 nations. Journal of Educational Psychology , 107(3), 631–644. DOI: https://doi.org/10.1037/edu0000005  

Morgan, D. L. (2014). Pragmatism as a paradigm for social research. Qualitative Inquiry , 20(8), 1045–1053. DOI: https://doi.org/10.1177/1077800413513733  

Morin, A. J., Bujacz, A., & Gagné, M. (2018). Person-centered methodologies in the organizational sciences: Introduction to the feature topic. Organizational Research Method , 21(4), 803–813. DOI: https://doi.org/10.1177/1094428118773856  

National Academy of Engineering. (2008). Changing the conversation: Messages for improving public understanding of engineering. Washington DC, National Academies Press. DOI: https://doi.org/10.17226/12187  

Oakley, A. (1998). Gender, methodology and people’s ways of knowing: Some problems with feminism and the paradigm debate in social science. Sociology , 32(4), 707–731. DOI: https://doi.org/10.1177/0038038598032004005  

Oberski, D. (2016) Mixture Models: Latent Profile and Latent Class Analysis. In J. Robertson, M. Kaptein (Eds.) Modern Statistical Methods for HCI . Human–Computer Interaction Series. Springer. DOI: https://doi.org/10.1007/978-3-319-26633-6_12  

Omi, M., & Winant, H. (2014). Racial formation in the United States (3rd ed.). Routledge. DOI: https://doi.org/10.4324/9780203076804-6  

Pallas, A. M. (2001) Preparing education doctoral students for epistemological diversity. Educational Researcher , 30(5), 1–6. DOI: https://doi.org/10.3102/0013189X030005006  

Pawley, A. L. (2017). Shifting the “default”: The case for making diversity the expected condition for engineering education and making whiteness and maleness visible. Journal of Engineering Education , 106(4), 531–533. DOI: https://doi.org/10.1002/jee.20181  

Pawley, A. L. (2018). Learning from small numbers: Studying ruling relations that gender and race the structure of US engineering education. Journal of Engineering Education , 108(1), 13–31. DOI: https://doi.org/10.1002/jee.20247  

Perdomo Meza, D. A. (2015). Topological data analysis with metric learning and an application to high-dimensional football data [Master’s thesis, Bogotá-Uniandes]. Retrieved from https://repositorio.uniandes.edu.co/bitstream/handle/1992/12963/u713491.pdf?sequence=1  

Qiu, L., Chan, S. H. M., & Chan, D. (2018). Big data in social and psychological science: theoretical and methodological issues. Journal of Computational Social Science , 1(1), 59–66. DOI: https://doi.org/10.1007/s42001-017-0013-6  

R Core Team. (2018). R: A language and environment for statistical computing . Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org .  

Ram, N., & Grimm, K. J. (2009). Methods and measures: Growth mixture modeling: A method for identifying differences in longitudinal change among unobserved groups. International journal of behavioral development , 33(6), 565–576. DOI: https://doi.org/10.1177/0165025409343765  

Ray, V. (2019). A theory of racialized organizations. American Sociological Review , 84(1), 26–53. DOI: https://doi.org/10.1177/0003122418822335  

Reed, I. A. (2010). Epistemology contextualized: Social-scientific knowledge in a postpositivist era. Sociological Theory , 28(1), 20–39. DOI: https://doi.org/10.1111/j.1467-9558.2009.01365.x  

Riley, D. (2017). Rigor/Us: Building boundaries and disciplining diversity with standards of merit. Engineering Studies , 9(3), 249–265. DOI: https://doi.org/10.1080/19378629.2017.1408631  

Scheurich, J. J., & Young, M. D. (1997). Coloring epistemologies: Are our research epistemologies racially biased? Educational researcher , 26(4), 4–16. DOI: https://doi.org/10.3102/0013189X026004004  

Secules, S., Gupta, A., Elby, A., & Turpen, C. (2018). Zooming out from the struggling individual student: An account of the cultural construction of engineering ability in an undergraduate programming class. Journal of Engineering Education , 107(1), 56–86. DOI: https://doi.org/10.1002/jee.20191  

Sellbom, M., & Tellegen, A. (2019). Factor analysis in psychological assessment research: Common pitfalls and recommendations. Psychological Assessment , 31(12), 1428–1441. DOI: https://doi.org/10.1037/pas0000623  

Sigle-Rushton, W. (2014). Essentially quantified? Towards a more feminist modeling strategy. In M. Evans, C. Hemmings, M. Henry, H. Johnstone, S. Madhok, A. Plomien, & S. Wearing (Eds.), The SAGE Handbook of Feminist Theory (pp. 431–445). SAGE. DOI: https://doi.org/10.4135/9781473909502.n29  

Slaton, A. E. (2015). Meritocracy, technocracy, democracy: Understandings of racial and gender equity in American engineering education. In International perspectives on engineering education (pp. 171–189). Springer. DOI: https://doi.org/10.1007/978-3-319-16169-3_8  

Slaton, A. E., & Pawley, A. L. (2018). The power and politics of engineering education research design: Saving the ‘Small N’. Engineering Studies , 10(2–3), 133–157. DOI: https://doi.org/10.1080/19378629.2018.1550785  

Sprague, J. (2005). How feminists count: Critical strategies for quantitative methods. In J. Sprague (Ed.), Feminist Methodology for Critical Researchers: Bridging Differences (1st ed., pp. 81–117). Rowman & Littlefield.  

Sprague, J., & Zimmerman, M. K. (1989). Quality and quantity: Reconstructing feminist methodology. The American Sociologist , 20(1), 71–86. DOI: https://doi.org/10.1007/BF02697788  

Streveler, R., & Smith, K. A. (2006). Rigorous research in engineering education. Journal of Engineering Education , 95(2), 103–105. DOI: https://doi.org/10.1002/j.2168-9830.2006.tb00882.x  

Su, R., & Rounds, J. (2015). All STEM fields are not created equal: People and things interests explain gender disparities across STEM fields. Frontiers in Psychology , 6(Article 189), 1–20. DOI: https://doi.org/10.3389/fpsyg.2015.00189  

Tashakkori, A., & Teddlie, C. (2008). Quality of inferences in mixed methods research: Calling for an integrative framework. In M. M. Bergman (Ed.), Advances in Mixed Methods Research (pp. 101–119), SAGE. DOI: https://doi.org/10.4135/9780857024329.d10  

Tuli, F. (2010). The basis of distinction between qualitative and quantitative research in social science: Reflection on ontological, epistemological and methodological perspectives. Ethiopian Journal of Education and Sciences , 6(1), 97–108. DOI: https://doi.org/10.4314/ejesc.v6i1.65384  

Tynjälä, P., Salminen, R. T., Sutela, T., Nuutinen, A., & Pitkänen, S. (2005). Factors related to study success in engineering education. European Journal of Engineering Education , 30(2), 221–231. DOI: https://doi.org/10.1080/03043790500087225  

Uhlar, J. R., & Secules, S. (2018). Butting heads: Competition and posturing in a paired programming team. In IEEE Frontiers in Education Conference, San Jose, CA. DOI: https://doi.org/10.1109/FIE.2018.8658654  

Verdín, D., Godwin, A., Kirn, A., Benson, L., & Potvin, G. (2018). Engineering women’s attitudes and goals in choosing disciplines with above and below average female representation. Social Sciences , 7(3), 44. DOI: https://doi.org/10.3390/socsci7030044  

Villanueva, I., Di Stefano, M., Gelles, L., Osoria, P. V., & Benson, S. (2019). A race re-imaged, intersectional approach to academic mentoring: Exploring the perspectives and responses of womxn in science and engineering research. Contemporary Educational Psychology , 59(2019), 101786. DOI: https://doi.org/10.1016/j.cedpsych.2019.101786  

Villanueva, I., Husman, J., Christensen, D., Youmans, K., Khan, M. T., Vicioso, P., Lampkins, S., & Graham, M. C. (2019). A cross-disciplinary and multi-modal experimental design for studying near-real-time authentic examination experiences. JoVE (Journal of Visualized Experiments) , (151), e60037. DOI: https://doi.org/10.3791/60037  

Walther, J., Pawley, A. L., & Sochacka, N. W. (2015). Exploring ethical validation as a key consideration in interpretive research quality. In ASEE Annual Conference & Exposition, Seattle, WA. DOI: https://doi.org/10.18260/p.24063  

Walther, J., Sochacka, N. W., Benson, L. C., Bumbaco, A. E., Kellam, N., Pawley, A. L., & Phillips, C. M. (2017). Qualitative research quality: A collaborative inquiry across multiple methodological perspectives. Journal of Engineering Education , 106(3), 398–430. DOI: https://doi.org/10.1002/jee.20170  

Walther, J., Sochacka, N. W., & Kellam, N. N. (2013). Quality in interpretive engineering education research: Reflections on an example study. Journal of Engineering Education, 102(4), 626–659. DOI: https://doi.org/10.1002/jee.20029  

Wang, M., Sinclair, R. R., Zhou, L., & Sears, L. E. (2013). Person-centered analysis: Methods, applications, and implications for occupational health psychology. In R. R. Sinclair, M. Wang, & L. E. Tetrick (Eds.), Research methods in occupational health psychology: Measurement, design, and data analysis (p. 349–373). Routledge/Taylor & Francis Group. DOI: https://doi.org/10.4324/9780203095249  

Wasserman, L. (2018). Topological data analysis. Annual Review of Statistics and Its Application , (5), 501–532. DOI: https://doi.org/10.1146/annurev-statistics-031017-100045  

Wickham, H. (2009). ggplot2: elegant graphics for data analysis. Springer. http://had.co.nz/ggplot2/book . Accessed: August, 5, 2014.  

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Appraising Quantitative Research in Health Education: Guidelines for Public Health Educators

Leonard jack, jr..

Associate Dean for Research and Endowed Chair of Minority Health Disparities, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, New Orleans, Louisiana 70125; Telephone: 504-520-5345; Fax: 504-520-7971

Sandra C. Hayes

Central Mississippi Area Health Education Center, 350 West Woodrow Wilson, Suite 3320, Jackson, MS 39213; Telephone: 601-987-0272; Fax: 601-815-5388

Jeanfreau G. Scharalda

Louisiana State University Health Sciences Center School of Nursing, 1900 Gravier Street, New Orleans, Louisiana 70112; Telephone: 504-568-4140; Fax: 504-568-5853

Barbara Stetson

Department of Psychological and Brain Sciences, 317 Life Sciences Building, University of Louisville, Louisville, KY 40292; Telephone: 502-852-2540; Fax: 502-852-8904

Nkenge H. Jones-Jack

Epidemiologist & Evaluation Consultant, Metairie, Louisiana 70002. Telephone: 678-524-1147; Fax: 504-267-4080

Matthew Valliere

Chronic Disease Prevention and Control, Bureau of Primary Care and Rural Health, Office of the Secretary, 628 North 4th Street, Baton Rouge, LA 70821-3118; Telephone: 225-342-2655; Fax: 225-342-2652

William R. Kirchain

Division of Clinical and Administrative Sciences, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, Room 121, New Orleans, Louisiana 70125; Telephone: 504-520-5395; Fax: 504-520-7971

Michael Fagen

Co-Associate Editor for the Evaluation and Practice section of Health Promotion Practice , Department of Community Health Sciences, School of Public Health, University of Illinois at Chicago, 1603 W. Taylor St., M/C 923, Chicago, IL 60608-1260, Telephone: 312-355-0647; Fax: 312-996-3551

Cris LeBlanc

Centers of Excellence Scholar, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, New Orleans, Louisiana 70125; Telephone: 504-520-5345; Fax: 504-520-7971

Many practicing health educators do not feel they possess the skills necessary to critically appraise quantitative research. This publication is designed to help provide practicing health educators with basic tools helpful to facilitate a better understanding of quantitative research. This article describes the major components—title, introduction, methods, analyses, results and discussion sections—of quantitative research. Readers will be introduced to information on the various types of study designs and seven key questions health educators can use to facilitate the appraisal process. Upon reading, health educators will be in a better position to determine whether research studies are well designed and executed.

Appraising the Quality of Quantitative Research in Health Education

Practicing health educators often find themselves with little time to read published research in great detail. Some health educators with limited time to read scientific papers may get frustrated as they get bogged down trying to understand research terminology, methods, and approaches. The purpose of appraising a scientific publication is to assess whether the study’s research questions (hypotheses), methods and results (findings) are sufficiently valid to produce useful information ( Fowkes and Fulton, 1991 ; Donnelly, 2004 ; Greenhalgh and Taylor, 1997 ; Johnson and Onwuegbuze, 2004 ; Greenhalgh, 1997 ; Yin, 2003; and Hennekens and Buring, 1987 ). Having the ability to deconstruct and reconstruct scientific publications is a critical skill in a results-oriented environment linked to increasing demands and expectations for improved program outcomes and strong justifications to program focus and direction. Health educators do must not solely rely on the opinions of researchers, but, rather, increase their confidence in their own abilities to discern the quality of published scientific research. Health educators with little experience reading and appraising scientific publications, may find this task less difficult if they: 1) become more familiar with the key components of a research publication, and 2) utilize questions presented in this article to critically appraise the strengths and weaknesses of published research.

Key Components of a Scientific Research Publication

The key components of a research publication should provide important information that is needed to assess the strengths and weaknesses of the research. Key components typically include the: publication title , abstract , introduction , research methods used to address the research question(s) or hypothesis, statistical analysis used, results , and the researcher’s interpretation and conclusion or recommended use of results to inform future research or practice. A brief description of these components follows:

Publication Title

A general heading or description should provide immediate insight into the intent of the research. Titles may include information regarding the focus of the research, population or target audience being studied, and study design.

An abstract provides the reader with a brief description of the overall research, how it was done, statistical techniques employed, key results,and relevant implications or recommendations.

Introduction

This section elaborates on the content mentioned in the abstract and provides a better idea of what to anticipate in the manuscript. The introduction provides a succinct presentation of previously published literature, thus offering a purpose (rationale) for the study.

This component of the publication provides critical information on the type of research methods used to conduct the study. Common examples of study designs used to conduct quantitative research include cross sectional study, cohort study, case-control study, and controlled trial. The methods section should contain information on the inclusion and exclusion criteria used to identify participants in the study.

Quantitative data contains information that is quantifiable, perhaps through surveys that are analyzed using statistical tests to determine if the results happened by chance. Two types of statistical analyses are used: descriptive and inferential ( Johnson and Onwuegbuze, 2004 ). Descriptive statistics are used to describe the basic features of the study data and provide simple summaries about the sample and measures. With inferential statistics, researchers are trying to reach conclusions that extend beyond the immediate data alone. Thus, they use inferential statistics to make inferences from the data to more general conditions.

This section presents the reader with the researcher’s data and results of statistical analyses described in the method section. Thus, this section must align closely with the methods section.

Discussion (Conclusion)

This section should explain what the data means thereby summarizing main results and findings for the reader. Important limitations (such as the use of a non-random sample, the absence of a control group, and short duration of the intervention) should be discussed. Researchers should discuss how each limitation can impact the applicability and use of study results. This section also presents recommendations on ways the study can help advance future health education and practice.

Critically Appraising the Strengths and Weaknesses of Published Research

During careful reading of the analysis, results, and discussion (conclusion) sections, what key questions might you ask yourself in order to critically appraise the strengths and weaknesses of the research? Based on a careful review of the literature ( Greenhalgh and Taylor, 1997 ; Greenhalgh, 1997 ; and Hennekens and Buring, 1987 ) and our research experiences, we have identified seven key questions around which to guide your assessment of quantitative research.

1) Is a study design identified and appropriately applied?

Study designs refer to the methodology used to investigate a particular health phenomenon. Becoming familiar with the various study designs will help prepare you to critically assess whether its selection was applied adequately to answer the research questions (or hypotheses). As mentioned previously, common examples of study designs frequently used to conduct quantitative research include cross sectional study, cohort study, case-control study, and controlled trail. A brief description of each can be found in Table 1 .

Definitions of Study Designs

2) Is the study sample representative of the group from which it is drawn?

The study sample must be representative of the group from which it is drawn. The study sample must therefore be typical of the wider target audience to whom the research might apply. Addressing whether the study sample is representative of the group from which it is drawn will require the researcher to take into consideration the sampling method and sample size.

Sampling Method

Many sampling methods are used individually or in combination. Keep in mind that sampling methods are divided into two categories: probability sampling and non-probability sampling ( Last, 2001 ). Probability sampling (also called random sampling) is any sampling scheme in which the probability of choosing each individual is the same (or at least known, so it can be readjusted mathematically to be equal). Non-probability sampling is any sampling scheme in which the probability of an individual being chosen is unknown. Typically, researchers should offer a rationale for utilizing non-probability sampling, and when utilized, be aware of its limitations. For example, use of a convenience sample (choosing individuals in an unstructured manner) can be justified when collecting pilot data around which future studies employing more rigorous sampling methods will be utilized.

Sample Size

Established statistical theories and formulas are used to generate sample size calculations—the recommended number of individuals necessary in order to have sufficient power to detect meaningful results at a certain level of statistical significance. In the methods section, look for a statement or two confirming whether steps where taken to obtain the appropriate sample size.

3) In research studies using a control group, is this group adequate for the purpose of the study?

Source of controls.

In case-control and cohort studies, the source of controls should be such that the distribution of characteristics not under investigation are similar to those in the cases or study cohort.

In case-control studies both cases and controls are often matched on certain characteristics such as age, sex, income, and race. The criteria used for including and excluding study participants must be adequately described and examined carefully. Inclusion and exclusion criteria may include: ethnicity, age of diagnosis, length of time living with a health condition, geographic location, and presence or absence of complications. You should critically assess whether matching across these characteristics actually occurred.

4) What is the validity of measurements and outcomes identified in the study?

Validity is the extent to which a measurement captures what it claims to measure. This might take the form of questions contained on a survey, questionnaire or instrument. Researchers should address one or more of the following types of validity: face, content, criterion-related, and construct ( Last, 2001 ; William and Donnelly, 2008).

Face validity

Face validity assures that, upon examination, the variable of interest can measure what it intends to measure. If the researcher has chosen to study a variable that has not been studied before, he/she usually will need to start with face validity.

Content validity

Content validity involves comparing the content of the measurement technique to the known literature on the topic and validating the fact that the tool (e.g., survey, questionnaire) does represent the literature accurately.

Criterion-related validity

Criterion-related validity involves making sure the measures within a survey when tested proves to be effective in predicting criterion or indicators of a construct.

Construct validity

Construct validity deals with the validation of the construct that underlies the research. Here, researchers test the theory that underlies the hypothesis or research question.

5) To what extent is a common source of bias called blindness taken into account?

During data collection, a common source of bias is that subjects and/or those collecting the data are not blind to the purpose of the research. This can likely be the result of researchers going the extra mile to make sure those in the experimental group benefit from the intervention ( Fowkes and Fulton, 1991 ). Inadequate blindness can be a problem in studies utilizing all types of study designs. While total blindness is not possible, appraising whether steps were taken to be sure issues related to ensure blindness occurred is essential.

6) To what extent is the study considered complete with regard to drop outs and missing data?

Regardless of the study design employed, one must assess not only the proportion of drop outs in each group, but also why they dropped out. This may point to possible bias, as well as determine what efforts were taken to retain participants in the study.

Missing data

Despite the fact that missing data are a part of almost all research, it should still be appraised. There are several reasons why the data may be missing. The nature and extent to which data is missing should be explained.

7) To what extent are study results influenced by factors that negatively impact their credibility?

Contamination.

In research studies comparing the effectiveness of a structured intervention, contamination occurs when the control group makes changes based on learning what those participating in the intervention are doing. Despite the fact that researchers typically do not report the extent to which contamination occurs, you should nevertheless try to assess whether contamination negatively impacted the credibility of study results.

Confounding factors

A confounding factor in a study is a variable which is related to one or more of the measurements (measures or variables) defined in a study. A confounding factor may mask an actual association or falsely demonstrate an apparent association between the study variables where no real association between them exists. If confounding factors are not measured and considered, study results may be biased and compromised.

The guidelines and questions presented in this article are by no means exhaustive. However, when applied, they can help health education practitioners obtain a deeper understanding of the quality of published research. While no study is 100% perfect, we do encourage health education practitioners to pause before taking researchers at their word that study results are both accurate and impressive. If you find yourself answering ‘no’ to a majority of the key questions provided, then it is probably safe to say that, from your perspective, the quality of the research is questionable.

Over time, as you repeatedly apply the guidelines presented in this article, you will become more confident and interested in reading research publications from beginning to end. While this article is geared to health educators, it can help anyone interested in learning how to appraise published research. Table 2 lists additional reading resources that can help improve one’s understanding and knowledge of quantitative research. This article and the reading resources identified in Table 2 can serve as useful tools to frame informative conversations with your peers regarding the strengths and weaknesses of published quantitative research in health education.

Publications on How to Read, Write and Appraise Quantitative Research

Contributor Information

Leonard Jack, Jr., Associate Dean for Research and Endowed Chair of Minority Health Disparities, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, New Orleans, Louisiana 70125; Telephone: 504-520-5345; Fax: 504-520-7971.

Sandra C. Hayes, Central Mississippi Area Health Education Center, 350 West Woodrow Wilson, Suite 3320, Jackson, MS 39213; Telephone: 601-987-0272; Fax: 601-815-5388.

Jeanfreau G. Scharalda, Louisiana State University Health Sciences Center School of Nursing, 1900 Gravier Street, New Orleans, Louisiana 70112; Telephone: 504-568-4140; Fax: 504-568-5853.

Barbara Stetson, Department of Psychological and Brain Sciences, 317 Life Sciences Building, University of Louisville, Louisville, KY 40292; Telephone: 502-852-2540; Fax: 502-852-8904.

Nkenge H. Jones-Jack, Epidemiologist & Evaluation Consultant, Metairie, Louisiana 70002. Telephone: 678-524-1147; Fax: 504-267-4080.

Matthew Valliere, Chronic Disease Prevention and Control, Bureau of Primary Care and Rural Health, Office of the Secretary, 628 North 4th Street, Baton Rouge, LA 70821-3118; Telephone: 225-342-2655; Fax: 225-342-2652.

William R. Kirchain, Division of Clinical and Administrative Sciences, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, Room 121, New Orleans, Louisiana 70125; Telephone: 504-520-5395; Fax: 504-520-7971.

Michael Fagen, Co-Associate Editor for the Evaluation and Practice section of Health Promotion Practice , Department of Community Health Sciences, School of Public Health, University of Illinois at Chicago, 1603 W. Taylor St., M/C 923, Chicago, IL 60608-1260, Telephone: 312-355-0647; Fax: 312-996-3551.

Cris LeBlanc, Centers of Excellence Scholar, College of Pharmacy, Xavier University of Louisiana, 1 Drexel Drive, New Orleans, Louisiana 70125; Telephone: 504-520-5345; Fax: 504-520-7971.

• Fowkes FG, Fulton PM. Critical appraisal of published research: introductory guidelines. British Medical Journal. 1991; 302 :1136–40. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Donnelly RA. The Complete Idiots Guide to Statistics. Alpha Books; New York, NY: 2004. pp. 6–7. [ Google Scholar ]
• Greenhalgh T, Taylor R. How to read a paper: Papers that go beyond numbers (qualitative research) British Medical Journal. 1997; 315 :740–743. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Greenhalgh T. How to read a paper: Assessing the methodological quality of published papers. British Medical Journal. 315 :305–308. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Johnson RB, Onwuegbuze AJ. Mixed methods research: A research paradigm whose time has come. Educational Researcher. 2004; 33 :14–26. [ Google Scholar ]
• Hennekens CH, Buring JE. Epidemiology in Medicine. Little, Brown and Company; Boston, Massachusetts: 1987. pp. 106–108. [ Google Scholar ]
• Last JM. A dictionary of epidemiology. 4. Oxford University Press, Inc; New York, New York: 2001. [ Google Scholar ]
• Trochim WM, Donnelly J. Research methods knowledge base. 3. Atomic Dog; Mason, Ohio: 2008. pp. 6–8. [ Google Scholar ]

Advertisement

• Previous Issue
• Previous Article
• Next Article

Clarifying the Research Purpose

Methodology, measurement, data analysis and interpretation, tools for evaluating the quality of medical education research, research support, competing interests, quantitative research methods in medical education.

Submitted for publication January 8, 2018. Accepted for publication November 29, 2018.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

John T. Ratelle , Adam P. Sawatsky , Thomas J. Beckman; Quantitative Research Methods in Medical Education. Anesthesiology 2019; 131:23–35 doi: https://doi.org/10.1097/ALN.0000000000002727

Download citation file:

• Ris (Zotero)
• Reference Manager

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Image: J. P. Rathmell and Terri Navarette.

There has been an explosion of research in the field of medical education. A search of PubMed demonstrates that more than 40,000 articles have been indexed under the medical subject heading “Medical Education” since 2010, which is more than the total number of articles indexed under this heading in the 1980s and 1990s combined. Keeping up to date requires that practicing clinicians have the skills to interpret and appraise the quality of research articles, especially when serving as editors, reviewers, and consumers of the literature.

While medical education shares many characteristics with other biomedical fields, substantial particularities exist. We recognize that practicing clinicians may not be familiar with the nuances of education research and how to assess its quality. Therefore, our purpose is to provide a review of quantitative research methodologies in medical education. Specifically, we describe a structure that can be used when conducting or evaluating medical education research articles.

Clarifying the research purpose is an essential first step when reading or conducting scholarship in medical education. 1   Medical education research can serve a variety of purposes, from advancing the science of learning to improving the outcomes of medical trainees and the patients they care for. However, a well-designed study has limited value if it addresses vague, redundant, or unimportant medical education research questions.

What is the research topic and why is it important? What is unknown about the research topic? Why is further research necessary?

What is the conceptual framework being used to approach the study?

What is the statement of study intent?

What are the research methodology and study design? Are they appropriate for the study objective(s)?

Which threats to internal validity are most relevant for the study?

What is the outcome and how was it measured?

Can the results be trusted? What is the validity and reliability of the measurements?

How were research subjects selected? Is the research sample representative of the target population?

Was the data analysis appropriate for the study design and type of data?

What is the effect size? Do the results have educational significance?

Fortunately, there are steps to ensure that the purpose of a research study is clear and logical. Table 1   2–5   outlines these steps, which will be described in detail in the following sections. We describe these elements not as a simple “checklist,” but as an advanced organizer that can be used to understand a medical education research study. These steps can also be used by clinician educators who are new to the field of education research and who wish to conduct scholarship in medical education.

Steps in Clarifying the Purpose of a Research Study in Medical Education

Literature Review and Problem Statement

A literature review is the first step in clarifying the purpose of a medical education research article. 2 , 5 , 6   When conducting scholarship in medical education, a literature review helps researchers develop an understanding of their topic of interest. This understanding includes both existing knowledge about the topic as well as key gaps in the literature, which aids the researcher in refining their study question. Additionally, a literature review helps researchers identify conceptual frameworks that have been used to approach the research topic. 2  

When reading scholarship in medical education, a successful literature review provides background information so that even someone unfamiliar with the research topic can understand the rationale for the study. Located in the introduction of the manuscript, the literature review guides the reader through what is already known in a manner that highlights the importance of the research topic. The literature review should also identify key gaps in the literature so the reader can understand the need for further research. This gap description includes an explicit problem statement that summarizes the important issues and provides a reason for the study. 2 , 4   The following is one example of a problem statement:

“Identifying gaps in the competency of anesthesia residents in time for intervention is critical to patient safety and an effective learning system… [However], few available instruments relate to complex behavioral performance or provide descriptors…that could inform subsequent feedback, individualized teaching, remediation, and curriculum revision.” 7  

This problem statement articulates the research topic (identifying resident performance gaps), why it is important (to intervene for the sake of learning and patient safety), and current gaps in the literature (few tools are available to assess resident performance). The researchers have now underscored why further research is needed and have helped readers anticipate the overarching goals of their study (to develop an instrument to measure anesthesiology resident performance). 4  

The Conceptual Framework

Following the literature review and articulation of the problem statement, the next step in clarifying the research purpose is to select a conceptual framework that can be applied to the research topic. Conceptual frameworks are “ways of thinking about a problem or a study, or ways of representing how complex things work.” 3   Just as clinical trials are informed by basic science research in the laboratory, conceptual frameworks often serve as the “basic science” that informs scholarship in medical education. At a fundamental level, conceptual frameworks provide a structured approach to solving the problem identified in the problem statement.

Conceptual frameworks may take the form of theories, principles, or models that help to explain the research problem by identifying its essential variables or elements. Alternatively, conceptual frameworks may represent evidence-based best practices that researchers can apply to an issue identified in the problem statement. 3   Importantly, there is no single best conceptual framework for a particular research topic, although the choice of a conceptual framework is often informed by the literature review and knowing which conceptual frameworks have been used in similar research. 8   For further information on selecting a conceptual framework for research in medical education, we direct readers to the work of Bordage 3   and Irby et al. 9  

To illustrate how different conceptual frameworks can be applied to a research problem, suppose you encounter a study to reduce the frequency of communication errors among anesthesiology residents during day-to-night handoff. Table 2 10 , 11   identifies two different conceptual frameworks researchers might use to approach the task. The first framework, cognitive load theory, has been proposed as a conceptual framework to identify potential variables that may lead to handoff errors. 12   Specifically, cognitive load theory identifies the three factors that affect short-term memory and thus may lead to communication errors:

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Intrinsic load: Inherent complexity or difficulty of the information the resident is trying to learn ( e.g. , complex patients).

Extraneous load: Distractions or demands on short-term memory that are not related to the information the resident is trying to learn ( e.g. , background noise, interruptions).

Germane load: Effort or mental strategies used by the resident to organize and understand the information he/she is trying to learn ( e.g. , teach back, note taking).

Using cognitive load theory as a conceptual framework, researchers may design an intervention to reduce extraneous load and help the resident remember the overnight to-do’s. An example might be dedicated, pager-free handoff times where distractions are minimized.

The second framework identified in table 2 , the I-PASS (Illness severity, Patient summary, Action list, Situational awareness and contingency planning, and Synthesis by receiver) handoff mnemonic, 11   is an evidence-based best practice that, when incorporated as part of a handoff bundle, has been shown to reduce handoff errors on pediatric wards. 13   Researchers choosing this conceptual framework may adapt some or all of the I-PASS elements for resident handoffs in the intensive care unit.

Note that both of the conceptual frameworks outlined above provide researchers with a structured approach to addressing the issue of handoff errors; one is not necessarily better than the other. Indeed, it is possible for researchers to use both frameworks when designing their study. Ultimately, we provide this example to demonstrate the necessity of selecting conceptual frameworks to clarify the research purpose. 3 , 8   Readers should look for conceptual frameworks in the introduction section and should be wary of their omission, as commonly seen in less well-developed medical education research articles. 14  

Statement of Study Intent

After reviewing the literature, articulating the problem statement, and selecting a conceptual framework to address the research topic, the final step in clarifying the research purpose is the statement of study intent. The statement of study intent is arguably the most important element of framing the study because it makes the research purpose explicit. 2   Consider the following example:

This study aimed to test the hypothesis that the introduction of the BASIC Examination was associated with an accelerated knowledge acquisition during residency training, as measured by increments in annual ITE scores. 15  

This statement of study intent succinctly identifies several key study elements including the population (anesthesiology residents), the intervention/independent variable (introduction of the BASIC Examination), the outcome/dependent variable (knowledge acquisition, as measure by in In-training Examination [ITE] scores), and the hypothesized relationship between the independent and dependent variable (the authors hypothesize a positive correlation between the BASIC examination and the speed of knowledge acquisition). 6 , 14  

The statement of study intent will sometimes manifest as a research objective, rather than hypothesis or question. In such instances there may not be explicit independent and dependent variables, but the study population and research aim should be clearly identified. The following is an example:

“In this report, we present the results of 3 [years] of course data with respect to the practice improvements proposed by participating anesthesiologists and their success in implementing those plans. Specifically, our primary aim is to assess the frequency and type of improvements that were completed and any factors that influence completion.” 16  

The statement of study intent is the logical culmination of the literature review, problem statement, and conceptual framework, and is a transition point between the Introduction and Methods sections of a medical education research report. Nonetheless, a systematic review of experimental research in medical education demonstrated that statements of study intent are absent in the majority of articles. 14   When reading a medical education research article where the statement of study intent is absent, it may be necessary to infer the research aim by gathering information from the Introduction and Methods sections. In these cases, it can be useful to identify the following key elements 6 , 14 , 17   :

Population of interest/type of learner ( e.g. , pain medicine fellow or anesthesiology residents)

Independent/predictor variable ( e.g. , educational intervention or characteristic of the learners)

Dependent/outcome variable ( e.g. , intubation skills or knowledge of anesthetic agents)

Relationship between the variables ( e.g. , “improve” or “mitigate”)

Occasionally, it may be difficult to differentiate the independent study variable from the dependent study variable. 17   For example, consider a study aiming to measure the relationship between burnout and personal debt among anesthesiology residents. Do the researchers believe burnout might lead to high personal debt, or that high personal debt may lead to burnout? This “chicken or egg” conundrum reinforces the importance of the conceptual framework which, if present, should serve as an explanation or rationale for the predicted relationship between study variables.

Research methodology is the “…design or plan that shapes the methods to be used in a study.” 1   Essentially, methodology is the general strategy for answering a research question, whereas methods are the specific steps and techniques that are used to collect data and implement the strategy. Our objective here is to provide an overview of quantitative methodologies ( i.e. , approaches) in medical education research.

The choice of research methodology is made by balancing the approach that best answers the research question against the feasibility of completing the study. There is no perfect methodology because each has its own potential caveats, flaws and/or sources of bias. Before delving into an overview of the methodologies, it is important to highlight common sources of bias in education research. We use the term internal validity to describe the degree to which the findings of a research study represent “the truth,” as opposed to some alternative hypothesis or variables. 18   Table 3   18–20   provides a list of common threats to internal validity in medical education research, along with tactics to mitigate these threats.

Threats to Internal Validity and Strategies to Mitigate Their Effects

Experimental Research

The fundamental tenet of experimental research is the manipulation of an independent or experimental variable to measure its effect on a dependent or outcome variable.

True Experiment

True experimental study designs minimize threats to internal validity by randomizing study subjects to experimental and control groups. Through ensuring that differences between groups are—beyond the intervention/variable of interest—purely due to chance, researchers reduce the internal validity threats related to subject characteristics, time-related maturation, and regression to the mean. 18 , 19  

Quasi-experiment

There are many instances in medical education where randomization may not be feasible or ethical. For instance, researchers wanting to test the effect of a new curriculum among medical students may not be able to randomize learners due to competing curricular obligations and schedules. In these cases, researchers may be forced to assign subjects to experimental and control groups based upon some other criterion beyond randomization, such as different classrooms or different sections of the same course. This process, called quasi-randomization, does not inherently lead to internal validity threats, as long as research investigators are mindful of measuring and controlling for extraneous variables between study groups. 19  

Single-group Methodologies

All experimental study designs compare two or more groups: experimental and control. A common experimental study design in medical education research is the single-group pretest–posttest design, which compares a group of learners before and after the implementation of an intervention. 21   In essence, a single-group pre–post design compares an experimental group ( i.e. , postintervention) to a “no-intervention” control group ( i.e. , preintervention). 19   This study design is problematic for several reasons. Consider the following hypothetical example: A research article reports the effects of a year-long intubation curriculum for first-year anesthesiology residents. All residents participate in monthly, half-day workshops over the course of an academic year. The article reports a positive effect on residents’ skills as demonstrated by a significant improvement in intubation success rates at the end of the year when compared to the beginning.

This study does little to advance the science of learning among anesthesiology residents. While this hypothetical report demonstrates an improvement in residents’ intubation success before versus after the intervention, it does not tell why the workshop worked, how it compares to other educational interventions, or how it fits in to the broader picture of anesthesia training.

Single-group pre–post study designs open themselves to a myriad of threats to internal validity. 20   In our hypothetical example, the improvement in residents’ intubation skills may have been due to other educational experience(s) ( i.e. , implementation threat) and/or improvement in manual dexterity that occurred naturally with time ( i.e. , maturation threat), rather than the airway curriculum. Consequently, single-group pre–post studies should be interpreted with caution. 18  

Repeated testing, before and after the intervention, is one strategy that can be used to reduce the some of the inherent limitations of the single-group study design. Repeated pretesting can mitigate the effect of regression toward the mean, a statistical phenomenon whereby low pretest scores tend to move closer to the mean on subsequent testing (regardless of intervention). 20   Likewise, repeated posttesting at multiple time intervals can provide potentially useful information about the short- and long-term effects of an intervention ( e.g. , the “durability” of the gain in knowledge, skill, or attitude).

Observational Research

Unlike experimental studies, observational research does not involve manipulation of any variables. These studies often involve measuring associations, developing psychometric instruments, or conducting surveys.

Association Research

Association research seeks to identify relationships between two or more variables within a group or groups (correlational research), or similarities/differences between two or more existing groups (causal–comparative research). For example, correlational research might seek to measure the relationship between burnout and educational debt among anesthesiology residents, while causal–comparative research may seek to measure differences in educational debt and/or burnout between anesthesiology and surgery residents. Notably, association research may identify relationships between variables, but does not necessarily support a causal relationship between them.

Psychometric and Survey Research

Psychometric instruments measure a psychologic or cognitive construct such as knowledge, satisfaction, beliefs, and symptoms. Surveys are one type of psychometric instrument, but many other types exist, such as evaluations of direct observation, written examinations, or screening tools. 22   Psychometric instruments are ubiquitous in medical education research and can be used to describe a trait within a study population ( e.g. , rates of depression among medical students) or to measure associations between study variables ( e.g. , association between depression and board scores among medical students).

Psychometric and survey research studies are prone to the internal validity threats listed in table 3 , particularly those relating to mortality, location, and instrumentation. 18   Additionally, readers must ensure that the instrument scores can be trusted to truly represent the construct being measured. For example, suppose you encounter a research article demonstrating a positive association between attending physician teaching effectiveness as measured by a survey of medical students, and the frequency with which the attending physician provides coffee and doughnuts on rounds. Can we be confident that this survey administered to medical students is truly measuring teaching effectiveness? Or is it simply measuring the attending physician’s “likability”? Issues related to measurement and the trustworthiness of data are described in detail in the following section on measurement and the related issues of validity and reliability.

Measurement refers to “the assigning of numbers to individuals in a systematic way as a means of representing properties of the individuals.” 23   Research data can only be trusted insofar as we trust the measurement used to obtain the data. Measurement is of particular importance in medical education research because many of the constructs being measured ( e.g. , knowledge, skill, attitudes) are abstract and subject to measurement error. 24   This section highlights two specific issues related to the trustworthiness of data: the validity and reliability of measurements.

Validity regarding the scores of a measurement instrument “refers to the degree to which evidence and theory support the interpretations of the [instrument’s results] for the proposed use of the [instrument].” 25   In essence, do we believe the results obtained from a measurement really represent what we were trying to measure? Note that validity evidence for the scores of a measurement instrument is separate from the internal validity of a research study. Several frameworks for validity evidence exist. Table 4 2 , 22 , 26   represents the most commonly used framework, developed by Messick, 27   which identifies sources of validity evidence—to support the target construct—from five main categories: content, response process, internal structure, relations to other variables, and consequences.

Sources of Validity Evidence for Measurement Instruments

Reliability

Reliability refers to the consistency of scores for a measurement instrument. 22 , 25 , 28   For an instrument to be reliable, we would anticipate that two individuals rating the same object of measurement in a specific context would provide the same scores. 25   Further, if the scores for an instrument are reliable between raters of the same object of measurement, then we can extrapolate that any difference in scores between two objects represents a true difference across the sample, and is not due to random variation in measurement. 29   Reliability can be demonstrated through a variety of methods such as internal consistency ( e.g. , Cronbach’s alpha), temporal stability ( e.g. , test–retest reliability), interrater agreement ( e.g. , intraclass correlation coefficient), and generalizability theory (generalizability coefficient). 22 , 29  

Example of a Validity and Reliability Argument

This section provides an illustration of validity and reliability in medical education. We use the signaling questions outlined in table 4 to make a validity and reliability argument for the Harvard Assessment of Anesthesia Resident Performance (HARP) instrument. 7   The HARP was developed by Blum et al. to measure the performance of anesthesia trainees that is required to provide safe anesthetic care to patients. According to the authors, the HARP is designed to be used “…as part of a multiscenario, simulation-based assessment” of resident performance. 7  

Content Validity: Does the Instrument’s Content Represent the Construct Being Measured?

To demonstrate content validity, instrument developers should describe the construct being measured and how the instrument was developed, and justify their approach. 25   The HARP is intended to measure resident performance in the critical domains required to provide safe anesthetic care. As such, investigators note that the HARP items were created through a two-step process. First, the instrument’s developers interviewed anesthesiologists with experience in resident education to identify the key traits needed for successful completion of anesthesia residency training. Second, the authors used a modified Delphi process to synthesize the responses into five key behaviors: (1) formulate a clear anesthetic plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize one’s limits. 7 , 30  

Response Process Validity: Are Raters Interpreting the Instrument Items as Intended?

In the case of the HARP, the developers included a scoring rubric with behavioral anchors to ensure that faculty raters could clearly identify how resident performance in each domain should be scored. 7  

Internal Structure Validity: Do Instrument Items Measuring Similar Constructs Yield Homogenous Results? Do Instrument Items Measuring Different Constructs Yield Heterogeneous Results?

Item-correlation for the HARP demonstrated a high degree of correlation between some items ( e.g. , formulating a plan and modifying the plan under changing conditions) and a lower degree of correlation between other items ( e.g. , formulating a plan and identifying performance improvement opportunities). 30   This finding is expected since the items within the HARP are designed to assess separate performance domains, and we would expect residents’ functioning to vary across domains.

Relationship to Other Variables’ Validity: Do Instrument Scores Correlate with Other Measures of Similar or Different Constructs as Expected?

As it applies to the HARP, one would expect that the performance of anesthesia residents will improve over the course of training. Indeed, HARP scores were found to be generally higher among third-year residents compared to first-year residents. 30  

Consequence Validity: Are Instrument Results Being Used as Intended? Are There Unintended or Negative Uses of the Instrument Results?

While investigators did not intentionally seek out consequence validity evidence for the HARP, unanticipated consequences of HARP scores were identified by the authors as follows:

“Data indicated that CA-3s had a lower percentage of worrisome scores (rating 2 or lower) than CA-1s… However, it is concerning that any CA-3s had any worrisome scores…low performance of some CA-3 residents, albeit in the simulated environment, suggests opportunities for training improvement.” 30  

That is, using the HARP to measure the performance of CA-3 anesthesia residents had the unintended consequence of identifying the need for improvement in resident training.

Reliability: Are the Instrument’s Scores Reproducible and Consistent between Raters?

The HARP was applied by two raters for every resident in the study across seven different simulation scenarios. The investigators conducted a generalizability study of HARP scores to estimate the variance in assessment scores that was due to the resident, the rater, and the scenario. They found little variance was due to the rater ( i.e. , scores were consistent between raters), indicating a high level of reliability. 7  

Sampling refers to the selection of research subjects ( i.e. , the sample) from a larger group of eligible individuals ( i.e. , the population). 31   Effective sampling leads to the inclusion of research subjects who represent the larger population of interest. Alternatively, ineffective sampling may lead to the selection of research subjects who are significantly different from the target population. Imagine that researchers want to explore the relationship between burnout and educational debt among pain medicine specialists. The researchers distribute a survey to 1,000 pain medicine specialists (the population), but only 300 individuals complete the survey (the sample). This result is problematic because the characteristics of those individuals who completed the survey and the entire population of pain medicine specialists may be fundamentally different. It is possible that the 300 study subjects may be experiencing more burnout and/or debt, and thus, were more motivated to complete the survey. Alternatively, the 700 nonresponders might have been too busy to respond and even more burned out than the 300 responders, which would suggest that the study findings were even more amplified than actually observed.

When evaluating a medical education research article, it is important to identify the sampling technique the researchers employed, how it might have influenced the results, and whether the results apply to the target population. 24  

Sampling Techniques

Sampling techniques generally fall into two categories: probability- or nonprobability-based. Probability-based sampling ensures that each individual within the target population has an equal opportunity of being selected as a research subject. Most commonly, this is done through random sampling, which should lead to a sample of research subjects that is similar to the target population. If significant differences between sample and population exist, those differences should be due to random chance, rather than systematic bias. The difference between data from a random sample and that from the population is referred to as sampling error. 24  

Nonprobability-based sampling involves selecting research participants such that inclusion of some individuals may be more likely than the inclusion of others. 31   Convenience sampling is one such example and involves selection of research subjects based upon ease or opportuneness. Convenience sampling is common in medical education research, but, as outlined in the example at the beginning of this section, it can lead to sampling bias. 24   When evaluating an article that uses nonprobability-based sampling, it is important to look for participation/response rate. In general, a participation rate of less than 75% should be viewed with skepticism. 21   Additionally, it is important to determine whether characteristics of participants and nonparticipants were reported and if significant differences between the two groups exist.

Interpreting medical education research requires a basic understanding of common ways in which quantitative data are analyzed and displayed. In this section, we highlight two broad topics that are of particular importance when evaluating research articles.

The Nature of the Measurement Variable

Measurement variables in quantitative research generally fall into three categories: nominal, ordinal, or interval. 24   Nominal variables (sometimes called categorical variables) involve data that can be placed into discrete categories without a specific order or structure. Examples include sex (male or female) and professional degree (M.D., D.O., M.B.B.S., etc .) where there is no clear hierarchical order to the categories. Ordinal variables can be ranked according to some criterion, but the spacing between categories may not be equal. Examples of ordinal variables may include measurements of satisfaction (satisfied vs . unsatisfied), agreement (disagree vs . agree), and educational experience (medical student, resident, fellow). As it applies to educational experience, it is noteworthy that even though education can be quantified in years, the spacing between years ( i.e. , educational “growth”) remains unequal. For instance, the difference in performance between second- and third-year medical students is dramatically different than third- and fourth-year medical students. Interval variables can also be ranked according to some criteria, but, unlike ordinal variables, the spacing between variable categories is equal. Examples of interval variables include test scores and salary. However, the conceptual boundaries between these measurement variables are not always clear, as in the case where ordinal scales can be assumed to have the properties of an interval scale, so long as the data’s distribution is not substantially skewed. 32  

Understanding the nature of the measurement variable is important when evaluating how the data are analyzed and reported. Medical education research commonly uses measurement instruments with items that are rated on Likert-type scales, whereby the respondent is asked to assess their level of agreement with a given statement. The response is often translated into a corresponding number ( e.g. , 1 = strongly disagree, 3 = neutral, 5 = strongly agree). It is remarkable that scores from Likert-type scales are sometimes not normally distributed ( i.e. , are skewed toward one end of the scale), indicating that the spacing between scores is unequal and the variable is ordinal in nature. In these cases, it is recommended to report results as frequencies or medians, rather than means and SDs. 33  

Consider an article evaluating medical students’ satisfaction with a new curriculum. Researchers measure satisfaction using a Likert-type scale (1 = very unsatisfied, 2 = unsatisfied, 3 = neutral, 4 = satisfied, 5 = very satisfied). A total of 20 medical students evaluate the curriculum, 10 of whom rate their satisfaction as “satisfied,” and 10 of whom rate it as “very satisfied.” In this case, it does not make much sense to report an average score of 4.5; it makes more sense to report results in terms of frequency ( e.g. , half of the students were “very satisfied” with the curriculum, and half were not).

Effect Size and CIs

In medical education, as in other research disciplines, it is common to report statistically significant results ( i.e. , small P values) in order to increase the likelihood of publication. 34 , 35   However, a significant P value in itself does necessarily represent the educational impact of the study results. A statement like “Intervention x was associated with a significant improvement in learners’ intubation skill compared to education intervention y ( P < 0.05)” tells us that there was a less than 5% chance that the difference in improvement between interventions x and y was due to chance. Yet that does not mean that the study intervention necessarily caused the nonchance results, or indicate whether the between-group difference is educationally significant. Therefore, readers should consider looking beyond the P value to effect size and/or CI when interpreting the study results. 36 , 37  

Effect size is “the magnitude of the difference between two groups,” which helps to quantify the educational significance of the research results. 37   Common measures of effect size include Cohen’s d (standardized difference between two means), risk ratio (compares binary outcomes between two groups), and Pearson’s r correlation (linear relationship between two continuous variables). 37   CIs represent “a range of values around a sample mean or proportion” and are a measure of precision. 31   While effect size and CI give more useful information than simple statistical significance, they are commonly omitted from medical education research articles. 35   In such instances, readers should be wary of overinterpreting a P value in isolation. For further information effect size and CI, we direct readers the work of Sullivan and Feinn 37   and Hulley et al. 31  

In this final section, we identify instruments that can be used to evaluate the quality of quantitative medical education research articles. To this point, we have focused on framing the study and research methodologies and identifying potential pitfalls to consider when appraising a specific article. This is important because how a study is framed and the choice of methodology require some subjective interpretation. Fortunately, there are several instruments available for evaluating medical education research methods and providing a structured approach to the evaluation process.

The Medical Education Research Study Quality Instrument (MERSQI) 21   and the Newcastle Ottawa Scale-Education (NOS-E) 38   are two commonly used instruments, both of which have an extensive body of validity evidence to support the interpretation of their scores. Table 5 21 , 39   provides more detail regarding the MERSQI, which includes evaluation of study design, sampling, data type, validity, data analysis, and outcomes. We have found that applying the MERSQI to manuscripts, articles, and protocols has intrinsic educational value, because this practice of application familiarizes MERSQI users with fundamental principles of medical education research. One aspect of the MERSQI that deserves special mention is the section on evaluating outcomes based on Kirkpatrick’s widely recognized hierarchy of reaction, learning, behavior, and results ( table 5 ; fig .). 40   Validity evidence for the scores of the MERSQI include its operational definitions to improve response process, excellent reliability, and internal consistency, as well as high correlation with other measures of study quality, likelihood of publication, citation rate, and an association between MERSQI score and the likelihood of study funding. 21 , 41   Additionally, consequence validity for the MERSQI scores has been demonstrated by its utility for identifying and disseminating high-quality research in medical education. 42  

Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007. 2  

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The NOS-E is a newer tool to evaluate the quality of medication education research. It was developed as a modification of the Newcastle-Ottawa Scale 43   for appraising the quality of nonrandomized studies. The NOS-E includes items focusing on the representativeness of the experimental group, selection and compatibility of the control group, missing data/study retention, and blinding of outcome assessors. 38 , 39   Additional validity evidence for NOS-E scores includes operational definitions to improve response process, excellent reliability and internal consistency, and its correlation with other measures of study quality. 39   Notably, the complete NOS-E, along with its scoring rubric, can found in the article by Cook and Reed. 39  

A recent comparison of the MERSQI and NOS-E found acceptable interrater reliability and good correlation between the two instruments 39   However, noted differences exist between the MERSQI and NOS-E. Specifically, the MERSQI may be applied to a broad range of study designs, including experimental and cross-sectional research. Additionally, the MERSQI addresses issues related to measurement validity and data analysis, and places emphasis on educational outcomes. On the other hand, the NOS-E focuses specifically on experimental study designs, and on issues related to sampling techniques and outcome assessment. 39   Ultimately, the MERSQI and NOS-E are complementary tools that may be used together when evaluating the quality of medical education research.

Conclusions

This article provides an overview of quantitative research in medical education, underscores the main components of education research, and provides a general framework for evaluating research quality. We highlighted the importance of framing a study with respect to purpose, conceptual framework, and statement of study intent. We reviewed the most common research methodologies, along with threats to the validity of a study and its measurement instruments. Finally, we identified two complementary instruments, the MERSQI and NOS-E, for evaluating the quality of a medical education research study.

Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312–9.

Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7–166. e116.

Franenkel JR, Wallen NE, Hyun HH: How to Design and Evaluate Research in Education. 9th edition. New York, McGraw-Hill Education, 2015.

Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research. 4th edition. Philadelphia, Lippincott Williams & Wilkins, 2011.

Irby BJ, Brown G, Lara-Alecio R, Jackson S: The Handbook of Educational Theories. Charlotte, NC, Information Age Publishing, Inc., 2015

Standards for Educational and Psychological Testing (American Educational Research Association & American Psychological Association, 2014)

Swanwick T: Understanding medical education: Evidence, theory and practice, 2nd edition. Wiley-Blackwell, 2013.

Sullivan GM, Artino Jr AR: Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013; 5(4):541–2.

Sullivan GM, Feinn R: Using effect size—or why the P value is not enough. Journal of graduate medical education. 2012; 4(3):279–82.

Tavakol M, Sandars J: Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical teacher. 2014; 36(10):838–48.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

We Trust in Human Precision

20,000+ Professional Language Experts Ready to Help. Expertise in a variety of Niches.

API Solutions

• API Pricing
• Cost estimate
• Customer loyalty program
• Educational Discount
• Non-Profit Discount
• Green Initiative Discount1

Value-Driven Pricing

Unmatched expertise at affordable rates tailored for your needs. Our services empower you to boost your productivity.

• Special Discounts
• Enterprise transcription solutions
• Enterprise translation solutions
• Transcription/Caption API
• AI Transcription Proofreading API

Trusted by Global Leaders

GoTranscript is the chosen service for top media organizations, universities, and Fortune 50 companies.

GoTranscript

One of the Largest Online Transcription and Translation Agencies in the World. Founded in 2005.

Speaker 1: Welcome to this overview of quantitative research methods. This tutorial will give you the big picture of quantitative research and introduce key concepts that will help you determine if quantitative methods are appropriate for your project study. First, what is educational research? Educational research is a process of scholarly inquiry designed to investigate the process of instruction and learning, the behaviors, perceptions, and attributes of students and teachers, the impact of institutional processes and policies, and all other areas of the educational process. The research design may be quantitative, qualitative, or a mixed methods design. The focus of this overview is quantitative methods. The general purpose of quantitative research is to explain, predict, investigate relationships, describe current conditions, or to examine possible impacts or influences on designated outcomes. Quantitative research differs from qualitative research in several ways. It works to achieve different goals and uses different methods and design. This table illustrates some of the key differences. Qualitative research generally uses a small sample to explore and describe experiences through the use of thick, rich descriptions of detailed data in an attempt to understand and interpret human perspectives. It is less interested in generalizing to the population as a whole. For example, when studying bullying, a qualitative researcher might learn about the experience of the victims and the experience of the bully by interviewing both bullies and victims and observing them on the playground. Quantitative studies generally use large samples to test numerical data by comparing or finding correlations among sample attributes so that the findings can be generalized to the population. If quantitative researchers were studying bullying, they might measure the effects of a bully on the victim by comparing students who are victims and students who are not victims of bullying using an attitudinal survey. In conducting quantitative research, the researcher first identifies the problem. For Ed.D. research, this problem represents a gap in practice. For Ph.D. research, this problem represents a gap in the literature. In either case, the problem needs to be of importance in the professional field. Next, the researcher establishes the purpose of the study. Why do you want to do the study, and what do you intend to accomplish? This is followed by research questions which help to focus the study. Once the study is focused, the researcher needs to review both seminal works and current peer-reviewed primary sources. Based on the research question and on a review of prior research, a hypothesis is created that predicts the relationship between the study's variables. Next, the researcher chooses a study design and methods to test the hypothesis. These choices should be informed by a review of methodological approaches used to address similar questions in prior research. Finally, appropriate analytical methods are used to analyze the data, allowing the researcher to draw conclusions and inferences about the data, and answer the research question that was originally posed. In quantitative research, research questions are typically descriptive, relational, or causal. Descriptive questions constrain the researcher to describing what currently exists. With a descriptive research question, one can examine perceptions or attitudes as well as more concrete variables such as achievement. For example, one might describe a population of learners by gathering data on their age, gender, socioeconomic status, and attributes towards their learning experiences. Relational questions examine the relationship between two or more variables. The X variable has some linear relationship to the Y variable. Causal inferences cannot be made from this type of research. For example, one could study the relationship between students' study habits and achievements. One might find that students using certain kinds of study strategies demonstrate greater learning, but one could not state conclusively that using certain study strategies will lead to or cause higher achievement. Causal questions, on the other hand, are designed to allow the researcher to draw a causal inference. A causal question seeks to determine if a treatment variable in a program had an effect on one or more outcome variables. In other words, the X variable influences the Y variable. For example, one could design a study that answered the question of whether a particular instructional approach caused students to learn more. The research question serves as a basis for posing a hypothesis, a predicted answer to the research question that incorporates operational definitions of the study's variables and is rooted in the literature. An operational definition matches a concept with a method of measurement, identifying how the concept will be quantified. For example, in a study of instructional strategies, the hypothesis might be that students of teachers who use Strategy X will exhibit greater learning than students of teachers who do not. In this study, one would need to operationalize learning by identifying a test or instrument that would measure learning. This approach allows the researcher to create a testable hypothesis. Relational and causal research relies on the creation of a null hypothesis, a version of the research hypothesis that predicts no relationship between variables or no effect of one variable on another. When writing the hypothesis for a quantitative question, the null hypothesis and the research or alternative hypothesis use parallel sentence structure. In this example, the null hypothesis states that there will be no statistical difference between groups, while the research or alternative hypothesis states that there will be a statistical difference between groups. Note also that both hypothesis statements operationalize the critical thinking skills variable by identifying the measurement instrument to be used. Once the research questions and hypotheses are solidified, the researcher must select a design that will create a situation in which the hypotheses can be tested and the research questions answered. Ideally, the research design will isolate the study's variables and control for intervening variables so that one can be certain of the relationships being tested. In educational research, however, it is extremely difficult to establish sufficient controls in the complex social settings being studied. In our example of investigating the impact of a certain instructional strategy in the classroom on student achievement, each day the teacher uses a specific instructional strategy. After school, some of the students in her class receive tutoring. Other students have parents that are very involved in their child's academic progress and provide learning experiences in the home. These students may do better because they received extra help, not because the teacher's instructional strategy is more effective. Unless the researcher can control for the intervening variable of extra help, it will be impossible to effectively test the study's hypothesis. Quantitative research designs can fall into two broad categories, experimental and quasi-experimental. Classic experimental designs are those that randomly assign subjects to either a control or treatment comparison group. The researcher can then compare the treatment group to the control group to test for an intervention's effect, known as a between-subject design. It is important to note that the control group may receive a standard treatment or may receive a treatment of any kind. Quasi-experimental designs do not randomly assign subjects to groups, but rather take advantage of existing groups. A researcher can still have a control and comparison group, but assignment to the groups is not random. The use of a control group is not required. However, the researcher may choose a design in which a single group is pre- and post-tested, known as a within-subjects design. Or a single group may receive only a post-test. Since quasi-experimental designs lack random assignment, the researcher should be aware of the threats to validity. Educational research often attempts to measure abstract variables such as attitudes, beliefs, and feelings. Surveys can capture data about these hard-to-measure variables, as well as other self-reported information such as demographic factors. A survey is an instrument used to collect verifiable information from a sample population. In quantitative research, surveys typically include questions that ask respondents to choose a rating from a scale, select one or more items from a list, or other responses that result in numerical data. Studies that use surveys or tests need to include strategies that establish the validity of the instrument used. There are many types of validity that need to be addressed. Face validity. Does the test appear at face value to measure what it is supposed to measure? Content validity. Content validity includes both item validity and sampling validity. Item validity ensures that the individual test items deal only with the subject being addressed. Sampling validity ensures that the range of item topics is appropriate to the subject being studied. For example, item validity might be high, but if all the items only deal with one aspect of the subjects, then sampling validity is low. Content validity can be established by having experts in the field review the test. Concurrent validity. Does a new test correlate with an older, established test that measures the same thing? Predictive validity. Does the test correlate with another related measure? For example, GRE tests are used at many colleges because these schools believe that a good grade on this test increases the probability that the student will do well at the college. Linear regression can establish the predictive validity of a test. Construct validity. Does the test measure the construct it is intended to measure? Establishing construct validity can be a difficult task when the constructs being measured are abstract. But it can be established by conducting a number of studies in which you test hypotheses regarding the construct, or by completing a factor analysis to ensure that you have the number of constructs that you say you have. In addition to ensuring the validity of instruments, the quantitative researcher needs to establish their reliability as well. Strategies for establishing reliability include Test retest. Correlates scores from two different administrations of the same test. Alternate forms. Correlates scores from administrations of two different forms of the same test. Split half reliability. Treats each half of one test or survey as a separate administration and correlates the results from each. Internal consistency. Uses Cronbach's coefficient alpha to calculate the average of all possible split halves. Quantitative research almost always relies on a sample that is intended to be representative of a larger population. There are two basic sampling strategies, random and non-random, and a number of specific strategies within each of these approaches. This table provides examples of each of the major strategies. The next section of this tutorial provides an overview of the procedures in conducting quantitative data analysis. There are specific procedures for conducting the data collection, preparing for and analyzing data, presenting the findings, and connecting to the body of existing research. This process ensures that the research is conducted as a systematic investigation that leads to credible results. Data comes in various sizes and shapes, and it is important to know about these so that the proper analysis can be used on the data. In 1946, S.S. Stevens first described the properties of measurement systems that allowed decisions about the type of measurement and about the attributes of objects that are preserved in numbers. These four types of data are referred to as nominal, ordinal, interval, and ratio. First, let's examine nominal data. With nominal data, there is no number value that indicates quantity. Instead, a number has been assigned to represent a certain attribute, like the number 1 to represent male and the number 2 to represent female. In other words, the number is just a label. You could also assign numbers to represent race, religion, or any other categorical information. Nominal data only denotes group membership. With ordinal data, there is again no indication of quantity. Rather, a number is assigned for ranking order. For example, satisfaction surveys often ask respondents to rank order their level of satisfaction with services or programs. The next level of measurement is interval data. With interval data, there are equal distances between two values, but there is no natural zero. A common example is the Fahrenheit temperature scale. Differences between the temperature measurements make sense, but ratios do not. For instance, 20 degrees Fahrenheit is not twice as hot as 10 degrees Fahrenheit. You can add and subtract interval level data, but they cannot be divided or multiplied. Finally, we have ratio data. Ratio is the same as interval, however ratios, means, averages, and other numerical formulas are all possible and make sense. Zero has a logical meaning, which shows the absence of, or having none of. Examples of ratio data are height, weight, speed, or any quantities based on a scale with a natural zero. In summary, nominal data can only be counted. Ordinal data can be counted and ranked. Interval data can also be added and subtracted, and ratio data can also be used in ratios and other calculations. Determining what type of data you have is one of the most important aspects of quantitative analysis. Depending on the research question, hypotheses, and research design, the researcher may choose to use descriptive and or inferential statistics to begin to analyze the data. Descriptive statistics are best illustrated when viewed through the lens of America's pastimes. Sports, weather, economy, stock market, and even our retirement portfolio are presented in a descriptive analysis. Basic terminology for descriptive statistics are terms that we are most familiar in this discipline. Frequency, mean, median, mode, range, variance, and standard deviation. Simply put, you are describing the data. Some of the most common graphic representations of data are bar graphs, pie graphs, histograms, and box and whisker graphs. Attempting to reach conclusions and make causal inferences beyond graphic representations or descriptive analyses is referred to as inferential statistics. In other words, examining the college enrollment of the past decade in a certain geographical region would assist in estimating what the enrollment for the next year might be. Frequently in education, the means of two or more groups are compared. When comparing means to assist in answering a research question, one can use a within-group, between-groups, or mixed-subject design. In a within-group design, the researcher compares measures of the same subjects across time, therefore within-group, or under different treatment conditions. This can also be referred to as a dependent-group design. The most basic example of this type of quasi-experimental design would be if a researcher conducted a pretest of a group of students, subjected them to a treatment, and then conducted a post-test. The group has been measured at different points in time. In a between-group design, subjects are assigned to one of the two or more groups. For example, Control, Treatment 1, Treatment 2. Ideally, the sampling and assignment to groups would be random, which would make this an experimental design. The researcher can then compare the means of the treatment group to the control group. When comparing two groups, the researcher can gain insight into the effects of the treatment. In a mixed-subjects design, the researcher is testing for significant differences between two or more independent groups while subjecting them to repeated measures. Choosing a statistical test to compare groups depends on the number of groups, whether the data are nominal, ordinal, or interval, and whether the data meet the assumptions for parametric tests. Nonparametric tests are typically used with nominal and ordinal data, while parametric tests use interval and ratio-level data. In addition to this, some further assumptions are made for parametric tests that the data are normally distributed in the population, that participant selection is independent, and the selection of one person does not determine the selection of another, and that the variances of the groups being compared are equal. The assumption of independent participant selection cannot be violated, but the others are more flexible. The t-test assesses whether the means of two groups are statistically different from each other. This analysis is appropriate whenever you want to compare the means of two groups, and especially appropriate as the method of analysis for a quasi-experimental design. When choosing a t-test, the assumptions are that the data are parametric. The analysis of variance, or ANOVA, assesses whether the means of more than two groups are statistically different from each other. When choosing an ANOVA, the assumptions are that the data are parametric. The chi-square test can be used when you have non-parametric data and want to compare differences between groups. The Kruskal-Wallis test can be used when there are more than two groups and the data are non-parametric. Correlation analysis is a set of statistical tests to determine whether there are linear relationships between two or more sets of variables from the same list of items or individuals, for example, achievement and performance of students. The tests provide a statistical yes or no as to whether a significant relationship or correlation exists between the variables. A correlation test consists of calculating a correlation coefficient between two variables. Again, there are parametric and non-parametric choices based on the assumptions of the data. Pearson R correlation is widely used in statistics to measure the strength of the relationship between linearly related variables. Spearman-Rank correlation is a non-parametric test that is used to measure the degree of association between two variables. Spearman-Rank correlation test does not assume any assumptions about the distribution. Spearman-Rank correlation test is used when the Pearson test gives misleading results. Often a Kendall-Taw is also included in this list of non-parametric correlation tests to examine the strength of the relationship if there are less than 20 rankings. Linear regression and correlation are similar and often confused. Sometimes your methodologist will encourage you to examine both the calculations. Calculate linear correlation if you measured both variables, x and y. Make sure to use the Pearson parametric correlation coefficient if you are certain you are not violating the test assumptions. Otherwise, choose the Spearman non-parametric correlation coefficient. If either variable has been manipulated using an intervention, do not calculate a correlation. While linear regression does indicate the nature of the relationship between two variables, like correlation, it can also be used to make predictions because one variable is considered explanatory while the other is considered a dependent variable. Establishing validity is a critical part of quantitative research. As with the nature of quantitative research, there is a defined approach or process for establishing validity. This also allows for the findings transferability. For a study to be valid, the evidence must support the interpretations of the data, the data must be accurate, and their use in drawing conclusions must be logical and appropriate. Construct validity concerns whether what you did for the program was what you wanted to do, or whether what you observed was what you wanted to observe. Construct validity concerns whether the operationalization of your variables are related to the theoretical concepts you are trying to measure. Are you actually measuring what you want to measure? Internal validity means that you have evidence that what you did in the study, i.e., the program, caused what you observed, i.e., the outcome, to happen. Conclusion validity is the degree to which conclusions drawn about relationships in the data are reasonable. External validity concerns the process of generalizing, or the degree to which the conclusions in your study would hold for other persons in other places and at other times. Establishing reliability and validity to your study is one of the most critical elements of the research process. Once you have decided to embark upon the process of conducting a quantitative study, use the following steps to get started. First, review research studies that have been conducted on your topic to determine what methods were used. Consider the strengths and weaknesses of the various data collection and analysis methods. Next, review the literature on quantitative research methods. Every aspect of your research has a body of literature associated with it. Just as you would not confine yourself to your course textbooks for your review of research on your topic, you should not limit yourself to your course texts for your review of methodological literature. Read broadly and deeply from the scholarly literature to gain expertise in quantitative research. Additional self-paced tutorials have been developed on different methodologies and techniques associated with quantitative research. Make sure that you complete all of the self-paced tutorials and review them as often as needed. You will then be prepared to complete a literature review of the specific methodologies and techniques that you will use in your study. Thank you for watching.