quality improvement in healthcare research paper

The Journal for Healthcare Quality (JHQ)

• Subscribe to journal Subscribe
• Get new issue alerts Get alerts
• Become a NAHQ Member

Secondary Logo

Journal logo, current issue.

July/August 2024 - Volume 46 - Issue 4

• Editor-in-Chief: Cathy E. Duquette, PhD, MBA, RN, CPHQ, FNAHQ
• ISSN: 1062-2551
• Online ISSN: 1945-1474
• Frequency: 6 per year
• Ranking: Health Care Sciences & Services: 97/105; Health Policy & Services 82/87
• Impact Factor: 0.9

Call for Papers “Developing Workforce Competency to Deliver on Value”

The Journal for Healthcare Quality (JHQ) is seeking high-quality original research, robust quality improvement, and systematic review manuscripts for the 2021 special issue, which will focus on workforce competency.  Please download our overview for more information.

Call for Papers

​ JHQ is seeking high-quality articles for a special issue focused on strategies to reduce health disparities and improve health equity. Submission deadline is Decemnber 10th. Download our Call for Pa​pers​ to learn more.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Current Issue Highlights

• Original Article

Differential Mortality Among Heart Failure Patients Across Different COVID-19 Surges in New York City

Journal for Healthcare Quality. 46(4):197-202, July/August 2024.

• Abstract Abstract
• Permissions

Quality and Safety in Nursing: Recommendations From a Systematic Review

Journal for Healthcare Quality. 46(4):203-219, July/August 2024.

Go to Full Text of this Article

Iron Deficiency Among Hospitalized Patients With Congestive Heart Failure

Journal for Healthcare Quality. 46(4):220-227, July/August 2024.

Improving First Case Operating Room Efficiency

Journal for Healthcare Quality. 46(4):228-234, July/August 2024.

EHR Smart Phrases Used as Enrollment Mechanism in Diabetes Self-Management Support Programs: Preliminary Outcomes

Journal for Healthcare Quality. 46(4):235-244, July/August 2024.

Multimodal Quality Initiatives in Sepsis Care: Assessing Impact on Core Measures and Outcomes

Journal for Healthcare Quality. 46(4):245-250, July/August 2024.

Ambulatory Quality Improvement Despite COVID-19: Blueprint for a Successful System for Continuous Improvement

Journal for Healthcare Quality. 46(4):251-258, July/August 2024.

Implementation of a Patient-Reported Outcome Measure: A Quality Improvement Project

Journal for Healthcare Quality. 46(4):e20-e25, July/August 2024.

Pilot Process Evaluation of the Supporting Older Adults at Risk Model: A RE-AIM Approach

Journal for Healthcare Quality. 46(4):e26-e39, July/August 2024.

Implementing Behavioral Optimization and Outcomes Support Team in a Medical/Surgical Telemetry Unit

Journal for Healthcare Quality. 46(4):e40-e48, July/August 2024.

Quality Coordinators’ Perspectives on Quality Improvement in Primary Healthcare in Kosovo: A Qualitative Study

Journal for Healthcare Quality. 46(4):e49-e55, July/August 2024.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Research and Quality Improvement: How Can They Work Together?

Affiliations.

• 1 Director, Data Science, Quality Insights, Williamsburg, VA.
• 2 Chair, ANNA Research Committee.
• 3 President, ANNA's Old Dominion Chapter.
• 4 Instructor, Case Western Reserve University, Cleveland, OH.
• 5 Associate Degree Nursing Instructor, Northeast Wisconsin Technical College, Green Bay, WI.
• PMID: 35503694

Research and quality improvement provide a mechanism to support the advancement of knowledge, and to evaluate and learn from experience. The focus of research is to contribute to developing knowledge or gather evidence for theories in a field of study, whereas the focus of quality improvement is to standardize processes and reduce variation to improve outcomes for patients and health care organizations. Both methods of inquiry broaden our knowledge through the generation of new information and the application of findings to practice. This article in the "Exploring the Evidence: Focusing on the Fundamentals" series provides nephrology nurses with basic information related to the role of research and quality improvement projects, as well as some examples of ways in which they have been used together to advance clinical knowledge and improve patient outcomes.

Keywords: kidney disease; nephrology; quality improvement; research.

Copyright© by the American Nephrology Nurses Association.

PubMed Disclaimer

Conflict of interest statement

The authors reported no actual or potential conflict of interest in relation to this nursing continuing professional development (NCPD) activity.

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Improving care coordination between nephrology and primary care: a quality improvement initiative using the renal physicians association toolkit. Haley WE, Beckrich AL, Sayre J, McNeil R, Fumo P, Rao VM, Lerma EV. Haley WE, et al. Am J Kidney Dis. 2015 Jan;65(1):67-79. doi: 10.1053/j.ajkd.2014.06.031. Epub 2014 Aug 30. Am J Kidney Dis. 2015. PMID: 25183380
• Quality improvement in pediatric nephrology-a practical guide. Gaudreault-Tremblay MM, McQuillan RF, Parekh RS, Noone D. Gaudreault-Tremblay MM, et al. Pediatr Nephrol. 2020 Feb;35(2):199-211. doi: 10.1007/s00467-018-4175-0. Epub 2019 Jan 5. Pediatr Nephrol. 2020. PMID: 30612204 Review.
• The Health and Safety of Nephrology Nurses and the Environments in Which They Work: Important for Nurses, Patients, and Organizations. Ulrich BT, Kear TM. Ulrich BT, et al. Nephrol Nurs J. 2018 Mar-Apr;45(2):117-168. Nephrol Nurs J. 2018. PMID: 30303636
• The learning health system for pediatric nephrology: building better systems to improve health. Varnell CD Jr, Margolis P, Goebel J, Hooper DK. Varnell CD Jr, et al. Pediatr Nephrol. 2023 Jan;38(1):35-46. doi: 10.1007/s00467-022-05526-0. Epub 2022 Apr 20. Pediatr Nephrol. 2023. PMID: 35445971 Free PMC article. Review.
• Search in MeSH

Grants and funding

• K23 NR019744/NR/NINR NIH HHS/United States

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals

You are here

• Volume 11, Issue 2
• Quality improvement research: understanding the science of change in health care
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• R Baker 1 ,
• 1 Guest Editors, Quality Improvement Research Series
• 2 Editor in Chief, Quality and Safety in Health Care
• Correspondence to:
 Professor R Grol, Centre for Research on Quality in Health Care (WOK), PO Box 9101, 6500 HB Nijmegen, The Netherlands;
 R.Grol{at}hsv.kun.nl

https://doi.org/10.1136/qhc.11.2.110

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• quality improvement research
• change management

Essential for all who want to improve health care.

Expectations of healthcare services are ever increasing and those delivering care no longer hold the monopoly of opinion on what constitutes good or best care. To earn the label “good enough”, care must meet standards expected by consumers as well those of expert providers. Headlines in newspapers, statements in policy documents, and many analyses, surveys and reports repeatedly highlight serious problems in healthcare delivery related to underuse, overuse, or misuse of care. 1 Health systems are sometimes unsafe and frequently we harm patients who have trusted us with their care. There is an endemic failure to engage patients with decisions about their care. We know there are problems; we just need to change so that care can be made safer and better.

Everyone—authorities, policy makers, and professionals—seems to accept the need for change. New initiatives aiming to cure our ailing systems come in droves. This is an international phenomenon. Many initiatives are linked to programmes that capture a particular approach—for example, evidence based medicine; accreditation and (external) accountability; total quality management; professional development and revalidation; risk management and error prevention; organisational development and leadership enhancement; disease management and managed care; complex adaptive systems; and patient empowerment. They may differ in perspective. Some focus on changing professionals, others on changing organisations or interactions between parts of the system; some emphasise self-regulation, others external control and incentives; some advocate “bottom up” and others “top down” methods. Despite their differences, however, each aims to contribute to better patient care—and they might, but the evidence for understanding their likely impact is not robust and many seem based more on belief than rigorous research of value, efficacy, or feasibility. 2 From what we know, no quality improvement programme is superior and real sustainable improvement might require implementation of some aspects of several approaches—perhaps together, perhaps consecutively. We just do not know which to use, when to use them, or what to expect.

More evidence and understanding is required. At least 40 good systematic reviews and numerous controlled trials are available, 3, 4 but many of the trials can be criticised because, for example, randomisation or analysis was conducted at the patient level while the intervention focused on professionals or teams, and outcome parameters are often poorly chosen or are difficult to compare. Most studies were conducted in the USA, limiting generalisations to other systems. Some strategies are better studied than others. We know more about CME, audit and feedback, reminders and computerised decision support than about organisational, economic, administrative and patient mediated interventions. New methods including the effects of problem based education or portfolio learning, TQM, breakthrough projects, risk management methods, business process redesign, leadership enhancement, or sharing decisions with patients are not well studied. Studying the effects of specific strategies in controlled trials will provide some answers to some questions about effective change, but will not address some of the basic questions about the critical success factors in change processes. They need to be complemented by observational and qualitative studies.

Health care is becoming increasingly complex and the problems are large. It is unrealistic to expect that one specific approach can solve everything. A qualitative study by Solberg et al 5 of critical factors supporting implementation of change showed that a mixture of professional and organisational factors is crucial. “Give attention to many different factors and use multiple strategies” is the message. 6 Although we may know that multifaceted strategies combining different actions and measures linked to specific obstacles to change are usually more successful than single interventions, 7 we know little about which components of such complex interventions are effective in different target groups. So, while there is some general knowledge, there is little detailed understanding of the “black box” of change.

We need to learn about change in the real world of health care and the crucial determinants of successful improvement. New thinking about healthcare settings as complex adaptive systems emphasises the importance of experimenting with multiple approaches and discovering what works best. 8 Small changes can sometimes have large effects—but we have little understanding about which small changes to use in which settings and their likely impact.

For real change and sustained improvement a tailored research methodology is essential. The full range of methodology has yet to be established, but will include contributions from epidemiology, behavioural sciences, educational research, organisational and management studies, economics, and statistics (box 1). Theoretical models of evaluations of complex interventions propose a phased approach (theoretical phase, definition of the components of the intervention, small scale explanatory trial, followed by larger trials and research into long term implementation). Clearly, different research methods are required for different phases, 9 but it is essential that, despite the eclectic base of the research, researchers from different faculties and disciplines come together to collaborate in this complex field and that the vogue for “quick fixes” is replaced with sustained research.

Box 1 Some research approaches for quality improvement research

Observational studies of existing change processes

In-depth qualitative studies on critical success factors and barriers to change improvement programmes

Systematic reviews of both the impact of different strategies and the influence of specific factors on change

Well designed cluster randomised trials

Systematic sampling and interpretation of experiences of change

Methods for developing valid and sensitive indicators for measuring change

Meta-analyses of large samples of improvement projects

Methods for evaluation of large scale implementation and change programmes

Economic analyses of resources needed for effective change and improvement of care

Statistical process control

To stimulate and support debate about research on quality improvement and change management in health care we have commissioned a series of papers to provide an overview of some relevant methodologies. The first two papers are published in this issue and more will follow. Pope et al 10 explore some of the qualitative methods that can be used to gather information about the delivery of good quality care, and Wensing and Elwyn 11 consider some of the key issues related to measurement of patients' views. Forthcoming issues of QSHC will include papers that describe research methods for indicator development in primary care; a methodology for evaluating small scale improvement projects; methods for evaluating quality improvement programmes; research designs for randomised controlled trials in quality improvement; and economic evaluations of change management.

There is a recognised process for the development of new drugs, their introduction into routine practice, and their establishment in the treatment of defined conditions. As knowledge about a drug is accrued, new and better patterns of treatment gradually become established. Similar measured approaches are needed to help develop and establish better, safer systems of care. “Change management” is a discipline central to health care. The academic base that supports change management and quality improvement in health care should underpin all clinical and managerial learning programmes. The science of change management is not new, but there is a long way to go before we will understand enough to be able to transform care so that it is “good enough” to meet everyone's expectations of quality and safety.

• ↵ Bodenheimer T . The American health care system. The movement for improved quality in health care. N Engl J Med 1999 ; 340 : 488 –92. OpenUrl CrossRef PubMed Web of Science
• ↵ Grol R . Beliefs and evidence in changing clinical practice. BMJ 1997 ; 315 : 418 –21. OpenUrl FREE Full Text
• ↵ Grol R . Improving the quality of medical care. Building bridges among professional price, payer profit, and patient satisfaction. JAMA 2001 ; 286 : 2578 –85. OpenUrl CrossRef PubMed Web of Science
• ↵ Grimshaw JM , Shirran L, Thomas R, et al . Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001 ; 39 (8 Suppl 2): II2 –45. OpenUrl CrossRef PubMed Web of Science
• ↵ Solberg L , Brekke M, Fasio J, et al . Lessons from experienced guideline implementers: attend to many factors and use multiple strategies. Jt Comm J Qual Improv 2000 ; 26 : 171 –88. OpenUrl PubMed
• ↵ Solberg L . Guideline implementations: what the literature doesn't tell us. Jt Comm J Qual Improv 2000 ; 26 : 525 –37. OpenUrl PubMed
• ↵ Wensing M , van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract 1998 ; 48 : 991 –7. OpenUrl Abstract / FREE Full Text
• ↵ Plsek PE , Greenhalgh T. The challenge of compexity in health care. BMJ 2001 ; 323 : 625 –8. OpenUrl FREE Full Text
• ↵ Campbell M , Fitzpatrick R, Haines A, et al . Framework for design and evaluation of complex interventions to improve health. BMJ 2000 ; 321 : 694 –6. OpenUrl FREE Full Text
• ↵ Pope C , van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care 2002 ; 11 : 148 –52. OpenUrl Abstract / FREE Full Text
• ↵ Wensing M , Elwyn G. Research on patients' views on the evaluation and improvement of quality of care. Qual Saf Health Care 2002 ; 11 : 153 –7. OpenUrl Abstract / FREE Full Text

Read the full text or download the PDF:

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• Supplements
• French Abstracts
• Portuguese Abstracts
• Spanish Abstracts
• Author Guidelines
• Submission Site
• Open Access
• About International Journal for Quality in Health Care
• About the International Society for Quality in Health Care
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Contact ISQua
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, conclusions and recommendations, acknowledgements, appendix a—participant survey for 2016/17 telfer qps course, appendix b—criteria to assess projects.

• < Previous

Development and evaluation of a quality improvement framework for healthcare

• Article contents
• Figures & tables
• Supplementary Data

Samantha Hamilton, Alison Jennings, Alan J Forster, Development and evaluation of a quality improvement framework for healthcare, International Journal for Quality in Health Care , Volume 32, Issue 7, August 2020, Pages 456–463, https://doi.org/10.1093/intqhc/mzaa075

• Permissions Icon Permissions

Develop and evaluate a framework for quality improvement which will provide a common approach, terminology and greater likelihood of success in achieving demonstrable and sustained improvement.

Descriptive with mixed methods evaluation.

Tertiary care academic hospital in Ottawa, Canada.

Students enrolled in the Quality and Patient Safety Leadership Executive Program at the local university.

The quality improvement (QI) framework was developed though comparison and coding of key strengths across four commonly applied quality improvement frameworks. Effectiveness of the framework was evaluated through a satisfaction survey of students who were taught the framework, and independent assessment of student improvement initiatives that followed the QI Framework versus initiatives from a prior year who did not.

Acceptability and effectiveness of the QI Framework.

All participants found the QI Framework to be useful and easy to follow. Independent evaluation of improvement initiatives following the QI Framework, as compared to those that did not, scored higher (95% CI: 3.0 ± 0.49) than the projects completed without the QI Framework (95% CI: 2.2 ± 0.30), P value < 0.01. Scores were higher for cohort 2017/18 in all criteria except scaling and spreading, in which there was no change (2015/16 95% CI: 1.2 ± 0.24, 2017/18 95% CI: 1.2 ± 0.38).

The method we have developed is acceptable and helpful to users, and overall application resulted in higher quality initiatives. We believe this method, which we have branded as The Ottawa Hospital Innovation Framework, can be beneficial in healthcare settings for a variety of change initiatives.

A quality improvement framework is a stepwise approach to executing quality improvement (QI) projects. A structured framework provides consistency, common thinking and language across organizations. This approach is particularly helpful in healthcare organizations which are complex, adaptive systems, with multiple moving parts and factors influencing care, activities, events and outcomes [ 1 ].

There are number of different philosophies, tools and frameworks for health system change. Each of these has pros and cons and varied applicability depending on the healthcare sector and the types of problems it addresses [ 2 ]. Among them, we can cite internal continuous improvement models which originate from industry (e.g. Kaizen, Lean, Total Quality Management and Six Sigma) and campaigns , grounded in the theory of large-scale improvement, focusing on predetermined aspects of healthcare in the community. One of the most widely used models is the Model for Improvement methodology [ 1 ], which uses rapid cycles of change (Plan-Do-Study-Act [PDSA]) and is the current approach encouraged by the Institute for Healthcare Improvement and others [ 3 , 4 ].

The literature evaluating QI Frameworks is limited [ 5 , 6 ]. Most studies focus on the evidence of achieving improved outcomes instead of effectiveness and utility of the framework itself [ 7 ]. While outcomes are extremely important, it is also important to understand how best to achieve these outcomes and transfer successful interventions into other settings [ 8 ]. Evaluation of these aspects is challenging as a result of the heterogeneity in interventions and contexts [ 8–10 ]. There is, however, some evidence that these models lack practical utility and are inconsistently applied in healthcare [ 6 , 7 ].

Simple approaches, such as the Model for Improvements’ PDSA cycles or Lean, are appealing but insufficient at addressing complexities [ 6 , 7 ] and have yet to demonstrate improvement through application [ 11 ]. Other more complex models, such as Six Sigma, or combinations of other models with Six Sigma, are highly rigorous, and although application of the methodology is observed in very specific instances, there is no evidence to date that this method is a widely applicable overall method for QI within an organization [ 11 ]. There are also some niche approaches such as the Comprehensive Unit-Based Safety Program (CUSP), but CUSP is typically not applied beyond surgical specialties, and even among those, other methods are employed to address particular quality problems [ 12 ].

The different philosophies, approaches, terminology and tools can result in confusion and, as such, can be a barrier to widespread uptake of QI. To address this concern, we sought to create one simple, effective framework to undertake QI initiatives at our institution. The objective of this paper is to describe our framework as well as how it was developed, disseminated and evaluated. We endeavor to demonstrate that our framework performs well, is acceptable to users and offers key advantages over existing frameworks typically used by healthcare organizations.

Framework development

We reviewed the most common quality improvement frameworks used at our hospital, namely, the Model for Improvement by the Institute for Healthcare Improvement, Lean, Six Sigma and CUSP by Johns Hopkins Medicine. We reviewed the frameworks in a side-by-side manner to align common components and highlight differences. We coded the key strengths for each framework and grouped these into concepts which became the core components (or steps) of our framework ( Fig. 1 ). This work was also done with careful consideration of the contextual application and varying size and complexity of QI initiatives across our organization.

The TOH Innovation Framework and alignment to existing methods for improvement.

Once the core components were identified, we enhanced the Framework in two ways: (1) the addition of ‘gates’ and (2) the mapping of key tools. Gates are formal reviews between the project sponsor and the project lead. The criteria at each gate are used to evaluate if and how the project moves forward. The stage-gate approach ensures constant communication and buy-in from stakeholders and sponsors. We identified and aligned the most common, practical tools to each step in the framework to help individuals distinguish tools from the overall framework itself. The tools provided are suggested based on the type of problem being addressed. For example, efficiency and access-related quality issues may be best addressed through use of value stream mapping to identify the root causes. The resulting framework is a step-by-step approach to addressing any type, size or complexity of quality issue encountered. Components of the framework are summarized in Table 1 .

Summary of The Ottawa Hospital Innovation Framework components

Gate = check-in and approval to proceed to the next step.

Framework roll-out

The developed framework was put into use immediately by our hospital Quality Improvement Coordinators who are responsible for supporting quality projects. During the pilot, the Coordinators solicited real-time unstructured feedback from users throughout their projects. In addition, the framework was taught to a small group of hospital leaders as part of an internal leadership program. They applied the framework to their own projects, and again, feedback was sought on the teaching and practical application of the framework. All feedback was used to refine the content and method of delivery.

The revised framework was disseminated to the entire hospital in October 2017, during a hospital-wide information session. The session introduced the framework along with an online platform for all staff to access the framework, tools training and help (among other functions). Monthly ‘Quality & Innovation Showcases’ after the launch highlighted the framework and its components as well as local quality projects that used the framework.

The framework was introduced as part of the curriculum of a Quality and Patient Safety Leadership Executive Program offered through the School of Management at the local university in October 2017. In addition, the framework was introduced as part of the core curriculum of a 1-day course for clinical and administrative staff involved in QI projects as well as a more intensive 4-day course for leaders.

Our experience in applying the framework to a variety of improvement initiatives has garnered support for this framework as a model for innovation, innovation being larger scale change, yet still benefiting from a disciplined methodical approach. For these reasons the framework has been branded as The Ottawa Hospital (TOH) Innovation Framework.

We undertook a two-part evaluation of the acceptability and effectiveness of our framework. Part 1 was a survey of participants who had been taught and applied the new framework. Part 2 was an independent review of quality project that used the new framework compared to projects that did not use the framework.

Setting and Participants

All participants were students enrolled in the Quality and Patient Safety Leadership Executive Program at a local university. This program has been running annually since 2010 and is designed for physician and health service leaders in healthcare organizations who have an interest in improving quality and patient safety. Program participants are sponsored by their home institutions. The program spans 7 months with a total of 63 classroom hours. Each student must commit additional time to undertake a project to improve quality in their organization and prepare a poster to present their initiative and results.

The course content was historically based on The Improvement Guide [ 13 ] and did not teach substantial QI methodology. In 2017/18 the new TOH Innovation Framework was incorporated into the program curriculum. Training on the framework was offered in an applied, just-in-time fashion to course participants helping them to learn and then apply the components as they progressed through the execution of their selected QI initiative over the course of the program. As such, the training is a blend of didactic and applied content.

Part 1—Participant survey. We conducted an anonymous survey of all students who were enrolled in the 2017/18 cohort of the Quality and Patient Safety Leadership Executive Program. The survey was disseminated through LimeSurvey [ 14 ]. Completion of the survey implied consent. The survey questions were designed to assess the acceptability and effectiveness of the framework (questions in   Appendix A ).

Part 2—Independent quality project review. Four independent reviewers reviewed and assessed the projects from a year where the QI Framework was not incorporated (2015/16) and the most recent year in which the QI Framework was incorporated (2017/18). The final poster presentation from each project was anonymized and uploaded to a central repository. Standard criteria, described below, were used to assess each project. Reviewers were not blinded to whether the projects were products of the new framework.

Criteria to be used for both 2015/16 and 2017/18 cohort projects were rated on a Likert scale of 1–5, where 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence). The specific questions are described in   Appendix B .

The four reviewers had diverse backgrounds with varying degrees of QI knowledge. One reviewer was a Physician and former Hospital Quality Executive, one was a Research Coordinator, one was a Manager in the Quality Department, and one was a Quality Improvement Coordinator. We attempted to address potential bias by using multiple diverse reviewers and applying the structured review process described.

All survey results are presented descriptively. For the independent project review, we calculated mean scores and confidence intervals for each criterion. Mean score across cohorts were compared using t -tests assuming both equal and unequal variance. The non-parametric Wilcoxon signed-rank test was also used to compare medians across cohorts.

Participant survey

We sent the survey to 22 participants and received responses from 14 (64%). Most respondents were physicians ( n  = 11, 79%) with only a small number of administrators ( n  = 2, 14%) and 1 nurse (7%). Very few participants had any prior training in QI ( n  = 3, 21%).

Survey responses ( Fig. 2 ) showed that all respondents ( n  = 14, 100%) either strongly agreed or agreed that (a) taking the course alleviated anxiety regarding the right way to undertake a QI initiative; (b) the framework provided a simplified stepwise approach to undertaking a QI initiative; (c) the framework was easy to understand and follow; (d) the framework training and the tools taught according to step heightened the participants understanding of concepts, tools and approaches for QI in healthcare; (e) the training enhanced the participants understanding of QI philosophies, frameworks and tools; and (f) the training clarified how philosophies, frameworks and tools can work together for achieving better outcomes to QI initiative efforts. While 67% of respondents (8/12, 2 did not answer) agreed that when ‘compared to previous QI training, the QI Framework, tools and training were superior’, only three participants identified having some previous QI training. Of these, two agreed that the framework was superior compared to their previous training, and one was neutral. The majority of participants ( n  = 13, 93%) either strongly agreed or agreed that (a) following the framework helped the participant implement a sustainable improvement; (b) following the framework has provided the participant with greater confidence in leading QI initiatives; and (c) following the framework helped the participant communicate the results of my QI initiative.

Survey responses of the 2017/18 QI Executive Program participants regarding use of the TOH Innovation Framework, n  = 14.

Independent assessments of quality improvement projects

Each reviewer independently evaluated 13 projects from the 2015/16 cohort and 13 projects form the 2017/18 cohort. The mean scores for all assessment criterion are shown in Fig. 3 . Overall, the projects completed in the 2017/18 cohort (i.e. taught the TOH Innovation Framework) scored higher (95% CI: 3.0 ± 0.49) than the projects completed in cohort 2015/16 (95% CI: 2.2 ± 0.30), P value < 0.01 (two-sample t -tests assuming both equal and non-equal variance). The Wilcoxon signed-rank test also demonstrated significant difference across the medians of the two cohorts (Wilcoxon value = 5, P  < 0.01). Mean scores were higher for cohort 2017/18 in all criteria except scaling and spreading in which there was no change (2015/16 95% CI: 1.2 ± 0.24; 2017/18 95% CI: 1.2 ± 0.38). The biggest change (increase in the mean score) was in the presence of a plan for hardwiring (1.4), followed by the extent to which an appreciable change was seen (1.1) and the extent to which the interventions trialed were understood (1.0).

Independent evaluation of QI initiatives across cohort 1 (2015–2016) not taught TOH Innovation Framework, n  = 13, and Cohort 2 (2017–2018) taught TOH Innovation Framework, n  = 13.

Results are presented as mean criterion score for all projects from each cohort. 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence). Overall, evaluators of the projects in the 2017/18 (i.e. those students having been trained on the framework) found that these projects were easier to understand, and one evaluator noted that ‘there were definite improvements in projects in 2017/18 as compared to those from 2015/16’.

Healthcare professionals require a simple yet effective method for addressing quality problems. Making improvement happen within the healthcare environment is highly complex, but having a consistent, step-by-step framework to rely upon is an essential component to achieving successful outcomes. Despite several frameworks being available, there are limited studies looking at the effectiveness of QI methods in healthcare. We developed and tested a framework at our institution and performed an evaluation to determine its acceptability and effectiveness. We did so in order to share a common language for QI, to apply rigor and discipline in our improvement efforts and ultimately to be as successful as possible in demonstrating improvements in quality outcomes. We found that in the application described, our framework, being a blend of the best of the methods available globally, coupled with adjustments for the realities of healthcare delivery is an easy to apply and effective method for addressing QI initiatives in the healthcare environment.

The framework we developed calls for disciplined steps in understanding the causes of the quality problem to be solved prior to attempting any change ideas. Change ideas are linked to one or more selected root cause(s) and are then trialed in a controlled way—through application PDSA cycles. As such there are fewer, more targeted PDSAs applied through the framework. The positioning of targeted PDSAs only after a comprehensive understanding of the causes of the overarching problem results in fewer test cycles. Accordingly, Reed and Card’s review of PDSAs argues that a deeper understanding and framing of the problem prior to commencing use of PDSAs is essential [ 15 ].

Further, our framework calls for an explicit review at the conclusion of each step with the executive sponsor of the QI initiative in order to share findings, discuss and seek approval and the support necessary to successfully undertake the next step in the framework. These ‘gates’ allow for the stoppage of any QI initiative that is not well positioned to succeed or is unable, for various reasons, to continue. This accountable approach to project continuation is rarely seen in practice where projects often continue that should not.

A strength of our framework is within its development and content; consisting of the best of existing models yet incorporating/or adjusting for the real-life challenges of the healthcare environment. Another strength of the framework is its applicability to quality issues of various sizes and types (e.g. patient safety, effectiveness, efficiency, access or patient-centered related). Our study cohort included different professionals from different institutions, who applied the framework to a variety of projects. This may suggest the framework was applicable across different applications; however our sample size was too small to draw any conclusions. In addition, though our assessment of the new framework evaluated only one improvement effort, it is worth noting that despite not having the benefit of experience with the framework, students were able to apply it with greater project success than the previous year.

Our study is important because we evaluated the framework to gauge its acceptability and effectiveness. We undertook a two-pronged evaluation which included a survey of course participants and a comparison of projects that used and did not use the framework. As with all surveys, there is a risk of non-response bias, which is almost impossible to determine impact. However, we did have a very good response rate and very consistent responses among those who completed the survey. With respect to the comparison between course cohorts, there are also several limitations. Though we performed independent reviews of the completed QI projects, reviewers were not blinded to the framework used, and as such, this may have introduced a bias. Blinding was not possible since the project write-ups for the new framework followed a clear structure. We minimized the impact of possible bias by having multiple independent reviewers and using a structured review process. The cohorts themselves may also have differed; for example, there may have been variation in participant knowledge prior to starting the course. However, students were not granted registration on the basis on knowledge or experience in QI, only a common desire to realize change. It should also be acknowledged that improvements seen in the second cohort may simply have been related to the universal application of a framework and not necessarily our specific framework. Further evaluation of the framework in larger samples and different contexts is required.

We demonstrated that the Innovation Framework we developed to guide quality improvement was acceptable to users and performed well at improving quality in healthcare organizations among the participant group that applied it. Of the limited studies to date, there is no clear method for QI that has been evaluated to show enhanced quality outcomes. Rather, current methods such as the PDSA method are argued to be insufficient outside of a broader problem-solving method.

We have formally adopted this framework at our institution. Certainly, in our organization, having one common approach to QI has provided the benefits of common language and enabled healthcare providers across a variety of disciplines to work in a consistent, disciplined manner. Having one common approach will also better enable our organization to continue to perfect our skills in making improvement happen and should ultimately lead to better quality outcomes over time. As next steps, we plan to further evaluate the framework across different disciplines and contexts and hope to evaluate the applicability, utility and overall generalizability of the framework in different institutions.

The authors would like to thank Julie Lawson for her assistance in the review and submission of this paper and the Quality Improvement Team at the Ottawa Hospital for their assistance with the management of the project evaluation.

Braithwaite J . Changing how we think about healthcare improvement . BMJ 2018 ; 361 : k2014 . doi: 10.1136/bmj.k2014 .

Google Scholar

Øvretveit J. What Are the Advantages and Limitations of Different Quality and Safety Tools for Health Care? 2005 . http://www.euro.who.int/document/e87577.pdf ( 20 March 2019, last date accessed ).

Health Quality Ontario . Quality Improvement Science . 2013 . www.hqontario.ca ( 30 January 2019 last date accessed ).

Patient Safety Institute . Improvement Frameworks: Getting Started Kit . 2015 . www.saferhealthcarenow.ca ( 30 January 2019 last date accessed ).

Spencer E , Walshe K . National quality improvement policies and strategies in European healthcare systems . Qual Saf Health Care 2009 ; 18 : i22 – 7 . doi: 10.1136/qshc.2008.029355 .

Kraft S , Carayon P , Weiss J et al.  A simple framework for complex system improvement . Am J Med Qual 2015 ; 30 : 223 – 31 . doi: 10.1177/1062860614530184 .

Taylor MJ , McNicholas C , Nicolay C et al.  Systematic review of the application of the plan-do-study-act method to improve quality in healthcare . BMJ Qual Saf 2014 ; 23 : 290 – 8 . doi: 10.1136/bmjqs-2013-001862 .

Øvretveit J , Gustafson D . Evaluation of quality improvement programmes . Qual Saf Health Care 2002 ; 11 : 270 – 5 . doi: 10.1136/QHC.11.3.270 .

Walshe K , Freeman T . Effectiveness of quality improvement: learning from evaluations . Qual Saf Health Care 2002 ; 11 : 85 – 7 . doi: 10.1136/QHC.11.1.85 .

Busetto L , Luijkx K , Vrijhoef HJM . Development of the COMIC model for the comprehensive evaluation of integrated care interventions . Int J Care Coord 2016 ; 19 : 47 – 58 . doi: 10.1177/2053434516661700 .

Deblois S , Lepanto L . Lean and six sigma in acute care: a systematic review of reviews . Int J Health Care Qual Assur 2016 ; 29 : 192 – 208 . doi: 10.1108/IJHCQA-05-2014-0058 .

Brown R , Grehan P , Brennan M et al.  Using lean six sigma to improve rates of day of surgery admission in a national thoracic surgery department . International J Qual Health Care 2019 ; 31 (Supplement_1): 14 – 21 . doi: 10.1093/intqhc/mzz083 .

Langley GL, Moen R, Nolan KM et al.  The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd Edition) . San Francisco, California, USA: Jossey-Bass Publishers; 2009 .

Limesurvey GmbH . LimeSurvey: An Open Source survey tool . http://www.limesurvey.org . https://community.limesurvey.org/licence-trademark/#citation .

Reed JE , Card AJ . The problem with plan-do-study-act cycles . BMJ Qual Saf 2016 ; 25 : 147 – 52 . doi: 10.1136/bmjqs-2015-005076 .

Q1. My role is (select one):

a) physician

c) health professional

d) administrator (leader in some health administration capacity)

Q2. I have taken other trainings in QI in healthcare previously (select all that apply).

a) IDEAS—introductory course

b) IDEAS—multiple day, applied course

c) Lean yellow belt

d) Lean green belt

e) Other (i.e. value stream mapping, root cause analysis, etc.)—list (free text)

Q3. Compared to previous training, I found the QI Framework and tools taught according to step to be superior.

a) Strongly agree

d) Disagree

e) Strongly disagree

Q4. Taking the course has alleviated my anxiety regarding the right way to undertake a QI initiative.

Q5. The QI Framework provided a simplified stepwise approach to undertaking a QI initiative.

Q6. Following the QI Framework helped me implement a sustainable improvement.

Q7. The QI Framework is easy to understand and follow.

Q8. The QI Framework training and the tools taught according to step heightened my understanding of concepts, tools and approaches for QI in healthcare.

Q9. The training enhanced my understanding of QI philosophies, frameworks and tools.

Q10. The training enhanced my understanding of QI philosophies, frameworks and tools.

Q11. The training clarified how philosophies, frameworks and tools can work together for achieving better outcomes to QI initiative efforts.

Q12. Following the QI Framework has provided me with greater confidence in leading QI initiatives.

Q13. Following the QI Framework helped me communicate the results of my QI initiative.

Questions * :

Rate the extent to which you understood the problem they were trying to solve.

Rate the extent to which you clearly understand interventions trialed.

What evidence was there that the findings or the lessons learned led to iterative improvement cycles, i.e. adopt, abandon, adapt?

Rate the extent to which you see appreciable changes as a result [of the work].

Was there a plan for hardwiring?

Was there a plan for scaling and spreading?

* Rated on a Likert scale of 1–5, where 1 = totally unclear (or evidence unclear) and 5 = fully clear (or clear evidence).

• quality improvement

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1464-3677
• Print ISSN 1353-4505
• Copyright © 2024 International Society for Quality in Health Care and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• News & Views
• How to improve...

How to improve healthcare improvement—an essay by Mary Dixon-Woods

Read the full collection.

• Related content
• Peer review
• Mary Dixon-Woods , director
• THIS Institute, Cambridge, UK
• director{at}thisinstitute.cam.ac.uk

As improvement practice and research begin to come of age, Mary Dixon-Woods considers the key areas that need attention if we are to reap their benefits

In the NHS, as in health systems worldwide, patients are exposed to risks of avoidable harm 1 and unwarranted variations in quality. 2 3 4 But too often, problems in the quality and safety of healthcare are merely described, even “admired,” 5 rather than fixed; the effort invested in collecting information (which is essential) is not matched by effort in making improvement. The National Confidential Enquiry into Patient Outcome and Death, for example, has raised many of the same concerns in report after report. 6 Catastrophic degradations of organisations and units have recurred throughout the history of the NHS, with depressingly similar features each time. 7 8 9

More resources are clearly necessary to tackle many of these problems. There is no dispute about the preconditions for high quality, safe care: funding, staff, training, buildings, equipment, and other infrastructure. But quality health services depend not just on structures but on processes. 10 Optimising the use of available resources requires continuous improvement of healthcare processes and systems. 5

The NHS has seen many attempts to stimulate organisations to improve using incentive schemes, ranging from pay for performance (the Quality and Outcomes Framework in primary care, for example) to public reporting (such as annual quality accounts). They have had mixed results, and many have had unintended consequences. 11 12 Wanting to improve is not the same as knowing how to do it.

In response, attention has increasingly turned to a set of approaches known as quality improvement (QI). Though a definition of exactly what counts as a QI approach has escaped consensus, QI is often identified with a set of techniques adapted from industrial settings. They include the US Institute for Healthcare Improvement’s Model for Improvement, which, among other things, combines measurement with tests of small change (plan-do-study-act cycles). 8 Other popular approaches include Lean and Six Sigma. QI can also involve specific interventions intended to improve processes and systems, ranging from checklists and “care bundles” of interventions (a set of evidence based practices intended to be done consistently) through to medicines reconciliation and clinical pathways.

QI has been advocated in healthcare for over 30 years 13 ; policies emphasise the need for QI and QI practice is mandated for many healthcare professionals (including junior doctors). Yet the question, “Does quality improvement actually improve quality?” remains surprisingly difficult to answer. 14 The evidence for the benefits of QI is mixed 14 and generally of poor quality. It is important to resolve this unsatisfactory situation. That will require doing more to bring together the practice and the study of improvement, using research to improve improvement, and thinking beyond effectiveness when considering the study and practice of improvement.

Uniting practice and study

The practice and study of improvement need closer integration. Though QI programmes and interventions may be just as consequential for patient wellbeing as drugs, devices, and other biomedical interventions, research about improvement has often been seen as unnecessary or discretionary, 15 16 particularly by some of its more ardent advocates. This is partly because the challenges faced are urgent, and the solutions seem obvious, so just getting on with it seems the right thing to do.

But, as in many other areas of human activity, QI is pervaded by optimism bias. It is particularly affected by the “lovely baby” syndrome, which happens when formal evaluation is eschewed because something looks so good that it is assumed it must work. Five systematic reviews (published 2010-16) reporting on evaluations of Lean and Six Sigma did not identify a single randomised controlled trial. 17 18 19 20 21 A systematic review of redesigning care processes identified no randomised trials. 22 A systematic review of the application of plan-do-study-act in healthcare identified no randomised trials. 23 A systematic review of several QI methods in surgery identified just one randomised trial. 56

The sobering reality is that some well intentioned, initially plausible improvement efforts fail when subjected to more rigorous evaluation. 24 For instance, a controlled study of a large, well resourced programme that supported a group of NHS hospitals to implement the IHI’s Model for Improvement found no differences in the rate of improvement between participating and control organisations. 25 26 Specific interventions may, similarly, not survive the rigours of systematic testing. An example is a programme to reduce hospital admissions from nursing homes that showed promise in a small study in the US, 27 but a later randomised implementation trial found no effect on admissions or emergency department attendances. 28

Some interventions are probably just not worth the effort and opportunity cost: having nurses wear “do not disturb” tabards during drug rounds, is one example. 29 And some QI efforts, perversely, may cause harm—as happened when a multicomponent intervention was found to be associated with an increase rather than a decrease in surgical site infections. 30

Producing sound evidence for the effectiveness of improvement interventions and programmes is likely to require a multipronged approach. More large scale trials and other rigorous studies, with embedded qualitative inquiry, should be a priority for research funders.

Not every study of improvement needs to be a randomised trial. One valuable but underused strategy involves wrapping evaluation around initiatives that are happening anyway, especially when it is possible to take advantage of natural experiments or design roll-outs. 31 Evaluation of the reorganisation of stroke care in London and Manchester 32 and the study of the Matching Michigan programme to reduce central line infections are good examples. 33 34

It would be impossible to externally evaluate every QI project. Critically important therefore will be increasing the rigour with which QI efforts evaluate themselves, as shown by a recent study of an attempt to improve care of frail older people using a “hospital at home” approach in southwest England. 35 This ingeniously designed study found no effect on outcomes and also showed that context matters.

Despite the potential value of high quality evaluation, QI reports are often weak, 18 with, for example, interventions so poorly reported that reproducibility is frustrated. 36 Recent reporting guidelines may help, 37 but some problems are not straightforward to resolve. In particular, current structures for governance and publishing research are not always well suited to QI, including situations where researchers study programmes they have not themselves initiated. Systematic learning from QI needs to improve, which may require fresh thinking about how best to align the goals of practice and study, and to reconcile the needs of different stakeholders. 38

Using research to improve improvement

Research can help to support the practice of improvement in many ways other than evaluation of its effectiveness. One important role lies in creating assets that can be used to improve practice, such as ways to visualise data, analytical methods, and validated measures that assess the aspects of care that most matter to patients and staff. This kind of work could, for example, help to reduce the current vast number of quality measures—there are more than 1200 indicators of structure and process in perioperative care alone. 39

The study of improvement can also identify how improvement practice can get better. For instance, it has become clear that fidelity to the basic principles of improvement methods is a major problem: plan-do-study-act cycles are crucial to many improvement approaches, yet only 20% of the projects that report using the technique have done so properly. 23 Research has also identified problems in measurement—teams trying to do improvement may struggle with definitions, data collection, and interpretation 40 —indicating that this too requires more investment.

Improvement research is particularly important to help cumulate, synthesise, and scale learning so that practice can move forward without reinventing solutions that already exist or reintroducing things that do not work. Such theorising can be highly practical, 41 helping to clarify the mechanisms through which interventions are likely to work, supporting the optimisation of those interventions, and identifying their most appropriate targets. 42

Research can systematise learning from “positive deviance,” approaches that examine individuals, teams, or organisations that show exceptionally good performance. 43 Positive deviance can be used to identify successful designs for clinical processes that other organisations can apply. 44

Crucially, positive deviance can also help to characterise the features of high performing contexts and ensure that the right lessons are learnt. For example, a distinguishing feature of many high performing organisations, including many currently rated as outstanding by the Care Quality Commission, is that they use structured methods of continuous quality improvement. But studies of high performing settings, such as the Southmead maternity unit in Bristol, indicate that although continuous improvement is key to their success, a specific branded improvement method is not necessary. 45 This and other work shows that not all improvement needs to involve a well defined QI intervention, and not everything requires a discrete project with formal plan-do-study-act cycles.

More broadly, research has shown that QI is just one contributor to improving quality and safety. Organisations in many industries display similar variations to healthcare organisations, including large and persistent differences in performance and productivity between seemingly similar enterprises. 46 Important work, some of it experimental, is beginning to show that it is the quality of their management practices that distinguishes them. 47 These practices include continuous quality improvement as well as skills training, human resources, and operational management, for example. QI without the right contextual support is likely to have limited impact.

Beyond effectiveness

Important as they are, evaluations of the approaches and interventions in individual improvement programmes cannot answer every pertinent question about improvement. 48 Other key questions concern the values and assumptions intrinsic to QI.

Consider the “product dominant” logic in many healthcare improvement efforts, which assumes that one party makes a product and conveys it to a consumer. 49 Paul Batalden, one of the early pioneers of QI in healthcare, proposes that we need instead a “service dominant” logic, which assumes that health is co-produced with patients. 49

More broadly, we must interrogate how problems of quality and safety are identified, defined, and selected for attention by whom, through which power structures, and with what consequences. Why, for instance, is so much attention given to individual professional behaviour when systems are likely to be a more productive focus? 50 Why have quality and safety in mental illness and learning disability received less attention in practice, policy, and research 51 despite high morbidity and mortality and evidence of both serious harm and failures of organisational learning? The concern extends to why the topic of social inequities in healthcare improvement has remained so muted 52 and to the choice of subjects for study. Why is it, for example, that interventions like education and training, which have important roles in quality and safety and are undertaken at vast scale, are often treated as undeserving of evaluation or research?

How QI is organised institutionally also demands attention. It is often conducted as a highly local, almost artisan activity, with each organisation painstakingly working out its own solution for each problem. Much improvement work is conducted by professionals in training, often in the form of small, time limited projects conducted for accreditation. But working in this isolated way means a lack of critical mass to support the right kinds of expertise, such as the technical skill in human factors or ergonomics necessary to engineer a process or devise a safety solution. Having hundreds of organisations all trying to do their own thing also means much waste, and the absence of harmonisation across basic processes introduces inefficiencies and risks. 14

A better approach to the interorganisational nature of health service provision requires solving the “problem of many hands.” 53 We need ways to agree which kinds of sector-wide challenges need standardisation and interoperability; which solutions can be left to local customisation at implementation; and which should be developed entirely locally. 14 Better development of solutions and interventions is likely to require more use of prototyping, modelling and simulation, and testing in different scenarios and under different conditions, 14 ideally through coordinated, large scale efforts that incorporate high quality evaluation.

Finally, an approach that goes beyond effectiveness can also help in recognising the essential role of the professions in healthcare improvement. The past half century has seen a dramatic redefining of the role and status of the healthcare professions in health systems 54 : unprecedented external accountability, oversight, and surveillance are now the norm. But policy makers would do well to recognise how much more can be achieved through professional coalitions of the willing than through too many imposed, compliance focused diktats. Research is now showing how the professions can be hugely important institutional forces for good. 54 55 In particular, the professions have a unique and invaluable role in working as advocates for improvement, creating alliances with patients, providing training and education, contributing expertise and wisdom, coordinating improvement efforts, and giving political voice for problems that need to be solved at system level (such as, for example, equipment design).

Improvement efforts are critical to securing the future of the NHS. But they need an evidence base. Without sound evaluation, patients may be deprived of benefit, resources and energy may be wasted on ineffective QI interventions or on interventions that distribute risks unfairly, and organisations are left unable to make good decisions about trade-offs given their many competing priorities. The study of improvement has an important role in developing an evidence-base and in exploring questions beyond effectiveness alone, and in particular showing the need to establish improvement as a collective endeavour that can benefit from professional leadership.

Mary Dixon-Woods is the Health Foundation professor of healthcare improvement studies and director of The Healthcare Improvement Studies (THIS) Institute at the University of Cambridge, funded by the Health Foundation. Co-editor-in-chief of BMJ Quality and Safety , she is an honorary fellow of the Royal College of General Practitioners and the Royal College of Physicians. This article is based largely on the Harveian oration she gave at the RCP on 18 October 2018, in the year of the college’s 500th anniversary. The oration is available here: http://www.clinmed.rcpjournal.org/content/19/1/47 and the video version here: https://www.rcplondon.ac.uk/events/harveian-oration-and-dinner-2018

This article is one of a series commissioned by The BMJ based on ideas generated by a joint editorial group with members from the Health Foundation and The BMJ , including a patient/carer. The BMJ retained full editorial control over external peer review, editing, and publication. Open access fees and The BMJ ’s quality improvement editor post are funded by the Health Foundation.

Competing interests: I have read and understood BMJ policy on declaration of interests and a statement is available here: https://www.bmj.com/about-bmj/advisory-panels/editorial-advisory-board/mary-dixonwoods

Provenance and peer review: Commissioned; not externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• Neuburger J ,
• Hutchings A ,
• Stewart K ,
• Buckingham R
• Castelli A ,
• Verzulli R ,
• Allwood D ,
• Sunstein CR
• Healthcare Quality Improvement Partnership
• Shortell SM
• Martin GP ,
• Dixon-Woods M
• Donabedian A
• Himmelstein DU ,
• Woolhandler S
• Woolhandler S ,
• Himmelstein DU
• Dixon-Woods M ,
• Ioannidis JPA ,
• Marshall M ,
• Pronovost P ,
• Glasgow JM ,
• Scott-Caziewell JR ,
• Nicolay CR ,
• Deblois S ,
• Moraros J ,
• Lemstra M ,
• Amaratunga T ,
• Dobranowski J
• van Leijen-Zeelenberg JE ,
• Elissen AMJ ,
• Taylor MJ ,
• McNicholas C ,
• Nicolay C ,
• Morgan DJ ,
• Diekema DJ ,
• Sepkowitz K ,
• Perencevich EN
• Benning A ,
• Ouslander JG ,
• Huckfeldt P ,
• Westbrook JI ,
• Hooper TD ,
• Middleton S ,
• Anthony T ,
• Murray BW ,
• Sum-Ping JT ,
• Portela MC ,
• Pronovost PJ ,
• Woodcock T ,
• Ramsay AIG ,
• Hoffman A ,
• Richardson A ,
• Hibbert P ,
• Matching Michigan Collaboration & Writing Committee
• Tarrant C ,
• Pearson M ,
• Hemsley A ,
• Blackwell R ,
• Custerson L
• Goodman D ,
• Watson SI ,
• Taylor CA ,
• Chazapis M ,
• Gilhooly D ,
• Liberati EG ,
• Davidoff F ,
• Leviton L ,
• Johnston M ,
• Abraham C ,
• “Psychological Theory” Group
• Clay-Williams R ,
• Braithwaite J
• Bradley EH ,
• Willars J ,
• Mahajan A ,
• Aveling EL ,
• Thibaut BI ,
• Ramtale SC ,
• Boozary AS ,
• Shojania KG
• Pronovost PJ
• Armstrong N ,
• Herbert G ,
• Purkayastha S ,
• Greenhalgh A ,

• DOI: 10.1108/ijhg-04-2024-0045
• Corpus ID: 271840241

What influences perceptions about the concept of return on investment from healthcare quality improvement programmes? An institutional theory perspective

• S'thembile Thusini , T. Soukup , Claire Henderson
• Published in International Journal of… 12 August 2024
• Business, Medicine, Economics

56 References

How is return on investment from quality improvement programmes conceptualised by mental healthcare leaders and why: a qualitative study, identifying and understanding benefits associated with return-on-investment from large-scale healthcare quality improvement programmes: an integrative systematic literature review, does lean muddy the quality improvement waters a qualitative study of how a hospital management team understands lean in the context of quality improvement, using institutional theory to analyse hospital responses to external demands for finance and quality in five european countries, lessons learned from measuring return on investment in public health quality improvement initiatives., a scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: state of the literature and future directions, sustaining improvement the 20-year jönköping quality improvement program revisited, systematic review of the application of the plan–do–study–act method to improve quality in healthcare, studying large-scale programmes to improve patient safety in whole care systems: challenges for research., what does ‘quality’ add towards an ethics of healthcare improvement, related papers.

Showing 1 through 3 of 0 Related Papers

• Introduction
• Conclusions
• Article Information

Outcomes are estimated from bivariate and multivariable generalized estimating equation models. aOR, indicates adjusted odds ratio; GAD-7, Generalized Anxiety Disorder 7-item scale; PHQ-9, Patient Health Questionnaire 9-item scale; whiskers, 95% CIs.

eTable 1. Survey Instruments

eTable 2. Prevalence of Exposure Over Time

eTable 3. Prevalence of Outcomes Over Time by Exposure Group

eTable 4. E-Value Calculation for Association Between Puberty Blockers or Gender-Affirming Hormones and Mental Health Outcomes

eTable 5. Examining Association Between Puberty Blockers or Gender-Affirming Hormones and Mental Health Outcomes Separately

eTable 6. Bivariate Model Restricted to Youths Ages 13 to 17 Years

eTable 7. Multivariable Model Restricted to 90 Youths Ages 13 to 17 Years

eTable 8. Sensitivity Analyses using Patient Health Questionnaire 8-item Scale Score of 10 or Greater for Moderate to Severe Depression

eFigure 1. Schematic of Generalized Estimating Equation Model

eFigure 2. Association Between Receipt of Gender-Affirming Hormones or Puberty Blockers and Mental Health Outcomes

eReferences

• Medical Groups Defend Patient-Physician Relationship and Access to Adolescent Gender-Affirming Care JAMA Medical News & Perspectives April 19, 2022 This Medical News article discusses physicians’ advocacy to protect patients and the patient-physician relationship amid efforts by politicians to limit access or criminalize gender-affirming care. Bridget M. Kuehn, MSJ
• As Laws Restricting Health Care Surge, Some US Physicians Choose Between Fight or Flight JAMA Medical News & Perspectives June 13, 2023 In this Medical News article, 13 physicians and health care experts spoke with JAMA about the increasing efforts to criminalize evidence-based medical care in the US. Melissa Suran, PhD, MSJ
• Data Errors in eTables 2 and 3 JAMA Network Open Correction July 26, 2022
• Improving Mental Health Among Transgender and Gender-Diverse Youth JAMA Network Open Invited Commentary February 25, 2022 Brett Dolotina, BS; Jack L. Turban, MD, MHS

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Tordoff DM , Wanta JW , Collin A , Stepney C , Inwards-Breland DJ , Ahrens K. Mental Health Outcomes in Transgender and Nonbinary Youths Receiving Gender-Affirming Care. JAMA Netw Open. 2022;5(2):e220978. doi:10.1001/jamanetworkopen.2022.0978

Manage citations:

© 2024

• Permissions

Mental Health Outcomes in Transgender and Nonbinary Youths Receiving Gender-Affirming Care

• 1 Department of Epidemiology, University of Washington, Seattle
• 2 Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle
• 3 School of Medicine, University of Washington, Seattle
• 4 Department of Psychiatry and Behavioral Medicine, Department of Adolescent and Young Adult Medicine, Seattle Children’s Hospital, Seattle, Washington
• 5 University of California, San Diego School of Medicine, Rady Children's Hospital
• 6 Division of Adolescent Medicine, Department of Pediatrics, Seattle Children’s Hospital, Seattle, Washington
• Invited Commentary Improving Mental Health Among Transgender and Gender-Diverse Youth Brett Dolotina, BS; Jack L. Turban, MD, MHS JAMA Network Open
• Medical News & Perspectives Medical Groups Defend Patient-Physician Relationship and Access to Adolescent Gender-Affirming Care Bridget M. Kuehn, MSJ JAMA
• Medical News & Perspectives As Laws Restricting Health Care Surge, Some US Physicians Choose Between Fight or Flight Melissa Suran, PhD, MSJ JAMA
• Correction Data Errors in eTables 2 and 3 JAMA Network Open

Question   Is gender-affirming care for transgender and nonbinary (TNB) youths associated with changes in depression, anxiety, and suicidality?

Findings   In this prospective cohort of 104 TNB youths aged 13 to 20 years, receipt of gender-affirming care, including puberty blockers and gender-affirming hormones, was associated with 60% lower odds of moderate or severe depression and 73% lower odds of suicidality over a 12-month follow-up.

Meaning   This study found that access to gender-affirming care was associated with mitigation of mental health disparities among TNB youths over 1 year; given this population's high rates of adverse mental health outcomes, these data suggest that access to pharmacological interventions may be associated with improved mental health among TNB youths over a short period.

Importance   Transgender and nonbinary (TNB) youths are disproportionately burdened by poor mental health outcomes owing to decreased social support and increased stigma and discrimination. Although gender-affirming care is associated with decreased long-term adverse mental health outcomes among these youths, less is known about its association with mental health immediately after initiation of care.

Objective   To investigate changes in mental health over the first year of receiving gender-affirming care and whether initiation of puberty blockers (PBs) and gender-affirming hormones (GAHs) was associated with changes in depression, anxiety, and suicidality.

Design, Setting, and Participants   This prospective observational cohort study was conducted at an urban multidisciplinary gender clinic among TNB adolescents and young adults seeking gender-affirming care from August 2017 to June 2018. Data were analyzed from August 2020 through November 2021.

Exposures   Time since enrollment and receipt of PBs or GAHs.

Main Outcomes and Measures   Mental health outcomes of interest were assessed via the Patient Health Questionnaire 9-item (PHQ-9) and Generalized Anxiety Disorder 7-item (GAD-7) scales, which were dichotomized into measures of moderate or severe depression and anxiety (ie, scores ≥10), respectively. Any self-report of self-harm or suicidal thoughts over the previous 2 weeks was assessed using PHQ-9 question 9. Generalized estimating equations were used to assess change from baseline in each outcome at 3, 6, and 12 months of follow-up. Bivariate and multivariable logistic models were estimated to examine temporal trends and investigate associations between receipt of PBs or GAHs and each outcome.

Results   Among 104 youths aged 13 to 20 years (mean [SD] age, 15.8 [1.6] years) who participated in the study, there were 63 transmasculine individuals (60.6%), 27 transfeminine individuals (26.0%), 10 nonbinary or gender fluid individuals (9.6%), and 4 youths who responded “I don’t know” or did not respond to the gender identity question (3.8%). At baseline, 59 individuals (56.7%) had moderate to severe depression, 52 individuals (50.0%) had moderate to severe anxiety, and 45 individuals (43.3%) reported self-harm or suicidal thoughts. By the end of the study, 69 youths (66.3%) had received PBs, GAHs, or both interventions, while 35 youths had not received either intervention (33.7%). After adjustment for temporal trends and potential confounders, we observed 60% lower odds of depression (adjusted odds ratio [aOR], 0.40; 95% CI, 0.17-0.95) and 73% lower odds of suicidality (aOR, 0.27; 95% CI, 0.11-0.65) among youths who had initiated PBs or GAHs compared with youths who had not. There was no association between PBs or GAHs and anxiety (aOR, 1.01; 95% CI, 0.41, 2.51).

Conclusions and Relevance   This study found that gender-affirming medical interventions were associated with lower odds of depression and suicidality over 12 months. These data add to existing evidence suggesting that gender-affirming care may be associated with improved well-being among TNB youths over a short period, which is important given mental health disparities experienced by this population, particularly the high levels of self-harm and suicide.

Transgender and nonbinary (TNB) youths are disproportionately burdened by poor mental health outcomes, including depression, anxiety, and suicidal ideation and attempts. 1 - 5 These disparities are likely owing to high levels of social rejection, such as a lack of support from parents 6 , 7 and bullying, 6 , 8 , 9 and increased stigma and discrimination experienced by TNB youths. Multidisciplinary care centers have emerged across the country to address the health care needs of TNB youths, which include access to medical gender-affirming interventions, such as puberty blockers (PBs) and gender-affirming hormones (GAHs). 10 These centers coordinate care and help youths and their families address barriers to care, such as lack of insurance coverage 11 and travel times. 12 Gender-affirming care is associated with decreased rates of long-term adverse outcomes among TNB youths. Specifically, PBs, GAHs, and gender-affirming surgeries have all been found to be independently associated with decreased rates of depression, anxiety, and other adverse mental health outcomes. 13 - 16 Access to these interventions is also associated with a decreased lifetime incidence of suicidal ideation among adults who had access to PBs during adolescence. 17 Conversely, TNB youths who present to care later in adolescence or young adulthood experience more adverse mental health outcomes. 18 Despite this robust evidence base, legislation criminalizing and thus limiting access to gender-affirming medical care for minors is increasing. 19 , 20

Less is known about the association of gender-affirming care with mental health outcomes immediately after initiation of care. Several studies published from 2015 to 2020 found that receipt of PBs or GAHs was associated with improved psychological functioning 21 and body satisfaction, 22 as well as decreased depression 23 and suicidality 24 within a 1-year period. Initiation of gender-affirming care may be associated with improved short-term mental health owing to validation of gender identity and clinical staff support. Conversely, prerequisite mental health evaluations, often perceived as pathologizing by TNB youths, and initiation of GAHs may present new stressors that may be associated with exacerbation of mental health symptoms early in care, such as experiences of discrimination associated with more frequent points of engagement in a largely cisnormative health care system (eg, interactions with nonaffirming pharmacists to obtain laboratory tests, syringes, and medications). 25 Given the high risk of suicidality among TNB adolescents, there is a pressing need to better characterize mental health trends for TNB youths early in gender-affirming care. This study aimed to investigate changes in mental health among TNB youths enrolled in an urban multidisciplinary gender clinic over the first 12 months of receiving care. We also sought to investigate whether initiation of PBs or GAHs was associated with depression, anxiety, and suicidality.

This cohort study received approval from the Seattle Children’s Hospital Institutional Review Board. For youths younger than age 18 years, caregiver consent and youth assent was obtained. For youths ages 18 years and older, youth consent alone was obtained. The 12-month assessment was funded via a different mechanism than other survey time points; thus, participants were reconsented for the 12-month survey. The study follows the Strengthening the Reporting of Observational Studies in Epidemiology ( STROBE ) reporting guideline.

We conducted a prospective observational cohort study of TNB youths seeking care at Seattle Children’s Gender Clinic, an urban multidisciplinary gender clinic. After a referral is placed or a patient self-refers, new patients, their caregivers, or patients with their caregivers are scheduled for a 1-hour phone intake with a care navigator who is a licensed clinical social worker. Patients are then scheduled for an appointment at the clinic with a medical provider.

All patients who completed the phone intake and in-person appointment between August 2017 and June 2018 were recruited for this study. Participants completed baseline surveys within 24 hours of their first appointment and were invited to complete follow-up surveys at 3, 6, and 12 months. Youth surveys were used to assess most variables in this study; caregiver surveys were used to assess caregiver income. Participation and completion of study surveys had no bearing on prescribing of PBs or GAHs.

We assessed 3 internalizing mental health outcomes: depression, generalized anxiety, and suicidality. Depression was assessed using the Patient Health Questionnaire 9-item scale (PHQ-9), and anxiety was assessed using the Generalized Anxiety Disorder 7-item scale (GAD-7). We dichotomized PHQ-9 and GAD-7 scores into measures of moderate or severe depression and anxiety (ie, scores ≥10). 26 , 27 Self-harm and suicidal thoughts were assessed using PHQ-9 question 9 (eTable 1 in the Supplement ).

Participants self-reported if they had ever received GAHs, including estrogen or testosterone, or PBs (eg, gonadotropin-releasing hormone analogues) on each survey. We conducted a medical record review to capture prescription of androgen blockers (eg, spironolactone) and medications for menstrual suppression or contraception (ie, medroxyprogesterone acetate or levonorgestrel-releasing intrauterine device) during the study period.

We a priori considered potential confounders hypothesized to be associated with our exposures and outcomes of interest based on theory and prior research. Self-reported gender was ascertained on each survey using a 2-step question that asked participants about their current gender and their sex assigned at birth. If a participant’s self-reported gender changed across surveys, we used the gender reported most frequently by a participant (3 individuals identified as transmasculine at baseline and as nonbinary on all follow-up surveys). We collected data on self-reported race and ethnicity (available response options were Arab or Middle Eastern; Asian; Black or African American; Latinx; Native American, American Indian, or Alaskan Native or Native Hawaiian; Pacific Islander; and White), age, caregiver income, and insurance type. Race and ethnicity were assessed as potential covariates owing to known barriers to accessing gender-affirming care among transgender youth who are members of minority racial and ethnic groups. For descriptive statistics, Asian and Pacific Islander groups were combined owing to small population numbers. We included a baseline variable reflecting receipt of ongoing mental health therapy other than for the purpose of a mental health assessment to receive a gender dysphoria diagnosis. We included a self-report variable reflecting whether youths felt their gender identity or expression was a source of tension with their parents or guardians. Substance use included any alcohol, marijuana, or other drug use in the past year. Resilience was measured by the Connor-Davidson Resilience Scale (CD-RISC) 10-item score developed to measure change in an individual’s state resilience over time. 28 Resilience scores were dichotomized into high (ie, ≥median) and low (ie, <median). Prior studies of young adults in the US reported mean CD-RISC scores ranging from 27.2 to 30.1. 29 , 30

We used generalized estimating equations to assess change in outcomes from baseline at each follow-up point (eFigure 1 in the Supplement ). We used a logit link function to estimate adjusted odds ratio (aOR) for the association between variables and each mental health outcome. We initially estimated bivariate associations between potential confounders and mental health outcomes. Multivariable models included variables that were statistically significant in bivariate models. For all outcomes and models, statistical significance was defined as 95% CIs that did not contain 1.00. Reported P values are based on 2-sided Wald test statistics.

Model 1 examined temporal trends in mental health outcomes, with time (ie, baseline, 3, 6, and 12 months) modeled as a categorical variable. Model 2 estimated the association between receipt of PBs or GAHs and mental health outcomes adjusted for temporal trends and potential confounders. Receipt of PBs or GAHs was modeled as a composite binary time-varying exposure that compared mean outcomes between participants who had initiated PBs or GAHs and those who had not across all time points (eTable 2 in the Supplement ). All models used an independent working correlation structure and robust standard errors to account for the time-varying exposure variable.

We performed several sensitivity analyses. Because our data were from an observational cohort, we first considered the degree to which they were sensitive to unmeasured confounding. To do this, we calculated the E-value for the association between PBs or GAHs and mental health outcomes in model 2. The E-value is defined as the minimum strength of association that a confounder would need to have with both exposure and outcome to completely explain away their association (eTable 4 in the Supplement ). 31 Second, we performed sensitivity analyses on several subsets of youths. We separately examined the association of PBs and GAHs with outcomes of interest, although we a priori did not anticipate being powered to detect statistically significant outcomes owing to our small sample size and the relatively low proportion of youths who accessed PBs. We also conducted sensitivity analyses using the Patient Health Questionnaire 8-item scale (PHQ-8), in which the PHQ-9 question 9 regarding self-harm or suicidal thoughts was removed, given that we analyzed this item as a separate outcome. Lastly, we restricted our analysis to minor youths ages 13 to 17 years because they were subject to different laws and policies related to consent and prerequisite mental health assessments. We used R statistical software version 3.6.2 (R Project for Statistical Computing) to conduct all analyses. Data were analyzed from August 2020 through November 2021.

A total of 169 youths were screened for eligibility during the study period, among whom 161 eligible youths were approached. Nine youths or caregivers declined participation, and 39 youths did not complete consent or assent or did not complete the baseline survey, leaving a sample of 113 youths (70.2% of approached youths). We excluded 9 youths aged younger than 13 years from the analysis because they received different depression and anxiety screeners. Our final sample included 104 youths ages 13 to 20 years (mean [SD] age, 15.8 [1.6] years). Of these individuals, 84 youths (80.8%), 84 youths, and 65 youths (62.5%) completed surveys at 3, 6, and 12 months, respectively.

Our cohort included 63 transmasculine youths (60.6%), 27 transfeminine youths (26.0%), 10 nonbinary or gender fluid youths (9.6%), and 4 youths who responded “I don’t know” or did not respond to the gender identity question on all completed questionnaires (3.8%) ( Table 1 ). There were 4 Asian or Pacific Islander youths (3.8%), 3 Black or African American youths (2.9%); 9 Latinx youths (8.7%); 6 Native American, American Indian, or Alaskan Native or Native Hawaiian youths (5.8%); 67 White youths (64.4%); and 9 youths who reported more than 1 race or ethnicity (8.7%). Race and ethnicity data were missing for 6 youth (5.8%).

At baseline, 7 youths had ever received PBs or GAHs (including 1 youth who received PBs, 4 youths who received GAHs, and 2 youths who received both PBs and GAHs). By the end of the study, 69 youths (66.3%) had received PBs or GAHs (including 50 youths who received GAHs only [48.1%], 5 youths who received PBs only [4.8%], and 14 youths who received PBs and GAHs [13.5%]), while 35 youths had not received either PBs or GAHs (33.7%) (eTable 3 in the Supplement ). Among 33 participants assigned male sex at birth, 17 individuals (51.5%) had received androgen blockers, and among 71 participants assigned female sex at birth, 25 individuals (35.2%) had received menstrual suppression or contraceptives by the end of the study.

A large proportion of youths reported depressive and anxious symptoms at baseline. Specifically, 59 individuals (56.7%) had baseline PHQ-9 scores of 10 or more, suggesting moderate to severe depression; there were 22 participants (21.2%) scoring in the moderate range, 11 participants (10.6%) in the moderately severe range, and 26 participants (25.0%) in the severe range. Similarly, half of participants had a GAD-7 score suggestive of moderate to severe anxiety at baseline (52 individuals [50.0%]), including 20 participants (19.2%) scored in the moderate range, and 32 participants (30.8%) scored in the severe range. There were 45 youths (43.3%) who reported self-harm or suicidal thoughts in the prior 2 weeks. At baseline, 65 youths (62.5%) were receiving ongoing mental health therapy, 36 youths (34.6%) reported tension with their caregivers about their gender identity or expression, and 34 youths (32.7%) reported any substance use in the prior year. Lastly, we observed a wide range of resilience scores (median [range], 22.5 [1-38], with higher scores equaling more resiliency). There were no statistically significant differences in baseline characteristics by gender.

In bivariate models, substance use was associated with all mental health outcomes ( Table 2 ). Youths who reported any substance use were 4-fold as likely to have PHQ-9 scores of moderate to severe depression (aOR, 4.38; 95% CI, 2.10-9.16) and 2-fold as likely to have GAD-7 scores of moderate to severe anxiety (aOR, 2.07; 95% CI, 1.04-4.11) or report thoughts of self-harm or suicide in the prior 2 weeks (aOR, 2.06; 95% CI, 1.08-3.93). High resilience scores (ie, ≥median), compared with low resilience scores (ie, <median), were associated with lower odds of moderate or severe anxiety (aOR, 0.51; 95% CI, 0.26-0.999).

There were no statistically significant temporal trends in the bivariate model or model 1 ( Table 2 and Table 3 ). However, among all participants, odds of moderate to severe depression increased at 3 months of follow-up relative to baseline (aOR, 2.12; 95% CI, 0.98-4.60), which was not a significant increase, and returned to baseline levels at months 6 and 12 ( Figure ) prior to adjusting for receipt of PBs or GAHs.

We also examined the association between receipt of PBs or GAHs and mental health outcomes in bivariate and multivariable models (eFigure 2 in the Supplement ). After adjusting for temporal trends and potential confounders ( Table 4 ), we observed that youths who had initiated PBs or GAHs had 60% lower odds of moderate to severe depression (aOR, 0.40; 95% CI, 0.17-0.95) and 73% lower odds of self-harm or suicidal thoughts (aOR, 0.27; 95% CI, 0.11-0.65) compared with youths who had not yet initiated PBs or GAHs. There was no association between receipt of PBs or GAHs and moderate to severe anxiety (aOR, 1.01; 95% CI, 0.41-2.51). After adjusting for time-varying exposure of PBs or GAHs in model 2 ( Table 4 ), we observed statistically significant increases in moderate to severe depression among youths who had not received PBs or GAHs by 3 months of follow-up (aOR, 3.22; 95% CI, 1.37-7.56). A similar trend was observed for self-harm or suicidal thoughts among youths who had not received PBs or GAHs by 6 months of follow-up (aOR, 2.76; 95% CI, 1.22-6.26). Lastly, we estimated E-values of 2.56 and 3.25 for the association between receiving PGs or GAHs and moderate to severe depression and suicidality, respectively (eTable 4 in the Supplement ). Sensitivity analyses obtained comparable results and are presented in eTables 5 through 8 in the Supplement .

In this prospective clinical cohort study of TNB youths, we observed high rates of moderate to severe depression and anxiety, as well as suicidal thoughts. Receipt of gender-affirming interventions, specifically PBs or GAHs, was associated with 60% lower odds of moderate to severe depressive symptoms and 73% lower odds of self-harm or suicidal thoughts during the first year of multidisciplinary gender care. Among youths who did not initiate PBs or GAHs, we observed that depressive symptoms and suicidality were 2-fold to 3-fold higher than baseline levels at 3 and 6 months of follow-up, respectively. Our study results suggest that risks of depression and suicidality may be mitigated with receipt of gender-affirming medications in the context of a multidisciplinary care clinic over the relatively short time frame of 1 year.

Our findings are consistent with those of prior studies finding that TNB adolescents are at increased risk of depression, anxiety, and suicidality 1 , 11 , 32 and studies finding long-term and short-term improvements in mental health outcomes among TNB individuals who receive gender-affirming medical interventions. 14 , 21 - 24 , 33 , 34 Surprisingly, we observed no association with anxiety scores. A recent cohort study of TNB youths in Dallas, Texas, found that total anxiety symptoms improved over a longer follow-up of 11 to 18 months; however, similar to our study, the authors did not observe statistically significant improvements in generalized anxiety. 22 This suggests that anxiety symptoms may take longer to improve after the initiation of gender-affirming care. In addition, Olson et al 35 found that prepubertal TNB children who socially transitioned did not have increased rates of depression symptoms but did have increased rates of anxiety symptoms compared with children who were cisgender. Although social transition and access to gender-affirming medical care do not always go hand in hand, it is noteworthy that access to gender-affirming medical care and supported social transition appear to be associated with decreased depression and suicidality more than anxiety symptoms.

Time trends were not significant in our study; however, it is important to note that we observed a transient and nonsignificant worsening in mental health outcomes in the first several months of care among all participants and that these outcomes subsequently returned to baseline by 12 months. This is consistent with findings from a 2020 study 36 in an academic medical center in the northwestern US that observed no change in TNB adolescents’ GAD-7 or PHQ-9 scores from intake to first follow-up appointment, which occurred a mean of 4.7 months apart. Given that receipt of PBs or GAHs was associated with protection against depression and suicidality in our study, it could be that delays in receipt of medications is associated with initially exacerbated mental health symptoms that subsequently improve. It is also possible that mental health improvements associated with receiving these interventions may have a delayed onset, given the delay in physical changes after starting GAHs.

Few of our hypothesized confounders were associated with mental health outcomes in this sample, most notably receipt of ongoing mental health therapy and caregiver support; however, this is not surprising given that these variables were colinear with baseline mental health, which we adjusted for in all models. Substance use was the only variable associated with all mental health outcomes. In addition, youths with high baseline resilience scores were half as likely to experience moderate to severe anxiety as those with low scores. This finding suggests that substance use and resilience may be additional modifiable factors that could be addressed through multidisciplinary gender-affirming care. We recommend more granular assessment of substance use and resilience to better understand support needs (for substance use) and effective support strategies (for resilience) for TNB youths in future research.

This study has a number of strengths. This is one of the first studies to quantify a short-term transient increase in depressive symptoms experienced by TNB youths after initiating gender-affirming care, a phenomenon observed clinically by some of the authors and described in qualitative research. 37 Although we are unable to make causal statements owing to the observational design of the study, the strength of associations between gender-affirming medications and depression and suicidality, with large aOR values, and sensitivity analyses that suggest that these findings are robust to moderate levels of unmeasured confounding. Specifically, E-values calculated for this study suggest that the observed associations could be explained away only by an unmeasured confounder that was associated with both PBs and GAHs and the outcomes of interest by a risk ratio of 2-fold to 3-fold each, above and beyond the measured confounders, but that weaker confounding could not do so. 31

Our findings should be interpreted in light of the following limitations. This was a clinical sample of TNB youths, and there was likely selection bias toward youths with supportive caregivers who had resources to access a gender-affirming care clinic. Family support and access to care are associated with protection against poor mental health outcomes, and thus actual rates of depression, anxiety, and suicidality in nonclinical samples of TNB youths may differ. Youths who are unable to access gender-affirming care owing to a lack of family support or resources require particular emphasis in future research and advocacy. Our sample also primarily included White and transmasculine youths, limiting the generalizability of our findings. In addition, the need to reapproach participants for consent and assent for the 12-month survey likely contributed to attrition at this time point. There may also be residual confounding because we were unable to include a variable reflecting receipt of psychotropic medications that could be associated with depression, anxiety, and self-harm and suicidal thought outcomes. Additionally, we used symptom-based measures of depression, anxiety, and suicidality; further studies should include diagnostic evaluations by mental health practitioners to track depression, anxiety, gender dysphoria, suicidal ideation, and suicide attempts during gender care. 2

Our study provides quantitative evidence that access to PBs or GAHs in a multidisciplinary gender-affirming setting was associated with mental health improvements among TNB youths over a relatively short time frame of 1 year. The associations with the highest aORs were with decreased suicidality, which is important given the mental health disparities experienced by this population, particularly the high levels of self-harm and suicide. Our findings have important policy implications, suggesting that the recent wave of legislation restricting access to gender-affirming care 19 may have significant negative outcomes in the well-being of TNB youths. 20 Beyond the need to address antitransgender legislation, there is an additional need for medical systems and insurance providers to decrease barriers and expand access to gender-affirming care.

Accepted for Publication: January 10, 2022.

Published: February 25, 2022. doi:10.1001/jamanetworkopen.2022.0978

Correction: This article was corrected on July 26, 2022, to fix minor errors in the numbers of patients in eTables 2 and 3 in the Supplement.

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2022 Tordoff DM et al. JAMA Network Open .

Corresponding Author: Diana M. Tordoff, MPH, Department of Epidemiology, University of Washington, UW Box 351619, Seattle, WA 98195 ( [email protected] ).

Author Contributions : Diana Tordoff had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Diana Tordoff and Dr Wanta are joint first authors. Drs Inwards-Breland and Ahrens are joint senior authors.

Concept and design: Collin, Stepney, Inwards-Breland, Ahrens.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Tordoff, Wanta, Collin, Stepney, Inwards-Breland.

Critical revision of the manuscript for important intellectual content: Wanta, Collin, Stepney, Inwards-Breland, Ahrens.

Statistical analysis: Tordoff.

Obtained funding: Inwards-Breland, Ahrens.

Administrative, technical, or material support: Ahrens.

Supervision: Wanta, Inwards-Breland, Ahrens.

Conflict of Interest Disclosures: Diana Tordoff reported receiving grants from the National Institutes of Health National Institute of Allergy and Infectious Diseases unrelated to the present work and outside the submitted work. No other disclosures were reported.

Funding/Support: This study was supported Seattle Children’s Center for Diversity and Health Equity and the Pacific Hospital Preservation Development Authority.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.8(2); 2016 May

How to Write Up Your Quality Improvement Initiatives for Publication

The Journal of Graduate Medical Education often receives submissions from trainees and educators highlighting work they do in quality improvement (QI). This is remarkably encouraging given the emphasis that the Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System has placed on integrating QI into the clinical learning environment. 1 A major challenge for editors reviewing these manuscripts is the inconsistency with which authors report QI initiatives. After reviewing a large number of these submissions, we have noted common problems that arise and have prepared the following guide to help prospective authors prepare QI reports for publication.

Consistent with the Journal 's common format for Original Research or Educational Innovation articles (ie, Introduction, Methods, Results, Discussion, and Conclusion), our suggestions will highlight what authors should explicitly emphasize within each of these manuscript sections as it pertains to their QI initiative. We realize that a number of other frameworks and guidelines exist, the most common being the SQUIRE guidelines, which were updated in the fall of 2015. 2 Our suggestions are synergistic with the updated SQUIRE 2.0 guidelines, yet they also provide a high-level view of the philosophies that underpin these guidelines to help authors not only at the time of writing, but also when planning and implementing their QI initiatives.

What Is the Quality Problem, Why Is It Important, and What Is Your Aim?

The Introduction section must be brief. This is not the time to provide an in-depth review of the literature on your quality problem of interest—which could be an important but separate paper. Instead, it is most important to articulate why this quality problem is relevant beyond your local institution . Is it a common safety problem ubiquitous to multiple care settings, such as the need to improve patient handoffs? Or perhaps there are legislative changes or financial incentives that promote interest in your issue, such as the linking of financial reimbursement to readmission rates. Whatever the reason, it is critical to make clear what the external impact of your QI initiative would be for other groups and the readers of the journal. If the QI problem can be framed only as a need specific to your own setting, then the results may be best published in a local newsletter rather than a national or international journal.

After succinctly outlining the importance and relevance of the QI problem, the Introduction must describe the gap between current practice and preferred practice . What prior QI strategies have or have not worked to address this quality problem? If there is no gap in our understanding on how to improve practice, then further study of the area would be of limited value. Authors must demonstrate understanding of the pertinent literature in order to briefly discuss prior strategies that have been attempted; this usually includes the strategy as well as the required resources and resulting outcomes. Replication of a successful intervention in a new setting can help to fill key gaps in understanding: here the evidence gap is whether a prior strategy can be replicated in a different setting, one that is dissimilar in important features to the initial study.

Finally, the Introduction must also make clear what you hoped to achieve by carrying out your QI initiative. In 1 or 2 sentences, the final paragraph should clearly state the primary aim of your QI project. There are numerous references providing guidance on how to write good “aim” statements, by making sure that they are “SMART” (specific, measurable, achievable, realistic, anytime bound). 3 A clear and concise statement of the primary aim, and any relevant sub-aims, will ground the readers in the main purpose of your QI project.

Describing the Proposed Intervention, Including a Theory for Change, and the Road Toward Improvement

For QI reports the Methods section is probably the most important section as it ensures that readers understand how they can translate your reported innovation into their own settings. To facilitate this, authors must attend to several important issues. These include the context in which the QI work was carried out (ie, setting and participants) and a detailed description of the implementation strategy . The strategy must also include a theory for why a specific intervention (or set of interventions) was chosen.

Let's start with the proposed intervention. In QI, too often authors simply reach for the first available solution off the shelf without first considering why that particular solution could address the problem at hand. Frequently, authors appear to believe that checklists and order sets will solve everything. The truth is that if authors do not articulate a theory or rationale for why their proposed intervention should fix the quality problem of interest, they run the risk of designing a suboptimal intervention or choosing the wrong approach altogether. For example, attempts to reduce physician prescribing of unnecessary antibiotics to children with upper respiratory tract infections (URTIs; eg, continuing medical education, postcard reminders summarizing treatment guidelines, etc) consistently fail because they primarily address provider awareness rather than the dominant driver, which is parental demand for antibiotics. Theorizing that the ideal intervention should address parental expectations, one would instead choose an approach such as implementing the use of delayed antibiotic prescriptions, which in a recent Cochrane review has been shown to significantly reduce antibiotic utilization for the treatment of URTIs in children. 4

Similar to the general call for more theory-based interventions in medical education, 5 there has been a recent call for more theory-based QI interventions. 6 Therefore, the preferred approach is to clearly articulate the link between the proposed solution and the problem it will solve. For example, a hospitalist team seeking to reduce unnecessary urinary catheter use might theorize that a key driver is that residents do not know whether a patient has a urinary catheter in situ . A sensible solution therefore might bypass the physician altogether. For example, instituting automatic stop orders 7 and nursing advanced directives 8 to remove urinary catheters are 2 interventions that have previously been shown to be effective.

It is also crucial that authors state not only what the intervention was and its underlying rationale, but also how it was iteratively tested, refined, and eventually implemented. A common framework used in QI is rapid cycle change methodology or PDSA (plan-do-study-act) cycles. 9 Unfortunately, many published QI reports, despite claiming to use PDSA cycles, demonstrate little evidence that they refined their intervention prior to implementation. 10 This is a problem because for QI initiatives the devil really is in the details. It is simply not good enough to say that “we implemented a checklist” or “we created a new care pathway.” Instead, authors need to report how changes were tested and refined; reflect on what worked, what did not, and why; and provide a description of the eventual intervention. Recognizing how challenging it can be to chronicle the evolution of the proposed intervention from start to finish in a concise manner, authors should consider the use of a figure or a table to summarize the key PDSA cycles, which will avoid excess word length while still providing a concise summary of what was actually done. Another option for providing more details is to include additional supplemental information for publication online. For QI projects it is imperative that at least 2 cycles, and usually more, are described in the Methods section.

The Local Context and Its Impact on the QI Initiative

In evaluative research, authors go to great lengths to describe how they have controlled for contextual factors to ensure that they have eliminated any bias that might unduly influence their outcomes of interest. QI is different in this regard: context is critical to understand and characterize, not control . Authors must include details about their context and how these might influence the implementation or outcomes of QI projects to sensitize readers to the contextual factors that require careful consideration when introducing the QI intervention to local institutions.

Batalden and Davidoff 11 described the importance of context in a brief commentary. They provided a framework for QI that links generalizable scientific evidence to a particular context in order to generate measured performance improvement. Importantly, they emphasized that the focus should be both on the context, as well as how the generalizable scientific evidence (or the proposed intervention) integrates within the particular local context. In QI, a detailed description of the context is just as important as a detailed description of the proposed intervention.

Consider the example of a QI project to reduce unnecessary lab ordering. The reader would want to know: Does the institution have computer-based or paper-based orders? If computer-based, is it easy to make changes to the order entry system to introduce clinical decision support? How engaged is the lab in clinical QI initiatives? What is the front line staff capacity and capability for QI? All of these contextual factors play heavily into the choice of the intervention, how the intervention gets implemented, and how it affects project outcomes.

This emphasis on context for QI reports parallels a similar need in reports of educational interventions. For example, educators implementing bedside procedure training must also account for and describe relevant contextual factors, such as whether a simulation lab is available, whether faculty have maintained competence in bedside procedures, whether a culture of direct observation and feedback exists, and whether there is a mechanism to track procedures and monitor for complications. Similar to medical education research, explicitly acknowledging the role of context is paramount in the reporting of any QI initiative.

What Is the Evaluation Plan?

Most QI initiatives rely on the Donabedian model of outcome, process, and balancing measures to evaluate the impact of their intervention. 12 While beyond the scope of this article to address the specifics of measurement in QI, we will offer several practical suggestions. First, most QI projects will focus on improving processes of care and may not be able to demonstrate downstream impact on clinical outcomes. This is acceptable, as long as the authors have selected process measures that are tightly coupled with the clinical outcome of interest. For example, an orthopedic surgery residency team aiming to improve venothromboembolism (VTE) prophylaxis rates could justifiably track VTE prophylaxis administration as a clinical process because hospitalized patients who receive VTE prophylaxis have a very low likelihood of developing VTEs. 13

Another useful process measure to report is one that measures the fidelity of the intervention. In other words, include a process measure that tracks how consistently or reliably your intervention is applied. For example, if your main intervention to improve VTE prophylaxis is the creation and implementation of a standardized order set, a measure of implementation fidelity would be to track whether residents and faculty actually used the order set. This is particularly informative for unsuccessful QI projects—interventions with high fidelity suggest that other contributing factors require attention, whereas interventions with low fidelity suggest that the evaluation may have been premature and more work is needed to increase uptake of the intervention before large-scale implementation and evaluation are undertaken.

Last but not least, balancing measures, which are intended to measure unintended consequences, often are missing from QI reports. A medical journal would not accept a clinical trial that reports only on the potential benefits and not the harms of a novel therapy, and we need to hold reports of QI interventions to a similar standard. Therefore, balancing measures of unintended consequence should be reported to ensure that the QI intervention improves care and does not create new problems. For example, if a QI initiative focuses on improving resident adherence to guidelines for a clinical area, such as diabetes care, does adherence to other guidelines, such as preventive screening, decline? If a new electronic handover tool is developed to support handoff communication, are there errors in the new document due to cut/paste activities? Selecting and reporting on sound balancing measures ensures a healthy respect for the law of unintended consequences in QI.

Providing Data With Greater Clarity

When it comes to displaying the data, it is best to avoid simple before-after comparisons. This evaluative approach is suboptimal because secular trends make it difficult to attribute observed differences to the intervention. Traditional approaches to research and evaluation would typically address this limitation through the inclusion of a contemporaneous control group or setting, which is also suitable for QI studies.

An alternative approach would be to display your outcome or process measures over time through the use of statistical process control. This methodology often utilizes run charts or control chart s to display data over time ( figure ). Following run chart or control chart “rules,” one can interpret the data plotted sequentially over time to identify instances when variation is not due to random chance (so-called special cause variation). Such handling of data enhances the ability to determine whether changes that occurred were a result of the interventions introduced, and greatly strengthens the evaluative approach as compared to aggregated before-after data. The BMJ Quality & Safety journal has published an overview on the use and interpretation of run charts. 14

Example of Control Chart to Display Quality Data Over Time

Note: This is an example of a control chart (specifically a P-chart). A typical control chart has the quality measure of interest on the Y-axis. The X-axis is always a time scale (in this case, consecutive months). As the team carries out the quality improvement initiative, they collect data prospectively over time and plot the data on a control chart. Using statistical process control software, several lines are plotted. The dotted line is the center line (CL) and is equivalent to the mean. The dashed lines on either side of the mean are the upper and lower control limits (UCL and LCL; approximately 3 standard deviations, or sigmas, on either side of the mean). Using this information, the statistical process control software can identify segments of the chart where nonrandom variation is occurring (so-called special cause variation). On this graph, the 2 times where nonrandom variation are occurring are indicated by the triangle and circle markers, suggesting that modification to the order set, and not education, was likely responsible for the improvement seen in venothromboembolism (VTE) prophylaxis ordering.

What Are the Implications of the Work? What Are the Next Steps?

For QI papers, the Discussion section will be similar to papers describing educational innovations or research. This section should concisely summarize the main findings of the QI project, relate the key findings to what is already known in the published literature, reflect on the broader implications of the findings, discuss how important limitations could have affected the findings, and briefly introduce next steps to further understand the field.

Perhaps most important are the reflections on lessons learned and future directions. In particular, reflections on the influence of the local context on project implementation and outcomes are highly relevant as readers will need to understand this if they want to replicate the intervention within their local context.

Well-conducted QI interventions that produced “negative” results (ie, did not achieve their intended outcomes) are still important and worthwhile for dissemination. Your reflections on why the intervention did not work can be helpful to others who might consider a similar initiative. In some cases, the problem may be the intervention itself, which signals the need to consider an alternative approach to addressing the QI problem of interest. More commonly, the implementation of the intervention lacked fidelity, or the integration of the intervention within the local context was suboptimal. In these instances, your QI report will still be helpful to others who can build on your work.

The Conclusion section of the report is also similar to Original Research and Educational Innovation articles. This short paragraph succinctly summarizes the most important findings from the study, without speculating beyond the results. Conclusions should be appropriately conservative in relation to the study findings. See the table for a summary of elements essential for QI reports.

Quality Improvement (QI) Reports: Recommended Elements and Common Pitfalls

By providing this overview of the approach to writing up QI initiatives, we hope to clarify, up front, those aspects of your initiative that require the most emphasis. The considerations presented here can serve as a high-level guide to authors, with the goal of disseminating QI reports that are more useful for other programs. QI studies that involve residents, faculty, or the general graduate medical education environment as key elements of the context or intervention are appropriate for submission to the Journal of Graduate Medical Education . We look forward to publishing reports that inform programs and educators about effective faculty and learner engagement in QI activities within the graduate medical education learning environment.