Even good journals sometimes make mistakes.
Impact factor is sometimes used problematically.
Beginner researchers might not have expertise to judge.
Citation counts can be misleading.
Sometimes older information is still relevant.
Essential Science Conversations
Have you heard about the initiative at the National Institutes of Health (NIH) to improve the peer review of research project grant and fellowship applications? Join us as NIH describes the steps the agency is taking to simplify its process of assessing the scientific and technical merit of applications, better identify promising scientists for training opportunities, and mitigate elements that have the potential to introduce bias in review.
This program does not offer CE credit.
Valerie Durrant, PhD
A discussion on best practices as a mentor, including mentoring scholars of color.
January 2023 On Demand Webinar
McGuire and a panel of psychological scientists discuss how your research and expertise can shape policy, garner funding for essential programs, and improve human lives.
October 2022 On Demand Webinar
Discusses what reporters and editors want and need from scientist sources, how psychological scientists can present their work in compelling ways, and the do’s and don’ts when working with the news media.
September 2022 On Demand Webinar
Discusses the need to shift psychology’s focus from the individual researcher, and their career and work, to a truly publicly-engaged science, where the public is involved in all steps of the research process.
April 2022 On Demand Webinar
BMC Medical Research Methodology volume 24 , Article number: 184 ( 2024 ) Cite this article
209 Accesses
1 Altmetric
Metrics details
Digital mental health interventions (DMHIs) overcome traditional barriers enabling wider access to mental health support and allowing individuals to manage their treatment. How individuals engage with DMHIs impacts the intervention effect. This review determined whether the impact of user engagement was assessed in the intervention effect in Randomised Controlled Trials (RCTs) evaluating DMHIs targeting common mental disorders (CMDs).
This systematic review was registered on Prospero (CRD42021249503). RCTs published between 01/01/2016 and 17/09/2021 were included if evaluated DMHIs were delivered by app or website; targeted patients with a CMD without non-CMD comorbidities (e.g., diabetes); and were self-guided. Databases searched: Medline; PsycInfo; Embase; and CENTRAL. All data was double extracted. A meta-analysis compared intervention effect estimates when accounting for engagement and when engagement was ignored.
We identified 184 articles randomising 43,529 participants. Interventions were delivered predominantly via websites (145, 78.8%) and 140 (76.1%) articles reported engagement data. All primary analyses adopted treatment policy strategies, ignoring engagement levels. Only 19 (10.3%) articles provided additional intervention effect estimates accounting for user engagement: 2 (10.5%) conducted a complier-average-causal effect (CACE) analysis (principal stratum strategy) and 17 (89.5%) used a less-preferred per-protocol (PP) population excluding individuals failing to meet engagement criteria (estimand strategies unclear). Meta-analysis for PP estimates, when accounting for user engagement, changed the standardised effect to -0.18 95% CI (-0.32, -0.04) from − 0.14 95% CI (-0.24, -0.03) and sample sizes reduced by 33% decreasing precision, whereas meta-analysis for CACE estimates were − 0.19 95% CI (-0.42, 0.03) from − 0.16 95% CI (-0.38, 0.06) with no sample size decrease and less impact on precision.
Many articles report user engagement metrics but few assessed the impact on the intervention effect missing opportunities to answer important patient centred questions for how well DMHIs work for engaged users. Defining engagement in this area is complex, more research is needed to obtain ways to categorise this into groups. However, the majority that considered engagement in analysis used approaches most likely to induce bias.
Peer Review reports
One in four people experience a mental health problem every year [ 1 ]. However, an estimated 70% with mental ill health are unable to access treatment [ 2 ]. App and web-based tools, collectively digital mental health interventions (DMHIs), are low cost, scalable [ 3 ], and have potential for overcoming traditional barriers to treatment access, such as physical access (flexibility in treatment location), confidentiality (providing anonymity), and stigma [ 4 ]. In recent years, the number of available DMHIs has rapidly increasd [ 5 ], the Apple App Store alone has over 10,000 behavioural apps [ 6 ]. This rapid increase combined with the complex nature of DMHIs has meant safety and effectiveness regulations have lagged behind [ 7 ]. Additionally, many DMHIs are developed for commercial purposes and marketed to the public without scientific evidence [ 8 ]. The current National Institute for Health and Care Excellence (NICE) guidelines [ 9 ] for digital health technologies, advocate for use of randomised controlled trials (RCTs) to evaluate the effectiveness of digital interventions in specific conditions such as mental health. Promisingly, the number of digital interventions evaluated in RCTs over the last decade has more than doubled [ 10 ].
Many DMHIs are developed through the digitalisations of existing services, such as online self-led formats of conventional therapist-delivered treatments. However, in contrary to conventional therapist-led treatments, DMHIs offer flexible anytime access for individuals [ 11 ]. This change in delivery means existing evidence of risk-benefit balance from structured therapist-delivered interventions is not translatable. DMHIs are potential solutions to provide more individuals with much needed treatment access, but they are not without challenges. In 2018 the James Lind Alliance (JLA) patient priority setting group for DMHIs set out the top 10 challenges to address [ 12 ]. Overcoming these challenges is essential for DMHIs to successfully improve treatment access and health outcomes in mental health [ 13 , 14 ]. One theme that emerged from across the priorities was the importance of improving methods for evaluating DMHIs including the impact of user engagement.
The impact user engagement has on DMHIs efficacy is poorly understood [ 6 , 15 , 16 ]. Although DMHIs are widely available, user engagement with DMHIs is typically low [ 17 ]. For multi-component DMHIs (commonly including psychoeducation, cognitive exercises, self-monitoring diary), a minimally sufficient engagement in DMHIs is often crucial for establishing behavioural changes and thus improved health outcomes [ 18 ]. However, achieved sustained behavioural changes by engaging with DMHIs is a multidimensional construct that is both challenging to assess and the pathway for patients to achieve this is complex [ 19 , 20 ]. Unlike other interventions, DMHIs are unique in that web-based or app-based interventions can capture interactions from individuals. User engagement can be measured and recorded using automatically captured indicators (e.g., pageviews, proportion of content/modules completed, or number of logins). However, the large variety in measurable indicators across different DMHIs [ 16 , 21 ] further compounds challenges to understanding pathways to sustained behaviour changes.
For RCTS, the latest estimand framework in the ICH E9 R1 addendum [ 22 ] provides guidance on defining different estimands, which enables trialists to ensure the most important research questions of interest are evaluated. This includes guidance on handling post-randomisation events, such as user engagement with the DMHI, in efficacy analysis. For example, policy makers are likely to be most interested in a treatment policy estimand which provides an assessment of the benefit received on average under the new policy of prescribing the DMHI regardless of how it’s engaged with. For DMHIs typically engagement is poor, which means treatment policy estimands may underestimate the true intervention efficacy for those who engaged [ 23 ], so alternative estimands that address this may also be of interest to target. For example, the benefit received on average for individuals who would actively engage with the DMHI (a principal stratification estimand). However, to utilise available methods post-randomisation variables need to be clearly defined, but this is difficult for engagement with DMHIs because it is multifaceted with many different engagement indicators available to use.
This systematic review aimed to assess the current landscape of how RCTs for DMHIs are reported and analysed. The review primarily assessed how user engagement is described, what engagement indicators are reported and how, if at all, researchers assessed the impact of user engagement on efficacy. As the number of DMHIs evaluated in RCTs is ever increasing, this review is essential to identify current practice in trial reporting to inform further research to improve the quality of future trials. The specific research aims of interest were to: (1) examine trial design and characteristics of DMHIs; (2) summarise how user engagement had been defined and measured in RCTs of DMHIs; and (3) assess how often intervention efficacy was adjusted for user engagement and the impact of user engagement on efficacy estimates.
The protocol for this systematic review was prospectively published in Prospero [ 24 ], and PRISMA guidance was followed in reporting of this review.
We included RCTs examining the efficacy of DMHIs, excluding pilot and feasibility studies [ 25 ]. Search terms for RCT designs followed guidance from Glanville et al. [ 26 ]. We included trials of participants with common mental disorders (CMD) defined by Cochrane [ 27 ] excluding populations with non-CMD comorbidities, such as patients with depression and comorbid diabetes. Populations with multiple CMDs were not excluded as there were many transdiagnostic interventions targeting overlapping symptoms of different conditions. Both trials requiring a confirmed clinical diagnosis and trials where participants self-referred were included. For consistency in DMHIs included interventions must meet any criteria from items 1.1 (targeted communication on health information), 1.3 (client to client communication, e.g., peer forums), 1.4 (health tracking or self-monitoring) or 1.6 (access to own health information) from the WHO Classification of Digital Health Interventions [ 28 ]. DMHIs must have been delivered on a mobile app or through a web-browser and where the intervention was self-guided by participants, defined as an intervention where participants have full autonomy over how this is used. Search terms for interventions followed guidance from Ayiku et al. [ 29 ]. All publications must have been reported in English.
The search was performed on the 17th September 2021 and included trials published between 1st January 2016 to 17th September 2021. Search terms were adapted for each database: MEDLINE, Embase, PsycINFO and Cochrane CENTRAL (see supplemental table S1 for search strategy). Title and abstracts were independently screened by two reviewers (JE, RB, SO, LB, LM & VH), and again at the full text review stage. Covidence [ 30 ] was used to manage all stages, remove duplicates and resolve disagreements.
As a methodology review to examine how user engagement was described and analysed a risk of bias tool to assess trial quality was not undertaken [ 31 ]. However, key CONSORT items [ 32 ] were extracted to determine adherence to reporting guidance, including reporting of a protocol or trial registration (item 23/24), planned sample size (item 7a) and amendments to the primary analysis (item 3b). For all items self-reported data from articles was extracted.
A data extraction form was developed by the lead author (JE) and reviewed by VC, SC and JS. Summary data extracted covered: trial characteristics (e.g., design and sample size); intervention and comparator descriptions (e.g., delivery method or primary function); participant demographics (e.g., age or gender); reporting of user engagement (e.g., indicators reported); and point estimates, confidence intervals and P-values of analysis results unadjusted and adjusted for user engagement. In trials with multiple arms the first active arm mentioned was included. No restriction was applied to the control arm in the trial. The full extraction sheet, including CONSORT items, is in the table S2 of the supplementary material.
The analysis was predominantly descriptive and used mean and standard deviations, or medians and interquartile ranges (IQRs) to describe continuous variables. Frequencies and percentages summarized categorical variables. User engagement captured through engagement indicators (e.g., pageviews and total logins) and methods to encourage user engagement (e.g., automatic notifications) were summarised descriptively. Indicator data was summarised in four categories: duration of use (e.g., length of session), frequency of use (e.g., number of logins), milestone achieved (e.g., modules completed) and communication (e.g., messages to therapist). Descriptive summaries also assessed both the recommended user engagement definitions, the pre-specified minimum engagement level investigators told participants to use DMHIs, and active user definitions, the pre-specified engagement level of most interest to investigators for intervention effects accounting for user engagement. Both were summarised by indicators used in definitions.
To determine the impact of user engagement on intervention efficacy, restricted maximum likelihood random effects meta-analyses were conducted for articles that reported both intervention effect when user engagement was accounted for and when it wasn’t. Standardised effects were used due to outcomes and measures varying between articles. These were taken directly, where reported, otherwise calculated using guidance from Cochrane [ 33 ], and Cohen’s d formula for the standard deviation [ 34 ]. Articles were grouped by outcome domains (e.g., depression, anxiety or eating disorders) based on the reported primary clinical outcome used to evaluate efficacy. Analyses also group articles based on the analytical approach used for adjustment, those using statistical methods that retained all participants formed one group (recommend approaches) and those using statistical methods only retaining conventional per-protocol populations, i.e., exclude the data from those who did not comply, formed the other group (per-protocol approaches). All analysis was performed using Stata 17.
From a total of 6,042 articles identified, 184 were eligible and included in this review (see Fig. 1 ) randomising 43,529 participants. The most evaluated outcome domain was Depression, 74 (40.2%) articles, followed by Anxiety, 29 (15.8%) articles, and PTSD, 12 (6.5%) articles, see supplementary table S3 for full list. At least 123 unique interventions were assessed, however some interventions ( n = 39) were only described in general terms, such as internet delivered cognitive behaviour therapy for depression, so could not be distinguished as separate interventions and are excluded from the count. On average 30.7 (SD 7.7) articles were published each year, a more detailed breakdown by outcome domain is in supplementary figures s1 and s2 .
PRISMA flowchart for studies included in the systematic review
Extracted CONSORT items assessed trial reporting quality, 51 articles (27.7%) did not report their planned sample size and 36 articles (19.7%) did not clearly reference a trial protocol or trial registration number. For the 133 articles that reported both the planned and actual sample size, 43 (32.3%) failed to recruit to their target. The planned analysis approach was reportedly changed in 3 (1.6%) articles, one due to changes in the intervention [ 35 ] and the others due to high attrition [ 36 , 37 ].
Most articles used “traditional” trial designs with 170 (92.4%) opting for a parallel arm design and the majority assessed only one new intervention ( n = 134, 78.8%). Four articles (2.2%) used a factorial design allowing for the simultaneous evaluation of multiple treatments providing statistical efficiency by reducing the number of participants required in the trial. Two articles (1.1%) in the Body Dysmorphic Disorder outcome domain reported using a crossover design. However, the first had no wash-out period and instead those in the intervention arm were asked to stop engaging with the app after 16 days [ 38 ]. The second actually used a parallel arm design, where the control group received the intervention after 3 weeks [ 39 ]. Median delivery period for DMHIs was 56 days (IQR 42–84) post-randomisation and the median total follow-up time for primary outcome collection was 183 days post-randomisation (IQR 84–365).
Participants average age was 34.1 years (SD 11.1), and most participants were female (70.7%), see Table 1 . Ethnicity data was not extractable in 133 (72.3%) articles. Most trials required a confirmed diagnosis of a CMD, such as through a structured interview, for inclusion ( n = 110, 59.8%). Symptom severity could not be extracted in 97 (52.7%) trials, but where available the most common (49 trials, 56.3%) severity was a combination of both mild and moderate. Only 12 (6.5%) articles assessed participants with severe symptomatology in the depression domain ( n = 7, 58.3%), anxiety ( n = 1, 8.3%), psychological distress ( n = 1, 8.3%), general fatigue ( n = 1, 8.3%), post-traumatic stress disorder ( n = 1, 8.3%), or psychosis ( n = 1, 8.3%).
Most interventions were delivered through a website, 145 (78.8%), see Table 2 . There were 76 (41.3%) trials that adapted interventions from existing in-person therapist led interventions, and 84 (45.7%) interventions were newly developed. App delivered interventions were more likely to be newly developed, 23 (71.9%), compared to website interventions, 57 (39.3%). Most common choice of control arm was usual care, 126 (68.5%). For articles with usual care as control, most opted to use wait-lists, 94 (74.6%), where intervention access was provided either immediately after the intervention period, 62/94 (66.0%), or after the total follow-up period, 32/94 (34.0%).
Most articles, 136 (73.9%), reported using at least one approach to encourage participants to engage with the intervention. Methods of encouragement were automatic notifications, n = 49/136 (32.5%), contacting participants by telephone or email, n = 68/136 (45.0%), or automated feedback on homework exercises, n = 76/136 (50.3%). Most used only one method of encouragement, n = 85 (62.5%), with 6 (4.4%) articles using all 3 methods of encouragement. Although many articles encouraged engagement, only 23.9% ( n = 44) provided a recommended level of engagement to participants. Recommendations varied from using a rate to progress through content (e.g., one module per week or maximum of two modules per week), a specified duration to use the intervention (e.g., 1.5 h per week or 4 to 6 h per week), or specifying milestones to complete (e.g., complete one lesson every 1–2 weeks or complete daily homework assignments), a full list is in table s5 of the supplementary material.
User engagement data captured through indicators was reported in many articles, 76.1% ( n = 140), Fig. 2 . Typically, this included only reporting only one indicator ( n = 41, 29.3%) ranging up to eight indicators for one (0.7%) trial [ 40 ]. Across the 140 studies reporting user engagement data, most commonly indicators described the frequency of use, 150 (40.7%), followed by indicators to capture milestones achieved, 124 (33.6%), further detail is found in table s4 of the supplemental. A total of 150 unique indicators were reported across the 140 articles, the most popular measure used was modules completed, 51.3% ( n = 77), followed by the number of logins, 25.3% ( n = 38). In website only delivered interventions there were 102 unique indicators compared to 41 unique indicators reported in app-based interventions, and 7 unique indicators in interventions delivered as both an app and website.
Proportion of trials describing user engagement in methods section ( A ) or in results section ( B )
A) – How user engagement was reported in the methods section
Recommended – the participant was told how to use the intervention by the study team
Encouraged – reminders (e.g., notifications or emails) were sent to the participant
Active User – participants meeting a pre-specified engagement level set by the study team
B) – How user engagement data was reported in the results section
Reported – results describe activity for at least one engagement indicator
Analysis – results report an intervention effect where user engagement has been considered
Active user definitions, the engagement level of most interest to trial teams, was stated in the methods sections for 20.1% ( n = 37) of articles. Digital components of active user definitions included setting a minimum number of modules completed (e.g., 4 out of 5 modules), a proportion of content accessed (e.g., at least 25% of pages viewed), or the total time accessed (e.g., used app for 30 min per week), a full list of active user definitions is in table s6 of the supplemental. From the 37 articles reporting active user definitions, 27 (14.7%) described statistical methods to perform an analysis accounting for user engagement but only 19 (10.3%) reported intervention effect estimates.
All articles reporting effects from the analysis accounting for user engagement also reported effects not accounting for engagement so were included a meta-analysis, Table 3 . All articles used a treatment policy estimand (including all participants randomised regardless of the level of user engagement) for their primary outcome, where user engagement was not accounted for. In articles reporting an analysis accounting for user engagement, all outcome domains reported an increase in overall effect size favouring the intervention in comparison to estimates from analysis not accounting for user engagement. The largest increase in intervention efficacy was in the distress domain ( n = 1) where the standardised mean effect size increased from − 0.61 (95% CI -0.86 to -0.36) to -0.88 (95% CI -1.17 to -0.59).
The results comparing changes in the intervention effect by the analysis approach used (recommended versus per-protocol) is in Table 4 . From the 19 articles included in the analysis, 17 (89.5%) used a conventional per-protocol (i.e., exclude the data from those who did not comply) approach for the analysis accounting for user engagement [ 41 ]. A consequence of which is that the average sample size decreased to 76.9% (IQR 67.7–87.6%) of the original size, in the active arm the average size decreased by 61.8% (IQR 38.1–75.4%). The overall standardised intervention effect increased from − 0.14 (95% CI -0.24 to -0.03, n = 17), p = .01, to -0.18 (95% CI -0.32 to -0.04, n = 17), p = .01, but was also less precise. Two trials used a Complier Average Causal Effect (CACE) analysis [ 42 ], a recommended approach where assumptions hold, with all participants randomised included in the analysis. The overall standardised intervention effect increased in the meta-analysis with an overall change from − 0.16 (95% CI -0.38 to 0.06, n = 2), p = .16, to -0.19 (95%CI -0.42 to 0.03, n = 2), p = .09, with no decrease in sample size and slightly less impact on the precision of the estimate.
This systematic review found that in trials of DMHIs for CMDs, promisingly many articles reported user engagement as summaries of automatically captured indicators, but the reported intervention effect rarely accounted for this. Overall, trials were not well reported, almost 30% did not reference a trial protocol and only 27% of articles had available data on ethnicity. The JLA patient priority group set user engagement as a research priority in 2018 and this review, including publications between 2016 and 2021, supports evidence that engagement data has been poorly utilised where only 10% ( n = 19) of articles had available estimates to evaluate the impact of user engagement on intervention efficacy. Many (> 70%) articles reported summarised engagement data highlighting plenty of opportunities to better utilise this data and understand the relationship between user engagement and efficacy, a question of particular interest to the individual using DMHIs to know the true intervention efficacy.
Many articles reported at least one method used to encourage participants to engage with the intervention, however very few articles were able to specify what the recommended level of engagement should be for individuals. Additionally, only a small proportion of trials assessed the impact of user engagement on the intervention efficacy through active user definitions, but these were broad ranging and used a variety of different engagement indicators. This highlights the complex and challenging task to properly assess user engagement where currently there is little guidance available. This also shows how difficult it is for researchers to identify what the minimum required engagement to the intervention, active user definitions, should be due to the heterogeneity in both the individuals being treated and how the intervention is being delivered (e.g., timeliness and access to other support).
Most articles performing an analysis that accounted for engagement used a conventional per-protocol population. Although the per-protocol population can be unbiased under the strong assumption that user engagement is independent from treatment allocation [ 43 ], typically use of this population causes bias in the estimated intervention effect [ 44 ] and the underlying estimand cannot be determined, i.e. unclear precisely what is being estimated. User engagement is a post-randomisation variable and the estimand framework [ 22 ] suggests using more appropriate strategies for handling post-randomisation events. For example, conducting a complier average causal effect analysis [ 42 ] under the principal stratification strategy estimated using instrumental variable regression [ 45 ] with randomised treatment allocation used as the instrumental variable. Alternative statistical methods can also be used to implement the estimand framework [ 46 ], but due to large variation in the reported engagement indicators and therefore difficulties in how engagement as a post-randomisation variable should be defined comparisons between trials remain challenging.
Establishing better methods in how user groups are defined, based on all available engagement measures, for example by using clustering algorithms combining all engagement measures, are needed. Secondly, once groups are defined existing statistical methods available to implement the estimand framework need to be assessed to determine the optimal approach to analyse the impact of engagement on the efficacy analysis. This is now the focus of our future work.
The JLA priority setting partnership occurred in 2018, meaning this review of publications between 2016 and 2021, includes very few trials recruiting after 2018. Therefore, implementation of the JLA priorities cannot be assessed. However, this review has shown user engagement data was available, showing potential for more trials to explore engagement in efficacy analysis. An update of this systematic review should be performed for the next 5 years (2021–2026) to assess whether issues identified in this review around user engagement have been improved. More trials exploring engagement in efficacy analysis will mean the pathway of sustained behaviour changes through engagement with DMHIs is better understood. Additionally, reporting of user engagement varied greatly, and although the CONSORT extension of e-health [ 47 ] outlines some detail on engagement reporting, more directed guidance is needed. Improvements should include reporting what and how many indicators were available and better guidance on how indicator data should be summarised. Additionally, trial publications varied greatly in quality of reported results and particularly for key demographic information such as ethnicity. CONSORT trial reporting guidance has been around since 1996 and more journals should enforce its implemented to ensure robust reporting of trials.
Finally, where data was available, participants were mostly female, white ethnicity and young, demographics consistent with another systematic review of DMHI trials [ 48 ] and the most recent 2014 Adult Psychiatric Morbidity Survey (APMS) for who is most likely to receive treatment [ 49 ]. However, the APMS 2014 also shows that individuals from black or mixed ethnicities are more likely to experience a CMD than those from white ethnicities. This supports other literature [ 50 , 51 ] and highlights differences in those recruited into trials and those who experience a CMD and not represented in DMHI efficacy estimates.
This systematic review assessed a wide-ranging number of outcome domains, providing an overview for all current DMHIs evaluated, including articles from CMDs with lots of active research, such as anxiety and depression, to CMDs with very few published results. Additionally, this review collected detailed information on engagement indicators, how these were reported, and how they were utilised in the analysis of the intervention effect, providing a rich database of the typical indicators available across a wide range of DMHIs.
As the focus of this review was to assess user engagement the review does not analyse the temporal differences of when primary outcome data for the intervention effect were collected. This means the review ignores that differences of the intervention effects across articles could partly be due to temporal differences in when they were collected, assuming the intervention effect changes over time. However, comparisons of adjusted and unadjusted intervention effects are measured at the same timepoints within each article. Additionally, as very few studies reported analysis adjusted for user engagement there was limited data to assess the impact of user engagement on the intervention efficacy in most outcome domains. Further, as most studies assessing engagement used a similar approach, per-protocol population, a formal comparison of methods was not possible. Finally, as this review only focused on appraising how engagement was reported and statistical methods used to analyse engagement, we don’t consider the impact of loss to follow-up has on the efficacy of interventions but must acknowledge that DMHIs typically have high drop-out rates from studies with very low proportions of individuals completing the intervention [ 52 ].
This review assessed reporting of user engagement and how authors considered engagement in the efficacy analysis of digital mental health interventions. While many articles reported at least one measure of engagement, very few articles used the data to analyse how engagement affects intervention efficacy, making it difficult to draw conclusions on the impact of engagement. In the small proportion of articles that reported this analysis, nearly all used statistical methods at high risk of bias. There is a clear need to improve the methods used to define active users by using all available engagement measures. This will help ensure a more consistent approach to how user engagement as a post-randomisation variable is defined. Once these methods are established trialists can then utilise existing statistical methods to target alternative estimands, such as principal stratification, that mean the impact of user engagement with the intervention efficacy can be explored.
The study protocol is already available on Prospero (CRD42021249503), datasets used are available from the corresponding author on reasonable request after the NIHR fellowship from which this project comes from is completed (April 2025). Any researchers interested in using the data extracted can contact the lead author using the shared correspondence information.
England N. The Five Year Forward View For Mental Health. 82NHS, NHS England, 2016.
Henderson C, Evans-Lacko S, Thornicroft G. Mental Illness Stigma, help seeking, and Public Health Programs. Am J Public Health. 2013;103:777–80.
Article PubMed PubMed Central Google Scholar
Muñoz RF, et al. Massive Open Online interventions: a Novel Model for delivering behavioral-health services Worldwide. Clin Psychol Sci. 2015;4:194–205.
Article Google Scholar
Ferwerda M, et al. What patients think about E-health: patients’ perspective on internet-based cognitive behavioral treatment for patients with rheumatoid arthritis and psoriasis. 2013.
Koh J, Tng GA-O, Hartanto A. Potential and Pitfalls of Mobile Mental Health Apps in traditional treatment: an Umbrella Review. LID – https://doi.org/10.3390/jpm12091376 LID – 1376. J Personalized Med. 2022;12.
Torous J, et al. Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry. 2019;18:97–8.
Capital FC. NICE and MHRA will review regulation of digital mental health tools. Volume 2023. Future Care Capital Website; 2022.
Torous J, Haim A. Dichotomies in the Development and Implementation of Digital Mental Health Tools. 2018.
NICE. NICE Evidience Standards For Digital Health. NICE, https://www.nice.org.uk/ , 2019.
Koneska EA-O, Appelbe DA-O, Williamson PA-O, Dodd SA. -O. usage Metrics of web-based interventions evaluated in Randomized controlled trials. Systematic Review; 2020.
Patel SA-O, et al. The Acceptability and Usability of Digital Health Interventions for Adults With Depression, Anxiety, and Somatoform Disorders: Qualitative Systematic Review and Meta-Synthesis. 2020.
Hollis C, et al. Identifying research priorities for digital technology in mental health care: results of the James Lind Alliance Priority setting Partnership. Lancet Psychiatry. 2018;5:845–54.
Article PubMed Google Scholar
Torous JB, et al. A hierarchical Framework for evaluation and informed decision making regarding smartphone apps for Clinical Care. Technol Mental Health. 2018;69:498–500.
Google Scholar
Donker T, et al. Smartphones for smarter delivery of mental health programs: a systematic review. 2013.
Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Mental Health. 2018;21:116.
Doherty K, Doherty G. Engagement in HCI: Conception, Theory and Measurement. ACM Comput Surv. 2018;51:99.
Lipschitz J, et al. Adoption of mobile apps for depression and Anxiety: cross-sectional survey study on patient interest and barriers to Engagement. JMIR Ment Health. 2019;6:e11334.
Michie SA-O, Yardley LA-OX, West RA-O, Patrick KA-O, Greaves FA-O. Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. Journal of Medical Internet Research. 2017;19.
Haine-Schlagel R, Walsh NE. A review of parent participation engagement in child and family mental health treatment. Clin Child Fam Psychol Rev 2015.
Saleem MA-O et al. Understanding Engagement strategies in Digital Interventions for Mental Health Promotion: scoping review. JMIR Mental Health. 2021;8.
Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7:254–67.
Use C. f.M.P.f.H. ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. European Medicines Agency; 2020.
Eysenbach G. The law of attrition. J Med Internet Res. 2005;7.
Elkes J. A systematic review to evaluate how user engagement is described and analysed in randomised controlled trials for digital mental health interventions. PROSPERO Int Prospective Register Syst Reviews. 2021.
Eldridge SM, et al. Defining feasibility and Pilot studies in Preparation for Randomised controlled trials: development of a conceptual Framework. PLoS ONE. 2016;11:e0150205.
Glanville J, et al. Translating the Cochrane EMBASE RCT filter from the Ovid interface to Embase.com: a case study. Health Inform Libr J. 2019;36:264–77.
Cochrane. Glossary of Cochrane Common Mental Disorders. Vol. 2021 A glossary of the definitions of common mental disorders. 2021.
World Health O. Classification of digital health interventions v1.0: a shared language to describe the uses of digital technology for health. Geneva: World Health Organization; 2018.
Ayiku L, et al. The NICE MEDLINE and Embase (Ovid) health apps search filters: development of validated filters to retrieve evidence about health apps. Int J Technol Assess Health Care. 2021;37:e16.
Innovation VH. Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. www.covidence.org . 2022.
Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22:139.
Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
Cochrane. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane. 2023.
Larry V, Hedges, Olkin I. Statistical methods for meta-analysis. 2014.
Fitzsimmons-Craft EE, et al. Effectiveness of a Digital Cognitive Behavior Therapy-guided self-help intervention for eating disorders in College Women. A Cluster Randomized Clinical Trial; 2020.
Salamanca-Sanabria AA-O, et al. A culturally adapted cognitive behavioral internet-delivered intervention for depressive symptoms. Randomized Controlled Trial; 2020.
Richards D, et al. Effectiveness of an internet-delivered intervention for generalized anxiety disorder in routine care: A randomised controlled trial in a student population. 2016.
Cerea S, et al. Cognitive behavioral training using a Mobile Application reduces body image-related symptoms in high-risk Female University students: a randomized controlled study. Behav Ther. 2021;52:170–82.
Glashouwer KA, Neimeijer RAM, de Koning ML, Vestjens M, Martijn C. Evaluative conditioning as a body image intervention for adolescents with eating disorders. 2018.
Milgrom JA-O, et al. Internet Cognitive Behavioral Therapy for Women With Postnatal Depression: A Randomized Controlled Trial of MumMoodBooster. 2016.
Hernán MA, Hernández-Díaz S. Beyond the intention-to-treat in comparative effectiveness research. Clin Trails. 2011;9:48–55.
Dunn G, Maracy M, Fau - Tomenson B, Tomenson B. Estimating treatment effects from randomized clinical trials with noncompliance and loss to follow-up: the role of instrumental variable methods. 2005.
Kahan BC, White IR, Edwards M, Harhay MO. Using modified intention-to-treat as a principal stratum estimator for failure to initiate treatment. Clin Trails. 2023;20:269–75.
Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: Intention-to-treat versus per-protocol analysis. 2016.
Lipkovich I, et al. Using principal stratification in analysis of clinical trials. Stat Med. 2022;41:3837–77.
Parra CO, Rhian M, Daniel, Bartlett JW. Hypothetical estimands in clinical trials: a unification of causal inference and missing data methods. Arxiv - Stat Methodol. 2021.
Eysenbach G, Group C-E. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res. 2011;13:e126.
Sin J, et al. Digital Interventions for Screening and Treating Common Mental disorders or symptoms of Common Mental illness in adults: systematic review and Meta-analysis. J Med Internet Res. 2020;22:e20581.
McManus S, Jenkins BP, Brugha R T, editors. Mental health and wellbeing in England: Adult Psychiatric Morbidity Survey 2014. in Adult Psychiatric Morbidity Survey 405 (Leeds, NHS Digital) 2016.
Iflaifel M, et al. Widening participation - recruitment methods in mental health randomised controlled trials: a qualitative study. 2023.
Coss NA, et al. Does clinical research account for diversity in deploying digital health technologies? Npj Digit Med. 2023;6:187.
Karyotaki E, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med. 2015;45:2717–26.
Article CAS PubMed Google Scholar
Download references
This work was funded by the NIHR Doctoral Fellowship (NIHR301810). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funder had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The TMRP Health Informatics working group, which the lead author is a member of, was essential in finding members (SOC, LY, LB) to join this project and support the work.
This work was funded by the NIHR Doctoral Fellowship (NIHR301810).
Jacqueline Sin and Victoria Cornelius contributed equally to this work.
Imperial Clinical Trials Unit, Imperial College London, White City Campus, Stadium House, 68 Wood Lane, London, W12 7RH, UK
Jack Elkes, Suzie Cro & Victoria Cornelius
University of Oxford, Oxford, UK
Rachel Batchelor, Ly-Mee Yu & Victoria Harris
Florence Nightingale Faculty of Nursing, Midwifery and Palliative Care, King’s College London, London, UK
Siobhan O’Connor
Leeds Institute of Clinical Trials Research, University of Leeds, Leeds, LS2 9JT, UK
Lauren Bell
City St Geroge’s, University of London, London, UK
Jacqueline Sin
You can also search for this author in PubMed Google Scholar
Conceptualization, JE, VC, SC and JS.; Methodology, JE, VC, SC and JS.; Software, JE.; Validation, JE, VC, SC, JS, SO, LB, RB, LY, and VH.; Formal Analysis, JE.; Investigation, JE, VC, SC, JS, SO, LB, RB, LY, and VH.; Resources, JE, VC, SC, and JS.; Data Curation, JE.; Writing – Original Draft, JE, VC, SC, and JS.; Writing – Reviewing & Editing, JE, VC, SC, JS, SO, LB, RB, LY, and VH.; Visualisation, JE, VC, SC, and JS.; Supervision, VC, SC, and JS.; Project Administration, JE, VC, and SC.; Funding Acquisition, JE, VC, SC, and JS.
Correspondence to Jack Elkes .
Ethics approval and consent to participate.
Not applicable as all data was publicly available.
Not applicable as no participants were recruited for this research.
JE was recently a collaborator on a NIHR HTA grant (NIHR132896) for long term effectiveness of a video feedback intervention for parents. JE is also on the trial steering committee for a trial (NIHR302349) that is part of an NIHR Doctoral Fellowship called Restore-B. JE is also on the programme steering committee (NIHR204413) for a trial called ATTEND. SC was previously awarded funding for an NIHR advanced fellowship (NIHR300593) between Stepember 2020 and December 2023. VC was also involved in the NIHR HTA (NIHR132896) funded trial of long-term follow-up of the video feedback intervention for parents. VC is also on the trial steering committee for a problem solving intervention for adults with dementia and depression, a steering committee member for a trial called ADVANCE and the chair of a NIHR HTA funded data monitoring committee (NIHR132808) called BAY. No other competing interests are reported for all other authors (RB, SOC, LMY, LB, VH and JS).
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
Rights and permissions.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Reprints and permissions
Cite this article.
Elkes, J., Cro, S., Batchelor, R. et al. User engagement in clinical trials of digital mental health interventions: a systematic review. BMC Med Res Methodol 24 , 184 (2024). https://doi.org/10.1186/s12874-024-02308-0
Download citation
Received : 02 May 2024
Accepted : 14 August 2024
Published : 24 August 2024
DOI : https://doi.org/10.1186/s12874-024-02308-0
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1471-2288
Introduction COVID-19 showed that countries must strengthen their operational readiness (OPR) capabilities to respond to an imminent pandemic threat rapidly and proactively. We conducted a rapid scoping evidence review to understand the definition and critical elements of OPR against five core sub-systems of a new framework to strengthen the global architecture for Health Emergency Preparedness Response and Resilience (HEPR).
Methods We searched MEDLINE, Embase, and Web of Science, targeted repositories, websites, and grey literature databases for publications between 1 January 2010 and 29 September 2021 in English, German, French or Afrikaans. Included sources were of any study design, reporting OPR, defined as immediate actions taken in the presence of an imminent threat, from groups who led or responded to a specified health emergency. We used prespecified and tested methods to screen and select sources, extract data, assess credibility and analyse results against the HEPR framework.
Results Of 7005 sources reviewed, 79 met the eligibility criteria, including 54 peer-reviewed publications. The majority were descriptive reports (28%) and qualitative analyses (30%) from early stages of the COVID-19 pandemic. Definitions of OPR varied while nine articles explicitly used the term ‘readiness’, others classified OPR as part of preparedness or response. Applying our working OPR definition across all sources, we identified OPR actions within all five HEPR subsystems. These included resource prepositioning for early detection, data sharing, tailored communication and interventions, augmented staffing, timely supply procurement, availability and strategic dissemination of medical countermeasures, leadership, comprehensive risk assessment and resource allocation supported by relevant legislation. We identified gaps related to OPR for research and technology-enabled manufacturing platforms.
Conclusions OPR is in an early stage of adoption. Establishing a consistent and explicit framework for OPRs within the context of existing global legal and policy frameworks can foster coherence and guide evidence-based policy and practice improvements in health emergency management.
Data are available on reasonable request. The rapid scoping review protocol can be publicly accessed on the Open Science Framework (OSF) platform ( https://osf.io/39q4b/ ). The datasets used and/or analysed during the scoping review are available from the corresponding author on reasonable request.
This is an open access article distributed under the terms of the Creative Commons Attribution IGO License ( CC BY 3.0 IGO ), which permits use, distribution, and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.
https://doi.org/10.1136/bmjgh-2023-014379
Request permissions.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Operational readiness (OPR) has emerged as a crucial but relatively unexplored concept in the context of health emergencies.
OPR is in an early stage of adoption with variable understandings of what it entails. This study highlights a need for conceptual clarity and consistency in describing OPR to build a coherent body of evidence that can underpin policy and practice. Key OPR actions aligned with five core subsystems of Health Emergency Preparedness Response and Resilience (a global, integrated framework for health emergency management) are identified.
Instruments to evaluate country-level preparedness under the International Health Regulations require evidence of readiness planning. The most recent global policy framework to strengthen the global architecture for health emergencies also signposts the critical role of readiness. This scoping review has provided a foundation for global expert deliberations and agreement on OPR, which is an important step forward towards a coherent body of evidence and to advance policy and practice for improved health emergency management.
A key lesson learnt from the global and national response to COVID-19 is the critical importance of early action. COVID-19 caught many countries off guard, and the consequences of delayed responses were severe in terms of public health as well as socioeconomic impacts. To prevent and mitigate the impact of future events, countries must strengthen their capabilities for rapid mobilisation to proactively respond in anticipation of an imminent threat. To this end, operational readiness (OPR) has emerged as an important part of efforts to strengthen the global architecture for health emergency preparedness, response and resilience (HEPR). 1 HEPR, WHO’s new strategic framework, is intended to guide, inform and resource collective efforts to strengthen the key interlinked national, regional and global multisectoral capacities sitting at the intersection of health security, primary healthcare and health promotion.
In the context of the health emergency cycle, OPR arises at the intersection between preparedness planning and response. 2 By promptly mobilising specific resources and strategies in the face of a high-priority and imminent threat, countries can enhance their ability to respond swiftly and efficiently by strategic deployment of well-defined capabilities, plans and actions that are tailored to the specific threat. The importance of this neglected phase in the health emergency cycle has catalysed related global policy initiatives. Instruments to evaluate country preparedness for emergency response under the International Health Regulations (IHR) require evaluation of country-level OPR planning, as seen in the Joint External Evaluation (JEE) 3.0’s Health Emergency Management Capacity, which targets risk-based plans for readiness and existence of an emergency readiness assessment. 3 The WHO’s proposals for a strengthened HEPR architecture across core domains of governance, finance and systems require OPR and capacities in five core subsystems: Collaborative Surveillance; Community Protection; Safe and Scalable Care; Access to Countermeasures and Emergency Coordination, along with OPR plans in Emergency Coordination. 1 4 Currently, there is no WHO guidance related to standardised emergency readiness assessments and readiness planning. To achieve the promise of strengthened OPR policy and practice, closer specification is needed to define what OPR involves and how it works, and the methodologies and approaches used to implement and operationalise it.
To underpin WHO technical products for OPR, we conducted a rapid scoping evidence review to examine the definitions and critical elements of OPR for public health emergencies caused by new or re-emerging infectious diseases and other public health threats in the context of the latest global policy frameworks for health emergency management. 5 This review is important given the absence of a standardised checklist of ‘must haves’ to inform the development of a country contingency plan in the face of an emergency.
Objectives of our review were (a) to identify how OPR has been conceptualised and defined; (b) to elicit critical elements of ‘OPRs’ in the context of key global policy frameworks, such as the WHO Global Health Security Framework, HEPR and JEE 3.0. 3 4 6 Anticipating a large and diverse body of evidence and given the need for a rapid output from this work, we conducted a rapid, scoping review following well-recognised methods. 7–9 We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews 10 checklist for reporting. Our study protocol is published 5 and registered (doi:10.17605/OSF.IO/6SYAH).
We included articles that:
Reported on OPRs, defined for this review, as those immediate action(s) required to preposition response actions to acute, proximal or imminent hazards and/or threats (eg, an infectious disease outbreak or a natural disaster threat), that is, an all-hazards approach 5 in the context of health emergencies, that is, disasters and major incidents (natural and otherwise) including emerging and re-emerging infectious disease threats with the potential to significantly impact a population’s health; and described actions of emergency response groups or organisations at national, regional or global levels.
Types: English, German, French or Afrikaans language peer-reviewed original articles or reviews published between 1 January 2010 and 29 September 2021, publicly available policy frameworks and programme reports, published conference reports or electronic theses, relevant grey literature and documents for which full texts or abstracts were available.
We excluded articles that:
Focused exclusively on longer-term preparedness actions (ie, an imminent threat was not explicitly defined) or response actions (ie, actions to respond to an active public health emergency), reported on contexts beyond health emergencies or did not focus on disease prevention and control.
We developed and ran a search structured by population (health systems/community), concept (readiness/preparedness/risk/planning) and context (emergencies/diseases/natural disasters) in MEDLINE, Embase and Web of Science databases (see online supplemental table S1 for detailed search strategies for the electronic databases). We searched various targeted repositories, websites and databases for grey literature 11 (see online supplemental box S1 ). We also used forward and backward citation tracking.
Selection of sources.
Search outcomes were imported into Rayyan V.0.1.0 software (Rayyan Systems, Massachusetts, USA) for screening, checking of duplicates and final selection. 8 12 Our approach to citation screening aimed to balance rigour and speed, consistent with rapid reviews and adapted from the Cochrane Rapid Reviews Methods Group’s guidance for rapid reviews, 8 including guidance on addressing the methodological challenges faced during COVID-19 rapid reviews. 9
Screening occurred at three levels (title, abstract and full report). The review team agreed on screening decisions upfront and agreed on guidelines after piloting for consistency. 8 13 For piloting, two reviewers (MP and MYC) independently and in duplicate screened 100 titles and abstracts, followed by discussion with three senior authors (RE, QL and MM) to refine screening decisions. Category coding by study design and keywords for excluded articles at the title and abstract level were agreed and set in Rayyan.
After this, one reviewer (MP) screened 20% of the initially identified titles and screened abstracts to remove irrelevant reports. A second reviewer (MYC) verified excluded titles and abstracts. 8 Conflicts and uncertainties were resolved by discussion with senior authors (RE, HG or QL). To ensure that all texts could be assessed in detail against the eligibility criteria within the limited time frame of the rapid review, 9 full-text screening was independently conducted by eight reviewers (MYC, MP, KB, JCYN, CJ, QL, RE and HG) with the yield divided among them. Discrepancies were resolved through discussion.
Grey literature search outputs were screened at two levels (title and body of the report) and recorded by one reviewer (MP). A second content expert (HG) verified the included sample. 8 9
Two reviewers (MYC and QL) extracted data from journal articles, and one (MP) from grey literature; an additional reviewer (RE) checked for accuracy in both instances. 14 Data were deductively coded in ATLAS.ti V.9 (Scientific Software Development) ( https://atlasti.com/ ) and extracted into a custom-built, pilot-tested MS Excel spreadsheet, according to preset criteria The data extraction form was revised after pilot testing and consultation with WHO and amended 14 to reflect the study authors’ affiliations and the WHO region in which the study was conducted. Uncertainties were discussed by the full review team. It was not necessary to contact the study authors.
Credibility of evidence in the included articles was assessed based on the information source and type. 8–10 Two reviewers (MP and MYC) appraised the included sources for descriptive purposes and incorporated the results narratively in the reflective summaries of the charting findings.
To analyse data, we (QL, RE, HG, CJ and JCYN) used qualitative thematic analysis with deductive synthesis, 15–17 against the following preidentified thematic categories: leadership, governance and coordination; country risk assessment; operational planning and coordination; contingency finance; health facility capacity and service delivery; health workforce/human resources; early warning or surveillance and health information systems; community resilience and risk communications; logistics or supply chain for access to essential medicines; WHO readiness and partner readiness. New themes were also identified. A revision of this analysis (HC and NG) used the new HEPR global architecture as an organising frame. 4
As this study presents a scoping review of already published literature, patient and public involvement was not applicable.
Of 7005 citations identified in the database (n=6827) and grey literature (n=178) searches, we included 78 (54 peer-reviewed publications; 25 grey literature) ( figure 1 ). The study characteristics are highlighted in online supplemental table 2A, B .
PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews.
Online supplemental table S2A characteristics of peer-reviewed studies on the definitions of OPR according to emergency type.
Online supplemental table S2B characteristics of grey literature publications on the definitions of OPR according to emergency type.
Descriptions of OPR lacked clarity and consistency in definition and use. Nine primary research papers and one grey literature document provided explicit definitions of ‘readiness’ and/or ‘preparedness’ for infectious disease emergencies ( online supplemental table S3 ). 18–27 Of these, three 18 21 24 explicitly defined ‘readiness’ while the others used the term ‘preparedness’ in a way that was congruent with our working definition of OPR. The term readiness was used interchangeably with concepts of preparedness, response and recovery. In other included articles, the concept of readiness was reflected implicitly, as per our working definition.
Some included articles suggested that preparedness indicators, using tools like the State Party Self-Assessment Annual reporting tool (SPAR), could be used to indicate gaps for the purposes of targeting OPR actions. 24 28 Others suggested that a country’s OPR and response capacity depends on the strengths of its preparedness, with regular testing and updating of plans and capacities assessing country OPR. 22 26 However, some authors noted that countries’ responses to COVID-19 highlighted an incongruence between IHR compliance scores and response performance; for example, some countries with lower IHR scores demonstrated a better ability to contain COVID-19 at the early stages of the pandemic. 21 29 A lack of recently updated and tested plans and a lack of large-scale training and refresher courses or key actions for OPR, have been identified as reasons for inconsistency and weakness in previous responses. 23 25 Others have identified the activities they had taken as a result of lessons learned from similar diseases as a reason for more successful responses. 29 30 For example, rapid training and simulation exercises and leveraging specific expertise and experiences were considered important in preventing or mitigating an outbreak. 20 28 31
The nature of the imminent threat also influenced the scale and speed of OPR actions, along with the proximity to the hazard. 18–21 OPR could thus be considered the ‘operationalisation’ of hazard-specific capacities aimed at mitigation of a specific, identified risk. Triggering rapid action in response to an imminent threat was noted as a way to feedback and strengthen country capacities while effectively cutting costs of ‘firefighting’ public health emergencies.
While preparedness and OPR are used interchangeably by papers during this review, the reasonable abundance of literature dedicated to time-bound actions right before an event suggests they are different concepts. This observation prompts the necessity for a clear understanding of OPR and its differences from preparedness. Thus, OPR actions could build on overall preparedness levels but consist of time-sensitive activities focused on the imminent threat (eg, ensuring that the healthcare workforce has been recently trained for an imminent threat). These activities have been focused on ensuring that overall preparedness gaps are accounted for (eg, requesting international emergency medical teams (EMTs) to be ready to deploy if EMTs are unavailable in-country). In the following section, we detail the variety of OPR actions that have been taken in articles included in our review, in alignment with the HEPR subsystems. 20 28 31
Collaborative surveillance.
Previous emergencies highlighted the importance of a strong Early Warning System with capacity to improve disease outbreak detection for early action to localised health events. 32–34 Strong surveillance systems at all levels, rapid feedback of results and accessibility of information were described as critical for risk management and decision-making. 33 35 36 A critical review of epidemiological data linked with planning and decision-making to increase vigilance and real-time information sharing at all levels was viewed as critical to communicate changes in the incidence of disease, which could signal triggers. 22 35 37–40
Key OPR actions embedded in surveillance systems included updating case definitions for consistency in identifying and reporting cases, early investigation, proactive contract tracing training for all staff and rapidly updating guidance for clinicians. 18 19 23 41–44 Measures to rapidly ensure integration of various types of surveillance and to address gaps in information collection and sharing were noted. 19 37 40 Integration of human and animal health surveillance systems was viewed as critical, as was the importance of interoperability of surveillance systems. 38 40 The interconnectivity of surveillance systems has been stressed to ensure that actions taken, and information gathered in one part of the system are made aware to other parts. 22 45 For example, it was stressed that the occurrence of viral haemorrhagic fever in animals should activate enhanced surveillance. 38 The timely reconciliation of data from multiple sources has been noted as challenging without an escalation in trained staff, improved communication, information technology and accessibility to more remote locations. 32 The need to have epidemic data be open and transparent for decision-making was emphasised. 46
OPR actions taken for surveillance systems in anticipation of a disease outbreak were centred around detecting gaps and providing solutions, 19 47 improving case detection via procurement of supplies, distribution of case definitions and the deployment of screening teams, 28 44 47–49 improving reporting for Integrated Disease Surveillance and Response priority diseases 28 48 and strengthening specimen transportation and analysis. 47–49 Others included increased frequency of surveillance system results 36 and rapid delivery of updated training and mechanisms for data sharing. 28 29 50 Existing systems were leveraged for COVID-19 as a novel disease 37 or the private sector engaged to provide surge capacity. 31 Other efforts centred around digitising systems to improve flexibility of use and reporting times. 32 46 Contact tracing systems were established as OPR actions, 44 51 along with quarantine or isolation options, screening and referral pathways in community settings and dedicated transfers for suspect cases. 44 47 52
OPR actions for increasing diagnostics and laboratory capacity for surveillance included prepositioning laboratory supplies in high-risk areas which was described as key to facilitating the investigation of suspected cases (eg, specimen transportation containers, triple packages and gloves, transportation vehicles for specimens). 18 19 Electronic systems developed to improve laboratory results turnaround time, 19 the quick detection of hotspots 36 37 or digital contract tracing applications 37 were important developments implemented by countries by way of OPR actions. Lessons learnt from the digitalisation of contact tracing highlighted the importance of scaling up laboratory capacity to account for the increased demand for testing and to timeously ensure sufficient capability to test and process tests. 29 31 53 54 Mechanisms, if not available, should be rapidly instituted for sharing laboratory investigation data and establishing laboratory networks within and outside countries for timely diagnoses. 18 38
Included sources also signposted OPR actions for a collaborative approach to successful surveillance. For the rapid confirmation of novel influenza strains, for example, countries were successful in collaborating with WHO collaborating centres in their region. 35 Laboratory capacity in other countries were rapidly increased through the creation of laboratory networks. 18 42 In scenarios where a neighbouring country had a disease outbreak, cross-border surveillance teams have been established and the sharing of information between border countries improved and highlighted as a reason for the limited spill. 19 During COVID-19, surveillance was rapidly readied at the point of entries, including standard operating procedures for detected cases and awareness-raising sessions for personnel. 55 56
Included articles highlighted key actions to upscale for rapidly involving and engaging affected communities in anticipation of an imminent threat. 22 57 These include rapidly providing updated information about the threat, including on identifying symptoms and any known public health and social measures, disseminated through numerous mechanisms and in a variety of languages to those at risk. 19 23 31 33 46 These should be adapted for all literacy levels. 58 Value was found in daily communications to build public trust. 37 Community volunteers were trained to carry out communal and door-to-door health education 19 32 or public websites containing epidemic reports to keep communities informed. 46
Further recommendations highlighted risk communications and public health and social measures to be rapidly readied to contain any potential community transmission. 18 21 48 51 59–61 These communications should allow the public to have a proper understanding of the perceived risk. 35 Other recommendations included working with local influencers to disseminate trusted information 47 and creating specialised focus messages for high-risk populations. 26 62 Crucially, there should be strong efforts for engaging vulnerable populations. 28 31 57
Plans and protocols should be in place for community-specific risk assessments to fill gaps in community OPR. 28 These assessments should focus on community perception, knowledge, preferred and accessible communication channels and existing barriers preventing community members from adopting promoted behaviours. 47 Plans should further account for resources for social security to support vulnerable communities. 40 To support this, community-based measures such as leveraging the community health workforce and community-based actors should be considered. 52 In this way, community needs and realities can be accounted for in the development of risk communication and community protection interventions. Misconceptions in the community should be identified and efforts made to dispel misinformation. 44
Some papers highlighted the early identification of vulnerable and remote population groups to ensure that their unique needs are well understood and addressed both in the design of interventions and in mitigating the impact of response interventions. 28 57 Accordingly, planning OPR should involve the input of communities, particularly organisations representing vulnerable groups, to inform community OPR. 47 Plans for response action should additionally consider secondary impacts or unintended consequences. For example, a clear lesson from COVID-19 related to the need for social security policies to mitigate the impacts of restrictive public health and social measures. 63 Policies for implementation should incorporate social security safety nets for communities, such as social health protection schemes or providing financial assistance for quarantined populations. 40 Plans should further be supported by partners. 64 Indirect health impacts should also be considered when OPR actions are implemented. 65 For example, some countries rapidly scaled up their capabilities for mental health services by implementing psychiatric hotlines 66 or providing stress management protocols. 48 Other indirect health impacts could include food insecurity; to prevent this, doorstep delivery of daily essentials 31 or provision of prepackaged meals 39 were planned.
Numerous papers highlighted the need for public health and social measures to be available rapidly and as early as possible, such as (for respiratory disease outbreaks) mask usage in public places when the risk level was high 31 36 46 52 63 and access to water, sanitation and hygiene, 44 48 with additional measures in place for individuals at risk of complications at the household level, such as using physical barriers, proper wearing of masks and environmental cleaning. 52 If non-existent, a strategy should be in place to assist in accelerating the containment of disease through imposing various public health and social measures, such as limits on local and international travel, the wearing of masks in public places, 37 social distancing, 67 bans or limits on mass gathering events 33 48 and closing educational institutions. 36 48 These measures were all implemented to a varying degree during COVID-19, with analyses finding that the earlier efforts of containment generally resulted in better containment early in the pandemic. 21 The measures taken should be weighed against the possibility of improving detection and spread through other methods, such as a rapid expansion of laboratory testing. 63 Public health and social measures should additionally take into account other likely risks—for example, countries with hurricane-prone areas during COVID-19 had to quickly revise their strategies to ensure social distancing in shelters. 39 If vaccines are available, a prioritisation policy should be developed to avoid ethical and political conflicts. 23
For the health service to function during an emergency, they need a baseline quota of adequate staffing to perform core functions. 68 Included articles stressed OPR to surge additional healthcare personnel. 31 The healthcare workforce needs updated case definitions, transmission, clinical presentation, infection prevention and control (IPC), community surveillance and case management for the threat. 19 Capacity assessments can guide OPR to estimate the ability of health systems to contain the imminent threat 36 37 41 and to identify gaps. 29 36 Additional recommendations highlight that capacity modelling should integrate risks to the workforce during the response—previously, health workforce absenteeism has not always been considered in the development of staffing plans, leading to reduced response capacities. 57 When scaling up healthcare worker OPR for a threat, actions should also be taken to scale up the services to support them. 64 Health systems gaps have been addressed by increasing the space of intensive care unit beds in relevant facilities, human resource training and mobilisation 20 36 48 49 63 69–71 and reducing the workload (eg, patients with mild symptoms were managed at home in isolation). 46 Referral systems and safe pathways should be established. 36 42 52
COVID-19 highlighted the importance of maintaining essential health services during an emergency. Many studies under review did not immediately prioritise this when considering OPR for the imminent threat. Measures taken proactively to maintain essential health services and to reduce the stress on the health system were described, such as giving patients with chronic diseases a stockpile to prevent them from coming to the hospitals 31 72 and use of telemedicine. 31 40 It was recommended to establish referral systems and safe pathways to designated local isolation facilities and enhance case detection in healthcare facilities and the community. 47 Others emphasised their learnings from response to diseases before COVID-19 and maintaining the continuum of care 36 40 - for example, Korea created two systems (COVID-19 health system vs non-COVID-19 health system) to ensure continuity of non-COVID-19 needs and diverted the flow of patients through triage centres. 36 Measures were taken to safeguard hospitals not identified as part of the response, for example, using temperature checks or encouraging the use of masks. 33 40
Included articles also noted that staff protection and welfare should be strongly included in OPR planning, for example, to anticipate provision of personal protective equipment (PPE) and supplies for staff protection. 73 74 An IPC programme should be implemented before an outbreak. 33 38 63 Prepositioning of PPE supplies in high-risk districts has been recommended to enable a more rapid response, 19 or if the risk level is low, the availability of a regional reserve of PPE. 75 Where PPE was unavailable, production was quickly ramped up to be able to maintain inventory before the response 76 - others who did not do this noted that they suffered shortages during the response. 46 Regular training and simulation exercises were conducted for case management teams. 19 38 Psychosocial support and other interventions necessary to support staff welfare were also emphasised. 26 40 Others quickly put legislation into place to protect healthcare workers engaged in response from being attacked. 31
There were fewer descriptions of OPR in this HEPR subsystem in comparison with others. When gearing up for response, countries have increased production and procurement by procuring from local industry, working with manufacturing companies to increase supply by, for example, adapting manufacturing facilities or establishing warehouses and transportation. 18 31 Numerous studies noted that they had extreme difficulty in obtaining the supplies they needed, 40 46 due to limited stockpiles and lack of finances to maintain them. 23 OPR actions for an imminent threat would focus on scaling up manufacturing plans and to ensure that a stockpile is in place.
Prepositioning essential supplies is essential for OPR, with an adequate supply of medical equipment to the frontline identified as vital for reducing health emergency risks. 77 Additionally, measures to quickly acquire and distribute medical supplies using government-set prices, prioritise frontline health professionals and vulnerable populations for the disbursement of medical countermeasures and promote local manufacturing were identified. 20 Other countries described OPR actions to introduce therapeutics, diagnostics and vaccines. 37
One study identified research topics such as system OPR, knowledge, attitudes and practices of the health workforce, epidemiology of the disease at the national level, best practices at the points of entries and isolation centres and infection-control measures as important to inform OPR actions. 78 Research should also support decision-making, cost-effectiveness, intervention effectiveness and the impact of these on pandemic trajectories. 50 79 Competing demands can limit the volume of research conducted which was considered a missed opportunity. 32 Early convening of expert groups to advise government was identified as useful for managing health service responses and OPR, and their work should as far as possible be informed by evidence (eg, scenario planning). 33 Health systems researchers occupying the highest levels of oversight across the sectors were said to enhance the use of evidence and data for decision-making. 36 Another paper noted that lessons learnt by regions found that funding for research and investigations during OPR and response should also be in place. 39
We identified several critical and overarching governance-related elements that facilitated OPR within regions and countries. Lessons from OPR or responses to previous diseases have demonstrated the importance of a coordinating body at regional or national levels 19 35 36 41 42 46 48 75 78 80 81 led by high-level officials. 19 48 80 These structures should provide leadership and coordination, 42 46 62 82 guidance and action plans, 36 and communication of critical information. 48 80 Strong and skilled leadership was a notable enabler 29 32 36 54 83 and was marked by active OPR involvement of the responsible health departments, and effective coordination with multiple stakeholders as the planning or response evolved. 29 32 54 82 84 Flexibility and adaptation, particularly during OPR, were important. 32
Many included articles emphasised the timely activation of coordination mechanisms and risk assessments to inform plans. 18 19 31 34 38 47 54 69 75 83 85 86 This involved the establishment and operationalisation of intersectoral and/or interdisciplinary teams (eg, task teams, 19 33 75 80 special councils 41 42 46 and command centres 30 41 ) to provide technical expertise, 25 42 78 87 prepare and coordinate the implementation of policy decisions 32 80 87 and guide lower health system-level or governmental-level structures or actors. 28 32 88 An Incident Management System was adopted in several countries with a dedicated lead, 32 35 36 83 89 and this was further recommended in the grey literature. 72 90 91 When operationalising these aspects for an efficient and effective response, the early establishment of clear roles and responsibilities, with a clear lead was considered vital and instrumental for later response success. 28 32 The highest levels of government should be involved, with an all-of-society and/or all-of-government approach. 32 35 69 70 79 87–89
To successfully implement coordination and response to an emergency, workforce management is key for a successful response. Actions taken include recruitment of staff from the private sector, healthcare students or retired or non-practising trained workers, 31 40 42 48 78 89 92 community health workers and community-based organisations 19 31 40 48 73 or volunteers. 19 48 89 Grey literature emphasised, actions in support of cross-border response teams or surge teams with rapid staff registration and accreditation systems, staff redeployment and reallocation, 18 72 93 and appropriate training. 18 90 94 95 Also critical was ensuring the availability of emergency medical services for immediate response and the early deployment of multidisciplinary Rapid Response Teams in high-risk groups. 23 31 53 83 87 89 Some papers emphasised prioritising actions which enable rapid deployment of these teams. 53 83
Other important factors included threat-specific contingency planning at national and subnational levels for identifying preparedness gaps and actions to work around them, thus supporting rapid detection, response and containment. 18 19 35 83 89 Contingency plans helped to prioritise targeted actions 83 as well as identify and prioritise at-risk geographic areas and vulnerable communities. 40 57 Having recently updated or tested contingency plans in place was stated as essential to enhance OPR and effective response, 25 39 96 and these should support operations and logistics, help understand organisational structures and functions, and optimise resources. 44 68 93 They should further ensure critical infrastructure for health system functioning and ensure clinical and health service-level plans are detailed and able to assist in preparing for increased patient volumes or need for critical care services. 19 68 Contingency plans should incorporate past experiences and learnings from other outbreaks, changing contexts 52 and the results of simulation exercises conducted on the preparedness and response systems. 18 19 23 Countries with similar public health emergency experiences have been found to be better prepared than those without previous experience, 63 raising the importance of practice, via simulation exercises and training, for a new imminent threat. 19 23
Furthermore, country risk and vulnerability assessments should be available and guide risk assessment activities. 19 31 35 38 39 47 52 53 57 84 They were recommended to be focused on geographical areas with particularly high assessed risks 39 52 89 and related to prevention and control strategies. 19 47 84 The assessments should be conducted to ensure that the contingency plans contain appropriate OPR actions and consider local contexts 47 68 89 and can also be used to guide the prioritisation of actions. 47 89 Risk assessments for future waves or outbreaks should also be conducted, and updated worst-case scenarios incorporated into contingency plans. 39 63
OPR needs bespoke financial planning. 22 28 70 It was recommended that contingency funds be available for OPR, 83 ring-fenced and situated within a dedicated emergency programme. 19 50 70 There should be existing emergency financial management systems which allow for rapid, transparent and efficient use of funding. 40 42 Contingency funds were emphasised as particularly important as resources should not be diverted from necessary routine programmes. 25 50 Having contingency funds in place would ensure a few key capacities: first, earmarked resources for the hazard are ensured 22 and lead to rapid activation of key surveillance and early response activities. 25 50 Second, changes which may need to occur to financing healthcare services are already outlined, such as creating financial protection mechanisms for discontinued outpatient services or outlining how citizens or health insurance systems pay for screening and diagnostic testing. 42 Finally, contingency funds should cover workforce surge, including staff, supplies, training and workforce management. 73
This scoping review examined definitions and critical elements of OPR for public health emergencies. We sought to identify key actions that were mobilised in anticipation of an imminent threat framed in the latest conceptualisation of a global architecture for health emergency management. From 54 peer-reviewed publications and 24 grey literature sources, we found that the concept of OPR was in an early stage of adoption. Where the term was explicitly defined, these definitions lacked coherence and consistency and included articles that matched our working definition of OPR, often did not use the term. Our analysis highlights the important need for conceptual clarity regarding OPR. We agreed on a working definition of OPR at the outset of the review as those immediate actions taken in the presence of an imminent threat that is rapidly mobilised or prepositioned to respond to that threat. It was also often difficult to identify where the line between preparedness, OPR and response lay. For our purposes, these distinctions are relevant in so far as they can guide early detection and timely activation of key OPR capabilities in useful and practical ways. Put simply: when a hurricane is coming, you may rapidly begin to take measures to prevent damage to your house. These could include actions such as securing loose objects, protecting windows, turning off utilities and filling tubs with water. These actions, taken before a storm, would differ greatly from the years spent building and maintaining the house beforehand - ensuring the foundation is sound, and the roof has been well maintained. They would further differ from the actions would take immediately during and after the storm has hit.
This review was initiated during the dynamic and fast-moving context of a pandemic where important policy developments were advancing in parallel. To maximise the utility of this work, we reanalysed our findings to map to the HEPR framework once it became publicly available for wider discussion among WHO member states. Our analysis across the body of articles included in this review identified OPR actions that mapped to the five core subsystems considered critical to strengthen the global HEPR. Additionally, our review mostly identified national-level capabilities and provided less insight into key actions to activate subnational and local capabilities. This observation may reflect a limitation of our review, an under-reporting, or a need to further develop and define OPR at these levels.
Across articles included in this review, OPR actions were identified as those that aimed to fill gaps in a country’s capacities or to prepare for an early response. In this way, a key contribution of embedding OPR in health emergency management is in institutionalising prompt action as soon as a potential signal is detected. Of note are the many actions identified for emergency coordination, including strong, high-level leadership, governance and coordination, with clarity around the roles and responsibilities of the leaders and the coordination bodies. Collaborative surveillance that allows for early detection of signals is key for OPR in terms of triggering action. This is an underdeveloped part of readiness practice. Other important areas included rapid, integrated and interoperative health information systems for purposes such as surveillance, planning and decision-making, managing operations, and monitoring country responses. The ability to rapidly plan for, mobilise and manage resources (eg, human, PPE, financial) and scale-up services (eg, essential or laboratory) underpinned by supportive legislation were also identified. Clear and strong communication at the level of the policy-maker, within the services and in the community was also identified as crucial for optimal OPR. We note gaps related to research and manufacturing platforms enabled by technology and our analysis did not consider OPR actions at the intersection of the five subsystems, for example, the readiness of communities for early detection to support collaborative surveillance or for participation in clinical trials of novel medical countermeasures.
The review methodology has strengths and limitations. This work was done rapidly by a large team with the aim of underpinning practical technical products for OPR in health emergency risk management. A scoping review methodology was best suited to answer our research question, due to the broad base of evidence. 13 As far as possible, we followed expert group recommendations on the adaptations needed in the conduct of rapid reviews. 9 14 Our initial analysis mapped key thematic categories in the HEPR Framework. 1 To align with global policy developments that have led to the HEPR framework, we updated our analysis. In this process, we may have missed new articles that would add further insight into OPR experience. However, given the pragmatic focus for this review, and the global consensus work that has followed, it is unlikely that further updates to this review would significantly alter our key conclusions. Since this review, there has been significant progress in actions to strengthen the global HEPR architecture. A more thorough review of OPR, one for each of the subsystems, is needed to reflect the recently published breakdown of HEPR subsystems into capabilities. 4 Further, as OPR becomes engrained in health emergency response, a review to identify the optimal time frame needed to quickly and effectively operationalise the capabilities of every subsystem is needed. Additionally, the purpose of the review was not to identify how OPR actions have increased resilience. Future research is needed now that OPR has been defined to identify the OPR interventions which maximise populations; abilities to withstand an event and increase resilience. Finally, our review does not include a body of work on anticipatory actions, which aligns well with OPR. Anticipatory actions are defined as ‘actions taken ahead of predicted hazards to prevent or reduce acute humanitarian impacts before they fully unfold’. 97 They highlight OPR as part of emergency management, particularly for disaster management and in humanitarian contexts. 98 The outcome of these meetings reflects a growing consensus on the critical importance of OPR. The essence of OPR is to mobilise early action when a threat is on the horizon. The work reported in this paper is an important step to advancing this important and urgent agenda. Indeed, this work has now set a foundation for the more substantive and coherent development of the evidence in this important area and has provided input to readiness actions within the recently published IHR Benchmarks, and is informing the creation of readiness assessments and has informed the creation of a readiness course on OpenWHO. 99 100
Patient consent for publication.
Not applicable.
We would like to thank Professor Taryn Young for guidance regarding the methodology and Hilmar Luckoff for editing earlier versions of the paper. The rapid scoping review was commissioned by the WHO to inform an Operational Readiness Framework for the Country Readiness Strengthening Department in the World Health Emergencies Program in WHO (Reference #: 2021/1145765; Unit: MST; Cluster: QNF/SCI).
Handling editor Helen J Surana
Contributors Conceptualisation: RE, HG, QL, JCYN, NG and LLB; Data extraction: MYC and MP; Formal analysis: QL, RE, HG, CJ, JCYN, NG, HC and NE (synthesis); MYC, MP and KB (descriptive); Funding acquisition: RE; Methodology: QL, MM, KB, MP, MYC, CJ, NG and LLB; Project administration: JCYN; Software: MYC, MP, MM and KB; Source screening: MP, MYC, RE, HG, QL, KB and JCYN; Supervision: RE; Visualisation: QL and MYC; Writing–original draft preparation: KB, MYC, QL, MM, MP, CJ, RE, HG, JCYN, NG and LLB; Writing–review and editing: KB, MYC and NE; Writing–final version review: All authors have read and approved the final version of the report manuscript. RE is the nominated guarantor.
Funding This work was supported by the WHO (Reference: APW/RR/Readiness/2021/1145765). The manuscript development and publication were funded in part by the Wellcome Trust and the UK Foreign and Commonwealth Development Office under grant agreement 222037/A/20/Z and in part by the United States Agency for International Development (USAID) under grant agreement 720BHA21IO00300.
Disclaimer The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
BMC Health Services Research volume 24 , Article number: 998 ( 2024 ) Cite this article
Metrics details
The midwife-led model of care is woman-centered and based on the premise that pregnancy and childbirth are normal life events, and the midwife plays a fundamental role in coordinating care for women and linking with other health care professionals as required. Worldwide, this model of care has made a great contribution to the reduction of maternal and child mortality. For example, the global under-5 mortality rate fell from 42 deaths per 1,000 live births in 2015 to 39 in 2018. The neonatal mortality rate fell from 31 deaths per 1,000 live births in 2000 to 18 deaths per 1,000 in 2018. Even if this model of care has a pivotal role in the reduction of maternal and newborn mortality, in recent years it has faced many challenges.
To explore facilitators and barriers to a midwife-led model of care at a public health institution in Dire Dawa, Eastern Ethiopia, in 2021.
: A qualitative approach was conducted at Dire Dawa public health institution from March 1–April 30, 2022. Data was collected using a semi-structured, in-depth interview tool guide, focused group discussions, and key informant interviews. A convenience sampling method was implemented to select study participants, and the data were analyzed thematically using computer-assisted qualitative data analysis software Atlas.ti7. The thematic analysis with an inductive approach goes through six steps: familiarization, coding, generating themes, reviewing themes, defining and naming themes, and writing up.
Two major themes were driven from facilitators of the midwife-led model of care (professional pride and good team spirit), and seven major themes were driven from barriers to the midwife-led model of care (lack of professional development, shortage of resources, unfair risk or hazard payment, limited organizational power of midwives, feeling of demoralization absence of recognition from superiors, lack of work-related security).
The midwifery-led model of care is facing considerable challenges, both pertaining to the management of the healthcare service locally and nationally. A multidisciplinary and collaborative effort is needed to solve those challenges.
Peer Review reports
A midwife-led model of care is defined as care where “the midwife is the lead professional in the planning, organization, and delivery of care given to a woman from the initial booking to the postnatal period“ [ 1 ]. Within these models, midwives are, however, in partnership with the woman, the lead professional with responsibility for the assessment of her needs, planning her care, referring her to other professionals as appropriate, and ensuring the provision of maternity services. Most industrialized countries with the lowest mortality and morbidity rates of mothers and infants are those in which midwifery is a valued and integral pillar of the maternity care system [ 2 , 3 , 4 , 5 ].
Over the past 20 years, midwife-led model of care (MLC) has significantly lowered mother and infant mortality across the globe. In 2018, there were 39 deaths for every 1,000 live births worldwide, down from 42 in 2015. From 31 deaths per 1,000 live births in 2000 to 18 deaths per 1,000 in 2018, the neonatal mortality rate (NMR) decreased. The midwifery-led care approach is regarded as the gold standard of care for expectant women in many industrialized nations, including Canada, Australia, the United Kingdom, Sweden, the Netherlands, Norway, and Denmark. Evidence from those nations demonstrates that women and babies who get midwife-led care, as opposed to alternative types of care, experience favorable maternal outcomes, fewer interventions, and lower rates of fetal loss or neonatal death [ 6 , 7 , 8 ].
In Pakistan, the MLC was accompanied by many challenges. Some of the challenges were political threats, a lack of diversity (midwives had no opportunities for collaborating with other midwives outside their institutions), long duty hours and low remuneration, a lack of a career ladder, and a lack of socialization (the health centers are isolated from other parts of the country due to relative geographical inaccessibility, transportation issues, and a lack of infrastructure). Currently, in Pakistan, 276 women die for every 100,000 live births, and the infant mortality rate is 74/1000. But the majority of these deaths are preventable through the midwife-led care model [ 7 ].
The MLC in African countries has faced many challenges. Shortages of resources, work overload, low inter-professional collaboration between health facilities, lack of personal development, lack of a well-functioning referral system, societal challenges, family life troubles, low professional autonomy, and unmanageable workloads are the main challenges [ 8 ].
Due to the aforementioned challenges, Sub Saharan Africa (SSA) is currently experiencing the highest rate of infant mortality (1 in 13) and is responsible for 86% of all maternal fatalities worldwide. As a result, it is imperative to look at the MLC issues in low-income countries, which continue to be responsible for 99% of all maternal and newborn deaths worldwide [ 8 , 9 ].
Ethiopia’s has a Maternal mortality rate (MMR) and NMR of 412 per 100,000 live births and 33 per 1000 live births, respectively, remain high, making Ethiopia one of the largest contributors to the global burden of maternal and newborn deaths, placed 4th and 6th, although MLC could prevent a total of 83% of all neonatal and maternal fatalities in an environment that supports it. The MMR & infant mortality rate (IMR) in the research area were indistinguishable from that, at 150 per 100,000 live births and 67 fatalities per 1,000 live births, respectively [ 10 , 11 , 12 , 13 ].
Since the Federal Ministry of Health is currently viewing midwifery-led care as an essential tool in reducing the maternal mortality ratio and ending preventable deaths of newborns, exploring the facilitators and barriers of MLC may have a great contribution to make in reducing maternal and newborn mortality [ 14 ]. Since there has been no study done in Ethiopia or the study area regarding the facilitators and barriers of MLC, the aim of this research was to explore the facilitators and barriers of MLC in Dire Dawa City public health institutions.
In so doing, the research attempted to address the following research questions:
What were the facilitators for a midwife-led model of care at the Dire Dawa city public health institution?
What were the barriers to a midwife-led model of care at the Dire Dawa city public health institution?
Institutional based qualitative study was conducted from March 01-April 30, 2022 in Dire Dawa city. Dire Dawa city is one of the federal city administrations in Ethiopia which is located at the distance of 515killo meters away from Addis Ababa (the capital city) to the east. The city administration has 9 urban and 38 rural kebeles (kebeles are the smallest administrative unit in Ethiopia). There are 2 government hospitals, 5 private hospitals, 15 health centers, and 33 health posts. The current metro area population of Dire Dawa city is 426,129.Of which 49.8% of them are males and 50.2% females. The total number of women in reproductive age group (15–49 years) is 52,673 which account 15.4% of the total population. It has hot temperature with a mean of 25 degree centigrade [ 15 ].
The source population for this study included all midwives who worked at Dire Dawa City public health facilities as well as key informants from appropriate organizations (the focal person for the Ethiopian Midwives Association and maternal and child health (MCH) team leaders). The study encompassed basically 41 healthcare professionals who worked in Dire Dawa public health institutions in total, and the final sample size was decided based on the saturation of the data or information.
From the total 15 Health centers and 2 Governmental Hospitals found in Dire Dawa city administration, 8 Health centers and 2 Governmental Hospitals were selected by non-probability purposive sampling method. In addition to that a non-probability convenience sampling method was used to select midwives who were working in Dire Dawa city public health institutions and key informants from the relevant organization such as Ethiopian midwives association focal person and MCH team leaders. Midwives who were working for at least six months in the institution were taken as inclusion criteria while those who were working as a free service were excluded from the study.
Focus groups, in-depth interviews, and key informant interviews were used in collecting data. A voice recorder, a keynote-keeping, and a semi-structured interview tool were all used to conduct the interviews. Voluntary informed written consent was obtained from the study participant’s before they participated in the study. Then an in-depth interview and focus group discussion were held with midwives chosen from various healthcare organizations. The MCH department heads and the Dire Dawa branch of the Ethiopian Midwife Association served as the key informants. In-depth interview (IDI) and key informant interviews (KII) with participants took place only once and lasted for roughly 50–60 min. In the midwives’ duty room, the interview was held. Six to eight people participated in focus group discussions (FGD), which lasted 90 to 100 min. Two midwives with experience in gathering qualitative data gathered the information.
The qualitative design is prone for bias but open-ended questions were used to avoid acquiescence and 2 day proper training was given for the data collector regarding taking keynotes and recording using a tape recorder. For consistency and possible modification, a pretest was done in one FGD and In-depth interviews at non selected health institutions of Dire Dawa city administrations. A detailed explanation was given for the study participants about the objectives of the study prior to the actual data collections. All (FGDs, key informant interview and In-depth interviews) were taken in a silent place.
Atlas.ti7, a qualitative data analysis program, was used for analyzing the data thematically. An inductive approach to thematic analysis involves six steps: familiarization, coding, generation of themes, review of themes, defining and naming of themes, and writing up. By listening to the taped interview again, the data was transcribed. The participants’ well-spoken verbatim was used to extract and describe the inductive meanings of the statements. The data was then coded after that. Each code describes the concept or emotion made clear in that passage of text. Then we look at the codes we’ve made, search for commonalities, and begin to develop themes. To ensure the data’s accuracy and representation, the generated themes were reviewed. Themes were defined and named, and then the analysis of the data was written up.
Meeting standards of trustworthiness by addressing credibility, conformability, and transferability ensures the quality of qualitative research. Data triangulation, data collection from various sites and study participants, the use of multiple data collection techniques (IDI, KII, and FGD), multiple peer reviews of the proposal, and the involvement of more than two researchers in the coding, analysis, and interpretation decisions are all instances of the methods that were used in order to fulfill the criteria for credibility. To increase its transferability to various contexts, the study gave details of the context, sample size and sampling method, eligibility criteria, and interview processes. To ensure conformability, the research paths were maintained throughout the study in accordance with the work plan [ 16 , 17 ].
In this study, a total of 41 health care providers who are working in Dire Dawa public health facilities participated in the three FGDs, six KIIs, and fifteen IDIs. The years of experience of study participants range from one year to 12 years. The participants represented a wide age range (30–39 years), and the educational status of the respondents ranged from diploma to master’s degree. (Table 1 )
As shown in Table 2 , from the qualitative analysis of the data, two major themes were driven from facilitators of MLC, and seven major themes were driven from barriers to MLC. (Table 2 ).
Professional pride.
This study found that saving the lives of mothers and newborns was a strong facilitator. Specifically, it was motivational to have skills within the midwifery domain, such as managing the full continuum of care during pregnancy and labour, supporting women in having normal physiologic births, being able to handle complications, and building relationships with the women and the community, as mentioned below by one of the IDI participants.
“I am so proud since I am a midwife; nothing is more satisfying than seeing a pregnant mother give birth almost without complications. I always see their smile and happiness on their faces , especially in the postpartum period , and they warmly thank me and say , “Here is your child; he or she is yours.” They bless me a lot. Even sometimes , when they sew me in the transport area , cafeteria , or other area , they thank me warmly , and some of them also want to invite me to something else. The sum total of those things motivates me to be in this profession or to provide midwifery care.“ IDI participants.
This finding is also supported by other participants in FGD.
“We have learned and promised to work as midwives. We are proud of our profession , to help women and children’s health. The greatest motivation is that we are midwives , we love the profession , and we are contributing a great role in decreasing maternal and child mortality….” FGD discussant.
The research revealed that good midwifery teamwork and good social interaction within the staff have become facilitators of MLC. FGD participants share their experiences of working in a team.
“In our facility , all the midwives have good teamwork; we have good communication , and we share client information accurately and timely. In case a severe complication happens , we manage it as a team , and we try to cover the gap if some of our staff are absent. Further from that , we do have good social interactions in the case of weeding , funeral ceremonies , and other social activities. We do have good team spirit; we work as a team in the clinical area , and we also have good social relationships. “If some of our staff gets sick or if she or he has other social issues , the other free staff will cover her or his task.” FGD discussant.
Another participant from IDI also shared the same experience regarding their good teamwork and their social interactions.
“As a maternal and child health team , we do have a good team spirit , not only with midwives but also with other professions. We are not restricted by the ward that we assign. If there is a caseload in any unit , some midwives will volunteer to help the other team. Most of the time in the night , we admit more than 3 or 4 labouring mothers at the same time. Since in our health center only one midwife is assigned in the night , we always call nurses to help us. This is our routine experience.” IDI participants.
Lack of professional development.
This study revealed that insufficient opportunities for further education and updated training were the main barriers for MLC. Even the few trainings and update courses that were actually arranged were unavailable to them, either because they did not meet the criteria seated or because the people who work in administration were selected. Even though opportunities are not arranged for them to upgrade themselves through self-sponsored. One of the participants from IDI narrates her opinion about opportunities for further education as follows:
“Training and updates are not sufficient; currently we are almost working with almost old science. For example , the new obstetrics management protocol for 2021 has been released from the ministry of health , and many things have changed there. But we did not receive any training or even announcements. Even the few trainings and update courses that were truly organized and turned in to us are unavailable since the selection criteria are not fair. As a result , we miss those trainings either because we did not meet the selection criteria or because those who work in administration are prioritized.” IDI participant.
FGD discussants also support this idea. She mentioned that even though opportunities are not arranged for them to upgrade themselves through self-sponsorship,
“There is almost no educational opportunity in our institution. Every year , one or two midwives may get institutional sponsorship. Midwives that will be selected for this opportunity are those who have served for more than five to ten years. Imagine that to get this chance , every midwife is expected to serve five or more years. Not only this , even if staff want to learn or upgrade at governmental or private colleges through self-sponsored programmes , whether at night or in an extension programme , they are not cooperative. Let me share with you my personal experience. Before two years , I personally started my MSc degree at Dire Dawa University in a weekend programme , and I have repeatedly asked the management bodies to let me free on weekends and to compensate me at night or any time from Monday to Friday. Since they refuse to accept my concern , I withdraw from the programme.“ FGD discussant.
The finding indicates that a shortage of equipment, staff, and rooms or wards was a challenge for MLC. Midwives claimed they were working with few staff, insufficient essential supplies, and advanced materials. This lack of equipment endangers both the midwives and their patients. One of the participants from IDI narrates her opinion about the shortage of resources as follows:
“Of course there is a shortage of resources in our hospital , like gloves and personal protective devices. Even the few types of medical equipment available , like the autoclave , forceps , vacuum delivery couch , and BP apparatus , are outdated , and some of them are unfunctional. If you see the Bp apparatus we used in ANC , it is digital but full of false positives. When I worked in the ANC , I did not trust it and always brought the analogue one from other wards. This is the routine experience of every staff member.“ IDI participants.
Another participant from IDI also shared the same experience regarding the crowdedness of rooms or wards.
“In our health center , there are no adequate wards or rooms. For example , the delivery ward and postnatal ward are almost in one room. Postnatal mothers and neonates did not get enough rest and sleep because of the sound of laboring mothers. Not only is this , but even the antenatal care and midwifery duty rooms are also very narrow.“ IDI participants.
The study also revealed midwifery staff were pressured to work long hours because they were understaffed, which in turn affected the quality of midwifery care. The experience of a certain midwife is shared as follows:
“I did not think that the management bodies understood the risk and stress that we midwives face. They did not want to consider the risk of midwives even equal to that of other disciplines but lower than the others. For example , in our health centre , during the night , only one midwife is assigned for the next 12 hours , but if you see in the nurse department , two or more nurses are assigned at night in the emergency ward.” IDI participants.
The discussion affirms the fact that being understaffed and not having an adequate allocation of midwife professionals on night shifts are affecting labouring mothers’ ability to get sufficient health midwifery care. The above narration is also supported by the FGD discussant.
“In our case , only one midwife is assigned to the labour ward during the night shift. I think this is the main challenge for midwives that needs attention. Let me share with you my experience that happened months before. While I was on night shift , two labouring mothers were fully dilated within three or four minutes. It was very difficult for me , to manage two labouring mothers at the same time. Immediately , I call one of my nurse friends from the emergency department to help me. If my friend was so busy , what could happen to the labouring mother and also to me? This is not only my experience but also the routine experience of other midwives.” FGD discussant.
It is reported that the compensation amount paid for risk is lower than in other health professions. The health risks are not any less, but the remuneration system failed to capture the need to fairly compensate midwifery professionals. The narration from the FGD discussant regarding unfair payment is mentioned below.
“Only 470 ETB is paid for midwives as risk payments , which is incomparable with the risks that midwives are facing. But contrary to that , the risk payments for nurses (in emergencies) are about 1200 Ethiopian birr (ETB) , and Anesthesia is 1000 ETB. I did not want to compare my profession with other disciplines , but with the lowest cost , how the risk of midwifery cannot be equal to that of nursing and other professions. I did not know whose professionals made such types of unfair decisions and with what scientific background or base this calculation was done . ” FGD discussant.
The above finding is also supported by an IDI participant.
“………………………….Even though the midwifery profession is full of risks , with the current Ethiopian health care system , midwives are being paid the lowest risk payments compared to other disciplines…………….” IDI participants.
Midwives’ interviews reported that limited senior midwifery positions in the health system have become the challenge of midwifery care. This constrains the decision-making power and capability of midwives. This was compounded by limited opportunities for midwifery personnel to address their concerns to the responsible bodies, as stated by one of the key informants.
“Our staff has many concerns , especially professional-related concerns , which can contribute to the quality of midwifery care. Personally , as department head , I have tried to address those concerns in different management meetings at different times. But since the leadership positions are dominated by other disciplines , many of our staff concerns have not been solved yet. But let me tell you my personal prediction… If those concerns are not solved early and if this trend continues , the quality of midwifery care will be in danger.“ Participant from Key Informant.
The above finding is also supported by another IDI participant.
“In our hospital , at every hierarchal and structural level , midwives are not well represented. That is why all of our challenges or concerns have not been solved yet. For example , as a structure in the Dire Dawa Health Office (DDHO) , there is a team of management related to maternal and child health. But unfortunately , those professionals working there are not midwives. I was one of three midwives chosen to meet with Dr. X (former DDHO leader) to discuss this issue. At the time , we were reaching an agreement that two or three midwives would be represented on that team. But since a few months later the leader resigned , the issue has not gotten a solution yet.“ IDI participant.
One of the main concerns reported by the participants during the interviews was a feeling of demoralization induced by both their clients and their supervisors about barriers to midwifery care. They reported having been verbally abused by their patients, something that made them feel that their hard work was being undermined, as stated by an FGD participant.
“I don’t think there is any midwife who would be happy for anybody to lose their baby , or that there is any midwife who would want a woman to die. These things are accidents , but the patient and leaders will always blame the midwife.” FDG discussant.
A narration from an IDI participant also mentioned the following:
“……….If something happens , like a conflict with the patients or clients , the management is on the patient side. Not only that , the way in which they communicate with us is in an aggressive or disrespectful manner . ” IDI participant.
This study revealed that midwives experience a loss of motivation at work due to limited support from their superiors. Their effort is used only for reporting purposes. A midwife from FGD shared her experience as follows.
“In our scenario , till the nearest time , the maternal and child health services are provided in a good way. But this was not easy; it is the cumulative effort of midwives. But unfortunately , only those in managerial positions are recognized. Nothing was done for us despite our efforts. To me , our efforts are used only for reporting purposes.” FGD discussant.
This finding was also supported by IDI participants.
“Even though we have good achievements in the MCH services , there is no motivation mechanism done to motivate midwives.” But if something or a minor mistake happens , they are on the front lines to intimidate us or write a warning letter. Generally , their concern is a report or a number issue. We are tired of such types of scenarios.” IDI participant.
One of the main concerns reported by the participants during the interviews was the work related security, which has become a challenge for MLC. The midwives’ work environment was surrounded by insecurity, especially during night shifts, when midwives were facing verbal and even physical attack, as mentioned by participants.
“In the labour ward , especially at night , we face many security-related issues. The families of labouring mothers , especially those who are young , are very aggressive. Sometimes they even want to enter the delivery room. They did not hear what we told them to do , but if they hear any labour sounds from their family , they disturb the whole ward. This leads to verbal abuse , and sometimes we face physical abuse. There may be one or two security personnel at the main gate , but since the delivery ward is far from the main gate , they do not know what is happening in the delivery ward. When things become beyond our scope , we call security guards. Immediately after the security guards go back , similar things will continue. What makes it difficult to manage such situations is that only one midwife is assigned at night , and labouring mothers will not get quality midwifery care.” IDI participant.
FGD discussants also shared their experience that their working environment is full of insecurity.
“In case any complications occur , especially at night , it is very difficult to tell the labouring mother’s family or husband unless we call security personnel. It is not only swearing that we face but also that they intimidate us.” FDG discussant.
The aim of this study was to explore facilitators’ and barriers to a midwifery-led model of care at Dire Dawa public health facilities. In this study, professional pride was the main facilitator of the midwifery-led model of care. Another qualitative study that examined the midwifery care challenges and factors that motivate them to remain in their workplace lends confirmation to this conclusion. It was found that a strong feeling of love for their work was the main facilitator’s midwifery-led model of care [ 9 ]. Having a good team spirit was also another facilitator’s midwifery-led model of care in our study. Another study’s findings confirmed this one, which emphasizes that building relationships with the midwives, women, and community was the driving force behind providing midwifery care [ 7 , 18 ].
The midwives in this study expressed a need for additional professional training, updates, and competence as part of their continuing professional development. Similar findings have been reported in the worldwide literature that midwives were struggling for survival due to a lack of limited in-service training opportunities to improve their knowledge and skills [ 19 ]. This phenomenon does not seem to differ between settings in high-, middle-, and low-income countries [ 7 , 9 , 18 ], in which midwives experienced difficult work situations due to a lack of professional development to autonomously manage work tasks, which made them feel frustrated, guilty, and inadequate. As such, this can contribute to distress and burnout, which in turn prevent midwives from being able to provide quality care and can eventually cause them to leave the profession [ 19 ].
Shortages of resources (shortage of staff, lack of physical space, and equipment) were the other reported barriers to midwifery care explored in this study. They reported that they are working in an environment with a shortage of resources, which leads to poor patient outcomes. This finding is supported by many other studies conducted around the globe [ 20 , 21 , 22 , 23 ]. Another qualitative finding, which likewise supports the aforementioned finding, which emphasizes that a shortage of resources was reported as a barrier to providing adequate midwifery care [ 19 ]. Delivery attended by skilled personnel with appropriate supplies and equipment has been found to be strongly associated with a reduction in child and maternal mortality [ 24 ].
The feeling of demoralization and lack of motivation from their superiors were other barriers to midwifery care explored in this study. This finding is concurrent with other studies conducted around the globe [ 19 , 25 , 26 , 28 ]. The above finding is also is in accord with another qualitative narration, which emphasizes that feelings of demoralization and a lack of motivation were the main challenges of midwifery care [ 22 ]. Positive support from supervisors has been demonstrated to be important for the quality of services that health workers are able to deliver. In the World Health Organization’s report on improving performance in healthcare, the WHO stresses that supportive supervision can contribute to the improved performance of health workers [ 27 ].
Unfair risk payment was the other challenge identified by the current study. Even though there is no difference in the risk they face among health professionals, the risk payment for midwives is very low compared to others. This finding was in conformity with another qualitative narration, which emphasizes that the lack of an equitable remuneration system was experienced by the DRC midwives, and it has also been confirmed to be highly problematic in other studies in low- and middle-income settings [ 7 , 8 , 22 , 28 ], leading to serious challenges. In settings where salaries are extremely low or unpredictable, proper remuneration is seen as crucial to worker motivation and the quality of midwifery care [ 29 , 30 ].
The limited organizational power of midwives was another identified challenge of MLC. This finding was in step with other studies that emphasize that limited senior midwifery positions in the health system constrain the decision-making power and capability of midwives. This was compounded by limited opportunities for midwifery personnel to address their concerns to the responsible bodies. Hence, midwives need to take control of their own situations. When midwives are included in customizing their work environments, it has proven to result in improved quality of care for women and newborns around the globe [ 8 , 15 ].
Lack of work-related security was another barrier to MLC explored in this study, in which the midwives’ work environment was surrounded by insecurity, especially during night shifts, when midwives are facing verbal and even physical attack, as mentioned by participants. This finding is supported by many other studies conducted around the globe [ 22 , 23 , 25 , 31 ]. The above finding is also in agreement with another qualitative narration, which emphasizes that the midwives’ work environment was surrounded by insecurity, especially during night shifts due to a lack of available security personnel; they often felt frightened on their way to and from work [ 7 ]. In order for midwives to provide quality care, it is crucial to create supportive work environments by ensuring sufficient pre-conditions, primarily security issues [ 31 ].
The study findings contribute to a better understanding of the facilitators’ and barriers of a midwifery-led model of care in the case of Dire Dawa public health facilities. Professional pride and having good team spirit were the main facilitators of midwifery-led model care. Contrary to that, insufficient professional development, shortage of resources, feeling of demoralization, lack of motivation, limited organizational power of midwives, unfair risk payment, and lack of work-related security were the main barriers to a midwifery-led model of care in the case of Dire Dawa public health facilities. Generally, midwifery care is facing considerable challenges, both pertaining to the management of the healthcare service locally and nationally.
The findings of the study have implications for midwifery care practices in Eastern Ethiopia. Addressing these areas could potentially contribute to the reduction of IMR and MMR.
The first strength of the study is that the participants represented different healthcare facilities, both urban and rural, thereby offering deeper and more varied experiences and reflections. A second strength is using a midwife as a moderator. She or he understood the midwives’ situation, thereby making the participants feel more comfortable and willing to share their stories. However, focusing solely on the perspective of the midwives is a limitation.
To overcome the barriers of midwifery care, based on the result of this study and in accordance with the 2020 Triad Statement made by the International Council of Nurses, the International Confederation of Midwives, and the World Health Organization, it is suggested that policymakers, Ethiopian federal ministry of health, Dire dawa health office, and regulators in Dire Dawa city and settings with similar conditions coordinate actions in the following:
Should strengthen regular and continuous educational opportunities, trainings, and updates for midwives, prioritizing and enforcing policies to include adequate and reasonable remuneration and hazard payment for midwives. Support midwifery leadership at all levels of the health system to contribute to health policy development and decision-making.
Ensure decent working conditions and an enabling environment for midwives. This includes reasonable working hours, occupational safety, safe staffing levels, and merit-based opportunities for career progression. Special efforts must be made to ensure safe, respectful, and enabling workplaces for midwives operating on the night shift. Midwifery leaders should be involved in management bodies within an appropriate legal framework. Made regular mentorships on the functionality of different diagnostic instruments in respective health facilities.
Create an arena for dialogue and implement a more supportive leadership style at the respective health facilities. Should address professional-related concerns of midwives early. Ensure midwives’ representation at the management bodies. Ensure the selection criteria for educational opportunities and different trainings are fair and inclusive. Ensure the safety and security of midwives, especially those who work night shifts. Should assign adequate staff (midwives and security guards) to the night shifts.
Should influence different stakeholders to solve midwife’s concerns like hazards payment and educational opportunity.
All the datasets for this study are available from the corresponding author upon request.
Focused group discussion
In-depth interview
Infant mortality rate
Key informant interview
Maternal and child health
Midwives led model of care
Neonatal mortality rate
The midwives model of care. Midwives alliance North America, the MANA core documents, 2020.
WHO. Midwife-led care delivers positive pregnancy and birth outcomes. The global health work force alliance,2020.
ICM, Midwifery Led Care, the First Choice for All Women, Netherlands, 2017.
Alba R, Franco R, Patrizia B, Maria CB, Giovanna A, Chiara F, Isabella N. The midwifery-led care model: a continuity of care model in the birth path. Acta Bio Medica: Atenei Parmensis. 2019;90(Suppl 6):41.
Google Scholar
Dahl B, Heinonen K, Bondas TE. From midwife-dominated to midwifery-led antenatal care: a meta-ethnography. Int J Environ Res Public Health. 2020;17(23):8946.
Article PubMed PubMed Central Google Scholar
McConville F, Lavender DT. Quality of care and midwifery services to meet the needs of women and newborns. BJOG: Int J Obstet Gynecol. 2014;121.
Shahnaz S, Jan R, Lakhani A, Sikandar R. Factors affecting the midwifery-led service provider model in Pakistan. J Asian Midwives (JAM). 2015;1(2):33–45.
Bogren M, Grahn M, Kaboru BB, Berg M. Midwives’ challenges and factors that motivate them to remain in their workplace in the Democratic Republic of Congo—an interview study. Hum Resour Health. 2020;18:1–0.
Article Google Scholar
Bremnes HS, Wiig ÅK, Abeid M, Darj E. Challenges in day-to-day midwifery practice; a qualitative study from a regional referral hospital in Dar Es Salaam. Tanzan Global Health Action. 2018;11(1):1453333.
Yigzaw T, Abebe F, Belay L, Assaye Y, Misganaw E, Kidane A, Ademie D, van Roosmalen J, Stekelenburg J, Kim YM. Quality of midwife-provided intrapartum care in Amhara regional state, Ethiopia. BMC Pregnancy Childbirth. 2017;17:1–2.
Federal Democratic Republic of Ethiopia Mini Demographic and Health Survey. 2019 Ethiopian Public Health Institution, Addis Ababa The DHS Program ICF Rockville, Maryland, USA May 2021.
Federal Democratic Republic of Ethiopia. Demographic and Health Survey 2016 Central Statistical Agency Addis Ababa, Ethiopia The DHS Program ICF Rockville, Maryland, USA July 2017.
UNICEF for every child. Situation Analysis of children and women. Dire Dawa Administration; 2020.
Federal Ministry of. Health, Midwifery care process,2021.
Dire Dawa administration Regional Health Bureau. 2017 six months report [unpublished].
Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Educ Inform. 2004;22(2):63–75.
Irene K, Albine M, Series. Practical guidance to qualitative research. Trustworthiness and publishing. Eur J Gen Pract. 2018;24(1):120–4.
Behruzi R, Hatem M, Fraser W, Goulet L, Ii M, Misago C. Facilitators and barriers in the humanization of childbirth practice in Japan. BMC Pregnancy Childbirth. 2010;10:1–8.
Adatara P, Amooba PA, Afaya A, Salia SM, Avane MA, Kuug A, Maalman RS, Atakro CA, Attachie IT, Atachie C. Challenges experienced by midwives working in rural communities in the Upper East Region of Ghana: a qualitative study. BMC Pregnancy Childbirth. 2021;21:1–8.
Roets L. Independent midwifery practice: opportunities and challenges. Afr J Phys Health Educ Recreation Dance. 2014;20(3):1209–24.
Mselle LT, Moland KM, Mvungi A, Evjen-Olsen B, Kohi TW. Why give birth in health facility? Users’ and providers’ accounts of poor quality of birth care in Tanzania. BMC Health Serv Res. 2013;13:1–2.
Bogren M, Erlandsson K, Byrskog U. What prevents midwifery quality care in Bangladesh? A focus group enquiry with midwifery students. BMC Health Serv Res. 2018;18(1):639.
Mtegha MB, Chodzaza E, Chirwa E, Kalembo FW, Zgambo M. Challenges experienced by newly qualified nurse-midwives transitioning to practice in selected midwifery settings in northern Malawi. BMC Nurs. 2022;21(1):236.
Floyd L. Helping midwives in Ghana to reduce maternal mortality. Afr J Midwifery Women’s Health. 2013;7(1):34–8.
Filby A, McConville F, Portela A. What prevents quality midwifery care? A systematic mapping of barriers in low and middle income countries from the provider perspective. PLoS ONE. 2016;11(5):e0153391.
Prytherch H, Kagoné M, Aninanya GA, Williams JE, Kakoko DC, Leshabari MT, Yé M, Marx M, Sauerborn R. Motivation and incentives of rural maternal and neonatal health care providers: a comparison of qualitative findings from Burkina Faso, Ghana and Tanzania. BMC Health Serv Res. 2013;13:1–5.
World Health Organization. The world health report 2000: health systems: improving performance. World Health Organization; 2000.
Oyetunde MO, Nkwonta CA. Quality issues in midwifery: a critical analysis of midwifery in Nigeria within the context of the International Confederation of Midwives (ICM) global standards. Int J Nurs Midwifery. 2014;6(3):40–8.
Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, Adeyi O, Barker P, Daelmans B, Doubova SV, English M. High-quality health systems in the Sustainable Development goals era: time for a revolution. Lancet Global Health. 2018;6(11):e1196–252.
Article PubMed Google Scholar
Mathauer I, Imhoff I. Health worker motivation in Africa: the role of non-financial incentives and human resource management tools. Hum Resour Health. 2006;4:1–7.
World Health Organization. Global strategy on human resources for health: workforce 2030.
Download references
We are very grateful to Dire Dawa University for the financial support for this study and to the College of Medicine and Health for its monitoring ship. All study participants for their willingness to respond to our questionnaire.
this work has been funded by Dire Dawa University for data collection purposes. The Dire Dawa University College of Medicine and Health Sciences was involved in the project through monitoring and evaluation of the work from the beginning to the result submission. However, this organization was not involved in the design, analysis, critical review of its intellectual content, or manuscript preparation, and its budget did not include publication.
Authors and affiliations.
College of Medicine and Health Sciences, Dire Dawa University, Dire Dawa, Ethiopia
Mickiale Hailu, Aminu Mohammed, Daniel Tadesse, Neil Abdurashid, Legesse Abera, Tadesse Weldeamaniel, Tekleberhan Hailemariam, Netsanet Melkamu, Tewodros Getnet, Yibekal Manaye, Tariku Derese, Muluken Yigezu, Natnael Dechasa & Anteneh Atle
College of Health Sciences, Wachemo University, Hossana, Ethiopia
Samrawit Ali & Yesuneh Dejene
College of Health Sciences, Mekelle University, Mekelle, Ethiopia
Meklit Girma
You can also search for this author in PubMed Google Scholar
MH developed the study proposal, served as the primary lead for study implementation and data analysis/interpretation, and was a major contributor in writing and revising all drafts of the paper. AM, DT, NA, LA, and SA supported study implementation and data analysis, and contributed to writing the initial draft of the paper. YD, TW, MG, TH and, NM supported study recruitment and contributed to writing the final draft of the paper. TG, YM, TD, MY, ND and, AA conceptualized, acquired funding, and led protocol development for the study, co-led study implementation and data analysis/interpretation, and was a major contributor in writing and revising all drafts of the paper. All authors contributed to its content. All authors read and approved the final manuscript.
Correspondence to Mickiale Hailu .
Ethics approval and consent to participate.
All methods were followed in accordance with relevant guidelines and regulations. The institutional review board of Dire Dawa University has also examined and evaluated it for its methodological approach and ethical concerns. Ethical clearance was obtained from Dire Dawa University Institutional Review Board and an official letter from research affairs directorate office of Dire Dawa University was submitted to Dire Dawa health office and it was distributed to selected health institutions. Voluntary informed written consent was obtained from the study participant’s right after the objectives of the study were explained to the study participants and confidentiality of the study participants was assured throughout the study period. Participants were informed that they have the right to terminate the discussion (interview) or they can’t answer any questions they didn’t want to answer.
Not applicable.
The authors declare no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
Rights and permissions.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .
Reprints and permissions
Cite this article.
Hailu, M., Mohammed, A., Tadesse, D. et al. Facilitators and barriers of midwife-led model of care at public health institutions of dire Dawa city, Eastern Ethiopia, 2022: a qualitative study. BMC Health Serv Res 24 , 998 (2024). https://doi.org/10.1186/s12913-024-11417-x
Download citation
Received : 03 September 2023
Accepted : 09 August 2024
Published : 28 August 2024
DOI : https://doi.org/10.1186/s12913-024-11417-x
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1472-6963
BMC Medical Education volume 24 , Article number: 935 ( 2024 ) Cite this article
Metrics details
Traditional radiology education for medical students predominantly uses textbooks, PowerPoint files, and hard-copy radiographic images, which often lack student interaction. PACS (Picture Archiving and Communication System) is a crucial tool for radiologists in viewing and reporting images, but its use in medical student training remains limited.
This study investigates the effectiveness of using PACS (Picture Archiving and Communication System) for teaching radiology to undergraduate medical students compared to traditional methods.
Fifty-three medical students were divided into a control group (25 students) receiving traditional slide-based training and an intervention group (28 students) using PACS software to view complete patient images. Pre- and post-course tests and satisfaction surveys were conducted for both groups, along with self-evaluation by the intervention group. The validity and reliability of the assessment tools were confirmed through expert review and pilot testing.
No significant difference was found between the control and intervention groups regarding, gender, age, and GPA. Final multiple-choice test scores were similar (intervention: 10.89 ± 2.9; control: 10.76 ± 3.5; p = 0.883). However, the intervention group demonstrated significantly higher improvement in the short answer test for image interpretation (intervention: 8.8 ± 2.28; control: 5.35 ± 2.39; p = 0.001). Satisfaction with the learning method did not significantly differ between groups (intervention: 36.54 ± 5.87; control: 39.44 ± 7.76; p = 0.129). The intervention group reported high familiarity with PACS capabilities (75%), CT principles (71.4%), interpretation (64.3%), appropriate window selection (75%), and anatomical relationships (85.7%).
PACS-based training enhances medical students’ diagnostic and analytical skills in radiology. Further research with larger sample sizes and robust assessment methods is recommended to confirm and expand upon theses results
Peer Review reports
Radiology is a fundamental component in basic medical education, bridging the gap between anatomy and clinical practice. Like other fields of medical education, radiology education faces the challenge of transitioning from passive learning to interactive and experiential learning [ 1 , 2 ]. With the expansion of the field of radiology, radiology education has undergone a revolution. Doctors used to carry plain films and show them using projectors or view boxes because plain films were the only main diagnostic method in radiology during the 1970s. Since the introduction of computed tomography (CT) and magnetic resonance imaging (MRI) in the late 1980s, the increase in the amount of image data associated with these imaging modalities has led to a greater demand for compatible information storage systems. Therefore, the picture archiving and communication system (PACS), capable of storing, retrieving, distributing, analyzing, and digitally processing medical images, has become an essential tool in clinical work today [ 3 , 4 , 5 ]. However, due to hardware and software limitations, the use of PACS in radiology education remains somewhat limited [ 6 , 7 ]. Currently, most radiology education still relies heavily on textbooks and traditional computer media such as PowerPoint or Word files both of which lack student interaction. PACS offers advantages such as interactive image viewing, 3D reconstruction capabilities, and the ability to simulate real-life radiology practice, which traditional methods lack. These features enhance students’ understanding and interpretation of radiological images, addressing the shortcomings of conventional methods. There is a minimal probability for a medical student to see whole images like a real radiologist in class. It is often a challenge for them to understand 3D anatomical images, as well as a comprehensive view of diseases. Consequently, some students may attempt to independently identify abnormal findings and analyze and formulate radiological diagnoses. According to one study, only a limited number of final-year medical students had satisfactory basic radiology interpretation skills, which necessitates the search for a more effective method of training [ 8 ].
Recent advancements in radiology teaching methods have previously been reported in addition to face-to-face teaching, including problem-based learning (discussion of a case or scenario consistent with curriculum objectives and students’ independent research to complete subject knowledge and share findings), case-based learning (showing several radiographs of the same subject and discussing them), and team-based learning (student collaboration by creating learning groups) [ 8 ].
In contrast to these conventional methods, a new method was created under the concept of learning from experience. This virtual method is based on individual learning in the PACS software environment, enabling students in the role of radiologists to interpret and diagnose radiology in a simulation environment. All common items are shown to the student using PACS instead of selected specific images. Students are allowed to see the whole image, do basic reconstructions of the images freely, and find specific features of the image by themselves. During this process, students can access PACS and clinical information, integrating clinical knowledge and 3D reconstruction ability, essential to arriving at radiological diagnoses PACS enables efficient archiving and transfer of medical images. Initially developed in the U.S. in the 1980s, it later expanded to Europe and Asia, including China, Japan, and Korea [ 9 ]. Iran has also implemented PACS, improving its medical imaging infrastructure with global DICOM standards.
The goal of this learning method was to compare the effectiveness of practical radiology training through traditional face-to-face interactive lectures with the virtual practical radiology training method based on individual learning in the PACS software environment for medical students.
The use of PACS in healthcare in Iran has only recently become widespread, primarily for patient management and diagnosis, and is rarely used for educational purposes. Iran, as a country with a rapidly developing healthcare system, faces unique challenges in medical education. This study seeks to compare radiology education in Iran with existing literature and to understand its context in relation to the region and worldwide. Managing medical education effectively is a significant challenge. And this research addresses this by introducing innovative teaching methods. Specifically, current study investigates the effectiveness of using PACS on medical students radiology education compared to traditional methods.
The research population was the medical students of the Islamic Azad University of Mashhad during the academic year 2021–2022. The entry criteria were: being a medical trainee student, consent to enter the study, and the exclusion criteria were: students who had previously graduated in radiology or other medical sciences and students who had renewed their course in radiology. participation in the study was voluntary, and students were informed that it would not impact their end-of-section evaluation After obtaining informed consent, they participated in the study. Ethical approval for this study was obtained from the Virtual University of medical sciences with the reference number [IR.VUMS.REC.1400.022]. This proposal was implemented after being approved by the ethics committee and obtaining the code of ethics.
The sample size was calculated using power analysis to ensure the study had sufficient power to detect a statistically significant difference between the control and intervention groups. Assuming an effect size of 0.5, a significance level (alpha) of 0.05, and a power of 0.80, it was determined that at least 50 participants were needed. To account for potential dropouts and ensure robustness, a total of 53 students were included in the study. According to the calculated sample size, four rotations of radiology internship students were included in the study for each of the control and intervention groups (each rotation is about 5–10 students). Due to the prevention of contamination, the first four rotations were assigned to the control group and the next four rotations to the intervention group.
The validity of the tools used in this study was established through expert review and pilot testing. Content validity was confirmed by 10 faculty members specializing in radiology. Reliability was assessed using Cronbach’s Alpha, yielding a coefficient of 0.91, indicating high internal consistency. In this study, three tools were used: measuring the level of knowledge, measuring the level of performance, and measuring the satisfaction of students in both groups (Appendix 1 ) and self-evaluation for PACS learning in the intervention group (Appendix 2 ). After one month of class, the final exam was taken which was a combination of 20 multiple choice questions and 5 short answer type questions (description and image recognition). The scores of the questions were collected as an objective assessment. To provide a subjective assessment of radiology learning, all students were invited to complete a satisfaction questionnaire on how radiology was taught. Also, the students of the intervention group were invited to complete a questionnaire for their self-evaluation of the amount of PACS learning. A 5-point Likert scale was used in both researcher-made questionnaires. The questionnaire used was created for this study. Informed consent was obtained from each patient whose data was used in the study, ensuring they were fully aware of how their medical images would be utilized for educational purposes.
Before starting the study with the PACS system, students were given an introductory session that covered the basics of PACS functionality, including how to navigate the software, view and manipulate images, and use the various tools available for image analysis.
In the knowledge section, questions evaluated theoretical content, and the performance section involved diagnosing radiographic image. Students described the type of radiography, pathological signs, and the final diagnosis. Multiple-choice questions and short answer questions were used to assess knowledge and performance The specific type of radiography used in this study included plain radiographs, computed tomography (CT) scans, and magnetic resonance imaging (MRI). These imaging modalities were chosen to cover a broad spectrum of radiological techniques relevant to the medical curriculum. In the subject of knowledge, 20 multiple-choice questions were proposed based on the objectives of the lesson and the blueprint, which was approved by two colleagues of the radiology department, which must have been consistent with the objectives of the lesson. In the discussion of the performance of 5 of radiology images, which again corresponded to the objectives of the lesson and the blueprint, and it was approved by two colleagues of the radiology department that the objectives of the lesson were covered, they were provided to the students, and the students had to describe and diagnose the radiographies. The radiology images in both groups adequately covered the goals, but they were taught to the students in two different ways described.
This questionnaire aimed to determine student’s satisfaction with the educational method. It consisted of ten questions graded on a 5-point Likert scale the range of scores was between 10 and 50 and higher scores indicating greater satisfaction. The content and form validity were confirmed by 10 faculty members and reliability was obtained by Cronbach’s Alpha test of 0.91.
This questionnaire evaluated the learning rate of the PACS teaching method. It consisted of twelve questions graded on a 5-point Likert scale, and the range of scores was between 12 and 60, and higher scores indicate learning. Content and form validity were confirmed by 10 faculty members and reliability was assessed with a Cronbach’s Alpha of 0.91.
The teaching strategy involved traditional face-to-face interactive lectures using PowerPoint presentations. The practical part included demonstrating selected radiographic images on slides and discussing their interpretation.
This method aimed to develop the student’s ability to diagnose and interpret radiographs through structured lectures and guided discussions. A pre-test was conducted in the first session to determine the student’s initial knowledge and performance levels. The classes were held daily in person. After teaching the theoretical part with a PowerPoint presentation, radiographic images were shown to the control group for interpretation and discussion. This conventional method aimed to develop the ability to diagnose and interpret radiographs. The post-test to determine knowledge and performance was performed and the education satisfaction questionnaire was completed at the end of each rotation.
Bias caused by human factors during the teaching of the two groups was controlled by standardizing the teaching materials and methods across both groups. Additionally, the instructors were blinded to the group assignments to prevent any conscious or unconscious bias in teaching and assessment.
The stages of developing the training course using PACS software and DICOM were as follows: 1). Initial planning and curriculum alignment, 2) Selection of relevant radiographic cases, 3) Configuration of PACS workstations, 4) Training faculty on PACS software, and 5) Implementation of PACS-based learning sessions for students, followed by assessment and feedback.
After the control group, the rotations of the intervention group were included in the study, and the pre-test was administered to the students of the intervention group. Assessment of knowledge with multiple choice questions and performance with radiographic images was with short answer questions. The classes were held daily in person. In the intervention group, after participating in the theoretical part of the course, which was similar to the control group and was held face-to-face, for the practical part, they were trained in a virtual way with Adobe Connect software, and there was no face-to-face class for radiography images. In this way, students were given access to PACS Radiant software (installation on personal desktop). Following the teaching of the theoretical part, based on the goals of the radiology course for medical trainees, a number of images of the brain, lungs, bones, urinary tract, and digestive system (including radiography, CT and MRI) were assigned to the students of the intervention group, and the images of these patients were completely at their disposal.
The computers used were personal desktops with standardized configurations. Adjustments and calibrations were made to ensure all students could view images with consistent quality and brightness, replicating the clinical environment as closely as possible. This software enables students to perform basic operations with images, such as windowing, comparing different MRI sequences, and performing cross-sectional reconstruction (MPR) or 3D reconstruction, exactly as a radiologist does and has the facilities. After studying the material and checking the images, the students were required to announce the completion of their study to the teacher and they were given the opportunity to review the pictures, ask questions, and solve problems with the teacher in the virtual space.
The post-test to determine knowledge and performance was performed in the intervention group. The education satisfaction questionnaire was completed at the end of each rotation. The self-assessment questionnaire for PACS learning was completed at the end of each rotation.
The data was analyzed with SPSS-17 software, IBM, US. Central and dispersion indices were used in the descriptive statistics report, and a T-test was used in the analytical section, independent t-test, paired t-test and, chi-square test were used to compare the data. The confidence level was set at p < 0.05.
A total of 52 students entered this study, 28 students in the intervention group and 25 in the control group. The students were similar in terms of age, gender, and overall academic average ( p = 0.05) (Table 1 ). The average age in the control group is 26.04 ± 3.96 and in the intervention group is 24.29 ± 2.14. The result of the independent t-test shows that the average age in the two groups is not different ( P = 0.060). The average overall academic grade point average of the medical course in the control group is 15.73 and in the intervention group is 16.01, which has no difference ( P = 0.383) (Table 1 ).
The control group included 25 people, 16 of whom were women and 9 of whom were men, and the intervention group included 28 people of whom 16 were women and 12 were men. The result of the chi-square test shows that the two groups do not differ in terms of gender ( P = 0.610). Evaluation result: At the beginning of the exam, there were two parts of a multiple-choice test and a short answer for the interpretation of radiology images (pre-test). The same exam was done twice at the end of the one-month session (post-test). It was a multiple-choice test to check knowledge and a short answer test to check performance.
The result of the independent t-test shows that the score of the multiple-choice test before and after the intervention, as well as the changes in the test score, are not different in the two groups. ( P = 0.084, P = 0.883, P = 0.764) The result of the paired t-test shows that the multiple-choice test scores of the students before and after the intervention differ between the case and control groups, and it is higher after the intervention. ( P < 0.001, P < 0.001) (Table 2 ) The result of the independent t-test shows that the score of the student’s short answer test, which was for the interpretation of radiology images, is not different before and after the intervention ( P = 0.002 and P = 0.444, respectively). The changes in the test scores are different in the two groups and are more in the intervention group. ( P < 0.001) The result of the paired t-test shows that the score of the short answer test of the students before and after the intervention is different according to the case and control groups, and it is higher after the intervention. ( P < 0.001, P < 0.001)
The result of the independent t-test shows that there is no difference in the level of satisfaction with the teaching method between the two control groups with a score of 39.44 ± 7.76 and the intervention group with a score of 36.54 ± 5. ( P = 0.129) (Table 3 ).
The analysis of the satisfaction questionnaire in the intervention group showed that most students were satisfied with the organization (64%) and interaction of the learning activity (64%) (Table 3 ). Most students use this learning activity to learn radiology (85%). They found it useful. More importantly, a large percentage of students stated that PACS training encouraged personal interest in radiology (82%) as well as satisfaction with the quality of learning (71%). Also, in the intervention group, based on the self-evaluation form, they stated that with the abilities of PACS (75%), the principles of CT (71.4%) and its interpretation (64.3%), choosing the appropriate window (75%), the location of different organs in the image (82.9%) and their vicinity (85.7%) are familiar (Table 3 ). An evaluation of the impact of the intervention on participants’ knowledge is included, showing significant improvements in their understanding and diagnostic skills, highlighting the effectiveness of the PACS-based training method.
Traditional practical radiology training that continues to be used today provides only a cross-section of the entire routine imaging. While this teaching method may be useful in helping students manage the features of routine imaging, it may be inadequate for learning anatomy [ 10 ]. Hence, students may have difficulty interpreting images independently during clinical practice when they are expected to do so [ 11 ]. Although a variety of radiology educational models such as problem-based learning and the use of dynamic images can solve part of this problem, images of the main workplace are the most ideal learning method [ 12 , 13 ]. The experiential learning theory, developed by Dewey, Kolb and others provide explanations for how students learn things in their own way as they react to their perceptions of a real experiences. This concept is explained by principle of constructionism, which is the base of experiential learning [ 13 ].
During this study, a training course using PACS software and DICOM viewer was developed to simulate a work environment that reflects the typical clinical work of a radiologist. The results of the study indicated that this educational approach allows for better clinical guidance, which is necessary to help students form a holistic view of anatomy and pathology. Most importantly, this educational method helps students to develop critical thinking and a systematic approach to formulating imaging interpretation and differential diagnosis, which may be partially due to the exploratory atmosphere of the experiential learning mode. Apart from the objective improvement in imaging descriptions and interpretations, subjective improvements in self-confidence from students’ feedback to self-assessment questionnaires, as well as skills including determining the order of imaging reading, choosing the appropriate window, and also choosing the reconstruction method, which may result under the influence of direct activity during The course of learning and discussion should be free. In addition, the experiential approach allows for better interactions that increase interest in radiology [ 14 ].
To provide students with access to the Radiant PACS software (installed on their personal desktops), following the theoretical section and based on the objectives of the radiology course for medical trainees, a number of images from the brain, lungs, bones, urinary, and gastrointestinal systems (including radiography, CT, MRI) were assigned to the intervention group. These patient images were fully available to them. This software enables students to perform basic operations on images, such as window adjustment, comparing different MRI sequences, and performing multiplanar reconstruction (MPR) or 3D reconstruction, exactly as a radiologist does within the PACS system.
To resolve the issue of patient confidentiality, all patient identifiers were removed from the images before they were made accessible to students. Additionally, access to PACS was restricted to ensure that students could only view and analyze the images without accessing sensitive patient information.
Undergraduate students had limited access to PACS, ensuring they could not modify or delete any content. Additional software controls were implemented to restrict access and prevent any unauthorized changes. This ensured that the integrity of the medical images was maintained, and patient care data was not compromised.
Our study shows the effectiveness of PACS in training in the study of anatomical imaging. Anatomy is the basis of radiology training. In theory, reading CT and MRI images is a good way to study anatomy because continuous scanning helps students understand the three-dimensional concepts of the relative adjacencies of body parts [ 15 , 16 ]. Globally, they concluded that anatomical imaging increases the quality and efficiency of teaching human anatomy [ 17 ]. However, it is difficult to discern the entire anatomical structure from a single cross-section of the image, which increases students’ confusion [ 16 ]. The results of this study provide evidence that continuous scan reading improves students’ comprehensive understanding of anatomy. Furthermore, by using multiple reconstruction methods, 3D images are more comprehensively examined by students, which has been confirmed by other studies [ 18 ].
The integration of PACS in medical education has been shown to enhance the learning experience by providing students with interactive and practical tools for understanding radiological images. Recent advancements in healthcare technology acceptance highlight the importance of user-friendly interfaces and training for successful implementation [ 19 ]. Moreover, the current state of medical education in the UK emphasizes the adoption of advanced technologies like PACS to improve educational outcomes and prepare students for real-world clinical environments [ 20 ]. The utilization of big data technologies in conjunction with PACS further enhances the management and analysis of medical images, facilitating a more personalized and effective learning experience for medical students [ 21 ]. Additionally, recent market reports indicate a steady growth in the adoption of medical imaging technologies, including PACS, driven by advancements in AI and machine learning, which are poised to revolutionize medical education [ 22 ]. These developments collectively underscore the critical role of PACS in modernizing medical education and improving the quality of training for future healthcare professionals. Also, the implementation of PACS could significantly enhance radiology education by providing access to digital imaging resources that may otherwise be unavailable.
Compared to Chen et al.‘s study [ 1 ], the study was conducted on 101 students, but our study was on 52 students. Satisfaction with PACS training in Chen’s study was on average 80% and in our study, it was about 65%. The percentage of being interested in radiology in this study and Chen’s study was almost similar. Also, in our study, similar to Chen’s study, there was no difference in pre-test scores between the two intervention and control groups. Also, the final scores in Chen’s study and our study were not significantly different, but the scores of interpretations of pictures, which in our study were equivalent to a number of stereotypes in the form of PowerPoint with short answer questions, showed a significant difference in both our study and Chen’s study.
In the study of Restauri [ 6 ] and Soman [ 23 ], as in our study, PACS was used to teach medical students, and at the end of the course, only a survey form was filled by the students, and the impact of using PACS on the ability to interpret radiology images by students was not done. In the above two studies, after using PACS, students stated that they gained more confidence on interpreting images and would use PACS in the future, which was similar to the survey results in our study. It takes a lot of effort to do this kind of training. PACS and a suitable DICOM viewer represent basic software requirements for training and to protect patient privacy, DICOM data from PACS rather than linking to the original PACS. Copied In this way, a PACS simulation for medical education was obtained [ 6 ]. In addition, teacher guidance is a vital element in education. A minimum of 3 instructors with experience in standard radiology training is required for a class, as team discussion is a major component of the training. In experimental courses, students need educational help both to guide reading the picture and to answer the questions. Therefore, teaching professors need specific work experience in the radiology department. Having said that, the lack of a radiology professor prevents the use of this training and this training model acts as a limitation on a larger scale. There are several limitations to the study. First, due to the limited number of supervisors, the sample size was correspondingly limited. Secondly, it was a single study center. Thirdly, due to the limitation of the operation, some students did not answer some of the questions in the questionnaire. Although the probability is very low, it still has the chance to bias the result. Fourth, although we control for faculty and teaching standards between the two groups, human bias is still a factor that cannot be completely avoided in practice. Fifth, although we used objective assessment measures, the study also revealed the weakness of our assessment system in radiology education. The study instrument consisted of paper and pencil tests, with most questions consisting of objective items that test memory, such as multiple-choice questions and short answer questions. Furthermore, the mental items used to test application ability are limited. As a result, only a small part of the final test reflects the difference between the experimental training group and the control group. Other test forms such as bedside examinations and multi-station examinations should be used in the future for better evaluation [ 24 , 25 ]. In this study, according to the curriculum, students entered the radiology department with different numbers during different periods, and 4 periods of students were entered into the study for each group. The exams were held at the end of the one-month section, so the exam was held in the control group and in the intervention group at different times, although we tried to make the questions the same in terms of number and content similarity. In the study of Chen et al [ 8 ], the test was conducted at the end of the semester and simultaneously for two groups. If this study is conducted with a larger number of students and in multiple centers, the results will be more valid.
PACS-based training is beneficial for medical students, enhancing their diagnostic and analytical skills in radiology. Further research with larger sample sizes and robust assessment methods is recommended to confirm and expand upon theses results. We believe that our findings suggest that PACS which is used routinely in healthcare diagnostic context, can also be used in medical students’ education and healthcare can be integrated in education.
The demographic and clinical datasets generated and/or analyzed during the current study are available from the corresponding author (Dr. Farnood Rajabzadeh ) upon reasonable request.
Picture Archiving and Communication System
Computed Tomography
Magnetic Resonance Imaging
Grade Point Average
Digital Imaging and Communications in Medicine
Statistical Package for the Social Sciences
Multi planar Reconstruction
Artificial Intelligence
Bhogal P, Booth TC, Phillips AJ, Golding SJ. Radiology in the undergraduate medical curriculum -- who, how, what, when, and where? Clin Radiol. 2012;67(12):1146–52.
Article Google Scholar
Naeger DM, Webb EM, Zimmerman L, Elicker BM. Strategies for incorporating radiology into early medical school curricula. J Am Coll Radiol. 2014;11(1):74 – 9.
Forsberg D, Rosipko B, Sunshine JL. Factors affecting Radiologist’s PACS usage. J Digit Imaging. 2016;29(6):670–6.
Mirsadraee S, Mankad K, McCoubrie P, Roberts T, Kessel D. Radiology curriculum for undergraduate medical studies–a consensus survey. Clin Radiol. 2012;67(12):1155–61.
Arriero A, Bonomo L, Calliada F, Campioni P, Colosimo C, Cotroneo A, Cova M, Ettorre GC, Fugazzola C, Garlaschi G, Macarini L, Mascalchi M, Meloni GB, Midiri M, Mucelli RP, Rossi C, Sironi S, Torricelli P, Beomonte BZ, Zompatori M, Zuiani C. E-learning in radiology: an Italian multicentre experience. Eur J Radiol. 2012;81(12):3936–41.
Restauri N, Bang T, Hall B. Sachs P Development and Utilization of a Simulation PACS in Undergraduate Medical Education. J Am Coll Radiol. 2018;15(2):346–9.
Zafar S, Safdar S, Zafar AN. Evaluation of use of e-Learning in undergraduate radiology education: a review. Eur J Radiol. 2014;83(12):2277–87.
Chen Y, Zheng K, Ye S, Wang J, Xu L, Li Z, Meng Q, Yang J, Feng ST. Constructing an experiential education model in undergraduate radiology education by the utilization of the picture archiving and communication system (PACS). BMC Med Educ. 2019;19(1):383.
Huang -HK. Twenty-five years of Picture Archiving and Communication Systems (PACS). Dev Iran J Radiol. 2007;4(2):1.
Google Scholar
-Pascual TN, Chhem R, Wang SC, Vujnovic S. Undergraduate radiology education in the era of dynamism in medical curriculum: an educational perspective. Eur J Radiol. 2011;78(3):319–25.
-Sendra-Portero F, Torales-Chaparro OE, Ruiz-Gomez MJ, Martinez-Morillo. M.A pilot study to evaluate the use of virtual lectures for undergraduate radiology teaching. Eur J Radiol. 2013;82(5):888–93.
-Zhang S, Xu J, Wang H, Zhang D, Zhang Q, Zou L. Effects of problem-based learning in Chinese radiology education: a systematic review and metaanalysis. Med (Baltim). 2018;97(9):e0069.
Yardley S, Teunissen PW, Dornan T. Experiential learning: transforming theory into practice. Med Teach. 2012;34(2):161–4.
Branstetter BF. Humphrey AL,Schumann JB.The long-term impact of preclinical education on medical students’ pinions about radiology. Acad Radiol. 2008;15(10):1331–9.
Schober A, Pieper CC, Schmidt R, Wittkowski W. Anatomy and imaging: 10 years of experience with an interdisciplinary teaching project in preclinical medical education - from an elective to a curricular course. Rofo. 2014;186(5):458–65.
Jang HW, Oh CS, Choe YH, Jang DS. Use of dynamic images in radiology education: movies of CT and MRI in the anatomy classroom. Anat Sci Educ. 2018;11(6):547–53.
Grignon B, Oldrini G, Walter F. Teaching medical anatomy: what is the role of imaging today? Surg Radiol Anat. 2016;38(2):253–60.
Loke YH, Harahsheh AS, Krieger A, Olivieri LJ. Usage of 3D models of tetralogy of Fallot for medical education: impact on learning congenital heart disease. BMC Med Educ. 2017;17(1):54.
AlQudah, Adi A, Al-Emran M, Khaled Shaalan. Technology Acceptance in Healthcare: a systematic review Applied sciences 2021;11(22):10537. https://doi.org/10.3390/app112210537
GMC. (2022). The state of medical education and practice in the UK.
Geroski T, Jakovljević D, Filipović N. Big Data in multiscale modelling: from medical image processing to personalized models. J Big Data. 2023;10:72. https://doi.org/10.1186/s40537-023-00763-y .
-Visage Imaging. Medical imaging market size. Share & Growth Report; 2024.
Soman S, Amorosa JK, Mueller L, Hu J, Zou L, Masand A, Cheng C, Virk J, Rama H, Tseng I, Patel K, Connolly SE. Evaluation of medical student experience using medical student created StudentPACS flash based PACS simulator tutorials for learning radiological topics. Acad Radiol. 2010;17(6):799–807.
Reddy S, Straus CM, McNulty NJ, Tigges S, Ayoob A, Randazzo W, Neutze J, Lewis P. Development of the AMSER standardized examinations in radiology for medical students. Acad Radiol. 2015;22(1):130–4.
Monticciolo DL. The ACR diagnostic radiology in-training examination: evolution, current status, and future directions. J Am Coll Radiol. 2010;7(2):132–7.
Download references
Farbod Rajabzadeh for helping in data gathering, Ladan Goshayeshi for helping in editing, Lena Goshayeshi for helping in editing.
This study was supported by the Smart university of medical sciences and Mashhad Azad University of Medical Sciences.
Authors and affiliations.
Department of e-Learning in Medical Education, School of Medicine, Center of Excellence for E- learning in Medical Education, Tehran University of Medical Sciences, Tehran, Iran
Mojtahedzadeh Rita & Mohammadi Aeen
Department of Radiology, Faculty of Medicine, Mashhad Medical Sciences, Islamic Azad university, Mashhad, Iran
Farnood Rajabzadeh
Department of Community Medicine, University of Medical Sciences, Mashhad, Mashhad, Iran
Akhlaghi Saeed
You can also search for this author in PubMed Google Scholar
RM, FR, designed the study. FR was involved in the data gathering and interpretation of the results. AM and SA performed analyses. FR wrote the first draft of the manuscript. FR and RM edited the final version of the manuscript. All authors read and approved the final version of the manuscript.
Correspondence to Farnood Rajabzadeh .
Ethics approval and consent to participate.
This study was approved by the Ethics Committee of Smart University of Medical Sciences (ethics code: IR.VUMS.REC.1400.022, 4/12/2021) and conformed to the ethical principles contained in the Declaration of Helsinki. For experiments involving human participants the participants signed an informed consent form before the study.
Competing interests.
The authors declare no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
Supplementary material 2, rights and permissions.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .
Reprints and permissions
Cite this article.
Rita, M., Aeen, M., Rajabzadeh, F. et al. Using PACS for teaching radiology to undergraduate medical students. BMC Med Educ 24 , 935 (2024). https://doi.org/10.1186/s12909-024-05919-9
Download citation
Received : 05 December 2023
Accepted : 16 August 2024
Published : 28 August 2024
DOI : https://doi.org/10.1186/s12909-024-05919-9
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1472-6920
IMAGES
VIDEO
COMMENTS
Discussing negative aspects in a peer review requires tact and empathy. Try focusing on behaviors and actions rather than personal attributes, and use phrases that suggest areas for growth. For example: "While your dedication to the project is admirable, it might be beneficial to delegate some tasks to avoid burnout.".
The most common types are: Single-blind review. Double-blind review. Triple-blind review. Collaborative review. Open review. Relatedly, peer assessment is a process where your peers provide you with feedback on something you've written, based on a set of criteria or benchmarks from an instructor.
Think about structuring your review like an inverted pyramid. Put the most important information at the top, followed by details and examples in the center, and any additional points at the very bottom. Here's how your outline might look: 1. Summary of the research and your overall impression. In your own words, summarize what the manuscript ...
The good news is that published papers often now include peer-review records, including the reviewer comments and authors' replies. So here are two feedback examples from my own papers: Example Peer Review: Paper 1. Quantifying 3D Strain in Scaffold Implants for Regenerative Medicine, J. Clark et al. 2020 - Available here
The peer review template for patients and novice reviewers ( table 1) is a series of steps designed to create a workflow for the main components of peer review. A structured workflow can help a reviewer organise their thoughts and create space to engage in critical thinking. The template is a starting point for anyone new to peer review, and it ...
The peer review process is essential for evaluating the quality of scholarly works, suggesting corrections, and learning from other authors' mistakes. The principles of peer review are largely based on professionalism, eloquence, and collegiate attitude. As such, reviewing journal submissions is a privilege and responsibility for 'elite ...
components of peer review. A structured workflow can. help a reviewer organise their thoughts and create space. to engage in critical thinking. The template is a starting. point for anyone new to ...
Summary of the research and your overall impression. In your own words, summarize the main research question, claims, and conclusions of the study. Provide context for how this research fits within the existing literature. Discuss the manuscript's strengths and weaknesses and your overall recommendation. Evidence and examples.
You should now have a list of comments and suggestions for a complete peer review. The full peer-review document can comprise the following sections: 1. Introduction: Mirror the article, state ...
This paper by Amrhein et al. criticizes a paper by Bradley Efron that discusses Bayesian statistics ( Efron, 2013a ), focusing on a particular example that was also discussed in Efron (2013b). The example concerns a woman who is carrying twins, both male (as determined by sonogram and we ignore the possibility that gender has been observed ...
Here are 50+ peer review examples! Use these sample peer feedback phrases with your peers and help them grow professionally! ... To address the issue with the target demographic, it might be beneficial to integrate more specific market research data. I can share a few resources on market analysis that could provide some valuable insights for ...
Peer Review Examples on Professionalism and Work Ethics. "Noah's punctuality is an asset to the team. To maintain professionalism consistently, he should adhere to deadlines with unwavering dedication, setting a model example for peers.". "Grace's integrity and ethical standards are admirable.
The manuscript peer review process helps ensure scientific publications are credible and minimizes errors. Peer review is an essential element of the scientific publishing process that helps ensure that research articles are evaluated, critiqued, and improved before release into the academic community. Take a look at the significance of peer review in scientific publications, the typical steps ...
National Institutes of Health (NIH) Peer Review Policies and Practices. NIH resources about the regulations and processes that govern peer review, including management of conflicts of interest, applicant and reviewer responsibilities in maintaining the integrity in peer review, appeals, and more.
Peer review. A key convention in the publication of research is the peer review process, in which the quality and potential contribution of each manuscript is evaluated by one's peers in the scientific community. Like other scientific journals, APA journals utilize a peer review process to guide manuscript selection and publication decisions.
Peer Review. Peer Review in Three Minutes from NC State University Libraries on Vimeo. A peer reviewed or peer refereed journal or article is one in which a group of widely acknowledged experts in a field reviews the content for scholarly soundness and academic value.
Peer review feedback is a form of evaluative feedback that benefits both the person being reviewed and the reviewer. Unlike typical methods, this type of feedback focuses on strengths as well as areas for improvement. It may seem challenging at first, but it gets easier with practice! This article will go over some examples of what makes good peer review feedback, along with tips on giving it ...
The most common types are: Single-blind review. Double-blind review. Triple-blind review. Collaborative review. Open review. Relatedly, peer assessment is a process where your peers provide you with feedback on something you've written, based on a set of criteria or benchmarks from an instructor.
When peer reviewing, it is helpful to think from the point of view of three different groups of people: Authors. Try to review the manuscript as you would like others to review your work. When you point out problems in a manuscript, do so in a way that will help the authors to improve the manuscript. Even if you recommend to the editor that the ...
Peer-reviewed articles, also known as scholarly or refereed articles are papers that describe a research study. Why are peer-reviewed articles useful? They report on original research that have been reviewed by other experts before they are accepted for publication, so you can reasonably be assured that they contain valid information.
Here are examples of positive peer reviews phrases to use: I'm impressed with how you completed the website design task this week. I want you to keep implementing user-friendly designs and suggest areas where you think the process requires improvement. You excel at marketing products to clients. It may be beneficial for you to continue taking ...
Basic tenets of peer reviewing: There are 5 basic tenets that should be kept in mind: Decline the review if you have any conflicts of interest (COIs). Remember that you're advising the journal editor, not making the decision about whether to accept or reject. Try to be helpful and always respectful to the author.
Peer reviews on Zavvy: Questions and peer review example phrases. As part of a wider performance management system, peer reviews help an organization in the following ways:. 🎯 Can be used as a goal-setting opportunity.; 🔎 Peer feedback helps identify the strengths and weaknesses of individual employees, teams, and the company as a whole.; 🌱 Suggestions from peers can help employees ...
These articles go through a process known as "peer review" where the article is reviewed by a group of experts in the field and revised based on peer feedback before being accepted and published by a journal. ... These databases are examples of good subject-specific databases for researching the disciplines of Art, Education, and Psychology ...
Learning, Research & Engagement Librarian. Holly Surbaugh she/her Email Me. Fall 2024 Drop-In Hours Centennial Science & Engineering Library, L164 Mondays: 10 am - 12pm ... Does this publication venue conduct peer review? What is the publication process? Who edits the journal? What is the impact factor?
Have you heard about the initiative at the National Institutes of Health (NIH) to improve the peer review of research project grant and fellowship applications? Join us as NIH describes the steps the agency is taking to simplify its process of assessing the scientific and technical merit of applications, better identify promising scientists for ...
Introduction Digital mental health interventions (DMHIs) overcome traditional barriers enabling wider access to mental health support and allowing individuals to manage their treatment. How individuals engage with DMHIs impacts the intervention effect. This review determined whether the impact of user engagement was assessed in the intervention effect in Randomised Controlled Trials (RCTs ...
Results Of 7005 sources reviewed, 79 met the eligibility criteria, including 54 peer-reviewed publications. The majority were descriptive reports (28%) and qualitative analyses (30%) from early stages of the COVID-19 pandemic. Definitions of OPR varied while nine articles explicitly used the term 'readiness', others classified OPR as part of preparedness or response.
For example, the global under-5 mortality rate fell from 42 deaths per 1,000 live births in 2015 to 39 in 2018. ... Peer Review reports. Introduction. ... The MMR & infant mortality rate (IMR) in the research area were indistinguishable from that, at 150 per 100,000 live births and 67 fatalities per 1,000 live births, respectively [10,11,12,13].
Subjects. The research population was the medical students of the Islamic Azad University of Mashhad during the academic year 2021-2022. The entry criteria were: being a medical trainee student, consent to enter the study, and the exclusion criteria were: students who had previously graduated in radiology or other medical sciences and students who had renewed their course in radiology ...