research papers software quality

Systematic literature review on software quality for AI-based software

• Published: 17 March 2022
• Volume 27 , article number  66 , ( 2022 )

Cite this article

• Bahar Gezici   ORCID: orcid.org/0000-0001-6704-3134 1 &
• Ayça Kolukısa Tarhan 1  

3416 Accesses

13 Citations

1 Altmetric

Explore all metrics

There is a widespread demand for Artificial Intelligence (AI) software, specifically Machine Learning (ML). It is getting increasingly popular and being adopted in various applications we use daily. AI-based software quality is different from traditional software quality because it generally addresses distinct and more complex kinds of problems. With the fast advance of AI technologies and related techniques, how to build high-quality AI-based software becomes a very prominent subject. This paper aims at investigating the state of the art on software quality (SQ) for AI-based systems and identifying quality attributes, applied models, challenges, and practices that are reported in the literature. We carried out a systematic literature review (SLR) from 1988 to 2020 to (i) analyze and understand related primary studies and (ii) synthesize limitations and open challenges to drive future research. Our study provides a road map for researchers to understand quality challenges, attributes, and practices in the context of software quality for AI-based software better. From the empirical evidence that we have gathered by this SLR, we suggest future work on this topic be structured under three categories which are Definition/Specification, Design/Evaluation, and Process/Socio-technical.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Software Quality for AI: Where We Are Now?

Synergies Between Artificial Intelligence and Software Engineering: Evolution and Trends

AI-Oriented Software Engineering (AIOSE): Challenges, Opportunities, and New Directions

Explore related subjects.

• Artificial Intelligence

25000 I (2005) The iso/iec 25000 series of standards. https://iso25000.com/index.php/en/iso-25000-standards

25012:2008 I (2008) software engineering — software product quality requirements and evaluation (square) — data quality model. https://www.iso.org/standard/35736.html

26262-1:2018 I (2018) Road vehicles — functional safety. https://www.iso.org/standard/68383.html

29119-1:2013 I (2013) Software and systems engineering — software testing. https://www.iso.org/standard/45142.html

9126-1:2001 I (2001) Software engineering — product quality. https://www.iso.org/standard/22749.html

Aggarwal A, Lohia P, Nagar S, Dey K, Saha D (2019) Black box fairness testing of machine learning models. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 625–635

Alamin MAA, Uddin G (2021) Quality assurance challenges for machine learning software applications during software development life cycle phases. arXiv: 2105.01195

Ali Z Quality measurement challenges for artificial intelligence software

de Almeida Biolchini JC, Mian PG, Natali ACC, Conte T, Travassos GH (2007) Scientific research ontology to support systematic review in software engineering. Adv Eng Inform 21(2):133–151

Article   Google Scholar  

Arpteg A, Brinne B, Crnkovic-Friis L, Bosch J (2018) Software engineering challenges of deep learning. In: 2018 44Th euromicro conference on software engineering and advanced applications (SEAA). IEEE, pp 50–59

Borg M, Englund C, Wnuk K, Duran B, Levandowski C, Gao S, Tan Y, Kaijser H, Lönn H, Törnqvist J (2018) Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry. arXiv: 1812.05389

Bosch J, Olsson HH, Crnkovic I (2021) Engineering ai systems: a research agenda. In: Artificial intelligence paradigms for smart cyber-physical systems. IGI Global, pp 1–19

Bourque P, Dupuis R, Abran A, Moore JW, Tripp L (2004) Guide to the software engineering body of knowledge -

Braiek HB, Khomh F (2020) On testing machine learning programs. J Syst Softw 164:110542

Byrne C (2017) Development Workflows for Data Scientists. O’Reilly Media

Chen R, Bastani FB, Tsao TW (1995) On the reliability of ai planning software in real-time applications. IEEE Trans Knowl Data Eng 7(1):4–13

Cummaudo A, Vasa R, Grundy J, Abdelrazek M, Cain A (2019) Losing confidence in quality: Unspoken evolution of computer vision services. In: 2019 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 333–342

Deng L (2018) Artificial intelligence in the rising wave of deep learning: The historical path and future outlook [perspectives]. IEEE Signal Proc Mag 35(1):180–177

Forward A, Lethbridge TC (2008) A taxonomy of software types to facilitate search and evidence-based software engineering. In: Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds, pp 179–191

Garousi V, Felderer M, Mäntylä MV (2016) The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. In: Proceedings of the 20th international conference on evaluation and assessment in software engineering, pp 1–6

Geske F, Hofmann P, Lämmermann L, Schlatt V, Urbach N (2021) Gateways to artificial intelligence: Developing a taxonomy for ai service platforms. In: Twenty-ninth european conference on information systems (ECIS)

Gezici B, Tarhan AK (2019) Final pool. https://drive.google.com/file/d/1ve6BpJTrITsfo6auSoWKh48ajWbNb05n/view?usp=sharing

Hamada K, Ishikawa F, Masuda S, Matsuya M, Ujita Y (2020) Guidelines for quality assurance of machine learning-based artificial intelligence. In: SEKE2020: The 32nd international conference on software engineering & knowledge engineering, pp 335–341

Hannousse A (2021) Searching relevant papers for software engineering secondary studies: Semantic scholar coverage and identification role. IET Softw 15 (1):126–146

Henriksson J, Borg M, Englund C (2018) Automotive safety and machine learning: Initial results from a study on how to adapt the iso 26262 safety standard. In: 2018 IEEE/ACM 1St international workshop on software engineering for AI in autonomous systems (SEFAIAS). IEEE, pp 47–49

Hopgood AA (2005) The state of artificial intelligence. Adv Comput 65:1–75

Horkoff J (2019) Non-functional requirements for machine learning: Challenges and new directions. In: 2019 IEEE 27Th international requirements engineering conference (RE). IEEE, pp 386–391

Hyun Park S, Seon Shin W, Hyun Park Y, Lee Y (2017) Building a new culture for quality management in the era of the fourth industrial revolution. Total Qual Manag Bus Excell 28(9-10):934–945

Ishikawa F, Yoshioka N (2019) How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey. In: 2019 IEEE/ACM Joint 7th international workshop on conducting empirical studies in industry (CESI) and 6th international workshop on software engineering research and industrial practice (SER&IP). IEEE, pp 2–9

ISO/IEC (2011) Iso/iec 25010 (2011)-systems and software quality requirements and evaluation (square)-system and software quality models. International Standard ISO/IEC 25010 2(1):1–25

Google Scholar  

Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of technology evaluations. Empir Softw Eng 16(3):365–395

Kitchenham B (2004) Procedures for performing systematic reviews. keele, UK. Keele Univ 33(2004):1–26

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering -

Kuwajima H, Ishikawa F (2019) Adapting square for quality assessment of artificial intelligence systems. In: 2019 IEEE International symposium on software reliability engineering workshops (ISSREW). IEEE, pp 13–18

Kuwajima H, Yasuoka H, Nakae T (2018) Open problems in engineering and quality assurance of safety critical machine learning systems. arXiv: 1812.03057

Kuwajima H, Yasuoka H, Nakae T (2020) Engineering problems in machine learning systems. Mach Learn 109(5):1103–1126

Article   MathSciNet   MATH   Google Scholar  

Lakshen GA, Vraneš S., Janev V (2016) Big data and quality: A literature review. In: 2016 24Th telecommunications forum (TELFOR). IEEE, pp 1–4

Lenarduzzi V, Lomio F, Moreschini S, Taibi D, Tamburri DA (2021) Software quality for ai: Where we are now?. In: International conference on software quality. Springer, pp 43–53

Liu Y, Ma L, Zhao J (2019) Secure deep learning engineering: a road towards quality assurance of intelligent systems. In: International conference on formal engineering methods. Springer, pp 3–15

Lwakatare LE, Raj A, Crnkovic I, Bosch J, Olsson HH (2020) Large-scale machine learning systems in real-world industrial settings: a review of challenges and solutions. Inf Softw Technol 127:106368

Malik V, Singh S (2020) Artificial intelligent environments: risk management and quality assurance implementation. J Discret Math Sci Cryptogr 23 (1):187–195

Article   MATH   Google Scholar  

Mannarswamy S, Roy S, Chidambaram S (2020) Tutorial on software testing & quality assurance for machine learning applications from research bench to real world. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp 373–374

Martínez-Fernández S, Bogner J, Franch X, Oriol M, Siebert J, Trendowicz A, Vollmer AM, Wagner S (2021) Software engineering for ai-based systems: A survey. arXiv: 2105.01984

Masuda S, Ono K, Yasue T, Hosokawa N (2018) A survey of software quality for machine learning applications. In: 2018 IEEE International conference on software testing, verification and validation workshops (ICSTW). IEEE, pp 279–284

Murphy C, Kaiser GE, Arias M (2006) A framework for quality assurance of machine learning applications -

Nakajima S (2018) Quality assurance of machine learning software. In: 2018 IEEE 7Th global conference on consumer electronics (GCCE). IEEE, pp 601–604

Nakajima S (2019) Distortion and faults in machine learning software. In: International workshop on structured object-oriented formal language and method. Springer, pp 29–41

Nakamichi K, Ohashi K, Namba I, Yamamoto R, Aoyama M, Joeckel L, Siebert J, Heidrich J (2020) Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation. In: 2020 IEEE 28Th international requirements engineering conference (RE). IEEE, pp 260–270

Nascimento E, Nguyen-Duc A, Sundbø I, Conte T (2020) Software engineering for artificial intelligence and machine learning software: A systematic literature review. arXiv: 2011.03751

Nguyen-Duc A, Abrahamsson P (2020) Continuous experimentation on artificial intelligence software: a research agenda. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 1513–1516

Nishi Y, Masuda S, Ogawa H, Uetsuki K (2018) A test architecture for machine learning product. In: 2018 IEEE International conference on software testing, verification and validation workshops (ICSTW). IEEE, pp 273–278

Ongsulee P (2017) Artificial intelligence, machine learning and deep learning. In: 2017 15Th international conference on ICT and knowledge engineering (ICT&KE). IEEE, pp 1–6

Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Inf Softw Technol 64:1–18

Pons L, Ozkaya I (2019) Priority quality attributes for engineering ai-enabled systems. arXiv: 1911.02912

Poth A, Meyer B, Schlicht P, Riel A (2020) Quality assurance for machine learning–an approach to function and system safeguarding. In: 2020 IEEE 20Th international conference on software quality, reliability and security (QRS). IEEE, pp 22–29

Rahman MS, Reza H (2020) Systematic mapping study of non-functional requirements in big data system. In: 2020 IEEE International conference on electro information technology (EIT). IEEE, pp 025–031

Riccio V, Jahangirova G, Stocco A, Humbatova N, Weiss M, Tonella P (2020) Testing machine learning based systems: a systematic mapping. Empir Softw Eng 25(6):5193–5254

Rushby J (1988) Quality measures and assurance for AI software, vol 18. National Aeronautics and Space Administration, Scientific and Technical Information Division

Russel S, Norvig P (2009) Artificial intelligence: a modern approach, English

Samoili S, Cobo ML, Gomez E, De Prato G, Martinez-Plumed F, Delipetrev B (2020) Ai watch. defining artificial intelligence. towards an operational definition and taxonomy of artificial intelligence. In: JRC Technical reports. Joint research centre (seville site)

Siebert J, Joeckel L, Heidrich J, Nakamichi K, Ohashi K, Namba I, Yamamoto R, Aoyama M (2020) Towards guidelines for assessing qualities of machine learning systems. In: International conference on the quality of information and communications technology. Springer, pp 17–31

Taleb I, Serhani MA, Dssouli R (2018) Big data quality: a survey. In: 2018 IEEE International congress on big data (bigdata congress). IEEE, pp 166–173

Tao C, Gao J, Wang T (2019) Testing and quality validation for ai software–perspectives, issues, and practices. IEEE Access 7:120164–120175

Tao C, Hao C, Gao J, Wang T, Wen W (2017) A practical study on quality evaluation for age recognition systems. In: SEKE, pp 345–350

Tsintzira AA, Arvanitou EM, Ampatzoglou A, Chatzigeorgiou A (2020) Applying machine learning in technical debt management: Future opportunities and challenges. In: International conference on the quality of information and communications technology. Springer, pp 53–67

Turhan B, Kutlubay O (2007) Mining software data. In: 2007 IEEE 23Rd international conference on data engineering workshop. IEEE, pp 912–916

Vinayagasundaram B, Srivatsa S (2007) Software quality in artificial intelligence system. Inf Technol J 6(6):835–842

Vogelsang A, Borg M (2019) Requirements engineering for machine learning: Perspectives from data scientists. In: 2019 IEEE 27Th international requirements engineering conference workshops (REW). IEEE, pp 245–251

Wan Z, Xia X, Lo D, Murphy GC (2019) How does machine learning change software development practices? IEEE Transactions on Software Engineering

Wieringa RJ (2014) Design science methodology for information systems and software engineering. Springer

Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, pp 1–10

Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media

Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: survey, landscapes and horizons. IEEE Transactions on Software Engineering

Zhang P, Cao W, Muccini H (2020) Quality assurance technologies of big data applications: A systematic literature review. arXiv: 2002.01759

Download references

Author information

Authors and affiliations.

Institute of Science, Computer Engineering Department, Hacettepe University, Ankara, Turkey

Bahar Gezici & Ayça Kolukısa Tarhan

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Bahar Gezici .

Ethics declarations

Conflicts of interests/competing interests.

Please find attached the paper, “Systematic Literature Review on Software Quality for AI-based Software” by Bahar Gezici and Ayça Kolukısa Tarhan, which we would like to submit for possible publication to the Empirical Software Engineering. We confirm that this work is original and has not been published elsewhere nor is it currently under consideration for publication elsewhere.

For any information concerning this manuscript, please contact me preferably by e-mail at [email protected]. Thank you for your consideration of this manuscript.

Additional information

Communicated by: Paolo Tonella

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Gezici, B., Tarhan, A.K. Systematic literature review on software quality for AI-based software. Empir Software Eng 27 , 66 (2022). https://doi.org/10.1007/s10664-021-10105-2

Download citation

Accepted : 09 December 2021

Published : 17 March 2022

DOI : https://doi.org/10.1007/s10664-021-10105-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Machine learning
• Software quality
• Quality attributes
• Quality metrics
• Measurement
• Product quality model
• Find a journal
• Publish with us
• Track your research

Grab your spot at the free arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Software Engineering

Title: software quality: a historical and synthetic content analysis.

Abstract: Interconnected computers and software systems have become an indispensable part of people's lives, therefore software quality research is becoming more and more important. There have been multiple attempts to synthesize knowledge gained in software quality research, however, they were focused mainly on single aspects of software quality and not to structure the knowledge in a holistic way. The aim of our study was to close this gap. The software quality publications were harvested from the Scopus bibliographic database. The metadata was exported first to CRexlporer, which was employed to identify historical roots, and next to VOSViewer, which was used as a part of the synthetic content analysis. In our study we defined synthetic context analysis as a triangulation of bibliometrics and content analysis. Our search resulted in 14451 publications. The performance bibliometric study showed that the production of research publications relating to software quality is currently following an exponential growth trend and that the software quality research community is growing. The most productive country was the United States and the most productive Institution The Florida Atlantic University. The synthetic content analysis revealed that the published knowledge can be structured into 10 themes, the most important being the themes regarding software quality improvement with enhancing software engineering, advanced software testing, and improved defect and fault prediction with machine learning and data mining. According to the analysis of the hot topics, it seems that future research will be directed into developing and using a full specter of new artificial intelligence tools (not just machine learning and data mining) and focusing on how to assure software quality in agile development paradigms.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

DBLP - CS Bibliography

Bibtex formatted citation.

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Software Quality

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Last »
• Software Metrics Follow Following
• Software Testing Follow Following
• Software Quality Engineering Follow Following
• Software Maintenance Follow Following
• Software effort estimation Follow Following
• Information Systems Follow Following
• Software Quality Assurance Follow Following
• Information Technology Follow Following
• Software Measurement Follow Following
• Software Engineering Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Journals
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

This paper is in the following e-collection/theme issue:

Published on 14.8.2024 in Vol 26 (2024)

This is a member publication of University of Toronto

Leadership for AI Transformation in Health Care Organization: Scoping Review

Authors of this article:

• Abi Sriharan 1, 2 , MSc, DPhil   ; 
• Nigar Sekercioglu 2 , PhD   ; 
• Cheryl Mitchell 3 , PhD   ; 
• Senthujan Senkaiahliyan 2 , MHSc   ; 
• Attila Hertelendy 4 , PhD   ; 
• Tracy Porter 5 , PhD   ; 
• Jane Banaszak-Holl 6 , PhD  

1 Krembil Centre for Health Management and Leadership, Schulich School of Business, York University, Toronto, ON, Canada

2 Institute for Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

3 Gustavson School of Business, University of Victoria, Victoria, ON, Canada

4 College of Business, Florida International University, Florida, FL, United States

5 Department of Management, Cleveland State University, Cleveland, OH, United States

6 Department of Health Services Administration, School of Health Professions, University of Alabama Birmingham, Birmingham, OH, United States

Corresponding Author:

Abi Sriharan, MSc, DPhil

Krembil Centre for Health Management and Leadership

Schulich School of Business

York University

MB Room G315

4700 Keele St

Toronto, ON, M3J 1P3

Phone: 1 3658855898

Email: [email protected]

Background: The leaders of health care organizations are grappling with rising expenses and surging demands for health services. In response, they are increasingly embracing artificial intelligence (AI) technologies to improve patient care delivery, alleviate operational burdens, and efficiently improve health care safety and quality.

Objective: In this paper, we map the current literature and synthesize insights on the role of leadership in driving AI transformation within health care organizations.

Methods: We conducted a comprehensive search across several databases, including MEDLINE (via Ovid), PsycINFO (via Ovid), CINAHL (via EBSCO), Business Source Premier (via EBSCO), and Canadian Business & Current Affairs (via ProQuest), spanning articles published from 2015 to June 2023 discussing AI transformation within the health care sector. Specifically, we focused on empirical studies with a particular emphasis on leadership. We used an inductive, thematic analysis approach to qualitatively map the evidence. The findings were reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews) guidelines.

Results: A comprehensive review of 2813 unique abstracts led to the retrieval of 97 full-text articles, with 22 included for detailed assessment. Our literature mapping reveals that successful AI integration within healthcare organizations requires leadership engagement across technological, strategic, operational, and organizational domains. Leaders must demonstrate a blend of technical expertise, adaptive strategies, and strong interpersonal skills to navigate the dynamic healthcare landscape shaped by complex regulatory, technological, and organizational factors.

Conclusions: In conclusion, leading AI transformation in healthcare requires a multidimensional approach, with leadership across technological, strategic, operational, and organizational domains. Organizations should implement a comprehensive leadership development strategy, including targeted training and cross-functional collaboration, to equip leaders with the skills needed for AI integration. Additionally, when upskilling or recruiting AI talent, priority should be given to individuals with a strong mix of technical expertise, adaptive capacity, and interpersonal acumen, enabling them to navigate the unique complexities of the healthcare environment.

Introduction

Artificial intelligence in health care: overview.

Artificial intelligence (AI) technologies have gained significant momentum in health care, presenting a transformative potential across clinical processes, operational efficiency, decision-making, and workforce optimization [ 1 - 3 ]. The global AI market is projected to shift from US $14.6 billion in 2023 to a formidable estimate of US $102.7 billion by 2028 [ 4 ], unveiling a dynamic transformation of unprecedented scale. This investment, coupled with the engagement of nontraditional health care players such as Microsoft, Google, and Amazon and the convergence of technological prowess and health care innovation signaled by generative AI, will place the trajectory of AI in health care in a state of exponential growth [ 5 ].

Current investments in health care AI predominantly center on bolstering data capacity, enhancing computational power, and advancing methodological innovations in AI. This includes developing and testing AI models and algorithms tailored for precision medicine, drug discovery, clinical decision-making support, public health surveillance, operational optimization, and process improvement [ 6 , 7 ]. Notably, between August 2022 and July 2023, there were over 150 submissions of drug and biological applications incorporating AI and machine learning components to the US Food and Drug Administration, encompassing a wide array of therapeutic domains and developmental stages [ 8 ].

Yet the seamless integration of AI technologies into health care organizational settings presents a multifaceted challenge for health care leaders. This challenge arises from several factors, including the complex nature of AI models, the rapid pace of technological advancement, the imperative of regulatory adherence, ethical concerns surrounding data security and privacy, the risk of perpetuating racial and ethnic biases in data, the necessity of prioritizing human-centric approaches to patient care, and the intricate clinical workflows that must be navigated [ 9 - 15 ]. Furthermore, health care leaders are facing critical and intricate strategic decisions. They must discern which AI solutions merit investment while weighing the merits of in-house development against strategic partnerships with external vendors. Selecting the right vendors and defining the scope of collaboration is pivotal, as is devising a sustainable funding strategy to support both initial development and continuous innovation. Furthermore, they must confront the crucial question of whether to bring in new AI talent or bolster the expertise of their current workforce through upskilling. Each of these decisions will shape the trajectory of health care organizations as they navigate this transformative era. A report by Bain in 2023 revealed that although 75% of surveyed health system executives recognize AI’s potential to reshape the health care industry, only 6% have established concrete strategies related to AI [ 16 ].

The lack of strategy and strategic failures in AI integration not only have financial consequences for organizations but also erode trust among patients, providers, and organizations [ 17 ]. A prominent example is the collaboration between MD Anderson and IBM Watson, aimed at leveraging IBM Watson’s cognitive capabilities to combat cancer. This ambitious endeavor, however, incurred a substantial financial toll of over US $62 million for MD Anderson because of setbacks in clinical implementation [ 18 ].

Despite a growing body of AI literature, including toolkits such as Canada Health Infoway’s “Toolkit for AI Implementers” [ 19 ] and guidance from the US Department of Health and Human Services’ AI Task Force [ 20 ] and the UK National Strategy for AI in Health and Social Care [ 21 ], there is still insufficient scholarly attention on how leadership behavior guides AI transformation in health care. Existing reviews focus on AI in medical education [ 22 , 23 ], workforce impact [ 24 ], applications in clinical medicine [ 13 , 25 ], barriers to implementation [ 26 , 27 ], and ethical considerations [ 28 , 29 ]. However, no systematic mapping of empirical literature has clarified our understanding of leadership or identified gaps in research. Understanding leadership behavior is crucial for health care organizations considering AI because effective leadership shapes the strategic direction, adoption, and successful implementation of AI technologies.

Research Aim

To address this research gap and to establish a future research agenda this scoping review study aims to address two primary questions: (1) What role does leadership play in AI transformation within health care? and (2) What approaches can health care organizations use to empower their leaders in facilitating AI transformation?

Research Approach

This review follows scoping review methodology [ 30 ] to identify and analyze the current literature and report results following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Scoping Reviews; Multimedia Appendix 1 ) guidelines [ 31 ].

Key Definitions

In the context of this study, AI refers to combination of machine learning algorithms, large language models, robotics, and natural language processing systems designed to mimic human cognitive functions, enabling machines to perform tasks autonomously or with minimal human intervention.

AI transformation refers to the systematic changes in clinical, operational, or organizational processes and business models due to the introduction of AI systems to optimize decision-making, automate tasks, improve patient outcomes, and drive organizational change. This involves identifying opportunities for AI-related innovation, integrating them into processes, and developing strategies to operationalize implementation while ensuring organizational readiness. This is essential for getting health care organizations AI-ready.

Further, in the context of this study, drawing from seminal management and leadership theories, we view leadership as an effective management practice [ 32 ]. However, we recognize that leadership roles in health care occur at the clinical, organizational, and systems levels of health systems. At the clinical level, leadership emerges through health care professionals who steer patient care and treatment decisions. At the organizational level, leadership involves middle managers such as unit heads and division leaders guiding health care institutions, administrative units, and personnel toward their goals. At the systems level, leadership encapsulates C-suite leadership responsible for navigating regulatory complexities and organizational and structural silos within complex health systems.

Eligibility Criteria

The following inclusion and exclusion criteria guided our study: (1) focused on AI in health care, (2) contained an evaluation of leadership, (3) were written in English, (4) were published in a peer-reviewed journal, (5) published between January 2015 and June 2023, and (6) used research.

Information Sources and Search Strategy

We adopted comprehensive search strategies for the following electronic databases focused on the health care and business literature: MEDLINE (via Ovid), PsycINFO (via Ovid), CINAHL (via EBSCO), Business Source Premier (via EBSCO), and Canadian Business & Current Affairs (via ProQuest). An academic librarian developed these search strategies with input from the research team. We initially conducted the search in Ovid MEDLINE. We then reviewed our search results using the Peer Review of Electronic Search Strategies tool [ 33 ], a checklist for comparing, among other things, the types of errors found in articles and the relative fit of articles to the research question before translating the search strategy into other databases using their command language. Our search was limited to articles published from January 2015 (from the first use of AI-powered chatbots in health care [ 34 ] to June 2023. We then ran searches in 4 databases and exported the final search results into the EndNote reference management software (Clarivate), and we removed duplicate articles manually. To capture any papers that may have been missed during the search process, we did forward and reverse citation searches of systematic review articles related to AI [ 35 ]. However, we did not find any additional articles that met our criteria. Finally, we imported search results to Covidence (Veritas Health Innovation), a review management software for abstract and title screening, full-text screening, and data charting.

Selection of Sources of Evidence and Data Charting

To minimize selection bias, 2 independent screeners reviewed the titles and abstracts of articles identified via the search against the eligibility criteria using Covidence. We identified articles that met the eligibility criteria for a comprehensive full-text screening. Two independent reviewers then evaluated the full texts against the eligibility criteria using Covidence. In discrepancies between the reviewers, a third reviewer served as the consensus reviewer and used Covidence to resolve conflicts between reviewer 1 and reviewer 2. Following the exclusion of irrelevant articles, we used a predefined data extraction form aligned with our research objectives and guiding questions for systematic data collection. Data extraction categories included data on study characteristics (eg, citations and country); methods (eg, aim, data collection methods, and methodological quality); study context (eg, leadership role, ie, clinical, organizational, or systems); leadership practices (ie, behavior, enablers, and barriers to leadership success); results (ie, main results and author conclusion); and an open-ended reviewer note (ie, capture any relevant information that might aid in the data analysis stage). The data abstraction form was piloted on a random sample of 4 included articles and modified based on feedback from the team. Full data abstraction began only after sufficient agreement had been obtained. Two reviewers independently extracted the data using Covidence, and a third reviewer assessed the data extraction for quality and consensus. Three authors then held a group discussion to resolve any conflicts.

Risk of Bias Assessment

The focus of scoping reviews is to provide a comprehensive overview of the available literature, identifying the extent, range, and nature of research on a particular topic rather than assessing the methodological quality of individual studies [ 35 ]. Therefore, we did not perform risk of bias evaluations on the articles included in compliance with the guidelines for scoping reviews.

Data Analysis and Synthesis

Our data analysis was guided by a thematic analysis process [ 36 ]. To ensure the accuracy of the emerging themes, we conducted our analysis collaboratively in reviewer pairs [ 35 ].

We initially analyzed the extracted data using an open-coding method guided by our research questions. Subsequently, we grouped the codes into categories based on the emerging patterns in the data, which we then synthesized into leadership functional domains, capacities, and context.

In the context of our analysis, functional domains refer to distinct areas of responsibility that a leader must effectively manage a task or a role. Capacity, on the other hand, pertains to the abilities—skills, competencies, or behaviors—that a leader must demonstrate to achieve desired goals. Context refers to the environment, conditions, and situational factors that shape and influence leadership practices and decisions.

Study Selection

As described in Figure 1 , the original searches generated 3541 articles published from January 2015 to June 2023. After removing 728 duplicate articles in EndNote, 2813 unique articles were uploaded to Covidence. A total of 2813 relevant studies were then screened using Covidence using the articles’ titles and abstracts. We determined that 97 articles met the criteria for a full-text review for eligibility screening. Within these 97 articles, 75 were excluded as they were opinion articles or commentaries without objective data. After conducting the full-text screening, we found that 22 articles met the final inclusion criteria.

Study Characteristics

Of the 22 studies identified for final inclusion in our review, 12 involved qualitative methods [ 37 - 48 ] such as interviews and case studies, whereas 4 studies involved mixed methods research [ 49 - 52 ] with a qualitative and quantitative strand. There were 3 narrative reports [ 53 - 55 ] based on document synthesis, and 3 studies involved quantitative methods [ 56 - 58 ] such as surveys. These articles focused on clinical, organizational, and systems leadership and came from Canada, China, Finland, Saudi Arabia, Sweden, the Netherlands, the United Kingdom, and the United States. The included papers addressed a broad array of AI applications in health care, including studies focused on improving workflows, quality of care, patient safety, resource optimization, and patient experience. From a clinical domain, researchers focused on primary care, health care systems, radiology, or global health. From a population perspective, the papers covered leadership from the perspective of primary care physicians, radiologists, nurses, nurse managers, public health professionals, global health professionals, health care entrepreneurs, and health care leaders. Table 1 provides a summary of study characteristics.

a N/A: not applicable.

Leadership Tasks Essential for AI Transformation in Health Care

We mapped the themes from the included studies across 4 functional domains of leadership task responsibility—technological (AI innovation), strategic (vision and alignment), operational (process and oversight), and organizational (culture and work environment).

The technological functional domain garnered the most significant attention in the literature. The core themes that emerged under the technological domain primarily focused on applying subject matter expertise and AI technical skills to effectively identify AI opportunities, as well as to foster an innovation mindset to develop, tailor, and seamlessly implement AI-driven solutions to address key AI opportunities within health care organizations.

Within the strategic functional domain, the literature underscored the importance of change management and communication as strategic tools for consensus and collaboration related to the AI transformation process. Another core theme that emerged focused on the critical importance of integrating AI solutions into the existing clinical care processes. This strategic alignment is essential for getting support from the staff and ensuring smooth operations of patient care outcomes while embracing the potential of AI solutions. Although the significance of talent strategy related to the recruitment and retention of AI technical expertise within organizations was mentioned, it was not widely seen across the included papers.

Table 2 provides a summary of how the technological and strategic functional domains map across the papers and provides key themes that emerged with the domain area.

Emerging evidence in the operational functional domain highlights leaders’ need to navigate ethical and risk management issues by establishing robust governance structures prioritizing patient data privacy and security while ethically integrating AI technologies within existing workflows. Additionally, the literature emphasizes that implementing AI in health care will require leaders to ensure new AI solutions comply with existing regulatory and control systems. The literature highlighted that leaders need to pay attention to process agility through continuous monitoring to ensure AI solutions can adapt to contextual changes.

Finally, the organizational functional domain emerges from the thematic analysis as a pivotal area for AI leadership. The literature emphasizes the importance of stakeholder engagement in building collaboration. Furthermore, it underscores the importance of decision makers’ sense-making to enhance their trust in AI opportunities and ensure that AI integration is supported by individuals across the organization. Further, the literature underscored the importance of organizational culture readiness to support physicians and nurses through protected time and incentive pay to engage, innovate, and adopt AI solutions. Table 3 provides a summary of how operational and organizational functional domains map across the papers.

Leadership Skills and Behaviors for Preparing Health Care Organizations for AI Transformation

We categorized the themes related to skills and behaviors into 3 essential capacities that a leader must demonstrate to achieve desired goals—technical capacity, adaptive capacity, and interpersonal capacity. Technical capacity encompasses (1) AI literacy, (2) subject matter knowledge, (3) change leadership skills, and (4) innovation mindset to identify AI innovation opportunities. The interpersonal capacity involves several vital facets such as (1) the ability to foster partnerships among diverse stakeholders, (2) the ability to comprehend diverse stakeholder perspectives and deftly influence adoption, (3) the ability to build trust and collaboration, (4) self-awareness and humility to assemble teams with complementary skills, and (5) the integrity and accountability to embody ethical principles. The adaptive capacity encompasses (1) the foresight and sense-making abilities to discern emerging technologies and their implications within the health care sphere; (2) the agility to identify and capitalize on transformative opportunities, swiftly adapting and aligning strategies with evolving contexts; and (3) systems thinking to enable an understanding of how elements interconnect and how changes in 1 area can reverberate throughout the entire system.

Contextual Factors Influencing Leadership in AI Transformation

The emerging themes from our review reveal that dynamic environmental and situational factors, including regulatory, technology, and organizational contexts, shape AI transformation within health care organizations. For instance, the regulatory context and frameworks related to health professions and health care organizations play a critical role in how AI can be integrated within the organizations. Similarly, the technology context such as the availability of AI technical talent, the retention of technical expertise, the dynamic nature of AI maturity, and the presence of incentives and technological resources for AI innovation or adoption will significantly influence a leader’s ability to effectively drive AI readiness. Finally, the organization context is a critical influence on leaders’ capacity for AI adoption and implementation. Organizations that promote and reward innovation and that have transparent communication practices shape leaders’ ability to pursue AI opportunities.

Strategies for Empowering Health Care Leaders to Facilitate AI Transformation

For the technological domain, the included papers discussed approaches such as upskilling clinical experts with the necessary AI technical skills and ensuring the presence of specialized experts, such as computer scientists, to enable the subject matter experts to develop, test, and seamlessly integrate AI solutions. Further, the papers discussed collaborative strategies such as clinicians and computer scientists working together to effectively identify AI opportunities and develop, adopt, and implement AI solutions in clinical or operational areas.

For the strategic domain, organizational support was essential in supporting leaders to assess and identify AI opportunities that strategically align with organizational priorities and develop strategies to ensure AI transformation garners support from key stakeholders within the complex regulatory and environmental contexts. The literature also highlighted the competition for AI talent in health care and emphasized the significance of talent retention strategies to preserve the organization’s AI technical expertise.

Then, in the operational domain, the emphasis was on establishing governance structures to continuously monitor data quality, patient privacy, and patient care experiences and assess the feasibility and financial implications of AI transformation. These governance structures ensure effective oversight and management of AI initiatives within health care organizations.

Finally, for the organizational domain, the focus was on the pivotal role of organizational culture in AI leadership. Leaders require organizational support to cultivate an environment that fosters innovation and actively incentivizes clinical leaders, such as physicians and nurses, through protected time and incentive pay to innovate and adopt AI solutions. Transparent decision-making processes related to AI solutions are essential cultural elements that build trust in AI systems and promote collaboration among the diverse stakeholders involved in AI transformation within health care organizations.

Principal Findings

The purpose of a scoping review is not to draw definitive conclusions but to map the literature, identify emerging patterns, and develop critical propositions. As described in Figure 2 , analysis of current literature shows that leading organizations toward AI transformation requires multidimensional leadership. As such, health care organizations need to engage leaders in the technological, strategic, operational, and organizational domains to facilitate AI transformation in their organizations. Further, the reviewed papers suggest that individuals in AI-related leadership roles need to demonstrate (1) technical capacity to understand the technology and innovation opportunities, (2) adaptive capacity to respond to contextual changes, and (3) interpersonal capacity to navigate the human aspects of the AI transformation process effectively. Furthermore, our study illuminates that leaders in the AI-related leadership roles need to navigate regulatory context, the dynamic nature of changing technology context, and organization context.

Prior Research

Health care organizations are marked by multifaceted interdependencies among medical facilities, health care providers, patients, administrative units, technology, and the regulatory environment. Therefore, the leadership required for AI transformation—which includes identifying AI opportunities, implementing AI solutions, and achieving full-scale AI adaptation—is not a static role but a continuous and dynamic process. Effective leadership involves the capacity to continuously identify opportunities for AI transformation, influence the thoughts and actions of others, and navigate the complex dynamics of the health care setting and AI technology landscape simultaneously. However, the current literature has not fully articulated this multidimensionality, often focusing on leadership through a linear approach.

Further, multiple situational factors can shape AI transformation. First, the rapid growth of AI technologies introduces an element of uncertainty, making it challenging to anticipate the long-term impact and sustainability of specific AI solutions [ 6 ]. Second, AI implementation involves many stakeholders, from technical experts and domain specialists to clinicians, administrators, patients, vendors, and regulatory bodies. Each stakeholder group brings its unique perspectives, priorities, and control systems into the equation, necessitating leaders to navigate competing values, trade-offs, and paradoxes [ 27 ]. Third, once alignment is achieved, the integration of AI within an organization triggers a need for a cultural shift, altering work practices and decision-making processes [ 38 , 59 ]. Fourth, the effectiveness of AI solutions hinges on the availability of high-quality data for informed insights and decision-making. When implementing solutions originally developed within different contexts, local organizations must ensure data integrity and the solution’s adaptability to the organization’s unique context [ 18 ]. This challenge is compounded by emerging regulatory frameworks, which add a layer of complexity. Ensuring compliance and the responsible use of AI technologies has become a critical consideration [ 29 , 50 , 60 ]. Finally, introducing AI may provoke resistance from employees concerned about job displacement or disruptions to established workflows. This problem is further compounded when an organization transitions toward integrating multiple AI systems, as these changes can lead to periods of chaos and confusion [ 59 ].

Emerging key opinions and evidence from outside the health domain indicate that leaders must possess an understanding of data quality nuances, assess process risks, and manage AI as a new team member. Additionally, leaders should have a firm grasp of technology, articulate clear business objectives, define precise goals, uphold a long-term vision, prepare their teams for AI transformation, manage data resources effectively, and foster organizational collaboration [ 3 , 61 - 67 ].

Our findings on the leadership required for AI transformation in health care organizations reinforce this multidimensionality of leadership to effectively navigate the complexities of AI transformation and successfully leverage its potential to drive transformative change. Leaders must operate across different functional domains—technological, strategic, operational, and organizational—while demonstrating technical, adaptive, and interpersonal capacities.

Further, our findings show contingency leadership theories, complexity theory, and transformational leadership theory as relevant theoretical domains for further explaining the different facets of leadership behaviors needed to navigate the multidimensionality of leadership required for AI transformation.

Contingency theories suggest that leadership effectiveness depends on situational factors, which should be considered in future AI implementation studies in the context of AI adaptation and integration within health care organizations [ 68 , 69 ]. Complexity theory provides a framework for examining leadership behaviors in interconnected, dynamic environments where leaders must balance innovation and stability and demonstrate an adaptive approach to challenges, characterized by uncertainty and change [ 70 - 73 ]. Transformational leadership theory emphasizes motivating, empowering, and developing others by fostering trust and collaboration while challenging the status quo to drive organizational change and innovation [ 74 , 75 ]. These theories should be considered in future AI implementation studies within health care organizations.

Future research and training programs related to AI in health care should examine the leadership required for AI transformation through the lens of multidimensionality, providing insights into the interrelatedness of functional domains, leadership capacities, and contextual enablers and barriers, while exploring the key theoretical domains related to contingency, complexity, and transformational leadership to further understand the interpersonal dynamics shaping AI transformation in health care.

Limitations

Some limitations to our scoping review are worth noting. First, given the contextual variability in the included studies and the methodological variations, we could not establish firm correlations about specific leadership domains, capacities, and contextual factors; the effectiveness of leadership approaches; or the moderating effects of contextual factors. Consequently, we have presented only the overarching emergent themes.

Second, our study is limited by the significant variation in conceptual definitions of leadership and leadership competencies found in the current literature, which often lacks more standardized definitions or instruments for measurement. This variation caused conceptual inconsistencies. We addressed the inconsistencies by clearly defining what constitutes a functional domain, capacity, and context before our data analysis to address this. We iteratively coded the data into themes to ensure all relevant aspects were captured.

Third, our search strategy focused on MEDLINE-indexed journals, which may exclude some newer journals indexed in PubMed but not yet in MEDLINE. While this might limit the capture of the very latest advancements in digital health, it does not diminish the robustness of the review. Fourth, we retrieved only articles written in English, which possibly limited the comprehensiveness of our findings. Fifth, we looked at AI as a system and did not look at the relationship between the implementation of different types of AI tools and leadership behaviors which was beyond the scope of our review. Finally, our analysis used an inductive approach and was not informed by a predetermined theory to aid the mapping of the literature. This may have limited our analysis in capturing different elements of an umbrella theory.

Recommendations for Future Design and Research

Leading organizations toward AI transformation is an adaptive challenge influenced by a myriad of interwoven situational factors that create a dynamic and intricate environment. The body of literature related to AI in health care is rapidly expanding, and the recommendations imparted by this review, alongside the multidimensional leadership framework ( Figure 2 ), stand poised to guide research and practice to empower health care organizations in their AI transformation journey. Future research on AI transformation, which includes innovation identification, implementation, and scaling, can use this framework to understand the role of leadership in driving successful outcomes.

Further, future research must undergo methodological expansion by embracing qualitative and mixed methods approaches to illuminate the intricate temporal aspects of AI transformation and corresponding evolving leadership behaviors.

Conclusions

In summary, emerging evidence shows that multidimensional leadership plays crucial role in AI transformation in health care organization. Leaders must adeptly balance technology opportunities while demonstrating unwavering empathy for stakeholder needs and nimble adaptability to accommodating the ever-changing contextual landscape, which encompasses the regulatory frameworks, the evolution of technology, and the organization’s priorities.

Acknowledgments

This work is supported through a grant from the University of Toronto’s Connaught Global Challenges. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the study.

Data Availability

The main study data are the data extraction materials and quality ratings of included papers, most of which are included in the study tables. The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions

All authors were involved in conception and design of the study and approved the protocol. AS and NS were responsible for overseeing the search of databases and literature. AS, NS, and SS were involved in the screening of articles, data extraction and data verification, and analysis of data. All authors were involved in data interpretation, supported in the drafting of the paper, which was led by AS, and all authors supported in revising and formatting of the paper. All authors have provided final approval of the version of the paper submitted for publication, and all authors agree to be accountable for all aspects of the work.

Conflicts of Interest

None declared.

Updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) checklist.

• Fyfe I. AI predicts pancreatic cancer risk. Nat Rev Gastroenterol Hepatol. 2023;20(7):413. [ CrossRef ] [ Medline ]
• Haug CJ, Drazen JM. Artificial intelligence and machine learning in clinical medicine, 2023. N Engl J Med. 2023;388(13):1201-1208. [ CrossRef ] [ Medline ]
• Sharma M, Savage C, Nair M, Larsson I, Svedberg P, Nygren JM. Artificial intelligence applications in health care practice: scoping review. J Med Internet Res. 2022;24(10):e40238. [ FREE Full text ] [ CrossRef ] [ Medline ]
• AI in healthcare market | size, share, growth | 2022-2027. MarketWatch. 2023. URL: https://www.marketdataforecast.com/market-reports/artificial-intelligence-in-healthcare-market [accessed 2024-03-08]
• Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6(1):120. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. In: Bohr A, Memarzadeh K, editors. Artificial Intelligence in Healthcare. Cambridge. Academic Press; 2020:25-60.
• Kasula BY. AI applications in healthcare: a comprehensive review of advancements and challenges. Int J Manag Educ Sustain Dev. 2023;6(6). [ FREE Full text ]
• Artificial intelligence and machine learning (AI/ML)-enabled medical devices. U.S. Food & Drug Administration. 2023. URL: https:/​/www.​fda.gov/​medical-devices/​software-medical-device-samd/​artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices [accessed 2024-03-08]
• Minssen T, Vayena E, Cohen IG. The challenges for regulating medical use of ChatGPT and other large language models. JAMA. 2023;330(4):315-316. [ CrossRef ] [ Medline ]
• Capel T, Brereton M. What is human-centered about human-centered AI? a map of the research landscape. 2023. Presented at: CHI '23: CHI Conference on Human Factors in Computing Systems; April 23-28, 2023:1-23; Hamburg, Germany. [ CrossRef ]
• Matheny ME, Whicher D, Thadaney Israni S. Artificial intelligence in health care: a report from the National Academy of Medicine. JAMA. 2020;323(6):509-510. [ CrossRef ] [ Medline ]
• Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28(1):31-38. [ CrossRef ] [ Medline ]
• Shaheen MY. Applications of Artificial Intelligence (AI) in Healthcare: A Review. Saudia Arabia. ScienceOpen; 2021.
• Tipton K, Leas BF, Flores E, Jepson C, Aysola J, Cohen J, et al. Impact of healthcare algorithms on racial and ethnic disparities in health and healthcare. AHRQ. 2023. [ CrossRef ]
• Chin MH, Afsar-Manesh N, Bierman AS, Chang C, Colón-Rodríguez CJ, Dullabh P, et al. Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. JAMA Netw Open. 2023;6(12):e2345050. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Berger E, Dries M. Beyond hype: getting the most out of generative AI in healthcare today. Bain & Company. 2023. URL: https://www.bain.com/insights/getting-the-most-out-of-generative-ai-in-healthcare/ [accessed 2024-03-08]
• Khanna NN, Maindarkar MA, Viswanathan V, Fernandes JFE, Paul S, Bhagawati M, et al. Economics of artificial intelligence in healthcare: diagnosis vs. treatment. Healthcare (Basel). 2022;10(12):2493. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schmidt C. M.D. Anderson breaks with IBM Watson, raising questions about artificial intelligence in oncology. J Natl Cancer Inst. 2017;109(5). [ CrossRef ] [ Medline ]
• Toolkit for AI implementers. Canada Health Infoway. URL: https:/​/www.​infoway-inforoute.ca/​en/​digital-health-initiatives/​innovative-technologies/​artificial-intelligence/​toolkit-for-ai-implementers [accessed 2024-03-08]
• Liu Q, Huang R, Hsieh J, Zhu H, Tiwari M, Liu G, et al. Landscape analysis of the application of artificial intelligence and machine learning in regulatory submissions for drug development from 2016 to 2021. Clin Pharmacol Ther. 2023;113(4):771-774. [ CrossRef ] [ Medline ]
• The national strategy for AI in health and social care. NHS Transformation Directorate. URL: https:/​/transform.​england.nhs.uk/​ai-lab/​ai-lab-programmes/​the-national-strategy-for-ai-in-health-and-social-care/​ [accessed 2024-02-24]
• Chan KS, Zary N. Applications and challenges of implementing artificial intelligence in medical education: integrative review. JMIR Med Educ. 2019;5(1):e13930. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Han ER, Yeo S, Kim MJ, Lee YH, Park KH, Roh H. Medical education trends for future physicians in the era of advanced technology and artificial intelligence: an integrative review. BMC Med Educ. 2019;19(1):460. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chowdhury S, Dey P, Joel-Edgar S, Bhattacharya S, Rodriguez-Espindola O, Abadie A, et al. Unlocking the value of artificial intelligence in human resource management through AI capability framework. Hum Resour Manage Rev. 2023;33(1):100899. [ CrossRef ]
• Shah R, Chircu A. IOT and AI in healthcare: a systematic literature review. Issues Inf Syst. 2018;10:48009. [ CrossRef ]
• Chomutare T, Tejedor M, Svenning TO, Marco-Ruiz L, Tayefi M, Lind K, et al. Artificial intelligence implementation in healthcare: a theory-based scoping review of barriers and facilitators. Int J Environ Res Public Health. 2022;19(23):16359. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gama F, Tyskbo D, Nygren J, Barlow J, Reed J, Svedberg P. Implementation frameworks for artificial intelligence translation into health care practice: scoping review. J Med Internet Res. 2022;24(1):e32215. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morley J, Machado CC, Burr C, Cowls J, Joshi I, Taddeo M, et al. The ethics of AI in health care: a mapping review. Soc Sci Med. 2020;260:113172. [ CrossRef ] [ Medline ]
• Reddy S, Allan S, Coghlan S, Cooper P. A governance model for the application of AI in health care. J Am Med Inform Assoc. 2020;27(3):491-497. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19-32. [ CrossRef ]
• Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mintzberg H. Managing. California. Berrett-Koehler; 2009.
• McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc. 2020;92(4):807-812. [ CrossRef ] [ Medline ]
• Tricco CA, Langlois E, Straus ES. Rapid reviews to strengthen health policy and systems. WHO Organization. 2017. URL: https://apps.who.int/iris/bitstream/handle/10665/258698/9789241512763-eng.pdf [accessed 2024-03-08]
• Boyatzis RE. Transforming Qualitative Information: Thematic Analysis and Code Development. USA. Sage Publications; 1998.
• Barbour AB, Frush JM, Gatta LA, McManigle WC, Keah NM, Bejarano-Pineda L, et al. Artificial intelligence in health care: insights from an educational forum. J Med Educ Curric Dev. 2019;6:2382120519889348. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Darcel K, Upshaw T, Craig-Neil A, Macklin J, Steele Gray C, Chan TCY, et al. Implementing artificial intelligence in Canadian primary care: barriers and strategies identified through a national deliberative dialogue. PLoS One. 2023;18(2):e0281733. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dicuonzo G, Donofrio F, Fusco A, Shini M. Healthcare system: moving forward with artificial intelligence. Technovation. 2023;120:102510. [ CrossRef ]
• Ganapathi S, Duggal S. Exploring the experiences and views of doctors working with artificial intelligence in English healthcare; a qualitative study. PLoS One. 2023;18(3):e0282415. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gillan C. Healthcare Professional Association Agency in Preparing for Complex Technological Interventions: A Multiple-Case Study of Artificial Intelligence in Radiation Medicine and Medical Imaging (Dissertation). Canada. University of Toronto; 2022.
• Henriksen A, Bechmann A. Building truths in AI: making predictive algorithms doable in healthcare. Inf Commun Soc. 2020;23(6):802-816. [ CrossRef ]
• Laukka E, Hammarén M, Kanste O. Nurse leaders' and digital service developers' perceptions of the future role of artificial intelligence in specialized medical care: an interview study. J Nurs Manag. 2022;30(8):3838-3846. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Olaye IM, Seixas AA. The gap between AI and bedside: participatory workshop on the barriers to the integration, translation, and adoption of digital health care and AI startup technology into clinical practice. J Med Internet Res. 2023;25:e32962. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Petersson L, Larsson I, Nygren JM, Nilsen P, Neher M, Reed JE, et al. Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden. BMC Health Serv Res. 2022;22(1):850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ronquillo CE, Peltonen LM, Pruinelli L, Chu CH, Bakken S, Beduschi A, et al. Artificial intelligence in nursing: priorities and opportunities from an international invitational think-tank of the nursing and artificial intelligence leadership collaborative. J Adv Nurs. 2021;77(9):3707-3717. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Strohm L, Hehakaya C, Ranschaert ER, Boon WPC, Moors EHM. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur Radiol. 2020;30(10):5525-5532. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Upshaw TL, Craig-Neil A, Macklin J, Gray CS, Chan TCY, Gibson J, et al. Priorities for artificial intelligence applications in primary care: a Canadian deliberative dialogue with patients, providers, and health system leaders. J Am Board Fam Med. 2023;36(2):210-220. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hakim Z, Ierasts T, Hakim I, D'Penha A, Smith KPD, Caesar MCW. Connecting data to insight: a pan-Canadian study on AI in healthcare. Healthc Q. 2020;23(1):13-19. [ CrossRef ] [ Medline ]
• Morley J, Murphy L, Mishra A, Joshi I, Karpathakis K. Governing data and artificial intelligence for health care: developing an international understanding. JMIR Form Res. 2022;6(1):e31623. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Willis M, Duckworth P, Coulter A, Meyer ET, Osborne M. Qualitative and quantitative approach to assess of the potential for automating administrative tasks in general practice. BMJ Open. 2020;10(6):e032412. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang J, Luo B, Zhao C, Zhang H. Artificial intelligence healthcare service resources adoption by medical institutions based on TOE framework. Digit Health. 2022;8:20552076221126034. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dixit A, Quaglietta J, Gaulton C. Preparing for the future: how organizations can prepare boards, leaders, and risk managers for artificial intelligence. Healthc Manage Forum. 2021;34(6):346-352. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Galsgaard A, Doorschodt T, Holten AL, Müller FC, Ploug Boesen M, Maas M. Artificial intelligence and multidisciplinary team meetings; a communication challenge for radiologists' sense of agency and position as spider in a web? Eur J Radiol. 2022;155:110231. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sawers N, Bolster N, Bastawrous A. The contribution of artificial intelligence in achieving the sustainable development goals (SDGs): what can eye health can learn from commercial industry and early lessons from the application of machine learning in eye health programmes. Front Public Health. 2021;9:752049. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li X, Cheng M, Xu J. Leaders' innovation expectation and nurses' innovation behaviour in conjunction with artificial intelligence: the chain mediation of job control and creative self-efficacy. J Nurs Manag. 2022;30(8):3806-3816. [ CrossRef ] [ Medline ]
• Nasseef OA, Baabdullah AM, Alalwan AA, Lal B, Dwivedi YK. Artificial intelligence-based public healthcare systems: G2G knowledge-based exchange to enhance the decision-making process. Gov Inf Q. 2022;39(4):101618. [ CrossRef ]
• Ergin E, Karaarslan D, Şahan S, Çınar Yücel Ş. Artificial intelligence and robot nurses: from nurse managers' perspective: a descriptive cross-sectional study. J Nurs Manag. 2022;30(8):3853-3862. [ CrossRef ] [ Medline ]
• Hazarika I. Artificial intelligence: opportunities and implications for the health workforce. Int Health. 2020;12(4):241-245. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stanfill MH, Marc DT. Health information management: implications of artificial intelligence on healthcare data and information management. Yearb Med Inform. 2019;28(1):56-64. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davenport T, Foutty J. AI-driven leadership. Sloan Management Review. 2018. URL: https://sloanreview.mit.edu/article/ai-driven-leadership/ [accessed 2024-03-08]
• Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94-98. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Greenhill RG, Pearson JS, Schmidt RN, Stuart D, Rossettie S. Exploring healthcare leadership competencies for the fourth industrial revolution: a scoping review of the literature. J Health Adm Educ. 2021;38(3):695-708.
• Forum W. Empowering AI leadership: AI C-suite toolkit. World Economic Forum. 2022. URL: https://www3.weforum.org/docs/WEF_Empowering_AI_Leadership_2022.pdf [accessed 2024-03-08]
• Jones WA. Artificial intelligence and leadership: a few thoughts, a few questions. J Leadersh Stud. 2018;12(3):60-62. [ CrossRef ]
• Peifer Y, Jeske T, Hille S. Artificial intelligence and its impact on leaders and leadership. Procedia Comput Sci. 2022;200:1024-1030. [ CrossRef ]
• Smith AM, Green M. Artificial intelligence and the role of leadership. J of Leadership Studies. 2018;12(3):85-87. [ CrossRef ]
• Hersey P, Blanchard KH, Natemeyer WE. Situational leadership, perception, and the impact of power. Group Organ Manag. 2016;4(4):418-428. [ CrossRef ]
• Westbrook D, Kennerley H, Kirk J. Contingency theories of effective leadership. In: The SAGE Handbook of Leadership. London. SAGE; 2011:286-298.
• Uhl-Bien M, Marion R, McKelvey B. Complexity leadership theory: shifting leadership from the industrial age to the knowledge era. Leadersh Q. 2007;18(4):298-318. [ CrossRef ]
• Uhl-Bien M, Meyer D, Smith J. Complexity leadership in the nursing context. Nurs Adm Q. 2020;44(2):109-116. [ CrossRef ] [ Medline ]
• Heifetz AR, Grashow A, Linsky M. The Practice of Adaptive Leadership: Tools and Tactics for Changing Your Organization and the World. Cambridge. Harvard Business Press; 2009.
• Heifetz RA, Laurie DL. The Work of Leadership. New York. Harv Bus Rev; 2001.
• Bass BM, Riggio RE. Transformational Leadership. New York. Lawrence Erlbaum Associates; 2006.
• Bass B, Avolio B. Developing transformational leadership: 1992 and beyond. J Eur Ind Train. 1990;14(5). [ CrossRef ]

Abbreviations

Edited by T de Azevedo Cardoso, G Eysenbach; submitted 14.11.23; peer-reviewed by D Chrimes, TAR Sure, S Kommireddy, J Konopik, M Brommeyer; comments to author 20.02.24; revised version received 12.03.24; accepted 15.07.24; published 14.08.24.

©Abi Sriharan, Nigar Sekercioglu, Cheryl Mitchell, Senthujan Senkaiahliyan, Attila Hertelendy, Tracy Porter, Jane Banaszak-Holl. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Siemens PLM Software

Product lifecycle management

Teamcenter plm software.

Plan, develop and deliver innovative products with Teamcenter software. Discover why Teamcenter is a leading choice in product lifecycle management (PLM).

Take a sneak peek into the Teamcenter agenda for Realize LIVE Americas.

Why Teamcenter?

Leverage the digital twin.

Use digital twins to connect and optimize processes for designs, systems, software, simulation and visualization. Resolve problems before they become real.

Collaborate and innovate

Connect people across your business. Automate and streamline product lifecycle processes. Provide visibility for everyone to make innovative product decisions.

Rely on the leader

Start-ups to large enterprises - across industries - trust Teamcenter. Invest in the future with modern PLM of unmatched breadth, depth and usability.

Optimize your product lifecycle

Set the strategic direction and product definition to guide downstream decisions.

Design and document the multi-discipline product to leverage the digital twin.

Weave the digital thread to connect product development with manufacturing, service and suppliers.

PLM your way

Get Teamcenter delivered any way you want it – on premises or on the cloud – operated by your IT team or Siemens.

On premises

Host, manage and support your PLM software with your IT team. Explore the PLM software portfolio.

Choose your cloud provider, managed by your IT team. See all PLM solutions.

On cloud, SaaS by Siemens

Get started quickly using software as a service. Operation, maintenance and upgrades are included.

Teamcenter X SaaS grows with your business

See how even the smallest start-up business can grow with Teamcenter X. Cloud SaaS PLM is instant-on for fast time to value. You can manage your product digital twin efficiently to bring new and innovative products to market faster – and grow your business.

Engineering depth and business breadth

Adapt to business disruptions while driving product innovation. Teamcenter provides a single source of product information for every stakeholder across the business.

What PLM solution is best for you?

G2 ranks products based on customer reviews. G2 named Teamcenter the best PLM software for any size company – outperforming 99 others. Take a look at the PLM software comparison grid to see the top ranking for Teamcenter.

Top PLM software leadership recognition

Teamcenter is recognized by Forrester Research as a leader in PLM.

Teamcenter received the highest possible scores in more criteria than any other evaluated vendor in The Forrester Wave™: Product Lifecycle Management for Discrete Manufacturers, Q1 2023 report.

Northrop Grumman

Embracing digital transformation.

Company: Northrop Grumman

Industry: Aerospace & Defense

Siemens Software: Teamcenter