literature review open source software

7 open source tools to make literature reviews easy

Opensource.com

A good literature review is critical for academic research in any field, whether it is for a research article, a critical review for coursework, or a dissertation. In a recent article, I presented detailed steps for doing  a literature review using open source software .

The following is a brief summary of seven free and open source software tools described in that article that will make your next literature review much easier.

1. GNU Linux

Most literature reviews are accomplished by graduate students working in research labs in universities. For absurd reasons, graduate students often have the worst computers on campus. They are often old, slow, and clunky Windows machines that have been discarded and recycled from the undergraduate computer labs. Installing a flavor of GNU Linux will breathe new life into these outdated PCs. There are more than 100 distributions , all of which can be downloaded and installed for free on computers. Most popular Linux distributions come with a "try-before-you-buy" feature. For example, with Ubuntu you can make a bootable USB stick that allows you to test-run the Ubuntu desktop experience without interfering in any way with your PC configuration. If you like the experience, you can use the stick to install Ubuntu on your machine permanently.

Linux distributions generally come with a free web browser, and the most popular is Firefox . Two Firefox plugins that are particularly useful for literature reviews are Unpaywall and Zotero. Keep reading to learn why.

3. Unpaywall

Often one of the hardest parts of a literature review is gaining access to the papers you want to read for your review. The unintended consequence of copyright restrictions and paywalls is it has narrowed access to the peer-reviewed literature to the point that even Harvard University is challenged to pay for it. Fortunately, there are a lot of open access articles—about a third of the literature is free (and the percentage is growing). Unpaywall is a Firefox plugin that enables researchers to click a green tab on the side of the browser and skip the paywall on millions of peer-reviewed journal articles. This makes finding accessible copies of articles much faster that searching each database individually. Unpaywall is fast, free, and legal, as it accesses many of the open access sites that I covered in my paper on using open source in lit reviews .

Formatting references is the most tedious of academic tasks. Zotero can save you from ever doing it again. It operates as an Android app, desktop program, and a Firefox plugin (which I recommend). It is a free, easy-to-use tool to help you collect, organize, cite, and share research. It replaces the functionality of proprietary packages such as RefWorks, Endnote, and Papers for zero cost. Zotero can auto-add bibliographic information directly from websites. In addition, it can scrape bibliographic data from PDF files. Notes can be easily added on each reference. Finally, and most importantly, it can import and export the bibliography databases in all publishers' various formats. With this feature, you can export bibliographic information to paste into a document editor for a paper or thesis—or even to a wiki for dynamic collaborative literature reviews (see tool #7 for more on the value of wikis in lit reviews).

5. LibreOffice

Your thesis or academic article can be written conventionally with the free office suite LibreOffice , which operates similarly to Microsoft's Office products but respects your freedom. Zotero has a word processor plugin to integrate directly with LibreOffice. LibreOffice is more than adequate for the vast majority of academic paper writing.

If LibreOffice is not enough for your layout needs, you can take your paper writing one step further with LaTeX , a high-quality typesetting system specifically designed for producing technical and scientific documentation. LaTeX is particularly useful if your writing has a lot of equations in it. Also, Zotero libraries can be directly exported to BibTeX files for use with LaTeX.

7. MediaWiki

If you want to leverage the open source way to get help with your literature review, you can facilitate a dynamic collaborative literature review . A wiki is a website that allows anyone to add, delete, or revise content directly using a web browser. MediaWiki is free software that enables you to set up your own wikis.

Researchers can (in decreasing order of complexity): 1) set up their own research group wiki with MediaWiki, 2) utilize wikis already established at their universities (e.g., Aalto University ), or 3) use wikis dedicated to areas that they research. For example, several university research groups that focus on sustainability (including mine ) use Appropedia , which is set up for collaborative solutions on sustainability, appropriate technology, poverty reduction, and permaculture.

Using a wiki makes it easy for anyone in the group to keep track of the status of and update literature reviews (both current and older or from other researchers). It also enables multiple members of the group to easily collaborate on a literature review asynchronously. Most importantly, it enables people outside the research group to help make a literature review more complete, accurate, and up-to-date.

Wrapping up

Free and open source software can cover the entire lit review toolchain, meaning there's no need for anyone to use proprietary solutions. Do you use other libre tools for making literature reviews or other academic work easier? Please let us know your favorites in the comments.

Related Content

Start your free trial

Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.

REQUEST A FREE TRIAL

• Research Skills Blog

5 software tools to support your systematic review processes

By Dr. Mina Kalantar on 19-Jan-2021 13:01:01

Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating evidence was initially applied in healthcare, to set policies, create guidelines and answer medical questions.

Systematic reviews are large, complex projects and, depending on the purpose, they can be quite expensive to conduct. A team of researchers, data analysts and experts from various fields may collaborate to review and examine incredibly large numbers of research articles for evidence synthesis. Depending on the spectrum, systematic reviews often take at least 6 months, and sometimes upwards of 18 months to complete.

The main principles of transparency and reproducibility require a pragmatic approach in the organisation of the required research activities and detailed documentation of the outcomes. As a result, many software tools have been developed to help researchers with some of the tedious tasks required as part of the systematic review process.

hbspt.cta._relativeUrls=true;hbspt.cta.load(97439, 'ccc20645-09e2-4098-838f-091ed1bf1f4e', {"useNewLoader":"true","region":"na1"});

The first generation of these software tools were produced to accommodate and manage collaborations, but gradually developed to help with screening literature and reporting outcomes. Some of these software packages were initially designed for medical and healthcare studies and have specific protocols and customised steps integrated for various types of systematic reviews. However, some are designed for general processing, and by extending the application of the systematic review approach to other fields, they are being increasingly adopted and used in software engineering, health-related nutrition, agriculture, environmental science, social sciences and education.

Software tools

There are various free and subscription-based tools to help with conducting a systematic review. Many of these tools are designed to assist with the key stages of the process, including title and abstract screening, data synthesis, and critical appraisal. Some are designed to facilitate the entire process of review, including protocol development, reporting of the outcomes and help with fast project completion.

As time goes on, more functions are being integrated into such software tools. Technological advancement has allowed for more sophisticated and user-friendly features, including visual graphics for pattern recognition and linking multiple concepts. The idea is to digitalise the cumbersome parts of the process to increase efficiency, thus allowing researchers to focus their time and efforts on assessing the rigorousness and robustness of the research articles.

This article introduces commonly used systematic review tools that are relevant to food research and related disciplines, which can be used in a similar context to the process in healthcare disciplines.

These reviews are based on IFIS' internal research, thus are unbiased and not affiliated with the companies.

This online platform is a core component of the Cochrane toolkit, supporting parts of the systematic review process, including title/abstract and full-text screening, documentation, and reporting.

The Covidence platform enables collaboration of the entire systematic reviews team and is suitable for researchers and students at all levels of experience.

From a user perspective, the interface is intuitive, and the citation screening is directed step-by-step through a well-defined workflow. Imports and exports are straightforward, with easy export options to Excel and CVS.

Access is free for Cochrane authors (a single reviewer), and Cochrane provides a free trial to other researchers in healthcare. Universities can also subscribe on an institutional basis.

Rayyan is a free and open access web-based platform funded by the Qatar Foundation, a non-profit organisation supporting education and community development initiative . Rayyan is used to screen and code literature through a systematic review process.

Unlike Covidence, Rayyan does not follow a standard SR workflow and simply helps with citation screening. It is accessible through a mobile application with compatibility for offline screening. The web-based platform is known for its accessible user interface, with easy and clear export options.

Function comparison of 5 software tools to support the systematic review process

EPPI-Reviewer

EPPI-Reviewer is a web-based software programme developed by the Evidence for Policy and Practice Information and Co-ordinating Centre  (EPPI) at the UCL Institute for Education, London .

It provides comprehensive functionalities for coding and screening. Users can create different levels of coding in a code set tool for clustering, screening, and administration of documents. EPPI-Reviewer allows direct search and import from PubMed. The import of search results from other databases is feasible in different formats. It stores, references, identifies and removes duplicates automatically. EPPI-Reviewer allows full-text screening, text mining, meta-analysis and the export of data into different types of reports.

There is no limit for concurrent use of the software and the number of articles being reviewed. Cochrane reviewers can access EPPI reviews using their Cochrane subscription details.

EPPI-Centre has other tools for facilitating the systematic review process, including coding guidelines and data management tools.

CADIMA is a free, online, open access review management tool, developed to facilitate research synthesis and structure documentation of the outcomes.

The Julius Institute and the Collaboration for Environmental Evidence established the software programme to support and guide users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews.

CADIMA was initially developed for research questions in agriculture and environment but it is not limited to these, and as such, can be used for managing review processes in other disciplines. It enables users to export files and work offline.

The software allows for statistical analysis of the collated data using the R statistical software. Unlike EPPI-Reviewer, CADIMA does not have a built-in search engine to allow for searching in literature databases like PubMed.

DistillerSR

DistillerSR is an online software maintained by the Canadian company, Evidence Partners which specialises in literature review automation. DistillerSR provides a collaborative platform for every stage of literature review management. The framework is flexible and can accommodate literature reviews of different sizes. It is configurable to different data curation procedures, workflows and reporting standards. The platform integrates necessary features for screening, quality assessment, data extraction and reporting. The software uses Artificial Learning (AL)-enabled technologies in priority screening. It is to cut the screening process short by reranking the most relevant references nearer to the top. It can also use AL, as a second reviewer, in quality control checks of screened studies by human reviewers. DistillerSR is used to manage systematic reviews in various medical disciplines, surveillance, pharmacovigilance and public health reviews including food and nutrition topics. The software does not support statistical analyses. It provides configurable forms in standard formats for data extraction.

DistillerSR allows direct search and import of references from PubMed. It provides an add on feature called LitConnect which can be set to automatically import newly published references from data providers to keep reviews up to date during their progress.

The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to find the right tools for their research.

If you enjoyed this blog post, you may also be interested in our recently published blog post addressing the difference between a systematic review and a systematic literature review.

• FSTA - Food Science & Technology Abstracts
• IFIS Collections
• Resources Hub
• Diversity Statement
• Sustainability Commitment
• Company news
• Frequently Asked Questions
• Privacy Policy
• Terms of Use for IFIS Collections

Ground Floor, 115 Wharfedale Road,  Winnersh Triangle, Wokingham, Berkshire RG41 5RB

Get in touch with IFIS

© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved     |     Charity Reg. No. 1068176     |     Limited Company No. 3507902     |     Designed by Blend

A Literature Review for Open Source Software Studies

• Conference paper
• First Online: 01 January 2015
• Cite this conference paper

• Qiqi Jiang 15 ,
• Jianjun Qin 16 &
• Lele Kang 17  

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9191))

Included in the following conference series:

• International Conference on HCI in Business

5886 Accesses

1 Citations

In this work, we provided a comprehensive literature review of prior studies about OSS (open source software). In particular, we categorized those articles into three streams based on their research topics. In addition, the assessments of OSS success are also summarized. The future agenda and potential research gap are given in the end of the article.

You have full access to this open access chapter,  Download conference paper PDF

Similar content being viewed by others

Fifteen Years of Open Source Software Evolution

Open Source

• Open source software
• Literature review
• Assessment of OSS success

1 Introduction

Open source software (hereinafter OSS) is a user-driven, collaborative innovation produced by self-organizing teams of contributors dynamically formed through online interactions [ 1 ]. With the emergent development in the last decades, the OSS has become an important phenomenon in both economics and culture. Increasing number of IT giants has realized the value of OSS and created their own platforms for hosting the OSS projects. For instances, Google established Google Code on 2005 for providing fundamental tools for OSS developers to share their projects to the public; Microsoft subsequently set up the CodePlex for allowing the engineers and computer scientists to share their ideas and OSS projects although Microsoft used to strongly disagree the OSS campaigns [ 2 ].

Various benefits can be found with the emergence of OSS. We think three key benefits are predominantly important. First, OSS software can dramatically reduce the cost of development for IT company. Such cost does not only include the development cost but the innovation cost as well. The outstanding software received the effort and contribution from the talented developments throughout the world. Second, the OSS can facilitate the organizations to implement the information technology systems into their business process, especially for those small medium enterprise (hereinafter SMEs) or public institutes. There is no doubt that the IT can make the business better. However, due to the cost of IT implementations, the SMEs or public institutes may not implement the IT system due to their limited budget. Introducing the OSS systems can mitigate such dilemma. Last but not least, the information and knowledge can be diffused in terms of participating in OSS projects, which is beneficial for those people who want to have a good command of programming language since they can learn via practice.

To this end, great studies have been made to understand the OSS predominantly, with a focus on viewing OSS project as an organization for technological innovation. These investigations include the investigation of motivation to participate in the OSS development [ 3 – 6 ], the role of social network in promoting the OSS performance [ 7 – 10 , 27 ], the governance and management of an OSS project [ 11 , 12 ], and the influence of policy and legalization on OSS [ 13 – 15 ]. In addition, various measurements have been employed to assess to the extent the OSS success, such as number of downloads, frequency of CVS commits, extent of code reuse etc. Thus, it is imperative to provide a comprehensive extent of literature reviews on OSS studies. In this article, we surveyed the most representative OSS studies and summarized them into three streams based on their research topics. In addition, the assessments of OSS success are also summarized with their pros and cons. The future agenda and potential research gap are given in the end of the article.

2 Prior OSS Studies

In this section, a comprehensive extent of literature reviews is conducted. Based on the reviews, three subsections are listed by the research topics of prior OSS studies. In particular, the first stream includes the literatures studying the individual motivations to contribute or participate in the OSS projects. The second stream summarized the literatures discussing how the network characteristics influenced the OSS performances. In the third stream, we reviewed how the legalization, especially the OSS licenses, influences the OSS performances.

2.1 Individual Motivations

Thousands of individuals participate in OSS projects for diverse purposes. Their contributed product will be released to the public for free usage. Why are they willing to contribute to such OSS projects since their no economic returns from their contributed projects? What are their motivations? After surveying prior literatures, we found two key motivations, which can be concluded as intrinsic motivations and extrinsic motivations.

Intrinsic Motivation. The intrinsic motivations have been studied since 1970s. It is driven by an interest or enjoyment in the task itself and exists within the individual instead of depending on external pressure or desire for incentive reward [ 16 ]. The intrinsic motivation was widely regarded as a key motives for individuals to participate in the OSS projects. Previous literatures found more than half developers indicated that the enjoyment in programming and the sense of satisfaction originated form the participation in OSS software constantly motivate them to sustainably contribute to OSS projects. Besides the sense of enjoyment and satisfaction, Lakhani and Wolf [ 17 ] found the several individuals could show their creativity and new ideas in terms of implementing them into the OSS projects, which conferred great sense of accomplishment for them. Such sense was believed as another key motivation for constantly contribution. Although the intrinsic motivations were found as the original motives for breeding the OSS campaign, however, Roberts et al. [ 5 ] thought the intrinsic motivations had several defects, such as short effectiveness and strong self-direction, which might challenge the sustainability of OSS projects in future. Thus, it is imperative to unveil the extrinsic motivations for OSS participants.

Extrinsic Motivation. Extrinsic motivation denotes performing an activity is built upon the desired outcome like momentary incentives, reputations, and profits etc., which is the opposite of intrinsic motivation [ 16 ]. In other words, with the disappearance of external incentives, then the extrinsic motivations will decrease or even disappear. Von Krogh and his collaborators [ 18 ] investigated a large OSS project and interviewed several participants and summarized that the participation was built upon the pursuit of communal resources, which includes reputations, control of technology, and learning opportunities. The first dimension of extrinsic motivation is the acquisition of reputation. The reputation can be obtained in terms of (1) actively participating in the OSS projects, (2) providing solutions to the existing bugs or problems, (3) providing innovative revision or modification to the current OSS. With the increase of reputation, the participants will be conferred with higher authority, which will eventually enable him or her to dominate the entire project, such as the recruitment of developers or the decision power in the project management. To this end, the reputation in the OSS community is widely regarded as an extreme important extrinsic motivation for encouraging OSS developers to constantly contribute to OSS project. The second acquired communal resource is the “control over technology”. Such extrinsic motivation is formalized from the demand of self-usage. It is found that the key reason to participating in OSS project is to customize such project for own usage [ 3 ]. Although the novel and creative ideas can be realized through OSS project, however, the overall quality of OSS project cannot reach the industrial standard due to the relaxed management system and software testing procedures. Thus, those experienced users may compile their revised OSS and incidentally update it. In terms of long-term investment, those users may have a good command of this technology, which would benefit them in the future. The third communal resource is the learning opportunities. The participants of OSS project do not only include those experienced developers but those users or fresh programmers. By joining in OSS project, these people were provided a very good opportunity to learn (1) programming by collaborating with talented programmers in the world, and (2) knowledge about project management in software engineering as well. Besides these three key extrinsic motivations, prior literatures also identified several other extrinsic factors motivating individuals to contribute to the OSS project such as sense of reciprocity and job opportunities. Sense of reciprocity is prevailing in the OSS user support community, in which those end-users proactively respond or answer other’s questions in order to obtain the helps from others when they need in the future. In addition, prior literatures [ 19 , 20 ] found some IT companies might recruit some talented programmers from the OSS community, which encourage some participants to diligently work on their contributed OSS projects.

Besides the explanation from motivation theory, several studies found institutional management or leadership also affected individual motivations. For instances, by interviewing the developers from Debian project, O’Mahony and Ferraro [ 12 ] found the democratic management style outperformed the bureaucratic one thought the later was found to be efficient. In addition, Li and her collaborators [ 11 ] found individuals preferred to join those OSS project which were employed the transformational leadership style. In other words, such leadership style signals that everyone could be conferred as a project leader in the future.

2.2 Network Characteristics

In second stream, prior studies investigated how network structures affected the OSS performances [ 7 – 10 , 27 ]. The development of OSS project cannot be done without the collective actions. In this regard, prior literatures employed the social network analysis to articulate such collective actions, i.e. collaborative behaviors, which include inter-project individual-individual collaborative networks [ 7 ], project-affiliated networks [ 8 , 9 , 27 ], and intra-project communication or collaborative networks [ 10 ] etc. Thus, the social capital theory was mostly adopted as a theoretical underpinning to explicate such social relationships.

2.2.1 Inter-project Network

Due to the open nature of the OSS projects, the participants can freely contribute to multiple projects. In terms of such shared participants or concurrently contributed projects, two types of affiliated networks can be initialized. The first one is call project-affiliated network, in which the projects with shared participants are interconnected. The second one is called participant-affiliated network, in which the developers/project administrators who contributed to same projects are interconnected. Prior studies found the characteristics of participant-affiliated network, such as network distances or network density, had significant impact on the OSS performance [ 21 ]. After five-year observation on more than 2000 OSS projects hosted in Sourceforge.net, Singh and his collaborators [ 27 ] found internal collaborative cohesion had significantly positive impact on the OSS performances, manifested by the extent of CVS commits, but the external cohesion presented an inverted-U impact on the OSS performances. Hahn et al. [ 8 ] found the OSS developers preferred to join in those projects initialized by those who had collaborated before in terms of investigating the collaborative ties. Grewal et al. [ 7 ] argued the OSS performances was significantly influenced by the extent of network embeddedness in terms of studying the developer-affiliated network.

2.2.2 Intra-project Network

Besides the inter-project network, prior studies also found that the intra-project network also played a role on OSS performances [ 22 ]. Differing from the works studying OSS performances, the intra-project studies mainly discussed the individual collective behaviors. For instances, Conaldi et al. [ 10 ] established an analytical model for collaborative network and empirically verified it by using the debugging network from a large OSS project. Sowe et al. [ 23 ] categorized the participants from Debian (a leading Linux OS distributor) into three key roles, which included knowledge seekers, knowledge contributors, and knowledge brokers in terms of analyzing the emailing communication network, and argued that the knowledge brokers could facilitate the information flow and distribution. By studying the internal communication networks of two leading OSS projects, Singh and Tan [ 24 ] found the stability and efficiency cannot be concurrently reached, and advocated the OSS project administrators had to adjust the balance based on their key goals.

In sum, we can find the literatures on inter-project network mainly discussed how OSS projects could be outperformed in terms of network ties or vertex positions. The intra-project network studies mainly concentrated on particular behaviors or actions in OSS project development, such as debugging, communications, or collaborations. A brief summarization is given in Table  1 .

2.3 Policy and Legalization

Comparing with the works studying individual motivations or network characteristics influencing the success of OSS project, the studies in third stream mainly discussed how external policy or legalization affect the OSS performances. In particular, two sub-streams in this domain can be found, i.e. the characteristics of OSS licenses and the intellectual property lawsuits enforcement [ 13 – 15 ].

The OSS supports encouraged the source codes could be shared, revised, and redistributed, however, the rights of the intellectual property of the creators cannot be overlooked. In this regard, several OSS licenses have been given to restrict the right of usage or revisions. There are more than 60 different OSS licenses prevailing throughout the world. The extent of restrictiveness of those OSS licenses is different. For instances, comparing with BSD whose restrictiveness is very low, the GNU license is widely known for its restrictive copyleft. Prior studies found OSS project administrators and users preferred the highly restrictive OSS licenses, but the OSS developers did not welcome such license [ 13 ]. In addition, Sen et al. [ 14 ] found the OSS project with moderately restrictive OSS license cannot attract the developers and users. Besides the effectiveness of OSS licences, Wen et al. [ 15 ] found the intellectual property lawsuits enforcement would increase the OSS usage cost, which would reduce the OSS participants’ interests.

3 Assessment of OSS Success

Several metrics have be employed to assess the OSS success, and such assessments were developed in consideration of the research propose and the audiences, such as number of subscribers, number of active developers, number of downloads, frequency of CVS commits, and the extent of code reuse etc. [ 25 ]. After comprehensively reviewing prior literatures, the number of downloads and the frequency of CVS commits were most widely used to assess the OSS project success. However, we though either number of downloads or frequency of CVS commits cannot best represent the performance of OSS, i.e. success of OSS success. The number of downloads can only serve as a proxy indicator of software exposure, but cannot represent the success of an OSS due to the unknown conversion rate [ 25 ]. For instances, end users can simply remove their downloaded OSS after a quick trial, which cannot be reflected in the number of OSS downloads. To reconcile such defect, some scholars advocated to use the frequency of CVS commit to assess the success of OSS projects. Comparing with the number of downloads which represent the extent to which OSS prevails in end-users, the frequency of CVS commit serves as a proxy indicator of development vitality. The higher frequency of CVS commit indicates the developers constantly contributed to the OSS project. However, the limitation of such indicator is also obvious. The higher frequency of CVS commit may also indicate the fundamental quality of this OSS project is not good, and the developers had to constantly fix the bugs found in the newly distributed versions.

Encouraging the literatures in innovation and strategic management, we thought the extent of code reuse could be a representative indicator to assess the success of OSS project. Haefliger et al. [ 26 ] found the qualified sources codes were extensively reused in other OSS projects. In innovation literatures, the innovation capability of an organization is measured by the extent to which their patents are cited. In similar vein, we argued the extent of code reuse could serve as an important indicator to assess the OSS project success, i.e. the innovation performance.

4 Conclusion and Future Agenda

In this article, we provide a comprehensive literature reviews on prior OSS studies. In particular, we categorized the previous literatures into three streams based on the research topics, which included individual motivations to contribute or participate in the OSS projects, the relationship between network characteristics and OSS performances, and the external factors like OSS licenses and policy influencing the OSS performances. In addition, we also surveyed the prior literatures and summarized several key indicators assessing the OSS project success, and found the pros and cons of the existing measurement. Encouraged by the innovation literatures, we expected the future research could use the extent of code reuse to indicate the innovation quality of OSS.

Von Krogh, G., Von Hippel, E.: The promise of research on open source software. Manage. Sci. 52 (7), 975–983 (2006)

Article   Google Scholar  

Fitzgerald, B.: The transformation of open source software. Mis Q. 30 (3), 587–598 (2006)

Google Scholar  

Hertel, G., Niedner, S., Herrmann, S.: Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel. Res. Policy 32 (7), 1159–1177 (2003)

Shah, S.K.: Motivation, governance, and the viability of hybrid forms in open source software development. Manage. Sci. 52 (7), 1000–1014 (2006)

Roberts, J.A., Hann, I.H., Slaughter, S.A.: Understanding the motivations, participation, and performance of open source software developers: a longitudinal study of the Apache projects. Manage. Sci. 52 (7), 984–999 (2006)

Lakhani, K.R., Von Hippel, E.: How open source software works:“free” user-to-user assistance. Res. Policy 32 (6), 923–943 (2003)

Grewal, R., Lilien, G.L., Mallapragada, G.: Location, location, location: How network embeddedness affects project success in open source systems. Manage. Sci. 52 (7), 1043–1056 (2006)

Hahn, J., Moon, J.Y., Zhang, C.: Emergence of new project teams from open source software developer networks: impact of prior collaboration ties. Inf. Syst. Res. 19 (3), 369–391 (2008)

Singh, P.V.: The small-world effect: the influence of macro-level properties of developer collaboration networks on open-source project success. ACM Trans. Softw. Eng. Methodol. (TOSEM) 20 (2), 6 (2010)

Conaldi, G., Lomi, A., Tonellato, M.: Dynamic models of affiliation and the network structure of problem solving in an open source software project. Organ. Res. Methods 15 (3), 385–412 (2012)

Li, Y., Tan, C.H., Teo, H.H.: Leadership characteristics and developers’ motivation in open source software development. Inf. Manage. 49 (5), 257–267 (2012)

O’Mahony, S., Ferraro, F.: The emergence of governance in an open source community. Acad. Manage. J. 50 (5), 1079–1106 (2007)

Subramaniam, C., Sen, R., Nelson, M.L.: Determinants of open source software project success: a longitudinal study. Decis. Support Syst. 46 (2), 576–585 (2009)

Sen, R., Singh, S.S., Borle, S.: Open source software success: measures and analysis. Decis. Support Syst. 52 (2), 364–372 (2012)

Wen, W., Forman, C., Graham, S.J.: Research note-the impact of intellectual property rights enforcement on open source software project success. Inf. Syst. Res. 24 (4), 1131–1146 (2013)

Ryan, R.M., Deci, E.L.: Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55 (1), 68 (2000)

Lakhani, K., Wolf, R.G.: Why hackers do what they do: understanding motivation and effort in free/open source software projects (2003)

Von Krogh, G., Haefliger, S., Spaeth, S.: Collective action and communal resources in open source software development: the case of freenet. Academy of Management (2003)

Wu, C.G., Gerlach, J.H., Young, C.E.: An empirical analysis of open source software developers’ motivations and continuance intentions. Inf. Manage. 44 (3), 253–262 (2007)

Zhou, M., Mockus, A.: What make long term contributors: willingness and opportunity in OSS community. In: Proceedings of the 34th International Conference on Software Engineering, pp. 518–528. IEEE Press (2012)

Long, Y., Siau, K.: Social network structures in open source software development teams. J. Database Manage. (JDM) 18 (2), 25–40 (2007)

Mockus, A., Fielding, R.T., Herbsleb, J.D.: Two case studies of open source software development: apache and Mozilla. ACM Trans. Softw. Eng. Methodol. (TOSEM) 11 (3), 309–346 (2002)

Sowe, S., Stamelos, I., Angelis, L.: Identifying knowledge brokers that yield software engineering knowledge in OSS projects. Inf. Softw. Technol. 48 (11), 1025–1033 (2006)

Singh, P.V., Tan, Y.: Developer heterogeneity and formation of communication networks in open source software projects. J. Manage. Inf. Syst. 27 (3), 179–210 (2010)

Crowston, K., Annabi, H., Howison, J.: Defining open source software project success. In: ICIS 2003 Proceedings, p. 28 (2003)

Haefliger, S., Von Krogh, G., Spaeth, S.: Code reuse in open source software. Manage. Sci. 54 (1), 180–193 (2008)

Singh, P.V., Tan, Y., Mookerjee, V.: Network effects: the influence of structural capital on open source project success. MIS Q. 35 (4), 813–830 (2011)

Download references

Acknowledgement

This work was supported by the Tongji Talented Youth Project (grant number 2014KJ002); National Natural Science Foundation of China [grant number 51408438]; the Scientific Research Foundation for the Returned Overseas Chinese Scholars, State Education Ministry of the People’s Republic of China; and Shanghai Pujiang Program [grant number 14PJ1408300].

Author information

Authors and affiliations.

School of Economics and Management, Tongji University, Shanghai, China

Shanghai Institute of Disaster Prevention and Relief, Tongji University, Shanghai, China

Jianjun Qin

Department of Information Systems, City University of Hong Kong, Hong Kong, Hong Kong SAR

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jianjun Qin .

Editor information

Editors and affiliations.

Missouri University of Science and Technology, Rolla, Montana, USA

Fiona Fui-Hoon Nah

Department of Information Systems, City University of Hong Kong, Kowloon Tong, Hong Kong SAR

Chuan-Hoo Tan

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper.

Jiang, Q., Qin, J., Kang, L. (2015). A Literature Review for Open Source Software Studies. In: Fui-Hoon Nah, F., Tan, CH. (eds) HCI in Business. HCIB 2015. Lecture Notes in Computer Science(), vol 9191. Springer, Cham. https://doi.org/10.1007/978-3-319-20895-4_65

Download citation

DOI : https://doi.org/10.1007/978-3-319-20895-4_65

Published : 21 July 2015

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-20894-7

Online ISBN : 978-3-319-20895-4

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

10 Open Science Tools for Literature Review You Should Know about

Here are 10 literature search tools that will make your scientific literature search faster and more convenient. All of the presented literature review software is free and follows Open Science principles.

Traditionally, scientific literature has been tucked away behind paywalls of academic publishers. Not only is the access to papers often restricted, but subscriptions are required to use many scientific search engines. This practice discriminates against universities and institutions who cannot afford the licenses, e.g. in low-income countries. Closed publishing also makes it hard for persons not affiliated with research institutes, such as freelance journalists or the public, to learn about scientific discoveries. 

The proportion of research accessible publicly today at no cost varies between disciplines . While in the biomedical sciences and mathematics, the majority of research published between 2009 and 2015 was openly accessible, this held true only for around 15 percent of publications in chemistry. Luckily, the interest in open access publishing is steadily increasing and has gained momentum in the past decade or so.

Many governmental funding bodies around the world nowadays require science resulting from grant money they provided to be available publicly for free. The exact requirements vary and UNESCO is currently developing a framework that specifies standards for the whole area of Open Science. 

Once I started my research on the topic, I was astonished by just how many free Open Science tools for literature review already exist! Read on below for 10 literature search tools — from a search engines for research papers, over literature review software that helps you quickly find open access versions of papers, to tools that help you save the correct citation in one click.

Tools for Literature review

First, an overview of the literature search tools in this blog post:

ScienceOpen

• Citation Gecko
• Local Citation Network

ResearchRabbit

• Open Access Button
• EndNote Click

Read by QxMD

I divided the tools into four categories:

Search engines for research papers

• Literature review software based on citation networks
• Locating open access scientific papers, and
• Other tools that help in the literature review

Here, we go!

The best place to start a scientific literature search is with a search engine for research papers. Here are two you might not have heard of!

Want to perform a literature search and don’t want to pay for Web of Science or Scopus or perhaps you are tired of the limited functionality of the free Google Scholar ? ScienceOpen is many things, among others a search engine for research papers. Despite being owned by a private company, this scientific search engine is freely accessible with visually appealing and functional design. Search results are clearly labelled for type of publication, number of citations, altmetrics scores etc. and allow for filtering. You can also access citation metrics, i.e., display which publications have cited a certain paper.

Recommended by a reader of the blog (thank you!), the Lens is a search tool that doesn’t only allow you to search the scholarly literature but patents too! Millions of patents from over 95 jurisdictions can be searched. The Lens is run by the non-profit social enterprise Cambia. The search engine is free to use for the public, though charges occur for commercial use and to get additional functionality.

Literature Review software based on citation networks

The next category of tools we will be looking at are a bit more advanced than a simple search engine for research papers. These literature search tools help you discover scientific literature you may have missed by visualising citation networks.

Citation Gecko 

The literature search tool Citation Gecko is an open source web app that makes it easier to discover relevant scientific literature than your average keyword-based search engine for research papers. It works in the following way: First you upload about 5-6 “seed papers”. The program then extracts all references in and to these seed papers and creates a visual citation network. The nodes are displayed in different colours and sizes depending on whether the papers are citing a seed paper or are cited by it and how many, respectively. By combing through the citation network, you can discover new papers that may be relevant for your scientific literature search. You can also increase your citation network step by step by including more seed papers. 

This literature review tool was developed by Barney Walker , and the underlying citation data is provided by Crossref and Open Citations .

Local Citation Network 

Similar to Citation Gecko, Local Citation Network is an open source tool that works as a scientific search engine on steroids. Local Citation Network was developed by Physician Scientist Tim Wölfle. This literature review tool works best if you feed it with a larger library of seed papers than required for Citation Gecko. Therefore, Wölfle recommends using it at the end of your scientific literature search to identify papers you may have missed. 

As an alternative to the literature search tools Citation Gecko and Local Citation Network, a reader of the blog recommended ResearchRabbit . It’s free to use and looks like a versatile piece of literature review software helping you build your own citation network. ResearchRabbit lets you add labels to the entries in your citation network, download PDFs of papers and sign up for email alerts for new papers related to your research topic. Instead of a tool to use only once during your scientific literature search, ResearchRabbit seems to function more like a private scientific library storing (and connecting) all the papers in your field.

Run by (former) researchers and engineers, ResearchRabbit is partly financed through donations but their website does not state where the core funding of this literature review software originates from.

Locating open access scientific papers

You may face the problem in your scientific literature search that you don’t have access to every research paper you are interested in. I highly recommend installing at least one of the open access tools below so you can quickly locate freely accessible versions of the scientific literature if available anywhere.

Open Access Button 

Works like the scientific search engine Sci-hub but is legal: You enter the DOI, link or citation of a paper and the literature review tool Open Access Button displays it if freely accessible anywhere. To find an open access version, Open Access Button searches thousands of repositories, for example, preprint servers, authors’ personal pages, open access journals and other aggregators such as the COnnecting REpositories service based at The Open University in the UK ( CORE ), the EU-funded OpenAire infrastructure, and the US community initiative Share . 

If the article you are looking for isn’t freely available, Open Access Button asks the author to share it to a repository. You can enter your email address to be notified once it has become available. 

Open Access Button is also available as browser plugin, which means that a button appears next to an article whenever a free version is available. This search engine for research papers is funded by non-profit foundations and is open source. 

Unpaywall 

Unpaywall is a search engine for research papers similar to Open Access Button — but only available as browser plugin. If the article you are looking at is behind a paywall but freely accessible somewhere else, a green button appears on the right side of the article. I installed it recently and regret not having done it sooner, it works really smoothly! I think the plugin is a great help in your scientific literature search.

Unpaywall is run by the non-profit organisation Our Research who has created a fleet of open science tools.

EndNote Click 

Another browser extension that lets you access the scientific literature for free if available is EndNote Click (formerly Kopernio). EndNote Click claims to be faster than other search engines for research papers bypassing redirects and verification steps. I personally don’t find the Unpaywall or Open Access Button plugins inconvenient to use but I’d encourage you to try out all of these scientific search engines and see what works best for you. 

One advantage of EndNote Click that a reader of the blog told me about is the side bar that appears when opening a paper through the plugin. It lets you, for example, save citations quickly, avoiding time-consuming searches on publishers’ websites. 

As the reference manager, EndNote Click is part of the research analytics company Clarivate.  

Other tools for literature review

This last category of literature search tools features a tool that creates a personalised feed of scientific literature for you and another that makes citing the scientific literature effortless.

Available as an app or in a browser window, the literature review tool Read lets you create a personalised feed that is updated daily with new papers on research topics or from journals of your choice. If there is an openly accessible version of an article, you can read it with one click. If your institution has journal subscriptions, you can also link them to your Read profile. Read has been created by the company QxMD and is free to use. 

CiteAs 

You discovered a promising paper in your scientific literature search and want to cite it? CiteAs is a convenient literature review tool to obtain the correct citation for any publication, preprint, software or dataset in one click. Funded by the Alfred P. Sloan Foundation, CiteAs is operated partly by the non-profit Our Research . 

Beyond literature review tools

There you have it, 10 tools for literature review that are all completely free and follow Open Science principles.

Of course, finding a great literature review tool, such as a search engine for research papers or a citation tool, is only one essential part in the whole process of writing a scientific paper. If you would like to learn a complete process to write a scientific article step by step, then you’ll love our free training. Simply click on the orange button below to watch it now (or sign up to watch it later).

Share article

© Copyright 2018-2024 by Anna Clemens. All Rights Reserved. 

Photography by Alice Dix

• Open access
• Published: 08 November 2016

A systematic literature review of open source software quality assessment models

• Adewole Adewumi 1 ,
• Sanjay Misra   ORCID: orcid.org/0000-0002-3556-9331 1 , 2 ,
• Nicholas Omoregbe 1 ,
• Broderick Crawford 3 &
• Ricardo Soto 3  

SpringerPlus volume  5 , Article number:  1936 ( 2016 ) Cite this article

7925 Accesses

24 Citations

6 Altmetric

Metrics details

Many open source software (OSS) quality assessment models are proposed and available in the literature. However, there is little or no adoption of these models in practice. In order to guide the formulation of newer models so they can be acceptable by practitioners, there is need for clear discrimination of the existing models based on their specific properties. Based on this, the aim of this study is to perform a systematic literature review to investigate the properties of the existing OSS quality assessment models by classifying them with respect to their quality characteristics, the methodology they use for assessment, and their domain of application so as to guide the formulation and development of newer models. Searches in IEEE Xplore, ACM, Science Direct, Springer and Google Search is performed so as to retrieve all relevant primary studies in this regard. Journal and conference papers between the year 2003 and 2015 were considered since the first known OSS quality model emerged in 2003.

A total of 19 OSS quality assessment model papers were selected. To select these models we have developed assessment criteria to evaluate the quality of the existing studies. Quality assessment models are classified into five categories based on the quality characteristics they possess namely: single-attribute, rounded category, community-only attribute, non-community attribute as well as the non-quality in use models. Our study reflects that software selection based on hierarchical structures is found to be the most popular selection method in the existing OSS quality assessment models. Furthermore, we found that majority (47%) of the existing models do not specify any domain of application.

Conclusions

In conclusion, our study will be a valuable contribution to the community and helps the quality assessment model developers in formulating newer models and also to the practitioners (software evaluators) in selecting suitable OSS in the midst of alternatives.

Prior to the emergence of open source software (OSS) quality models, the McCall, Dromey and ISO 9126 models were already in existence (Miguel et al. 2014 ). These models however did not consider some quality attributes unique to OSS such as community—a body of users and developers formed around OSS who contribute to the software and popularize it (Haaland et al. 2010 ). This gap is what led to the evolution of OSS quality models. Majority of the OSS quality models that exist today are derived from the ISO 9126 quality model (Miguel et al. 2014 ; Adewumi et al. 2013 ). It defines six internal and external quality characteristics, which are functionality, reliability, usability, efficiency, maintainability and portability. ISO 25010 replaced the ISO 9126 in 2010 (ISO/IEC 9126 2001 ), it has the following product quality attributes (ISO/IEC 2501 0 2001): functional suitability, reliability, performance efficiency, operability, security, compatibility, maintainability and transferability. The ISO 25010 quality in use attributes includes effectiveness, efficiency, satisfaction, safety and usability.

It is important to note that ISO 25010 can serve as standard for OSS only in terms of product quality and quality in use. It does not address unique characteristics of OSS such as the community. A key distinguishing feature of OSS is that it is built and maintained by a community (Haaland et al. 2010 ). The quality of this community also determines the quality of the OSS (Samoladas et al. 2008 ). From the literature, community related quality characteristics include (Soto and Ciolkowski 2009 ): maintenance capacity, sustainability, and process maturity. Maintenance capacity refers to the number of contributors to an OSS project and the amount of time they are willing and able to contribute to the development effort as observed from versioning logs, mailing lists, discussion forums and bug report systems. Furthermore, sustainability refers to the ability of the community to grow in terms of new contributors and to regenerate by attracting and engaging new members to take the place of those leaving the community. In addition, process maturity refers to the adoption and use of standard practices in the development process such as submission and review of changes, peer review of changes, provision of a test suite, and planned releases.

Since the advent of the first OSS quality model in 2003 (Adewumi et al. 2013 ), a number of other models have since been derived leading to an increasing collection of OSS quality models. Quality models in general can be classified into three broad categories namely: definition, assessment and prediction models (Ouhbi et al. 2014 , 2015 ; Deissenboeck et al. 2009 ). Generally, OSS quality assessment models outline specific attributes that guide the selection of OSS. The assessment models are very significant because they can help software evaluators to select suitable OSS in the midst of alternatives (Kuwata et al. 2014 ). However, despite the numerous quality assessment models proposed, there is still little or no adoption of these models in practice (Hauge et al. 2009 ; Ali Babar 2010 ). In order to guide the formulation of newer models, there is need to understand the nature of the existing OSS quality assessment models. The aim of this study is to investigate the nature of the existing OSS quality assessment models by classifying them with respect to their quality characteristics, the methodology they use for assessment, and their domain of application so as to guide the formulation and development of newer models. Existing studies on OSS quality assessment models (Miguel et al. 2014 ; Adewumi et al. 2013 ) are largely descriptive reviews that did not seek to classify OSS quality assessment models along specific dimensions, or answer specific research questions. In contrast, this paper employs a methodical, structured, and rigorous analysis of existing literature in order to classify existing OSS quality assessment models and establish a template guide for model developers when they come up with new models. Thus, this study is a systematic literature review that investigates three research questions, namely: (1) what are the key quality characteristics possessed by the OSS assessment models? (2) What selection methods are employed for use in these assessment models? (3) What is the domain of application? In order to conduct this systematic review, the original guidelines proposed by Kitchenham ( 2004 ) have been followed.

The rest of this paper is structured as follows: “ Methods ” section describes the method of obtaining the existing OSS quality models. “ Results ” section presents the results obtained in the study, while “ Summary and discussion ” section discusses the findings of the study. “ Conclusion and future work ” section concludes the paper with a brief note.

This section outlines the research questions posed in this study and also explains in detail the rationale behind each question. It goes on to discuss the search strategy for retrieving the relevant papers; criteria for including any given paper in the study; quality assessment of the retrieved papers as well as how relevant information was extracted from each selected paper.

Research questions

This study aims at gaining insight into the existing OSS quality models and addresses three research questions. The three research questions alongside the rationale motivating each question is presented in Table  1 . These form the basis for defining the search strategy.

Search strategy

A search string was defined based on the keywords derived from the research question as follows: “(Open Source Software OR libre OR OSS or FLOSS or FOSS) AND (model OR quality model OR measurement model OR evaluation model)”.

In order to retrieve the primary studies containing OSS quality models we made use of Scopus digital library. It indexes several renowned scientific journals, books and conference proceedings (e.g. IEEE, ACM, Science Direct and Springer). We considered only papers from (2003 to 2015) since the first OSS quality model emerged in 2003 (Haaland et al. 2010 ; Adewumi et al. 2013 ). We also focused on journal papers and conference proceedings in the subject area of Computer Science that were written in English. A total of 3198 primary studies were initially retrieved. After checking through their titles and abstracts, the number was reduced to 209. To be sure that no paper had been left out, we also performed a search in IEEE Explore, ACM and Springer using the same search string. No new papers were retrieved from this search that had not already been seen from the search in Scopus. Furthermore, a search was performed using Google Search and two relevant articles were retrieved (Duijnhouwer and Widdows 2003 ; Atos 2006 ) and added to make a total of 211 retrieved papers. These papers were read in detail to determine their suitability for inclusion.

Inclusion criteria

Papers proposing cost models and conceptual models were removed. Also position papers and papers that did not present a model for assessing quality in OSS in order to guide selection in the midst of alternatives were also removed. A crosscheck was conducted through the reference list of candidate studies to ensure that no model had been left out. As a result, 19 primary studies were selected, which are further discussed in the next segment of this section.

Quality assessment

Each primary study was evaluated by using the criteria defined in Adewumi et al. ( 2013 ). The criteria are based on four quality assessment (QA) questions:

Are the model’s attributes derived from a known standard (this can be ISO 9126, ISO 25010 or CMMI)?

Is the evaluation procedure of the model adequately described?

Does a tool support the evaluation process?

Is a demonstration of quality assessment using the model provided?

The questions were scored as follows:

Y (yes), the model’s attribute are mostly derived from a known standard, P (Partly), only a few of the model’s attributes are derived from a known standard; N (no), the model’s attributes are not all derived from a known standard.

Y, the evaluation procedure of the model are adequately described; P, the evaluation procedure was described inadequately; N, the evaluation procedure of the model was not described at all.

Y, the evaluation process is fully supported by a tool; P, the evaluation process is partially supported by a tool; N no tool support is provided for the evaluation process.

Y a complete demonstration of quality assessment using the model is provided; P only a partial demonstration of quality assessment using the model is provided; N there is no demonstration of quality assessment using the model provided.

The scoring procedure was Y = 1, P = 0.5, N = 0. The first author coordinated the quality evaluation extraction process. The first author assessed every paper, and assigned 5 papers each to the second, third and fourth authors and 4 papers to the fifth author so they could assess independently. When there was a disagreement, we discussed the issues until we reached agreement.

Data extraction strategy

In this phase, the first author extracted the data while the other four authors checked the extraction. This approach though inconsistent with the medical standards summarized in Kitchenham’s guidelines ( 2004 ) has been found useful in practice (Brereton et al. 2007 ). The first author coordinated the data extraction and checking tasks, which involved all of the authors of this paper. Allocation was not randomized rather it was based on the time availability of the individual researchers. When there was a disagreement, we discussed the issues until we reached agreement.

The selected studies were gleaned to collect the data that would provide the set of possible answers to the research questions. Table  2 shows the data extraction form that was created as an Excel sheet and filled by the first author for each of the papers selected.

From Table  2 it can be observed that the information extracted includes: the Study Ref., title, and classification [publication outlet, publication year and research questions (RQ) 1, 2 and 3].

Quality characteristics that the models in the selected studies can possess include the product quality and the quality in use characteristics of the ISO 25010 namely: functional suitability, reliability, performance efficiency, operability, security, compatibility, maintainability, transferability, effectiveness, efficiency, satisfaction, safety and usability. We also include community related quality characteristics as described in the literature namely (Soto and Ciolkowski 2009 ): maintenance capacity, sustainability and process maturity.

The methods used by assessment models for selection can be classified as (Petersen et al. 2008 ; Wen et al. 2012 ):

Data mining technique such as: Artificial Neural Network, Case-Based Reasoning, Data Envelope Analysis (DEA), Fuzzy Logic etc.

Process: A series of actions, or functions leading to a selection result and performing operations on data

Tool based technique: A technique that greatly employs software tools to accomplish selection task

Model: A system representation that allows for selection based on investigation through a hierarchical structure

Framework: A real or conceptual structure intended to serve as support or guide for selection process

Other, e.g. guidelines

The domain of application can be classified as follows (Forward and Lethbridge 2008 ):

Data dominant software—i.e. consumer-oriented software, business-oriented software, design and engineering software as well as information display and transaction entry

Systems software—i.e. operating systems, networking/communications, device/peripheral drivers, support utilities, middleware and system components, software backplanes (e.g. Eclipse), servers and malware

Control-dominant software—i.e. hardware control, embedded software, real time control software, process control software (e.g. air traffic control, industrial process, nuclear plants)

Computation-dominant software—i.e. operations research, information management and manipulation, artistic creativity, scientific software and artificial intelligence

No domain specified

Synthesis method

The synthesis method was based on:

Counting the number of papers per publication outlet and the number of papers found on a year-wise basis,

Counting the primary studies that are classified in response to each research question,

Presenting charts and frequency tables for the classification results which have been used in the analysis,

Presenting in the discussion a narrative summary with which to recount the key findings of this study.

This section presents the results obtained in response to the research questions posed in this study. Table  3 is a summary of the OSS quality assessment models used in this study, their sources and year of publication. The first column of the table (Study Ref.) represents the reference number of each quality assessment model in ascending order. The table shows that 2009 has the most number of published papers—three publications in total. The year 2003, 2004, 2005 and 2012 have the lowest number of publications—one published paper each. All other years (2007, 2008, 2011, 2013, 2014, 2015) have two published papers.

The studies were assessed for quality using the criteria described in the previous section (see “ Quality assessment ” section). The score for each study is shown in Table  4 . The results of the quality analysis shows that all studies scored above 1 on the proposed quality assessment scale with only one study scoring less than 2. One study scored 4, five studies scored 3.5, five studies scored 3, five studies scored 2.5 and two studies scored 2.

Table  5 shows the summary of the response to the research questions from each of the selected articles. From the table, it can be observed that an assessment model can belong to more than one category for RQ1 (an example is the assessment model in Study Ref. 8 which is single-attribute model, a non-community attribute model and a non-quality in use model).

RQ1. What are the key quality characteristics possessed by the models?

To address RQ1, we performed a comparative study of each identified model against ISO 25010 as well as community related quality characteristics described in “ Background ” section. Based on our comparative study, which is presented in Table  6 , we classify the quality assessment models into five categories, which are discussed as follows:

Single-attribute models: This refers to models that only measure one quality characteristic. Qualification and Selection of Open Source software (QSOS) model (Atos 2006 , Deprez and Alexandre 2008 ), Mathieu and Wray model ( 2007 ), Sudhaman and Thangavel model ( 2015 ) and Open Source Usability Maturity Model (OS-UMM) model (Raza et al. 2012 ) fall into this category. QSOS possesses maintainability as its quality characteristic. Mathieu and Wray as well as Sudhaman and Thangavel models both possess efficiency as their singular quality characteristic. In addition, OS-UMM possesses usability as its singular quality characteristic.

Rounded category models: This refers to models that possess at least one quality characteristic in each of the three categories used for comparison (i.e. product quality, quality in use and community related characteristics). Open Source Maturity Model (OSMM) (Duijnhouwer and Widdows 2003 ), Open Business Readiness Rating (Open BRR) model (Wasserman et al. 2006 ), Source Quality Observatory for Open Source Software (SQO-OSS) model (Samoladas et al. 2008 ; Spinellis et al. 2009 ), Evaluation Framework for Free/Open souRce projecTs (EFFORT) model (Aversano and Tortorella 2013 ), Muller ( 2011 ) and Sohn et al. model ( 2015 ) fall into this category of models. OSMM possesses all the quality characteristics in the product quality category as well as in the community-related quality characteristics but only possesses usability in the quality in use category. Open BRR and EFFORT models both possess all the community-related quality characteristics, some of the product quality characteristics and usability from the quality in use category. SQO-OSS possesses all the community-related quality characteristics, three of the product quality characteristics and effectiveness from the quality in use category. Muller model possesses one characteristic each from the product quality and community-related categories. It also possesses efficiency and usability from the quality in use category. As for Sohn et al. model, it possesses two quality characteristics from the product quality category and one quality characteristic each from the quality in use and community-related quality categories.

Community-only attribute model: This refers to a model that only measures community-related quality characteristics. The only model that fits this description is the Kuwata et al. model ( 2014 ) as seen in Table  6 . The model does not possess any quality characteristic from the product quality or quality in use categories.

Non-community attribute model: This refers to models that do not measure any community-related quality characteristics. QSOS (Atos 2006 ), Sung et al. ( 2007 ), Raffoul et al. ( 2008 ), Alfonzo et al. ( 2008 ), Mathieu and Wray, Chirila et al. (Del Bianco et al. 2010a ), OS-UMM (Raza et al. 2012 ), Sudhaman and Thangavel, and Sarrab and Rehman (Sarrab and Rehman 2014 ) models fall into this category.

Non-quality in use models: This refers to models that do not include any quality in use characteristics in their structure. QSOS (Atos 2006 , Deprez and Alexandre 2008 ), QualOSS (Soto and Ciolkowski 2009 ), OMM (Petrinja et al. 2009 , Del Bianco et al. 2010b , Del Bianco et al. 2011 , Chirila et al. ( 2011 ), Adewumi et al. ( 2013 ), and Kuwata et al. models are the models in this category.

From our classification, it is possible for a particular model to belong to more than one category. QSOS for instance belongs to three of the categories (i.e. it is a single-attribute model, non-community attribute model and non-quality in use model). Mathieu and Wray model ( 2007 ), Chirila et al. model ( 2011 ), OS-UMM (Raza et al. 2012 ), Sudhaman and Thangavel model ( 2015 ), as well as Kuwata et al. model ( 2014 ) all belong to two categories respectively. Precisely, Mathieu and Wray model is a single-attribute model and non-community attribute model. Chirila et al. model is a non-community attribute model as well as a non-quality in use model. OS-UMM is a single attribute model and a non-community attribute model. Sudhaman and Thangavel model is both a single-attribute model and non-community attribute model. Kuwata et al. model is both a community-only attribute model and a non-quality in use model. All the other models belong to a single category and they include: OSMM (Duijnhouwer and Widdows 2003 ), Open BRR (Wasserman et al. 2006 ), Sung et al. ( 2007 ), QualOSS (Soto and Ciolkowski 2009 ), OMM (Petrinja et al. 2009 ), SQO-OSS (Samoladas et al. 2008 ), EFFORT (Aversano and Tortorella 2013 ), Raffoul et al. ( 2008 ), Alfonzo et al. ( 2008 ), Muller ( 2011 ), Adewumi et al. ( 2013 ), Sohn et al. as well as Sarrab and Rehman models ( 2014 ).

Table  6 is a comparative analysis between the OSS quality models presented in Table  3 and the ISO 25010 model. It also features community related characteristics and how they compare with the OSS quality models. Cells marked with ‘x’ indicate that the OSS quality model possesses such characteristic similar to ISO 25010. An empty cell simply means that the OSS quality model does not possess such characteristic as found in ISO 25010.

Figure  1 shows the frequency distribution of the ISO 25010 Product quality characteristics in the OSS quality models we considered. It shows that maintainability is measured by 55% of the existing OSS quality models making it the most common product quality characteristic measured by existing OSS quality models. This is followed by functional suitability, which is measured in 50% of the existing quality models. The least measured are operability, compatibility and transferability that are each measured by 30% of the existing quality models. From Fig.  1 , it can be inferred that the maintainability of a given OSS is of more importance than the functionality it possesses. This is because being an OSS; the code is accessible making it possible to incorporate missing features. However, such missing features can be difficult to implement if the code is not well documented, readable and understandable which are all attributes of maintainable code. Similar inferences can be made as regard the other quality characteristics. For instance, the reliability and security of an OSS can be improved upon if the code is maintainable. In addition, the performance efficiency, operability, compatibility and transferability can all be improved upon with maintainable code.

Frequency distribution of ISO 25010 product quality characteristics in OSS quality models

Figure  2 shows the frequency distribution of the ISO 25010 Quality in Use characteristics in the OSS quality models we considered. It shows that usability is measured by 50% of the existing OSS quality models making it the most commonly measured characteristic in this category. It is followed by effectiveness and efficiency, which are both considered by 15% of the existing OSS quality models. Satisfaction and safety on the other hand are not considered in any of the existing OSS quality models. From Fig.  2 , it can be easily inferred that usability is the most significant attribute under the quality in use category and hence all other attributes in this category add up to define it. In other words, usable OSS is one that is effective in accomplishing specific tasks, efficient in managing system resources, safe for the environment and provides satisfaction to an end-user.

Frequency distribution of ISO 25010 quality in use characteristics in OSS quality models

Figure  3 shows the frequency distribution of community related quality characteristics in the OSS quality models we considered. It shows that maintenance capacity is measured in 45% of the existing OSS quality models making it the most commonly measured attribute in this category. It is closely followed by sustainability that is measured by 40% of the existing OSS quality models. Process maturity is the least measured attribute in this category and is considered in 35% of the existing OSS quality models. It can be inferred from Fig.  3 that evaluators of an OSS quality via its community are mostly interested in the maintenance capacity of such a community in comparison to the sustainability of the community. Also, they are more concerned about the sustainability of the community than the maturity of the community’s processes.

Frequency distribution of community related quality characteristics in OSS quality models

RQ2. What are the methods applied for reaching selection decisions?

Figure  4 depicts the various selection methods adopted in the existing OSS quality models for reaching a decision in the midst of alternatives. The model approach, which entails making system representation that allows for selection based on investigation through a hierarchical structure is the most common selection method used in the existing literature and is used by six (32%) of the existing models. This is followed by the process approach that accounts for use in 21% (four) of the existing models. For the “other” category, three (16%) of the models use a form of guideline in the selection process. Framework approach accounts for 11% while the data mining approach, as well as the tool-based approach both account for 10% each of the existing OSS quality models. In general, it can be observed that more emphasis is placed on non-automated approaches in the existing quality models and so applying these models in real life selection scenarios is usually time-consuming and requires expertise to conduct (Hauge et al. 2009 ; Ali Babar 2010 ).

Selection methods used in OSS quality models

RQ3. What is the domain of application?

Figure  5 depicts the domain of application of the existing OSS quality assessment models. In general, majority of the models do not specify the domain of application. However, for those with specific domain of application, we observed that majority focus on measuring quality in data-dominant software that includes: business-oriented software such as Enterprise Resource Planning and Customer Relationship Management solutions; design and engineering software as well as information display and transaction systems such as issue tracking systems. System software evaluation accounts for 16% while computation-dominant software accounts for 11%.

Domains in which OSS quality models have been applied

Summary and discussion

Principal findings.

From the existing OSS quality models considered in this study, 20% of the models only measure a single quality attribute. Models in this category include: QSOS (which measures maintainability) (Atos 2006 ), Wray and Mathieu (Mathieu and Wray 2007 ) (which measures efficiency), OS-UMM (which measures usability) (Raza et al. 2012 ) and Sudhaman and Thangavel model (which measures efficiency) (Sudhaman and Thangavel 2015 ). Furthermore, 50% of the existing models do not measure community related quality characteristics even though community is what distinguishes OSS from their proprietary counterpart. Models in this category include: QSOS (Atos 2006 ), Sung et al. model ( 2007 ), Raffoul et al. model ( 2008 ), Alfonzo et al. model ( 2008 ), Wray and Mathieu model (Mathieu and Wray 2007 ), Chirila et al. model ( 2011 ), OS-UMM (Raza et al. 2012 ), Sudhaman and Thangavel model ( 2015 ) and Sarrab and Rehman model ( 2014 ). In addition, 35% of the models touch on all categories. They include: OSMM (Duijnhouwer and Widdows 2003 ), Open BRR (Wasserman et al. 2006 ), SQO-OSS (Spinellis et al. 2009 ), EFFORT (Aversano and Tortorella 2013 ), Müller model ( 2011 ) and Sohn et al. model ( 2015 ). Among these models a number of them have been applied to selection scenarios and reported in the literature. A notable example is the EFFORT model, which has been applied to evaluate OSS in the customer relationship management (CRM) domain (Aversano and Tortorella 2011 ) as well as in the enterprise resource-planning (ERP) domain (Aversano and Tortorella 2013 ).

From the existing OSS quality models, it is observed that in the aspect of product quality as defined by ISO 25010, maintainability is the most significant quality characteristic; Usability is the most significant quality in use characteristic in the existing OSS quality models while Maintenance capacity is the most significant community related characteristic in the OSS quality assessment models. Also worthy of note is that satisfaction and safety attributes of quality in use are never considered in the OSS quality models.

The model approach is the most adopted selection method in the existing OSS quality models. The least considered are the tool-based and data mining selection approaches. However, as newer publications emerge we expect to see other approaches and data mining gaining more ground.

Majority (47%) of the existing models do not specify any domain of application. As for those with specific domain of application, a greater percentage focus of data-dominant software especially enterprise resource planning software. Computation-dominant software is the least considered in this regard. Software in this category includes: operations research, information management and manipulation, artistic creativity, scientific software and artificial intelligence software.

From the this study, we also observed that none of the existing models evaluate all the criteria that we laid out, in terms of every quality characteristic under product quality, quality in use, and community related quality characteristics.

Implications of the results

Based on the comparison of the existing quality assessment models, there is clearly no suitable model—each model has its own limitations. As a result, the findings of this analysis have implications especially for practitioners who work towards coming up with new assessment models. They should note the following points in line with the research questions posed in this study:

Emphasis should shift from trying to build comprehensive models (containing all the possible software characteristics) to building models that include only essential quality characteristics. This study has shown that these essential quality characteristics include: maintainability, usability and maintenance capacity of software community. By narrowing down to these three essential quality characteristics, model developers would help to reduce the burden of OSS evaluation via existing quality assessment models, which has been referred to largely as being laborious and time consuming to conduct (Hauge et al. 2009 ; Ali Babar 2010 ).

Newer models should incorporate selection methods that are amenable to automation as this is not the case in most of the existing OSS quality assessment models reviewed in this study. The selection methods mostly adopted are the model (32%), process (21%) and other (16%) such as guidelines, which are not easily amenable to automation (Fahmy et al. 2012 ). Model developers should thus turn their focus to data mining techniques (Leopairote et al. 2013 ), framework or tool-based selection methods, which are currently among the least considered options. The advantage this offers is that it will help quicken the evaluation process resulting in faster decision-making. Following this advice could also bring about increased adoption of the models in practice (Wang et al. 2013 ). In addition, model developers can also consider modeling quality assessment as a multi-criteria decision-making (MCDM) problem so as to facilitate automation as seen in recent studies (Fakir and Canbolat 2008 ; Cavus 2010 , 2011 ). A MCDM problem in this context can be regarded as a process of choosing among available alternatives (i.e. different OSS alternatives) based on a number of attributes (quality criteria). Considering this option opens the model developer to several well-known MCDM methods that amenable to automation such as: DEA, Analytic Hierarchy Process (AHP), and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) to mention a few (Zavadskas et al. 2014 ).

From Fig.  5 , it can be observed that 47% of the quality assessment models considered do not mention the domain of application. This implies that most of the models were designed to be domain-independent. As such, domain-independence should be the focus of model developers (Wagner et al. 2015 ). A domain independent model is one that is able to assess quality in various category of OSS including those that are data-dominant, system software, control-dominant and computation-dominant. It should also be able to this with little or no customization. By following this particular consideration, the model proposed can tend to be widely adopted and possibly standardized.

Threats to validity

Construct threats to validity in this type of study is related to the identification of primary studies. In order to ensure that, as many relevant primary studies as possible were included, different synonyms for ‘open source software’ and ‘quality model’ were included in the search string. The first and second author conducted the automatic search for relevant literature independently and the results obtained were harmonized using a spreadsheet application and duplicates were removed. The reference sections of the selected papers were also scanned to ensure that all relevant references had been included. The final decision to include a study for further consideration depended on the agreement of all the authors. If a disagreement arose, then a discussion took place until consensus was reached.

Internal validity has to do with the data extraction and analysis. As previously mentioned, the first author carried out the data extraction of the primary studies and assigned them to the other authors to assess. The first author also participated in assessing all the primary studies and compared his results with those of the other authors and discrepancies in results were discussed until an agreement was reached. The assignment process of the primary studies to the other authors was not randomized because the sample size (number of primary studies) was relatively small and the time availability of each researcher needed to be considered. In order to properly classify the primary studies based on the quality characteristics they possessed, the authors adopted the ISO 25010 model ( 2001 ) as benchmark. All the authors were fully involved in the process of classifying the primary studies and all disagreements where discussed until a consensus was reached.

To mitigate the effects of incorrect data extraction, which can affect conclusion validity, the steps in the selection and data, extraction activity was clearly described as discussed in the previous paragraphs. The traceability between the data extracted and the conclusions was strengthened through the direct generation of charts and frequency tables from the data by using a statistical package. In our opinion, slight differences based on publication selection bias and misclassification would not alter the main conclusions drawn from the papers identified in this study.

As regards the external validity of this study, the results obtained apply specifically to quality assessment models within the OSS domain. Quality assessment models that evaluate quality in proprietary software are not covered. In addition, the validity of the inferences in this paper only concern OSS quality assessment models. This threat is therefore not present in this context. The results of this study may serve as starting point for OSS quality researchers to further identify and classify newer models in this domain.

Conclusion and future work

The overall goal of this study is to analyze and classify the existing knowledge as regards OSS quality assessment models. Papers dealing with these models were identified between 2003 and 2015. 19 papers were selected. The main publication outlets of the papers identified were journals and conference proceedings. The result of this study shows that maintainability is the most significant and ubiquitous product quality characteristic considered in the literature while usability is the most significant attribute in the quality in use category. Maintenance capacity of an OSS community is also a crucial quality characteristic under community related quality characteristics. The most commonly used selection method is the model approach and the least considered are the tool-based and data mining approaches. Another interesting result is that nearly half (47%) of the selected papers do not mention an application domain for the models in their research. More attention should be paid to building models that incorporate only essential quality characteristics. Also, framework, tool-based and data mining selection methods should be given more attention in future model proposals.

This study could help researchers to identify essential quality attributes with which to develop more robust quality models that are applicable in the various software domains. Also, researchers can compare the existing selection methods in order to determine the most effective. As future work, we intend to model OSS quality assessment as a MCDM problem. This will afford us the opportunity to choose from a range of MCDM methods one (or more) that can be used to evaluate quality in OSS across multiple domains.

Abbreviations

customer relationship management

Data Envelope Analysis

Evaluation Framework for Free/Open souRce projecTs

enterprise resource-planning

multi-criteria decision making

Open Business Readiness Rating

Open Source Maturity Model

Open Source Usability Maturity Model

open source software

quality assessment

Qualification and Selection of Open Source software

research question

Source Quality Observatory for Open Source Software

Technique for Order of Preference by Similarity to Ideal Solution

Adewumi A, Misra S, Omoregbe N (2013a) A review of models for evaluating quality in open source software. IERI Proc 4(1):88–92

Article   Google Scholar  

Adewumi A, Omoregbe N, Misra S (2013) Quantitative quality model for evaluating open source web applications: case study of repository software. In: 16th International conference on computational science and engineering (CSE), Dec 3 2013

Alfonzo O, Domínguez K, Rivas L, Perez M, Mendoza L, Ortega M (2008) Quality measurement model for analysis and design tools based on FLOSS. In: 19th Australian conference on software engineering, Perth, Australia, 26–28 March 2008

Atos (2006), Method for qualification and selection of open source software (QSOS) version 2.0. http://backend.qsos.org/download/qsos-2.0_en.pdf . Accessed 5 Jan 2015

Aversano L, Tortorella M (2011) Applying EFFORT for evaluating CRM open source systems. In: International conference on product-focused software process improvement, Springer, Heidelberg, pp 202–216

Aversano L, Tortorella M (2013) Quality evaluation of FLOSS projects: application to ERP systems. Inf Softw Technol 55(7):1260–1276

Brereton OP, Kitchenham BA, Budgen DT, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80:571–583

Cavus N (2010) The evaluation of learning management systems using an artificial intelligence fuzzy logic algorithm. Adv Eng Softw 41:248–254

Article   MATH   Google Scholar  

Cavus N (2011) The application of a multi-attribute decision-making algorithm to learning management systems evaluation. Br J Edu Technol 42:19–30

Chirila C, Juratoni D, Tudor D, Cretu V (2011) Towards a software quality assessment model based on open-source statical code analyzers. In: 6th IEEE international conference on computational intelligence and informatics (SACI), May 19 2011

Deissenboeck F, Juergens E, Lochman K, Wagner S (2009) Software quality models: purposes, usage scenarios and requirements. In: ICSE workshop on software quality, May 16 2009

Del Bianco V, Lavazza L, Morasca S, Taibi D, Tosi D (2010a) The QualiSPo approach to OSS product quality evaluation. In: 3rd International workshop on emerging trends in free/libre/open source software research and development, New York

Del Bianco V, Lavazza L, Morasca S, Taibi D, Tosi D (2010b) An investigation of the users’ perception of OSS quality. In: 6th International conference on open source systems, Springer Verlag, pp 15–28

Del Bianco V, Lavazza L, Morasca S, Taibi D (2011) A survey on open source software trustworthiness. IEEE Softw 28(5):67–75

Deprez JC, Alexandre S (2008) Comparing assessment methodologies for free/open source software: OpenBRR and QSOS. In: 9th international conference on product-focused software process improvement (PROFES‘08), Springer, Heidelberg, pp 189–203

Duijnhouwer F, Widdows C (2003) Open source maturity model. http://jose-manuel.me/thesis/references/GB_Expert_Letter_Open_Source_Maturity_Model_1.5.3.pdf Accessed: 5 Jan 2015

Fahmy S, Haslinda N, Roslina W, Fariha Z (2012) Evaluating the quality of software in e-book using the ISO 9126 model. Int J Control Autom 5:115–122

Google Scholar  

Fakir O, Canbolat MS (2008) A web-based decision support system for multi-criteria inventory classification using fuzzy AHP methodology. Expert Syst Appl 35:1367–1378

Forward A, Lethbridge TC (2008) A taxonomy of software types to facilitate search and evidence-based software engineering. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research, Oct 27 2008

Haaland K, Groven AK, Regnesentral N, Glott R, Tannenberg A, FreeCode AS (2010) Free/libre open source quality models—a comparison between two approaches. In: 4th FLOS international workshop on Free/Libre/Open Source Software, July 2010

Hauge Ø, Østerlie T, Sørensen CF, Gerea M (2009) An empirical study on selection of open source software—preliminary results. In: ICSE workshop on emerging trends in free/libre/open source software research and development, May 18 2009

ISO/IEC 9126 (2001) Software engineering—product quality—part 1: quality model. http://www.iso.org/iso/catalogue_detail.htm?csnumber=22749 Accessed 14 Nov 2015

ISO/IEC 25010 (2010) Systems and software engineering—systems and software product quality requirements and evaluation (SQuaRE)—system and software quality models. http://www.iso.org/iso/catalogue_detail.htm?csnumber=35733 Accessed 14 Oct 2016

Kitchenham BA (2004) Procedures for undertaking systematic reviews. http://csnotes.upm.edu.my/kelasmaya/pgkm20910.nsf/0/715071a8011d4c2f482577a700386d3a/$FILE/10.1.1.122.3308[1].pdf . Accessed 14 Oct 2016

Kuwata Y, Takeda K, Miura H (2014) A study on maturity model of open source software community to estimate the quality of products. Proc Comput Sci 35:1711–1717

Leopairote W, Surarerks A, Prompoon N (2013) Evaluating software quality in use using user reviews mining. In: 10th International joint conference on computer science and software engineering, May 29 2013

Mathieu R, Wray B (2007) The application of DEA to measure the efficiency of open source security tool production. In: AMCIS 2007 proceedings, Dec 31 2007

Miguel JP, Mauricio D, Rodríguez G (2014) A review of software quality models for the evaluation of software products. Int J Soft Eng Appl 5(6):31–53

Müller T (2011) How to choose an free and open source integrated library system. Int Digi Lib Perspect 27(1):57–78

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2014) Evaluating software product quality: a systematic mapping study. In: International conference on software process and product measurement, Oct 6 2014

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2015) Predicting software product quality: a systematic mapping study. Computación y Sistemas 19(3):547–562

Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: 12th International conference on evaluation and assessment in software engineering, Blekinge Institute of Technology, Italy, Jun 26 2008

Petrinja E, Nambakam R, Sillitti A (2009) Introducing the open source maturity model. In: Proceedings of the 2009 ICSE workshop on emerging trends in free/libre/open source software research and development, May 18 2009

Raffoul E, Domínguez K, Perez M, Mendoza LE, Griman AC (2008) Quality model for the selection of FLOSS-based Issue tracking system. In: Proceedings of the IASTED international conference on software engineering, Innsbruck, Austria, 12 Feb 2008

Raza A, Capretz LF, Ahmed F (2012) An open source usability maturity model (OS-UMM). Comput Hum Behav 28(4):1109–1121

Samoladas I, Gousios G, Spinellis D, Stamelos I (2008) The SQO-OSS quality model: measurement based open source software evaluation. In: IFIP International Conference on Open Source Systems. Springer, Milano, pp 237–248

Sarrab M, Rehman OMH (2014) Empirical study of open source software selection for adoption, based on software quality characteristics. Adv Eng Softw 69:1–11

Sohn H, Lee M, Seong B, Kim J (2015) Quality evaluation criteria based on open source mobile HTML5 UI framework for development of cross-platform. Int J Soft Eng Appl 9(6):1–12

Soto M, Ciolkowski M (2009) The QualOSS open source assessment model measuring the performance of open source communities. In: Proceedings of the 3rd international symposium on empirical software engineering and measurement, 15 Oct 2009

Spinellis D, Gousios G, Karakoidas V, Louridas P, Adams PJ, Samoladas I, Stamelos I (2009) Evaluating the quality of open source software. Elect Notes Theor Comp Sci 233:5–28

Stol KJ, Ali Babar, M (2010) Challenges in using open source software in product development: a review of the literature. In: Proceedings of the 3rd international workshop on emerging trends in free/libre/open source software research and development, May 8 2010

Sudhaman P, Thangavel C (2015) Efficiency analysis of ERP projects—software quality perspective. Int J of Proj Manag 33:961–970

Sung WJ, Kim JH, Rhew SY (2007) A quality model for open source software selection. In: Sixth international conference on advanced language processing and web information technology, 22 Aug 2007

Wagner S, Goeb A, Heinemann L, Kläs M, Lampasona C, Lochmann K, Mayr A, Plösch R, Seidl A, Streit J, Trendowicz A (2015) Operationalised product quality models and assessment: the Quamoco approach. Inf and Soft Tech 62:101–123

Wang D, Zhu S, Li T (2013) SumView: a web-based engine for summarizing product reviews and customer opinions. Expert Syst Appl 40:27–33

Wasserman AI, Pal M, Chan C (2006) Business readiness rating for open source. In: Proceedings of the EFOSS Workshop, Como, Italy, 8 Jun 2006

Wen J, Li S, Lin Z, Hu Y, Huang C (2012) Systematic literature review of machine learning based software development effort estimation models. Inf Softw Technol 54(1):41–59

Zavadskas EK, Turskis Z, Kildienė S (2014) State of art surveys of overviews on MCDM/MADM methods. Technol Econ Dev Econ 20:165–179

Download references

Authors’ contributions

AA is a Ph.D. student and has done a significant part of the work under the supervision of SM. SM—is main supervisor of AA and working with him since last 4 years for completion of the work. NO is co-supervisor of AA and provided his continuous guidance in completion of the work. BC and RS—are co-researchers with our software engineering cluster in CU. They both contributed a lot for improving the manuscript (reviewed and added valuable contributions) since the beginning of the work. All authors read and approved the final manuscript.

Acknowledgements

We are thankful to Dr. Olawande Daramola of Computer and Information Science Department for his valuable suggestions and comments for improvement of the work/paper.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

The dataset(s) supporting the conclusions of this article is included within the article in Tables  3 , 5 and 6 .

Author information

Authors and affiliations.

Covenant University, Ota, Nigeria

Adewole Adewumi, Sanjay Misra & Nicholas Omoregbe

Atilim University, Ankara, Turkey

Sanjay Misra

Pontificia Universidad Católica de Valparaíso, Valparaiso, Chile

Broderick Crawford & Ricardo Soto

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sanjay Misra .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Adewumi, A., Misra, S., Omoregbe, N. et al. A systematic literature review of open source software quality assessment models. SpringerPlus 5 , 1936 (2016). https://doi.org/10.1186/s40064-016-3612-4

Download citation

Received : 17 May 2016

Accepted : 27 October 2016

Published : 08 November 2016

DOI : https://doi.org/10.1186/s40064-016-3612-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Open source software
• Quality assessment models

• Oracle Mode
• Oracle Mode – Advanced
• Exploration Mode
• Simulation Mode
• Simulation Infrastructure

Join the movement towards fast, open, and transparent systematic reviews

ASReview LAB v1.5 is out!

By loading the video, you agree to YouTube's privacy policy. Learn more

Always unblock YouTube

ASReview uses state-of-the-art active learning techniques to solve one of the most interesting challenges in systematically screening large amounts of text : there’s not enough time to read everything!  

The project has grown into a vivid worldwide community of researchers, users, and developers. ASReview is coordinated at Utrecht University and is part of the official AI-labs at the university.

Free, Open and Transparent

The software is installed on your device locally. This ensures that nobody else has access to your data, except when you share it with others. Nice, isn’t it?

• Free and open source
• Local or server installation
• Full control over your data
• Follows the Reproducibility and Data Storage Checklist for AI-Aided Systematic Reviews

In 2 minutes up and running

With the smart project setup features, you can start a new project in minutes. Ready, set, start screening!

• Create as many projects as you want
• Choose your own or an existing dataset
• Select prior knowledge
• Select your favorite active learning algorithm

Three modi to choose from

ASReview LAB can be used for:

• Screening with the Oracle Mode , including advanced options
• Teaching using the Exploration Mode
• Validating algorithms using the Simulation Mode

We also offer an open-source research infrastructure to run large-scale simulation studies for validating newly developed AI algorithms.

Follow the development

Open-source means:

• All annotated source code is available 
• You can see the developers at work in open Pull Requests
• Open Pull Request show in what direction the project is developing
• Anyone can contribute!

Give a GitHub repo a star if you like our work.

Join the community

A community-driven project means:

• The project is a joined endeavor  
• Your contribution matters!

Join the movement towards transparent AI-aided reviewing

Beginner -> User -> Developer -> Maintainer

Organizations

Github stars

Join the ASReview Development Fund

Many users donate their time to continue the development of the different software tools that are part of the ASReview universe. Also, donations and research grants make innovations possible!

Navigating the Maze of Models in ASReview

Starting a systematic review can feel like navigating through a maze, with countless articles and endless…

ASReview LAB Class 101

ASReview LAB Class 101 Welcome to ASReview LAB class 101, an introduction to the most important…

Introducing the Noisy Label Filter (NLF) procedure in systematic reviews

The ASReview team developed a procedure to overcome replication issues in creating a dataset for simulation…

Seven ways to integrate ASReview in your systematic review workflow

Seven ways to integrate ASReview in your systematic review workflow Systematic reviewing using software implementing Active…

Active Learning Explained

Active Learning Explained The rapidly evolving field of artificial intelligence (AI) has allowed the development of…

the Zen of Elas

The Zen of Elas Elas is the Mascotte of ASReview and your Electronic Learning Assistant who…

Five ways to get involved in ASReview

Five ways to get involved in ASReview ASReview LAB is open and free (Libre) software, maintained…

Connecting RIS import to export functionalities

What’s new in v0.19? Connecting RIS import to export functionalities Download ASReview LAB 0.19Update to ASReview…

Meet the new ASReview Maintainer: Yongchao Ma

Meet Front-End Developer and ASReview Maintainer Yongchao Ma As a user of ASReview, you are probably…

UPDATED: ASReview Hackathon for Follow the Money

This event has passed The winners of the hackathon were: Data pre-processing: Teamwork by: Raymon van…

What’s new in release 0.18?

More models more options, available now! Version 0.18 slowly opens ways to the soon to be…

Simulation Mode Class 101

Simulation Mode Class 101 Have you ever done a systematic review manually, but wondered how much…

• DOI: 10.7275/JJHZ-SZ75
• Corpus ID: 53608075

How to Perform a Literature Review with Free and Open Source Software

• Joshua M. Pearce
• Published 8 May 2018
• Computer Science
• IRPN: Innovation Policy Studies (Topic)

13 Citations

How to conduct systematic literature reviews in management research: a guide in 6 steps and 14 decisions.

• Highly Influenced

The Rise of Platinum Open Access Journals with Both Impact Factors and Zero Article Processing Charges

Does open access citation advantage depend on paper topics, free and open source automated open access preprint harvesting, from open access to open science: the path from scientific reality to open scientific communication, how do self-archiving and author-pays models associate and contribute to oa citation advantage within hybrid journals, awareness, knowledge, and utilisation of online digital tools for literature review in educational research, analysis of free and open source software (foss) product in web based client-server architecture, design and biomimicry: a review of interconnections and creative potentials, a systematic approach to beginning a retrospective research study: a guide for surgical trainees, 77 references, performing systematic literature reviews in software engineering, article review of randolph (2006) ‘a guide to writing the dissertation literature review, open source database and website to provide free and open access to inactive u.s. patents in the public domain, bibliographic database searching by graduate students in language and literature: search strategies, system interfaces, and relevance judgments., how open source software works: 'free' user-to-user assistance.

• Highly Influential
• 16 Excerpts

A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research

The state of oa: a large-scale analysis of the prevalence and impact of open access articles, open source research in sustainability, do open-access articles have a greater research impact, organizing knowledge syntheses: a taxonomy of literature reviews, related papers.

Showing 1 through 3 of 0 Related Papers

Something went wrong when searching for seed articles. Please try again soon.

No articles were found for that search term.

Author, year The title of the article goes here

LITERATURE REVIEW SOFTWARE FOR BETTER RESEARCH

“Litmaps is a game changer for finding novel literature... it has been invaluable for my productivity.... I also got my PhD student to use it and they also found it invaluable, finding several gaps they missed”

Varun Venkatesh

Austin Health, Australia

As a full-time researcher, Litmaps has become an indispensable tool in my arsenal. The Seed Maps and Discover features of Litmaps have transformed my literature review process, streamlining the identification of key citations while revealing previously overlooked relevant literature, ensuring no crucial connection goes unnoticed. A true game-changer indeed!

Ritwik Pandey

Doctoral Research Scholar – Sri Sathya Sai Institute of Higher Learning

Using Litmaps for my research papers has significantly improved my workflow. Typically, I start with a single paper related to my topic. Whenever I find an interesting work, I add it to my search. From there, I can quickly cover my entire Related Work section.

David Fischer

Research Associate – University of Applied Sciences Kempten

“It's nice to get a quick overview of related literature. Really easy to use, and it helps getting on top of the often complicated structures of referencing”

Christoph Ludwig

Technische Universität Dresden, Germany

“This has helped me so much in researching the literature. Currently, I am beginning to investigate new fields and this has helped me hugely”

Aran Warren

Canterbury University, NZ

“I can’t live without you anymore! I also recommend you to my students.”

Professor at The Chinese University of Hong Kong

“Seeing my literature list as a network enhances my thinking process!”

Katholieke Universiteit Leuven, Belgium

“Incredibly useful tool to get to know more literature, and to gain insight in existing research”

KU Leuven, Belgium

“As a student just venturing into the world of lit reviews, this is a tool that is outstanding and helping me find deeper results for my work.”

Franklin Jeffers

South Oregon University, USA

“Any researcher could use it! The paper recommendations are great for anyone and everyone”

Swansea University, Wales

“This tool really helped me to create good bibtex references for my research papers”

Ali Mohammed-Djafari

Director of Research at LSS-CNRS, France

“Litmaps is extremely helpful with my research. It helps me organize each one of my projects and see how they relate to each other, as well as to keep up to date on publications done in my field”

Daniel Fuller

Clarkson University, USA

As a person who is an early researcher and identifies as dyslexic, I can say that having research articles laid out in the date vs cite graph format is much more approachable than looking at a standard database interface. I feel that the maps Litmaps offers lower the barrier of entry for researchers by giving them the connections between articles spaced out visually. This helps me orientate where a paper is in the history of a field. Thus, new researchers can look at one of Litmap's "seed maps" and have the same information as hours of digging through a database.

Baylor Fain

Postdoctoral Associate – University of Florida

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

5 literature review tools to ace your research (+2 bonus tools)

Table of Contents

Your literature review is the lore behind your research paper . It comes in two forms, systematic and scoping , both serving the purpose of rounding up previously published works in your research area that led you to write and finish your own.

A literature review is vital as it provides the reader with a critical overview of the existing body of knowledge, your methodology, and an opportunity for research applications.

Some steps to follow while writing your review:

• Pick an accessible topic for your paper
• Do thorough research and gather evidence surrounding your topic
• Read and take notes diligently
• Create a rough structure for your review
• Synthesis your notes and write the first draft
• Edit and proofread your literature review

To make your workload a little lighter, there are many literature review AI tools. These tools can help you find academic articles through AI and answer questions about a research paper.  

Best literature review tools to improve research workflow

A literature review is one of the most critical yet tedious stages in composing a research paper. Many students find it an uphill task since it requires extensive reading and careful organization .

Using some of the best literature review tools listed here, you can make your life easier by overcoming some of the existing challenges in literature reviews. From collecting and classifying to analyzing and publishing research outputs, these tools help you with your literature review and improve your productivity without additional effort or expenses.

1. SciSpace

SciSpace is an AI for academic research that will help find research papers and answer questions about a research paper. You can discover, read, and understand research papers with SciSpace making it an excellent platform for literature review. Featuring a repository with over 270 million research papers, it comes with your AI research assistant called Copilot that offers explanations, summaries , and answers as you read.

Get started now:

Find academic articles through AI

SciSpace has a dedicated literature review tool that finds scientific articles when you search for a question. Based on semantic search, it shows all the research papers relevant for your subject. You can then gather quick insights for all the papers displayed in your search results like methodology, dataset, etc., and figure out all the papers relevant for your research.

Identify relevant articles faster

Abstracts are not always enough to determine whether a paper is relevant to your research question. For starters, you can ask questions to your AI research assistant, SciSpace Copilot to explore the content and better understand the article. Additionally, use the summarize feature to quickly review the methodology and results of a paper and decide if it is worth reading in detail.

Learn in your preferred language

A big barrier non-native English speakers face while conducting a literature review is that a significant portion of scientific literature is published in English. But with SciSpace Copilot, you can review, interact, and learn from research papers in any language you prefer — presently, it supports 75+ languages. The AI will answer questions about a research paper in your mother tongue.

Integrates with Zotero

Many researchers use Zotero to create a library and manage research papers. SciSpace lets you import your scientific articles directly from Zotero into your SciSpace library and use Copilot to comprehend your research papers. You can also highlight key sections, add notes to the PDF as you read, and even turn helpful explanations and answers from Copilot into notes for future review.

Understand math and complex concepts quickly

Come across complex mathematical equations or difficult concepts? Simply highlight the text or select the formula or table, and Copilot will provide an explanation or breakdown of the same in an easy-to-understand manner. You can ask follow-up questions if you need further clarification.

Discover new papers to read without leaving

Highlight phrases or sentences in your research paper to get suggestions for related papers in the field and save time on literature reviews. You can also use the 'Trace' feature to move across and discover connected papers, authors, topics, and more.

SciSpace Copilot is now available as a Chrome extension , allowing you to access its features directly while you browse scientific literature anywhere across the web.

Get citation-backed answers

When you're conducting a literature review, you want credible information with proper references.  Copilot ensures that every piece of information provided by SciSpace Copilot is backed by a direct reference, boosting transparency, accuracy, and trustworthiness.

Ask a question related to the paper you're delving into. Every response from Copilot comes with a clickable citation. This citation leads you straight to the section of the PDF from which the answer was extracted.

By seamlessly integrating answers with citations, SciSpace Copilot assures you of the authenticity and relevance of the information you receive.

2. Mendeley

Mendeley Citation Manager is a free web and desktop application. It helps simplify your citation management workflow significantly. Here are some ways you can speed up your referencing game with Mendeley.

Generate citations and bibliographies

Easily add references from your Mendeley library to your Word document, change your citation style, and create a bibliography, all without leaving your document.

Retrieve references

It allows you to access your references quickly. Search for a term, and it will return results by referencing the year, author, or source.

Add sources to your Mendeley library by dragging PDF to Mendeley Reference Manager. Mendeley will automatically remove the PDF(s) metadata and create a library entry.‌

Read and annotate documents

It helps you highlight and comment across multiple PDFs while keep them all in one place using Mendeley Notebook . Notebook pages are not tied to a reference and let you quote from many PDFs.

A big part of many literature review workflows, Zotero is a free, open-source tool for managing citations that works as a plug-in on your browser. It helps you gather the information you need, cite your sources, lets you attach PDFs, notes, and images to your citations, and create bibliographies.

Import research articles to your database

Search for research articles on a keyword, and add relevant results to your database. Then, select the articles you are most interested in, and import them into Zotero.

Add bibliography in a variety of formats

With Zotero, you don’t have to scramble for different bibliography formats. Simply use the Zotero-Word plug-in to insert in-text citations and generate a bibliography.

Share your research

You can save a paper and sync it with an online library to easily share your research for group projects. Zotero can be used to create your database and decrease the time you spend formatting citations.

Sysrev is an AI too for article review that facilitates screening, collaboration, and data extraction from academic publications, abstracts, and PDF documents using machine learning. The platform is free and supports public and Open Access projects only.

Some of the features of Sysrev include:

Group labels

Group labels can be a powerful concept for creating database tables from documents. When exported and re-imported, each group label creates a new table. To make labels for a project, go into the manage -> labels section of the project.

Group labels enable project managers to pull table information from documents. It makes it easier to communicate review results for specific articles.

Track reviewer performance

Sysrev's label counting tool provides filtering and visualization options for keeping track of the distribution of labels throughout the project's progress. Project managers can check their projects at any point to track progress and the reviewer's performance.

Tool for concordance

The Sysrev tool for concordance allows project administrators and reviewers to perform analysis on their labels. Concordance is measured by calculating the number of times users agree on the labels they have extracted.

Colandr is a free, open-source, internet-based analysis and screening software used as an AI for academic research. It was designed to ease collaboration across various stages of the systematic review procedure. The tool can be a little complex to use. So, here are the steps involved in working with Colandr.

Create a review

The first step to using Colandr is setting up an organized review project. This is helpful to librarians who are assisting researchers with systematic reviews.

The planning stage is setting the review's objectives along with research queries. Any reviewer can review the details of the planning stage. However, they can only be modified by the author for the review.

Citation screening/import

In this phase, users can upload their results from database searches. Colandr also offers an automated deduplication system.

Full-text screening

The system in Colandr will discover the combination of terms and expressions that are most useful for the reader. If an article is selected, it will be moved to the final step.

Data extraction/export

Colandr data extraction is more efficient than the manual method. It creates the form fields for data extraction during the planning stage of the review procedure. Users can decide to revisit or modify the form for data extraction after completing the initial screening.

Bonus literature review tools

SRDR+ is a web-based tool for extracting and managing systematic review or meta-analysis data. It is open and has a searchable archive of systematic reviews and their data.

7. Plot Digitizer

Plot Digitizer is an efficient tool for extracting information from graphs and images, equipped with many features that facilitate data extraction. The program comes with a free online application, which is adequate to extract data quickly.

Final thoughts

Writing a literature review is not easy. It’s a time-consuming process, which can become tiring at times. The literature review tools mentioned in this blog do an excellent job of maximizing your efforts and helping you write literature reviews much more efficiently. With them, you can breathe a sigh of relief and give more time to your research.

As you dive into your literature review, don’t forget to use SciSpace ResearchGPT to streamline the process. It facilitates your research and helps you explore key findings, summary, and other components of the paper easily.

Frequently Asked Questions (FAQs)

1. what is rrl in research.

RRL stands for Review of Related Literature and sometimes interchanged with ‘Literature Review.’ RRL is a body of studies relevant to the topic being researched. These studies may be in the form of journal articles, books, reports, and other similar documents. Review of related literature is used to support an argument or theory being made by the researcher, as well as to provide information on how others have approached the same topic.

2. What are few softwares and tools available for literature review?

• SciSpace Discover

• Mendeley

• Zotero

• Sysrev

• Colandr

• SRDR+

3. How to generate an online literature review?

The Scispace Discover tool, which offers an excellent repository of millions of peer-reviewed articles and resources, will help you generate or create a literature review easily. You may find relevant information by utilizing the filter option, checking its credibility, tracing related topics and articles, and citing in widely accepted formats with a single click.

4. What does it mean to synthesize literature?

To synthesize literature is to take the main points and ideas from a number of sources and present them in a new way. The goal is to create a new piece of writing that pulls together the most important elements of all the sources you read. Make recommendations based on them, and connect them to the research.

5. Should we write abstract for literature review?

Abstracts, particularly for the literature review section, are not required. However, an abstract for the research paper, on the whole, is useful for summarizing the paper and letting readers know what to expect from it. It can also be used to summarize the main points of the paper so that readers have a better understanding of the paper's content before they read it.

6. How do you evaluate the quality of a literature review?

• Whether it is clear and well-written.

• Whether Information is current and up to date.

• Does it cover all of the relevant sources on the topic.

• Does it provide enough evidence to support its conclusions.

7. Is literature review mandatory?

Yes. Literature review is a mandatory part of any research project. It is a critical step in the process that allows you to establish the scope of your research and provide a background for the rest of your work.

8. What are the sources for a literature review?

• Reports

• Theses

• Conference proceedings

• Company reports

• Some government publications

• Journals

• Books

• Newspapers

• Articles by professional associations

• Indexes

• Databases

• Catalogues

• Encyclopaedias

• Dictionaries

• Bibliographies

• Citation indexes

• Statistical data from government websites

9. What is the difference between a systematic review and a literature review?

A systematic review is a form of research that uses a rigorous method to generate knowledge from both published and unpublished data. A literature review, on the other hand, is a critical summary of an area of research within the context of what has already been published.

Suggested reads!

Types of essays in academic writing Citation Machine Alternatives — A comparison of top citation tools 2023

QuillBot vs SciSpace: Choose the best AI-paraphrasing tool

ChatPDF vs. SciSpace Copilot: Unveiling the best tool for your research

You might also like

Consensus GPT vs. SciSpace GPT: Choose the Best GPT for Research

Literature Review and Theoretical Framework: Understanding the Differences

Types of Essays in Academic Writing - Quick Guide (2024)

10 Best Literature Review Tools for Researchers

Boost your research game with these Best Literature Review Tools for Researchers! Uncover hidden gems, organize your findings, and ace your next research paper!

Researchers struggle to identify key sources, extract relevant information, and maintain accuracy while manually conducting literature reviews. This leads to inefficiency, errors, and difficulty in identifying gaps or trends in existing literature.

Table of Contents

Top 10 Literature Review Tools for Researchers: In A Nutshell (2023)

#1. Semantic Scholar – A free, AI-powered research tool for scientific literature

By employing semantic analysis, users can explore scholarly articles based on context and meaning, making it a go-to resource for scholars across disciplines. 

Not all scholarly content may be indexed, and occasional false positives or inaccurate associations can occur. Furthermore, the tool primarily focuses on computer science and related fields, potentially limiting coverage in other disciplines. 

#2. Elicit – Research assistant using language models like GPT-3

Elicit is a game-changing literature review tool that has gained popularity among researchers worldwide. With its user-friendly interface and extensive database of scholarly articles, it streamlines the research process, saving time and effort. 

However, users should be cautious when using Elicit. It is important to verify the credibility and accuracy of the sources found through the tool, as the database encompasses a wide range of publications. 

Additionally, occasional glitches in the search function have been reported, leading to incomplete or inaccurate results. While Elicit offers tremendous benefits, researchers should remain vigilant and cross-reference information to ensure a comprehensive literature review.

#3. Scite.Ai – Your personal research assistant

Scite.Ai is a popular literature review tool that revolutionizes the research process for scholars. With its innovative citation analysis feature, researchers can evaluate the credibility and impact of scientific articles, making informed decisions about their inclusion in their own work. 

However, while Scite.Ai offers numerous advantages, there are a few aspects to be cautious about. As with any data-driven tool, occasional errors or inaccuracies may arise, necessitating researchers to cross-reference and verify results with other reputable sources. 

Rayyan offers the following paid plans:

#4. DistillerSR – Literature Review Software

Despite occasional technical glitches reported by some users, the developers actively address these issues through updates and improvements, ensuring a better user experience. 

#5. Rayyan – AI Powered Tool for Systematic Literature Reviews

However, it’s important to be aware of a few aspects. The free version of Rayyan has limitations, and upgrading to a premium subscription may be necessary for additional functionalities. 

#6. Consensus – Use AI to find you answers in scientific research

With Consensus, researchers can save significant time by efficiently organizing and accessing relevant research material.People consider Consensus for several reasons. 

Consensus offers both free and paid plans:

#7. RAx – AI-powered reading assistant

#8. lateral – advance your research with ai.

Additionally, researchers must be mindful of potential biases introduced by the tool’s algorithms and should critically evaluate and interpret the results. 

#9. Iris AI – Introducing the researcher workspace

Researchers are drawn to this tool because it saves valuable time by automating the tedious task of literature review and provides comprehensive coverage across multiple disciplines. 

#10. Scholarcy – Summarize your literature through AI

Scholarcy’s ability to extract key information and generate concise summaries makes it an attractive option for scholars looking to quickly grasp the main concepts and findings of multiple papers.

Scholarcy’s automated summarization may not capture the nuanced interpretations or contextual information presented in the full text. 

Final Thoughts

In conclusion, conducting a comprehensive literature review is a crucial aspect of any research project, and the availability of reliable and efficient tools can greatly facilitate this process for researchers. This article has explored the top 10 literature review tools that have gained popularity among researchers.

Q1. What are literature review tools for researchers?

Q2. what criteria should researchers consider when choosing literature review tools.

When choosing literature review tools, researchers should consider factors such as the tool’s search capabilities, database coverage, user interface, collaboration features, citation management, annotation and highlighting options, integration with reference management software, and data extraction capabilities. 

Q3. Are there any literature review tools specifically designed for systematic reviews or meta-analyses?

Meta-analysis support: Some literature review tools include statistical analysis features that assist in conducting meta-analyses. These features can help calculate effect sizes, perform statistical tests, and generate forest plots or other visual representations of the meta-analytic results.

Q4. Can literature review tools help with organizing and annotating collected references?

Integration with citation managers: Some literature review tools integrate with popular citation managers like Zotero, Mendeley, or EndNote, allowing seamless transfer of references and annotations between platforms.

By leveraging these features, researchers can streamline the organization and annotation of their collected references, making it easier to retrieve relevant information during the literature review process.

Leave a Comment Cancel reply

All-in-one Literature Review Software

Start your free trial.

Free MAXQDA trial for Windows and Mac

Your trial will end automatically after 14 days and will not renew. There is no need for cancelation.

MAXQDA The All-in-one Literature Review Software

MAXQDA is the best choice for a comprehensive literature review. It works with a wide range of data types and offers powerful tools for literature review, such as reference management, qualitative, vocabulary, text analysis tools, and more.

Document viewer

Your analysis.

As your all-in-one literature review software, MAXQDA can be used to manage your entire research project. Easily import data from texts, interviews, focus groups, PDFs, web pages, spreadsheets, articles, e-books, and even social media data. Connect the reference management system of your choice with MAXQDA to easily import bibliographic data. Organize your data in groups, link relevant quotes to each other, keep track of your literature summaries, and share and compare work with your team members. Your project file stays flexible and you can expand and refine your category system as you go to suit your research.

Developed by and for researchers – since 1989

Having used several qualitative data analysis software programs, there is no doubt in my mind that MAXQDA has advantages over all the others. In addition to its remarkable analytical features for harnessing data, MAXQDA’s stellar customer service, online tutorials, and global learning community make it a user friendly and top-notch product.

Sally S. Cohen – NYU Rory Meyers College of Nursing

Literature Review is Faster and Smarter with MAXQDA

Easily import your literature review data

With a literature review software like MAXQDA, you can easily import bibliographic data from reference management programs for your literature review. MAXQDA can work with all reference management programs that can export their databases in RIS-format which is a standard format for bibliographic information. Like MAXQDA, these reference managers use project files, containing all collected bibliographic information, such as author, title, links to websites, keywords, abstracts, and other information. In addition, you can easily import the corresponding full texts. Upon import, all documents will be automatically pre-coded to facilitate your literature review at a later stage.

Capture your ideas while analyzing your literature

Great ideas will often occur to you while you’re doing your literature review. Using MAXQDA as your literature review software, you can create memos to store your ideas, such as research questions and objectives, or you can use memos for paraphrasing passages into your own words. By attaching memos like post-it notes to text passages, texts, document groups, images, audio/video clips, and of course codes, you can easily retrieve them at a later stage. Particularly useful for literature reviews are free memos written during the course of work from which passages can be copied and inserted into the final text.

Find concepts important to your generated literature review

When generating a literature review you might need to analyze a large amount of text. Luckily MAXQDA as the #1 literature review software offers Text Search tools that allow you to explore your documents without reading or coding them first. Automatically search for keywords (or dictionaries of keywords), such as important concepts for your literature review, and automatically code them with just a few clicks. Document variables that were automatically created during the import of your bibliographic information can be used for searching and retrieving certain text segments. MAXQDA’s powerful Coding Query allows you to analyze the combination of activated codes in different ways.

Aggregate your literature review

When conducting a literature review you can easily get lost. But with MAXQDA as your literature review software, you will never lose track of the bigger picture. Among other tools, MAXQDA’s overview and summary tables are especially useful for aggregating your literature review results. MAXQDA offers overview tables for almost everything, codes, memos, coded segments, links, and so on. With MAXQDA literature review tools you can create compressed summaries of sources that can be effectively compared and represented, and with just one click you can easily export your overview and summary tables and integrate them into your literature review report.

Powerful and easy-to-use literature review tools

Quantitative aspects can also be relevant when conducting a literature review analysis. Using MAXQDA as your literature review software enables you to employ a vast range of procedures for the quantitative evaluation of your material. You can sort sources according to document variables, compare amounts with frequency tables and charts, and much more. Make sure you don’t miss the word frequency tools of MAXQDA’s add-on module for quantitative content analysis. Included are tools for visual text exploration, content analysis, vocabulary analysis, dictionary-based analysis, and more that facilitate the quantitative analysis of terms and their semantic contexts.

Visualize your literature review

As an all-in-one literature review software, MAXQDA offers a variety of visual tools that are tailor-made for qualitative research and literature reviews. Create stunning visualizations to analyze your material. Of course, you can export your visualizations in various formats to enrich your literature review analysis report. Work with word clouds to explore the central themes of a text and key terms that are used, create charts to easily compare the occurrences of concepts and important keywords, or make use of the graphical representation possibilities of MAXMaps, which in particular permit the creation of concept maps. Thanks to the interactive connection between your visualizations with your MAXQDA data, you’ll never lose sight of the big picture.

AI Assist: literature review software meets AI

AI Assist – your virtual research assistant – supports your literature review with various tools. AI Assist simplifies your work by automatically analyzing and summarizing elements of your research project and by generating suggestions for subcodes. No matter which AI tool you use – you can customize your results to suit your needs.

Free tutorials and guides on literature review

MAXQDA offers a variety of free learning resources for literature review, making it easy for both beginners and advanced users to learn how to use the software. From free video tutorials and webinars to step-by-step guides and sample projects, these resources provide a wealth of information to help you understand the features and functionality of MAXQDA for literature review. For beginners, the software’s user-friendly interface and comprehensive help center make it easy to get started with your data analysis, while advanced users will appreciate the detailed guides and tutorials that cover more complex features and techniques. Whether you’re just starting out or are an experienced researcher, MAXQDA’s free learning resources will help you get the most out of your literature review.

Free MAXQDA Trial for Windows and Mac

Get your maxqda license, compare the features of maxqda and maxqda analytics pro, faq: literature review software.

Literature review software is a tool designed to help researchers efficiently manage and analyze the existing body of literature relevant to their research topic. MAXQDA, a versatile qualitative data analysis tool, can be instrumental in this process.

Literature review software, like MAXQDA, typically includes features such as data import and organization, coding and categorization, advanced search capabilities, data visualization tools, and collaboration features. These features facilitate the systematic review and analysis of relevant literature.

Literature review software, including MAXQDA, can assist in qualitative data interpretation by enabling researchers to organize, code, and categorize relevant literature. This organized data can then be analyzed to identify trends, patterns, and themes, helping researchers draw meaningful insights from the literature they’ve reviewed.

Yes, literature review software like MAXQDA is suitable for researchers of all levels of experience. It offers user-friendly interfaces and extensive support resources, making it accessible to beginners while providing advanced features that cater to the needs of experienced researchers.

Getting started with literature review software, such as MAXQDA, typically involves downloading and installing the software, importing your relevant literature, and exploring the available features. Many software providers offer tutorials and documentation to help users get started quickly.

For students, MAXQDA can be an excellent literature review software choice. Its user-friendly interface, comprehensive feature set, and educational discounts make it a valuable tool for students conducting literature reviews as part of their academic research.

MAXQDA is available for both Windows and Mac users, making it a suitable choice for Mac users looking for literature review software. It offers a consistent and feature-rich experience on Mac operating systems.

When it comes to literature review software, MAXQDA is widely regarded as one of the best choices. Its robust feature set, user-friendly interface, and versatility make it a top pick for researchers conducting literature reviews.

Yes, literature reviews can be conducted without software. However, using literature review software like MAXQDA can significantly streamline and enhance the process by providing tools for efficient data management, analysis, and visualization.

Open Source Literature Review Search Tools

• A list of search tools help you find open source articles

Open Source Serch Tools

Browser extensions to help you identify open access articles, are you suspicious of a journal's authenticity is it a predatory journal, sammy chapman jr.

• Web Literacy for Student Fact Checkers

Use these browser extensions below to find open access articles The Chrome browser is recommended when using these extensions 

• CORE Discovery One-click access to free copies of research papers whenever you hit the paywall
• Open Access Button The Open Access Button is a browser bookmarklet which registers when people hit a paywall to an academic article and cannot access it. It is supported by Medsin UK and the Right to Research Coalition.
• Unpaywall Unpaywall searches the Unpaywall database which works by harvesting from Open Access sources to find OA content and then matching that content to the Crossref DOI, so for any DOI searched via the plugin, Unpaywall "knows about any OA versions that exist anywhere". The Unpaywall extension is already integrated into a number of discovery tools and platforms, including Scopus, Web of Science
• EndNote Click EndNote Click provides a link to the "best available full-text PDF" of an article and works in the same way as other OA browser extensions; by searching for an OA version on open databases on the Internet.
• Open Access Helper for iOS & macOS FIND OPEN ACCESS There are more than 25 million Open Access versions of otherwise “paywalled” scientific articles, however they are often not easy to find.Open Access Helper for iOS & macOS is designed to help you get easy access to these documents, with a lot of help from some amazing APIs.
• Google Scholar Button This extension for Firefox adds a browser button for easy access to Google Scholar from any web page. Click Scholar Settings (you may need to login with your libthe Scholar button to: Find full text on the web or in your university library. Select the title of the paper on the page you're reading, and click the Scholar button to find it. Transfer your query from web search to Scholar. Press the Scholar button to see top three results; click "full screen" in the lower left of the popup to see them all. Format references in widely used citation styles. Press the quote button under the result to see a formatted reference and copy it into the paper you're writing.Save the article to your Scholar library, so you can read it or cite it later. Press the blue star under the result to save it, or the gray star at the bottom to see all saved articles. Library links work best when you're on campus.To search the US case law, click the gear icon at the bottom of the popup, and configure your preferred collection in Google Scholar Settings.
• Google Scholar Button Extension for Chrome
• Cabell's Directories This link opens in a new window Are you suspicious of a journal's authenticity? Is it a predatory journal? Check the black list with Cabell's Directories
• Compass to Publish (Beta Version) helps you determine the degree of authenticity of open access journals requiring or hiding article processing charges (APCs) using a criteria-based evaluation
• QOAM /kju:əʊm/ abbr. Quality Open Access Market. QOAM is a market place for scientific and scholarly journals which publish articles in open access. It is important to realise that ‘quality’ in the context of QOAM relates to the quality of a journal’s service to authors, rather than to a hypothesized quality of a journal’s scientific and scholarly content as based on citation metrics. In QOAM, academic authors score the experience they have had with the journal’s peer review and editorial board from 1 to 5 via a concise journal score card. The QoS indicator of a journal is then defined as the product of the average score of the journal and the ‘robustness’ of this score.
• << Previous: A list of search tools help you find open source articles
• Next: Web Literacy for Student Fact Checkers >>
• Last Updated: Jun 11, 2024 1:38 PM
• URL: https://guides.pnw.edu/opensource

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

How to Perform a Literature Review with Free and Open Source Software

As it provides a firm foundation for advancing knowledge, a solid literature review is a critical feature of any academic investigation. Yet, there are several challenges in performing literature reviews including: i) lack of access to the literature because of costs, ii) fracturing of the literature into many sources, lack of access and comprehensive coverage in many databases and search engines, and iii) the use of proprietary software lock-in strategies for bibliographic software, which can make porting literature reviews between organizations cumbersome and costly. These challenges often result in poor quality literature reviews completed by a single researcher unfamiliar with the approaches to the same research in other sub-fields and static reviews that are often lost to the scientific community. In this paper, an open source approach will be expanded to the application of improving the quality of literature reviews by providing best practices. Although there are many types and goals of literature reviews, it is found that all of them can be improved using a tool chain of free and open source software (FOSS) and methods. Specifically, this paper will provide a clear framework for i) comprehensive searching and obtaining access to the literature, ii) the use of FOSS for all steps including browsing, bibliographic software, and writing and iii) documenting a literature review to encourage collaboration of a dynamic document that lives into the future. This approach solves the current challenges of literature reviews and provides benefits of lower labor and economic costs, improved researcher control, and increased potential for collaboration. Finally, the challenges of using this approach and methods to overcome them are reviewed and future work is described.

Related Papers

Burak Karakan

Die steigende Popularität des evidenz-basierten und empirischen Software-Engineerings hat dazu geführt, dass systematische Reviews sich in diesem Bereich als eine wichtige Forschungsmethode etabliert haben. Der systematische Charakter dieses Forschungsansatzes ermöglicht es, alle relevanten Beweise in Bezug auf ein bestimmtes Forschungsthema zu identifizieren, zu sammeln und zu interpretieren, wobei auch neue Forschungsaktivitäten zur weiteren Untersuchung durchgeführt werden können. Die Entwicklung eines umfassenden und detaillierten Review-Protokolls, das alle relevanten Methoden, Verfahren und Strategien für den Review dokumentiert, muss daher befolgt werden, um eine mögliche Verzerrung in der Bewertung durch den Forscher zu vermeiden und den wissenschaftlichen Wert der Ergebnisse zu erhöhen. Folglich sind die manuellen Tätigkeiten von systematischen Reviews sehr zeitaufwendig und erfordern einen hohen Arbeitsaufwand. Angesichts dieser Herausforderungen gibt es ein großes Potenzi...

Publications

Cherley C Du Plessis

The ability to conduct an explicit and robust literature review by students, scholars or scientists is critical in producing excellent journal articles, academic theses, academic dissertations or working papers. A literature review is an evaluation of existing research works on a specific academic topic, theme or subject to identify gaps and propose future research agenda. Many postgraduate students in higher education institutions lack the necessary skills and understanding to conduct in-depth literature reviews. This may lead to the presentation of incorrect, false or biased inferences in their theses or dissertations. This study offers scientific knowledge on how literature reviews in different fields of study could be conducted to mitigate against biased inferences such as unscientific analogies and baseless recommendations. The literature review is presented as a process that involves several activities including searching, identifying, reading, summarising, compiling, analysing, interpreting and referencing. We hope this article serves as reference material to improve the academic rigour in the literature review chapters of postgraduate students' theses or dissertations. This article prompts established scholars to explore more innovative ways through which scientific literature reviews can be conducted to identify gaps (empirical, knowledge, theoretical, methodological, application and population gap) and propose a future research agenda.

Jenine Beekhuyzen

It is important for researchers to efficiently conduct quality literature studies. Hence, a structured and efficient approach is essential. We overview work that has demonstrated the potential for using software tools in literature reviews. We highlight the untapped opportunities in using an end-to-end tool-supported literature review methodology. Qualitative data-analysis tools such as NVivo are immensely useful as a means to analyze, synthesize, and write up literature reviews. In this paper, we describe how to organize and prepare papers for analysis and provide detailed guidelines for actually coding and analyzing papers, including detailed illustrative strategies to effectively write up and present the results. We present a detailed case study as an illustrative example of the proposed approach put into practice. We discuss the means, value, and also pitfalls of applying tool-supported literature review approaches. We contribute to the literature by proposing a four-phased tool-supported methodology that serves as best practice in conducting literature reviews in IS. By viewing the literature review process as a qualitative study and treating the literature as the " data set " , we address the complex puzzle of how best to extract relevant literature and justify its scope, relevance, and quality. We provide systematic guidelines for novice IS researchers seeking to conduct a robust literature review.

Reference & User Services Quarterly

Virginia G Britt

Research outputs across the academic disciplines are almost exclusively published electronically. Organizing and managing these digital resources for purposes of review, and with the technical savvy to do so, are now essential skills for graduate study and life in academia. Paradoxically, digital and web-based technologies provide greater ease and efficiency with which to gather mass amounts of information, while at the same time presenting new challenges for reading, analyzing, organizing, and storing resources. Students, scholars, and the librarians who support them must adopt and refine practices to convert from paper-full to paperless literature review. This article proposes a methodical, reproducible, three-stage process that harnesses the power digital tools bring to the research cycle, regardless of the user’s preferred platform or operating system. Focusing just on the literature review phase, we develop a conceptual framework, illustrated with concrete tips and advice for s...

Francis Lapique

Oxymoron is a World Wide Web based knowledge capitalization and sharing tool, which was conceived and developed by a multidisciplinary team, comprised of adult education and distributed systems professionals from France and Switzerland. It aims to support and facilitate inter-peer work of students and researchers in the social sciences by providing them with a system where they can contribute and receive knowledge about the relevant readings in their fields of interest. Oxymoron is an extranet tool, the access to which is granted to various adult education institutions, in order to constitute an transdisciplinary knowledge repository as well as to facilitate distance learning and tutored pedagogy.

Systematic Reviews

Farhana Islam

Frances Slack

This article offers support and guidance for students undertaking a literature review as part of their dissertation during an undergraduate or Masters course. A literature review is a summary of a subject field that supports the identification of specific research questions. A literature review needs to draw on and evaluate a range of different types of sources including academic and professional journal articles, books, and web-based resources. The literature search helps in the identification and location of relevant documents and other sources. Search engines can be used to search web resources and bibliographic databases. Conceptual frameworks can be a useful tool in developing an understanding of a subject area. Creating the literature review involves the stages of: scanning, making notes, structuring the literature review, writing the literature review, and building a bibliography .

Amanda Bolderston

A literature review can be an informative, critical, and useful synthesis of a particular topic. It can identify what is known (and unknown) in the subject area, identify areas of controversy or debate, and help formulate questions that need further research. There are several commonly used formats for literature reviews, including systematic reviews conducted as primary research projects; reviews written as an introduction and foundation for a research study, such as a thesis or dissertation; and reviews as secondary data analysis research projects. Regardless of the type, a good review is characterized by the author’s efforts to evaluate and critically analyze the relevant work in the field. Published reviews can be invaluable, because they collect and disseminate evidence from diverse sources and disciplines to inform professional practice on a particular topic. This directed reading will introduce the learner to the process of conducting and writing their own literature review.

Australian Journal of Management

Martina Linnenluecke

Literature reviews play an essential role in academic research to gather existing knowledge and to examine the state of a field. However, researchers in business, management and related disciplines continue to rely on cursory and narrative reviews that lack a systematic investigation of the literature. This article details methodological steps for conducting literature reviews in a replicable and scientific fashion. This article also discusses bibliographic mapping approaches to visualise bibliometric information and findings from a systematic literature review. We hope that the insights provided in this article are useful for researchers at different stages of their careers-ranging from doctoral students who wish to assemble a broad overview of their field of interest to guide their work, to senior researchers who wish to publish authoritative literature reviews. JEL Classification: C18, C80, C88, M10, M20

Angelo D'Ambrosio

The exponential growth of scientific literature makes secondary literature abridgements increasingly demanding. We introduce a new open-source framework for systematic reviews that significantly reduces time and human resource allocation for collecting and screening scientific literature. The framework provides three main tools: 1) an automatic citation search engine and manager that collects records from multiple online sources with a unified query syntax, 2) a Bayesian, active machine learning, citation screening tool based on iterative human-machine interaction to increase predictive accuracy and, 3) a semi-automatic, data-driven query generator to create new search queries from existing citation data sets. To evaluate the automatic screener’s performance, we estimated the median posterior sensitivity and efficiency [90% Credible Intervals] using Bayesian simulation to predict the distribution of undetected potentially relevant records. Tested on an example topic, the framework c...

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

International Journal of P R O F E S S I O N A L Business Review

Information and Software Technology

Bedir Tekinerdogan

Zheng Huang

Journal of Asian Development

Erni Murniarti

F1000Research

Andrew Treloar

Alexandra Gheondea-Eladi

Review of Managerial Science

Marina Dabić

Rebekka Tunombili

Anayiaz Zaigmie

Helio Ferenhof , Roberto Fernandes

Auxiliadora Padilha

Isabel Pinho

The Qualitative Report

Janeen T Lamb

Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Érica Souza

Tatam Chiway , Abdullah Ramdhani , Muhammad Ali Ramdhani

Neal R Haddaway

Proceedings of the 23rd European Conference on Information Systems (ECIS 2015)

Ali Sunyaev

selorm kuffour

Campbell Systematic Reviews

Julia Littell

Management Information Systems Quarterly

Richard Watson

Communications of the Association for Information Systems

Murray Jennex

Alexandra Collins

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Grab your spot at the free arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Software Engineering

Title: systematic literature review of commercial participation in open source software.

Abstract: Open source software (OSS) has been playing a fundamental role in not only information technology but also our social lives. Attracted by various advantages of OSS, increasing commercial companies take extensive participation in open source development and have had a broad impact. This paper provides a comprehensive systematic literature review (SLR) of existing research on company participation in OSS. We collected 92 papers and organized them based on their research topics, which cover three main directions, i.e., participation motivation, contribution model, and impact on OSS development. We found the explored motivations of companies are mainly from economic, technological, and social aspects. Existing studies categorize companies' contribution models in OSS projects mainly through their objectives and how they shape OSS communities. Researchers also explored how commercial participation affects OSS development. We conclude with research challenges and promising research directions on commercial participation in OSS. This study contributes to a comprehensive understanding of commercial participation in OSS development.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Upload file
• Community portal
• Recent changes
• Contributions
• Create account
• Enable dark mode
• Enable read mode
• View history
• What links here
• Related changes
• Permanent link
• Page information
• Browse properties

How to perform a literature review with free and open source software

As it provides a firm foundation for advancing knowledge, a solid literature review is a critical feature of any academic investigation. Yet, there are several challenges in performing literature reviews including: i) lack of access to the literature because of costs, ii) fracturing of the literature into many sources, lack of access and comprehensive coverage in many databases and search engines, and iii) the use of proprietary software lock-in strategies for bibliographic software, which can make porting literature reviews between organizations cumbersome and costly. These challenges often result in poor quality literature reviews completed by a single researcher unfamiliar with the approaches to the same research in other sub-fields and static reviews that are often lost to the scientific community. In this paper, an open source approach will be expanded to the application of improving the quality of literature reviews by providing best practices. Although there are many types and goals of literature reviews, it is found that all of them can be improved using a tool chain of free and open source software (FOSS) and methods. Specifically, this paper will provide a clear framework for i) comprehensive searching and obtaining access to the literature, ii) the use of FOSS for all steps including browsing, bibliographic software, and writing and iii) documenting a literature review to encourage collaboration of a dynamic document that lives into the future. This approach solves the current challenges of literature reviews and provides benefits of lower labor and economic costs, improved researcher control, and increased potential for collaboration. Finally, the challenges of using this approach and methods to overcome them are reviewed and future work is described.

Keywords [ edit | edit source ]

See also [ edit | edit source ].

• MOST literature reviews
• Open Source Lab

Media [ edit | edit source ]

• 7 open source tools to make literature reviews easy - Open source

• MOST completed projects and publications
• Pages with no main image

Olivia A. Gallucci

Offensive Security, Open Source, and Glitter

Literature Reviews on Open Source

Literature reviews

In this post, I share the literature reviews I wrote for the Humanitarian Free and Open Source Software Development (HFOSS) course at Rochester Institute of Technology (RIT). I took this course in 2021, and the information in this post comes from the old HFOSS’21 blog. As I added more posts to my blog, I decided to combine my literature reviews into one post to make it clear that they were all written for an undergraduate class.

What are literature reviews

A literature review is an analysis and summary of existing research or publications on a particular topic. The purpose of a literature review is to provide an overview of what is known about the topic and to identify gaps or inconsistencies in the existing research.

🌸👋🏻 Join 10,000+ followers! Let’s take this to your inbox. You’ll receive occasional emails about whatever’s on my mind—offensive security, open source, academics, boats, software freedom, you get the idea.

Type your email…

Types of literature reviews

There are several types of literature reviews, including:

• Narrative Literature Review : This type of literature review involves a comprehensive and systematic review of the literature on a specific topic without using statistical methods to analyze the data.
• Systematic Literature Review : A systematic review is a more structured approach to reviewing the literature. It involves a comprehensive and rigorous search of the literature on a particular topic, with a specific methodology to identify, assess and analyze the literature.
• Meta-analysis : This type of literature review involves statistical analysis of the data from a collection of studies on a particular topic, with the goal of identifying patterns, relationships, and trends.
• Scoping Review : A scoping review is a type of literature review that aims to map the key concepts, theories, and sources of evidence on a particular topic, without necessarily answering a specific research question.
• Critical Review : A critical review is an in-depth analysis of the literature on a particular topic that includes an evaluation of the strengths and weaknesses of the studies reviewed. It aims to identify gaps in the existing research and to provide recommendations for future research.

This post consists of four scoping reviews. Here are the titles of my literature reviews:

The Organization for Ethical Source: Increasing the Adoptability of Ethical Licenses

How small businesses and organizations can manage open source components, new open source projects combating racism, defenders of free software in the legal sphere, open source open for business, literature review summary.

Ethical-Source Movement Opens New Open-Source Organization by Steven J. Vaughan-Nichols details the creation of the Organization for Ethical Source (OES). OES is a non-profit created by ethical source advocates. The goal of OES is to increase the adoption of Ethical Source licenses.

Why should you read this article?

Vaughan-Nichols provided ample context to the creation of ethical source licenses. Ethical source licensing has a long history, and it mostly consists of failures; OES’ goal is to change that. For example, some ethical licenses (i.e., Hippocratic License 2.1) use the MIT open-source license as a backbone; then, add human rights clauses in the United Nations Universal Declaration of Human Rights and Global Compact . The human rights clauses are what makes the license ethical. Vaughan-Nichols explanation of standard licensing, and ethical source licensing, helps the reader understand the differences between them.

Another benefit of the article is that Vaughan-Nichols explores the perspectives of OES’ legal and funding bodies. OES’ partners like the Corporate Accountability Lab, a pro bono legal team, and the Omidyar Network, a philanthropic investment firm, are quoted. This helps the reader understand the development and structure of OES as an organization.

The article also provides why some licenses in open source are incompatible with ethical licenses. The primary reason open source and ethical licenses are incompatible is because some open source licenses include Freedom Zero. Freedom Zero allows software to run for any purpose.

Freedom zero, the right to run the program for any purpose, comes first in the four freedoms because if users do not have that right with respect to computer programs they run, they ultimately do not have any rights in those programs at all. Efforts to give permission only for good uses, or to prohibit bad ones in the eyes of the licensor, violate the requirement to protect freedom zero. Thus they cannot be free software licenses, and cannot be “open source” licenses unless that category now includes licenses that don’t protect all the fundamental software freedoms. Eben Moglen, Columbia law professor, 2019

Freedom Zero allows software to be used for the creation of a bomb or for the work of a local charity. Ethical licenses want to restrict the usage of their software (i.e. the software should not help with the creation of a bomb, but can be used to help a local charity). Vaughan-Nichols decision to include the conflict between open and ethical source helps the reader understand that they do not perfectly overlap; this is critical for the reader to understand because many ethical source advocates promote open source software.

Questions and critiques

The article left me wondering about the effectiveness of Fortune 500 company ethics. Vaughan-Nichols notes that ethics are appearing in “numerous big tech companies” that produce artificial intelligence software like Google and Microsoft. Google and Microsoft have drowned in ethics-related controversies; Google’s secret search engine built for the Chinese government to Microsoft’s contract with ICE are microscopic compared to their overall collection of ethical violations. Given Google and Microsoft’s lack of credibility, it was not effective for Vaughan-Nichols to cite them as companies including ethics in their software production. Vaughan-Nichols should expand upon his reasoning for including immoral companies as promoters of ethical software production; at present, it is difficult for the reader to follow his train of thought.

Another issue is that the article does not explain how OES plans on increasing the adoptablity of ethical source licenses. Specifically, Vaughan-Nichols overlooks why ethical-source licensing has previously failed, and how OES will improve the adoptablity of ethical source licenses. This would give the reader insight as to why ethical source licensing has failed, and what OES plans on doing to improve the adoption of ethical source licenses. In short, Vaughan-Nichols disregards the topic of his article: what is OES doing to promote ethical source licenses?

Lastly, Vaughan-Nichols’ article should have included more information about the potential success and failures of OES. Vaughan-Nichols should provide his input, or the input of an OES advocate and OES detractor, to where they think the fate of OES and ethical source licenses will be.

• OES promotes ethical licenses because of companies like Google and Microsoft’s immoral software usage. Why does Vaughan-Nichols compare the ethical practices of OES to the immoral practices of Google and Microsoft?
• What do experts say about OES’ future?
• What is Vaughan-Nichols opinion of OES, and what are his predictions about the future success or failure of OES?

This article provides a meaningful update to what is going on in the ethical source community. The article provides a neutral standpoint on an ideologically driven issue, but does not provide expert opinions. Providing multiple opinions on the future of OES and ethical source licensing gives readers insight as to why some people support initiatives like OES, and why others reject ethical source licenses entirely.

Overall, I give this article 6/10 starts. (Bad=0-3; Good=4-6; Great=7-10).

7 Best Practices for Managing Open Source Components by Limor Wainstein is an overview of how to use community-driven open source software in a business environment. The article included actionable steps, which help companies organize their existing projects, so that they can be safely managed. These steps are policy, update promptly, empathize quality, use a binary repo manager, participate in the community, control build with tools, and fork when possible.

Wainstein explains how to implement each step, and provides caveats and hazards to consider at each step. The article is similar to something on WikiHow or in the Dummies book series; Wainstein takes a complex issue, and simplifies it to make the issue solvable for lay people.

Additionally, Wainstein leaves the steps open-ended, but provides enough information to make the caveats and hazards easily searchable. This way, readers can apply her solutions to their specific needs, and find the resources they need to succeed. Lastly, Wainstein’s article is concise and organized. Wainstein has a fabulous, cookie-cutter essay outline, which makes it easy for the reader to follow. Wainstein introduces the topic, supports her thesis, and concludes the article. Although essay formatting is taught in most schools, it is rarely perfected; Wainstein’s article is close to perfection.

The article’s link contains “5-best-practices-for-managing-open-source-components,” but the article lists seven reasons; this is a bit sloppy, but Altexsoft may not update URLs after the initial article has been published. Another issue with the article is the conclusion. Wainstein’s introduction and body are thorough, and provide the reader with lots of details. The conclusion, however, is very short and does a poor job summarizing the information provided in the introduction (mainly, her hypothesis) and body of the article.

• Is it normal for websites to keep the same URL if the article is updated?
• How often are guest writers featured on company blogs?
• Why does Altexsoft choose an independent writer when they could use their own writers to implement Altexsoft’s products and solutions into the article?

IBM, Call for Code, and the Linux Foundation announce new open source projects to combat racism by Tristan Greene discusses seven, innovative open source projects trying to combat racism. The seven projects are Fair Change, TakeTwo, Five Fifths Voter, Legit-Info, Incident Accuracy Reporting System, and Open Sentencing. Each project promotes racial equity through online mediums; examples include, fixing racially-biased facial recognition and artificial intelligence programs, promoting information accuracy, voting strategies, easy-to-read documentation of local laws and regulations, and collaborative-witness police reports.

Greene’s article is hyper-focused. It is similar to a secretary’s meeting notes because Greene only states the programs’ objectives and intentions; he does not give his analysis on the programs’ effectiveness or legitimacy. Articles like this are unusual because of the internet’s politically charged culture.

The article is structured in this format: introduction, project descriptions, and where to find more information. The project descriptions include their history and what they hope to accomplish. Most importantly, the project descriptions are thorough and simply worded.

Greene’s article does not leave much room for critiques or questions. He cites his sources, and he does not give his own take on the projects. This is important because many articles that push a specific view do not include a fair explanation of detractors’ views. However, articles like Greene’s are politically neutral. Overall, this allows readers to gauge their interest in the projects without feeling politically charged.

The only issue with this article is that it excludes how to get involved. For example, what demographic are the projects looking for (i.e. people of color, writing skills, bilingual, volunteers or paid workers, etc…)? Including this information would help the reader define their role in the projects if they wanted to join. At the end of the article, Greene includes a link to learn more , but it would be more effective to disbursed links throughout the article.

• Who can contribute? Are these projects looking for people with a specific skill set or background? Are these projects looking for volunteers, new employees, or are they only open to employees at IBM, Call for Code, and the Linux Foundation?
• What skills are necessary for each project?
• How can readers contribute to these projects?

The The New York Times published The Defenders of Free Software by Ashlee Vance in 2010. It is about an enthusiastic, free and open source software (FOSS) volunteer watchman named Armijn Hemel. Vance chronicles Hemel’s experience sending cease-and-desist letters to large companies–like Dell, Google, TiVo and Sony–who use FOSS, but do not follow the conditions of FOSS licenses. He explains that licensing enforcement is a recurring problem because large companies “often opt to piggyback on the work of others rather than going through the ordeal of building all of the software for their products from scratch.” Vance examines potential solutions to this problem; mainly, that some FOSS and open source groups, like the Linux Foundation , are creating programs to make it easier for companies to keep track of the licenses they are using, so they can avoid lawsuits.

Vance explains the lifestyle, intentions, and outcomes of Hemel’s work, which helps the reader understand why Hemel (and other FOSS volunteers) promote licensing enforcement. The article is personal, but also explains the broader implications of FOSS activism in the legal sphere. In addition, Vance mentions activist groups like the Software Freedom Law Center and gpl-violations.org . Name-dropping these organizations allows the reader to learn more about those groups.

Although Vance explains what legal FOSS volunteers undertake, he excludes how FOSS activists can contribute to the legal sphere. Additionally, he excludes information about what contributor qualifications projects desire. The article leaves out details about the effectiveness of the current volunteers, and does not predict if the solutions provided by the Linux Foundation will be effective in educating big companies on how to follow licenses. This point is important because most FOSS licenses are short and easy to follow. Lastly, some of the information Vance includes does not make sense for readers who do not understand business or law. For example, Vance states that “lawsuits are typically settled out of court,” but does not explain why a company would want to settle in or out of court.

• How can FOSS activists contribute to the FOSS legal sphere?
• What qualifications and experience does Hemel have? What about other volunteers in FOSS law?
• Why are FOSS lawsuits normally settled out of court?

Open Source Open for Business –written by Bill Brigge, Stefan Kircher, and Michael Bechtel–is about how large companies interact with open source software (OSS). Specifically, this article is about the current benefits and future potential of companies using OSS in business. The piece focuses on the advantages OSS offers and some of the untapped advantages OSS can provide companies. Deloitte Touche Tohmatsu Limited (Deloitte for short) sponsored the article, and The Wall Street Journal published the article.

I read this article because I am co-oping at Deloitte’s 30 Rockefeller Head Quarters this summer. I thought reading this would be a great way to learn about open source at Deloitte.

This article offers insight as to where large companies are spending money. For example, Deloitte and Datawheel created a joint research effort called Open Source Compass , and the article includes statistics from the research they conducted.

Another benefit of this article is that it examines ideas rarely discussed in open source communities (OSC). Specifically, the authors note how contributing to OSCs can increase productivity, growth, knowledge, and security. However, this is weird because many older companies–particularly those involved with finance and legal work–focus on profits and secrecy. Since profits drive the economy, it is interesting that a large company like Deloitte would endorse and contribute to OSCs. In addition, the authors describe how OSS is beneficial for auditing because all the code is publicly available.

Lastly, Deloitte acknowledges that participating in OSCs provides junior developers with opportunities to “read code written by more experienced codes and highly creative pioneers.” Although universities and OSCs acknowledge this benefit, it is rare to see large companies recommending OSS for junior developers to gain experience. Again, this is surprising because older private companies tend to favor proprietary software.

Critiques on Open Source Open for Business

The authors mention companies contributing to OSS, but exclude examples of companies contributing. The authors state that “for technology capabilities at the core of strategic differentiation, a healthy reluctance to depend on–let alone share expertise with–anyone outside the organization’s direct control is in order.” Although that notion makes sense, it conveys that a company’s “core of strategic differentiation” consists of all internally developed software. In other words, all internally developed code gives companies a competitive edge, and therefore, should not be shared.

• Why do the authors claim that it is beneficial for companies to contribute to OSS without providing any examples?
• Does Deloitte contribute to the OSS they use? If so, how?

Open source literature reviews

The literature reviews I wrote during the 2021 Humanitarian Free and Open Source Software Development (HFOSS) course at Rochester Institute of Technology provided basic insights into various aspects of open-source software development, as well as its applications in humanitarian contexts. Through my reviews, I learned about the challenges and opportunities associated with the development and use of humanitarian FOSS projects. Additionally, I learned about the importance of collaboration, community building, and user engagement in successful open-source development. Overall, I believe that the knowledge and skills I gained during this course were useful in my endeavors as a software developer. Additionally, I hope future HFOSS students can use my reviews as sample responses for the literature review assignments.

I hope you enjoyed this post on my open source literature reviews. If you want to learn more about open source or HFOSS , consider reading Contributing to Open Source at RIT .

Written by Olivia Gallucci

Olivia is an honors student at the Rochester Institute of Technology. She writes about security, open source software, and professional development.

Discover more from Olivia A. Gallucci

Subscribe now to keep reading and get access to the full archive.

Continue reading

Open Source Digital Library Software: A Literature Review

Proceedings of the National Seminar on 'Preservation and Conservation of Information Resources in Knowledge Society: Issues, Challenges and Trends', pp. 238-258, Manipur University, Canchipur, Imphal, March 3-4, 2009

18 Pages Posted: 13 Dec 2019

Hanadashisha Warr

Affiliation not provided to ssrn, prof p hangsing.

North-Eastern Hill University (NEHU) - Department of Library and Information Science

Date Written: 2009

Digital Library initiatives are gaining momentum and many open source digital library software emerged. These developments necessitate universal standard for digital library software. Conforming the software to universal standard means compromising local requirements. No matter how general the software may be, suitability of the digital library software to specific requirements of digital library project will differ from software to software. These variations in digital library software allow ample room for librarians to assess the digital library softwares with the local requirement criterion. To ease assessment/selection process, this paper attempts to evaluate four most popular open source digital library software. Various features of the softwares are comparatively tabled for ready reference.

Keywords: Digital Library, Digital Library Software, Open Source Digital Library Software, Library Digitization, Digital Archive

Suggested Citation: Suggested Citation

Paokholun Hangsing (Contact Author)

North-eastern hill university (nehu) - department of library and information science ( email ).

North-Eastern Hill University Shillong, ME India

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, io: productivity, innovation & technology ejournal.

Subscribe to this fee journal for more curated articles on this topic

Writing Technologies eJournal

Subscribe to this free journal for more curated articles on this topic

Legal Information, Technology & Law Librarianship eJournal

Economic & social impacts of innovation ejournal, diffusion of innovation ejournal, information technology & systems ejournal, information systems: behavioral & social methods ejournal, international political economy: globalization ejournal, information curation, management & organization ejournal, library services & librarianship ejournal.

6 Of The Best Open-Source PDF Editors Available To Try

PDFs have become quite a staple in our digital lives, and it's likely that you use them more often than you realize. Perhaps you're a student doing research for multiple school projects, or possibly you're a human resource specialist handling employee and company documents. Whatever the case may be, having apps that handle such files is a must.

If you mainly just read PDFs, internet browsers do a decent job of displaying them for you. However, if you're always modifying your PDFs or filling out PDF forms, then what you need is a dedicated PDF editor. Unfortunately, Adobe Acrobat — many people's go-to PDF editor — can be quite pricey. The good news is that you really don't need to spend a pretty penny just to edit PDFs on your computer. There's a bunch of  alternatives for Adobe Acrobat  out there, some of which are known as open-source PDF editors. Let's have a look at what this type of software is, and explore a handful of open-source PDF editors that you can install. 

What are open-source PDF editors?

Open-source PDF editors are publicly available pieces of software with unrestricted access. That means its source code can be viewed and modified freely. And you can share the software without repercussions. 

Similar to Adobe Acrobat, open-source PDF editors come with different tools for altering PDF files, from adding text to merging pages. Some of them are readily available on all major desktop operating systems — Windows, macOS, and Linux — while others only cater to one or two systems. Unlike the commercial and paid counterparts, open-source PDF editors don't have formal customer support. You either need to contact the developer, or submit an issue on the relevant GitHub page, if there is one. Alternatively, you may be able to find a community of other users online to help you with any concerns. 

It's also worth noting that some open-source PDF editors might become abandoned after a number of years with a lack of developers contributing updates to the software. Hence, they may no longer be compatible with modern systems. If, however, you're still happy using open-source PDF editors despite these disadvantages, here are six options you can try.

ONLYOFFICE is both a PDF editor and viewer

Available on Windows, Mac, and Linux computers, ONLYOFFICE is a great choice for when you need to edit the content of a PDF. This includes any available text, image, shape, or even table. The app lets you rewrite the text and change aspects, such as font size, font style, color, emphasis (e.g., bold, italics), and alignment (e.g., justify, align right), among other things. You can even apply a new text art and color fill to add more flair to your otherwise boring text. Several editing options like crop, rotate, opacity adjustment, and line color selection are also available for the images and shapes. This can save you time and effort as you won't need to extract the images from the PDF and insert them again if you're only making minor modifications to them.

Apart from the editing tools, ONLYOFFICE supports plugins too. You can install the Jitsi plugin that allows you to make calls within the app — ideal for collaborating on and editing documents in real time. There are also some translator and grammar checkers in the plugins library to make editing your PDF faster and more efficient. After modifying your work, you can switch to the PDF viewer mode, where you can adjust the zoom, and change the page view (e.g., fit to page, fit to width).

The downside of ONLYOFFICE is that you might experience changes in the PDF's format (e.g., font style, image placement) when editing your file.

LibreOffice Draw is a Microsoft Office-like PDF editor

You may already know LibreOffice Writer as one of the best free alternatives to Microsoft Word , but the LibreOffice suite actually has more to offer than just a word processor — there's a handy app for editing PDFs too. Designed like the Microsoft Office interface, LibreOffice Draw was originally meant for creating drawings and graphics. However, it provides an array of tools for changing the existing content of a PDF as well. For instance, you can edit the text, its style, size, color, and spacing and add highlights, underlines, and shadows. If you have any images on the page, you can also modify how they're arranged (e.g., bring to front, bring forward), how big they are, and whether they should be flipped, rotated, or made transparent.

Since LibreOffice Draw is essentially a graphics editor, you won't have trouble inserting all sorts of designs to your PDF if the need arises. There are individual tools for adding predefined shapes like callouts, banners, and arrows, or drawing new elements like lines, curves, and polygons. You can also insert typical document elements, including images, charts, tables, text boxes, and page numbers to your PDF. 

LibreOffice Draw is available for download on Windows, macOS, and Linux, so you're free to use it with whichever major OS you prefer. However, do note that even though Draw looks similar in style to Microsoft Office, it might be a bit overwhelming for new users as there are multiple tabs and tools cluttered all over the interface. 

Stirling PDF is a minimalist and user-friendly PDF editor

Created by a UK-based software developer, Stirling PDF is unlike other open-source PDF editors as it isn't a single app but a collection of tools instead. When you access Stirling PDF, you'll be greeted by over 40 different PDF manipulation options. Each comes with their own minimalist interface, which is a stark contrast to most PDF editors that cram all of the features onto one screen. This design makes the app a lot more user-friendly.

Now onto the tools. Stirling PDF includes the typical manipulation features like removing annotations, merging, splitting, and page rearranging. However, you can also find other handy tools in the app that allow you to crop the PDF, add page numbers, adjust the brightness, and delete unwanted pages. Stirling PDF is even equipped with file conversions. You can conveniently convert your PDF to files like JPEG, PNG, CSV, and RTF and convert your files like MD, DOCX, HTML, URL, and PNG to PDF. If you're looking for advanced PDF tools, there are some options available in Stirling PDF. There's a password removal for protected PDFs and a metadata handler for removing, adding, or editing the PDF's metadata.

One downside about using Stirling PDF is the installation. It can be pretty technical as it needs to run inside a Docker container which you'd have to set up first. If you're editing a PDF on Mac or Linux, you can use the local version of the app instead, but this means going through a time-consuming eight-step installation process. If you're using Windows, there's a downloadable .exe file you can easily run. However, this version doesn't have the full capabilities of the original app.

PDF4QT features tools for annotating and redacting PDFs

Developed for editing PDFs on Windows and Linux, PDF4QT is a great app if you typically annotate and redact PDFs. For annotation, you have a great selection of features, starting with content addition that allows you to insert the usual elements like text, lines, tick and cross marks, and signatures. PDF4QT also offers additional annotation elements, including stamps (e.g., Approved, Confidential, Expired) for your formal documents and sticky notes for your thoughts and feedback. When it comes to redaction, you have the option to redact entire pages, certain sections of a page, individually selected text, and automatically selected specific words.

More than being a PDF annotation and redaction app, PDF4QT comes complete with various advanced settings too. For one, you can encrypt the PDF using one of several encryption algorithms like RC4 128, AES 128, AES 256, and certificate encryption. There are also optimization settings that reduce the file size of your PDF. If you'll be mostly reading the PDF, you can readily change its color to inverted, grayscale, high contrast, or monochromatic for a more comfortable view. If, however, you prefer to listen instead of reading the PDF, you can find a text-to-speech tool in the app. PDF4QT even lets you convert the PDF to an audiobook for you to listen to anywhere.

A major drawback of PDF4QT , though, is its inability to modify text, images, shapes, and other elements already existing in the original PDF.

Xournal++ is an easy-to-use PDF annotator

Although an open-source note-taking app, Xournal++ can actually do more than just handwriting and sketching notes. It includes features for annotating PDFs too. You have two ways to annotate your files: using the freehand tools or using the content adders. With the freehand tools, you can draw freely on the PDF using straight, dashed, or dotted lines. This is quite handy for when you want to add arrows or markings to lecture notes, or when you need to sign business documents. Another freehand tool in Xournal++ is the highlighter, which is highly customizable. You can change its brush size from medium to fine or thick and use a different color — pick one of the 11 predefined colors or add a custom color that suits your needs better. To make your freehand annotations look cleaner, there are reference guides like a set square and compass you can temporarily add onto the page.

If you still find freehand annotation messy, you can just use the content adders. Xournal++ lets you insert text and images to your PDF and modify existing text by adding an underline, strikethrough, or highlight. Similar to other PDF editors, you can also insert new pages to a file. However, instead of just the typical blank page, Xournal++ features designed journal-like pages, including ruled, graphing, and dotted paper. The app is available on Windows, macOS, and Linux, and includes a Beta mobile version too.

Scribus is perfect for form filling and annotation

Scribus is a desktop publisher similar to Adobe InDesign and Microsoft Publisher, but while it isn't a dedicated PDF editor, it does a pretty decent job of editing your PDF files. One of its standout features is when opening a new PDF, it lets you import either all of the pages of a PDF, just a single page, or several individual pages. This means that if you want to extract and modify only certain pages of a file, you won't have to go through the typical process of splitting the PDF, saving your desired pages as another file, and then opening this new file in a PDF editor. Instead, you can directly extract them by simply opening the whole PDF on Scribus and then picking your pages.

Unlike other PDF editors, this app doesn't support editing existing content. You can't rewrite text or adjust how images look. However, this is somewhat of a plus point as you're sure the format and style of the original PDF will be retained. What you can do on Scribus , though, is add new elements. You have multiple such tools at your disposal. There's the Text Frame for inserting text — for instance if you're filling out a form, the Image Frame for adding pictures from your PC — which you can then edit further with the built-in image effects adder, and Freehand Line for drawing freely on the page when annotating. You can run Scribus on Windows, Mac, and Linux systems.

Recommended

• Introduction
• Conclusions
• Article Information

LMIC indicates low- and- middle-income country; SR, systematic review.

a This review included distinct conclusions about separate conditions and comparators, and so it appears in this map more than once.

eAppendix 1. Search Strategies

eAppendix 2. Excluded Studies

eAppendix 3. Evidence Table

eAppendix 4. Conditions in Previously Published Map in 2018 and Current Map

eReferences.

Data Sharing Statement

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Mak S , Allen J , Begashaw M, et al. Use of Massage Therapy for Pain, 2018-2023 : A Systematic Review . JAMA Netw Open. 2024;7(7):e2422259. doi:10.1001/jamanetworkopen.2024.22259

Manage citations:

© 2024

• Permissions

Use of Massage Therapy for Pain, 2018-2023 : A Systematic Review

• 1 Veterans Health Administration, Greater Los Angeles Healthcare System, Los Angeles, California
• 2 UCLA Fielding School of Public Health, University of California, Los Angeles
• 3 RAND Corporation, Santa Monica, California

Question   What is the certainty or quality of evidence in recent systematic reviews for use of massage therapy for painful adult health conditions?

Findings   This systematic review identified 129 systematic reviews in a search of the literature published since 2018; of these, 41 assessed the certainty or quality of evidence of their conclusions. Overall, 17 systematic reviews regarding 13 health conditions were mapped, and most reviews concluded that the certainty of evidence was low or very low.

Meaning   This study found that despite massage therapy having been the subject of hundreds of randomized clinical trials and dozens of systematic reviews about adult health conditions since 2018, there were few conclusions that had greater than low certainty of evidence.

Importance   Massage therapy is a popular treatment that has been advocated for dozens of painful adult health conditions and has a large evidence base.

Objective   To map systematic reviews, conclusions, and certainty or quality of evidence for outcomes of massage therapy for painful adult health conditions.

Evidence Review   In this systematic review, a computerized search was conducted of PubMed, the Allied and Complementary Medicine Database, the Cumulated Index to Nursing and Allied Health Literature, the Cochrane Database of Systematic Reviews, and Web of Science from 2018 to 2023. Included studies were systematic reviews of massage therapy for pain in adult health conditions that formally rated the certainty, quality, or strength of evidence for conclusions. Studies of sports massage therapy, osteopathy, dry cupping or dry needling, and internal massage therapy (eg, for pelvic floor pain) were ineligible, as were self-administered massage therapy techniques, such as foam rolling. Reviews were categorized as those with at least 1 conclusion rated as high-certainty evidence, at least 1 conclusion rated as moderate-certainty evidence, and all conclusions rated as low- or very low–certainty evidence; a full list of conclusions and certainty of evidence was collected.

Findings   A total of 129 systematic reviews of massage therapy for painful adult health conditions were found; of these, 41 reviews used a formal method to rate certainty or quality of evidence of their conclusions and 17 reviews were mapped, covering 13 health conditions. Across these reviews, no conclusions were rated as high certainty of evidence. There were 7 conclusions that were rated as moderate-certainty evidence; all remaining conclusions were rated as low- or very low–certainty evidence. All conclusions rated as moderate certainty were that massage therapy had a beneficial associations with pain.

Conclusions and Relevance   This study found that despite a large number of randomized clinical trials, systematic reviews of massage therapy for painful adult health conditions rated a minority of conclusions as moderate-certainty evidence and that conclusions with moderate- or high-certainty evidence that massage therapy was superior to other active therapies were rare.

Massage therapy is a popular and widely accepted complementary and integrative health modality for individuals seeking relief from pain. 1 This therapy is the practice of manual assessment and manipulation of the superficial soft tissues of skin, muscle, tendon, ligament, and fascia and the structures that lie within the superficial tissues for therapeutic purpose. 2 Individuals may seek massage therapy to address pain where conventional treatments may not always provide complete relief or may come with potential adverse effects. Massage therapy encompasses a range of techniques, styles, and durations and is intended to be delivered by uniquely trained and credentialed therapists. 3 Original research studies have reported on massage therapy delivered by a wide variety of health care professionals, such as physical therapists, physiotherapists, and nurses. 4 , 5 Despite massage therapy’s popularity and long history in practice, evidence of beneficial outcomes associated with massage therapy remains limited.

The Department of Veterans Affairs (VA) previously produced an evidence map of massage therapy for pain, which included systematic reviews published through 2018. 6 An evidence map is a form of systemic review that assesses a broad field to identify the state of the evidence, gaps in knowledge, and future research needs and that presents results in a user-friendly format, often a visual figure or graph. 7 To categorize this evidence base for use in decision-making by policymakers and practitioners, VA policymakers requested a new evidence map of reviews published since 2018 to answer the question “What is the certainty of evidence in systematic reviews of massage therapy for pain?”

This systematic review is an extension of a study commissioned by the VA. While not a full systematic review, this study nevertheless reports methods and results using the Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA ) reporting guideline where applicable and filed the a priori protocol with the VA Evidence Synthesis Program Coordinating Center. Requirements for review and informed consent were waived because the study was designated as not human participants research.

Literature searches were based on searches used for the evidence map of massage therapy completed in 2018. 8 We searched 5 databases for relevant records published from July 2018 to April 2023 using the search terms “massage,” “acupressure,” “shiatsu,” “myofascial release therapy,” “systematic*,” “metaanaly*,” and similar terms. The databases were PubMed, the Allied and Complementary Medicine Database, the Cumulated Index to Nursing and Allied Health Literature, the Cochrane Database of Systematic Reviews, and Web of Science. See eAppendix 1 in Supplement 1 for full search strategies.

Each title was screened independently by 2 authors for relevance (S.M., J.A., and P.G.S.). Abstracts were then reviewed in duplicate, with any discrepancies resolved by group discussion. To be included, abstracts or titles needed to be about efficacy or effectiveness of massage therapy for a painful adult health condition and be a systematic review with more than 1 study about massage therapy. A systematic review was defined as a review that had a documented systematic method for identifying and critically appraising evidence. In general, any therapist-delivered modality described as massage therapy by review authors was considered eligible (eg, tuina, acupressure, auricular acupressure, reflexology, and myofascial release). Sports massage therapy, osteopathy, dry cupping or dry needling, and internal massage therapy (eg, for pelvic floor pain) were ineligible, as were self-administered massage therapy techniques, like foam rolling. Reviews had to be about a painful condition for adults, and we excluded publications in low- and middle-income countries because of differences in resources for usual care or other active treatments for included conditions. Publications were required to compare massage therapy with sham or placebo massage, usual care, or other active therapies. Systematic reviews that covered other interventions were eligible if results for massage therapy were reported separately.

We next restricted eligibility to reviews that used formal methods to assess the certainty (sometimes called strength or quality) of the evidence for conclusions. In general, this meant using Grading of Recommendations, Assessment, Development, and Evaluations (GRADE). 9 However, other formal methods were also included, such as the approach used by the US Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program. To be included, a review had to state or cite the method used and report the certainty (or strength or quality) of evidence for each conclusion. After we applied this restriction, most health conditions had only 1 systematic review meeting the eligibility criteria, and we used this review for the map. Among conditions for which we identified more than 1 review meeting the eligibility criteria, we first assessed whether reviews differed in some other feature used to classify reviews on our map (eg, different comparators or type of massage therapy), which we would label with the appropriate designation (such as vs usual care or reflexology ). If there were multiple reviews about the same condition and they did not differ in some other feature, we selected the systematic review we judged as being most informative for readers. In general, this was the most recent review or the review with the greatest number of included studies.

Data on study condition, number of articles in a review, intervention characteristics, comparators, conclusions, and certainty, quality, or strength of evidence were extracted by 1 reviewer and then verified by a second reviewer (S.M., J.A., and P.G.S.). Our evidence mapping process produced a visual depiction of the evidence for massage therapy, as well as an accompanying narrative with an ancillary figure and table.

The visual depiction or evidence map uses a bubble plot format to display information on 4 dimensions: bubble size, bubble label, x-axis, and y-axis. This allowed us to provide the following types of information about each included systematic review:

Number of articles in systematic review (bubble size): The size of each bubble corresponds to the number of relevant primary research studies included in a systematic review.

Condition (bubble label): Each bubble is labeled with the condition discussed by that systematic review.

Shapes and colors: Intervention characteristics for each condition are presented in the form of colors (type of intervention) and shapes (comparators). For type of intervention, we included nonspecified massage therapy, tuina, myofascial release, reflexology, acupressure, and auricular acupressure. For comparators, we included mixed comparators with subgroups, mixed comparators with no subgroups, sham or placebo, and active therapy or usual care. A condition can appear more than once if multiple systematic reviews included different type of massage therapy or different comparators.

Strength of findings (rows): Each condition is plotted on the map based on the ratings of certainty of evidence statement as reported in the systematic reviews: high, moderate, low, or very low.

Outcome associated with massage therapy (columns): Each condition is plotted in potential benefit or no benefit as the outcome associated with massage therapy. Columns are not mutually exclusive. A review could have more than 1 conclusion, and conclusions could differ in the benefit associated with massage therapy. Both conclusions are included on the map.

Risk of bias is not part of the method of an evidence map. We assessed the quality of included reviews using criteria developed by the U S Preventive Services Task Force (USPSTF). Certainty of evidence as determined by the original authors of the systematic review was abstracted for each conclusion in each systematic review and tabulated.

The search identified 1164 potentially relevant citations. Among 129 full-text articles screened, 41 publications were retained for further review. Of these, 24 reviews were excluded from the map for the following reasons: only 1 primary study about interventions of interest (11 studies), outcomes associated with massage therapy could not be distinguished from other included interventions (5 studies), not an intervention of interest (3 studies), not a comparison of interest (2 studies), overlap with a more recent or larger review that was already included on the map (2 studies), and self-delivered therapy (1 study). We included 17 publications in this map covering 13 health conditions. 4 , 10 - 25 The literature flowchart ( Figure 1 ) summarizes results of the study selection process, and eAppendix 2 in Supplement 1 presents citations for all excluded reviews at full-text screening.

The total number of primary studies about massage therapy for pain in the included reviews ranged from 2 studies to 23 studies. There were 12 reviews that included fewer than 10 primary studies 4 , 11 - 17 , 20 - 23 and 5 reviews that included 10 to 25 studies about massage therapy for pain. 10 , 18 , 19 , 24 , 25 Of included reviews, 3 reviews were completed by the Cochrane Collaboration 4 , 19 , 23 and 2 reviews were completed by the AHRQ EPC program. 11 , 18

We categorized the included 17 reviews by health condition. These categories were cancer-related pain, 15 , 24 back pain (including chronic back pain, 25 chronic low back pain, 18 , 22 and low back pain 17 ), chronic neck pain, 18 fibromyalgia, 21 labor pain, 4 , 19 mechanical neck pain, 13 myofascial pain, 14 palliative care needs, 10 plantar fasciitis, 12 post–breast cancer surgery pain, 16 postcesarean pain, 23 postpartum pain, 20 and postoperative pain. 11

Of 17 included reviews, 3 reviews included more than 1 type of massage therapy and 14 reviews included 1 type of massage therapy. Reviews by Chou et al 11 and Smith et al 16 included acupressure and nonspecified massage therapy as interventions. The review by Candy et al 7 included reflexology and nonspecified massage therapy as interventions. Of the 14 reviews with 1 type of massage therapy, there were 5 reviews describing nonspecified massage therapy, 10 , 14 , 17 , 20 1 review about tuina, 22 5 reviews about myofascial release, 8 , 9 , 12 , 18 , 19 and 3 reviews about acupressure. 13 , 15 , 21

A variety of comparators were included in reviews. Of 9 reviews that included more than 1 comparator in analyses, 4 , 11 , 13 , 14 , 18 - 22 2 reviews did not conduct separate analyses by comparator (labeled mixed with no subgroups ) 13 , 14 and 3 reviews conducted separate analyses by comparator (labeled mixed with subgroups ). 4 , 21 , 22 The other 4 reviews included a mix of comparators with separate conclusions: sham or placebo and active therapy or usual care, 11 mixed with no subgroups and active therapy or usual care, 18 mixed with subgroups and active therapy or usual care, 20 and mixed with no subgroups, sham, and active therapy or usual care. 19 There were 8 reviews that included 1 comparator only in their analyses, 10 , 12 , 15 - 17 , 23 - 25 with 7 reviews that described interventions compared with active therapy or usual care only, 10 , 12 , 15 , 17 , 23 - 25 while 1 review limited inclusion to primary studies with a sham or placebo comparator. 16

There was substantial variation in the reporting of other details from primary studies in included reviews. Any study that did not specify the mode of delivery was included; studies that explicitly stated that massage therapy was self-delivered were excluded. Of the 17 included reviews, 5 reviews provided details of personnel who administered the therapy, including massage therapist, nurse, aromatherapist, physiotherapist, and reflexologist. 4 , 10 , 19 - 21 A total of 7 reviews presented length of sessions (eg, 5-minute or 90-minute sessions for massage therapy studies and 30-second or 5-minute sessions for acupressure studies). 10 , 16 , 18 , 20 - 23 With the exception of the review by He et al, 15 all reviews reported details about frequency, duration, or both when available. A total of 9 reviews included information about frequency of sessions (eg, 1 session or once every 3 weeks for massage therapy studies and 4 times per day or daily for acupressure studies), 10 , 12 , 16 - 18 , 20 - 23 and 9 reviews reported duration of sessions (eg, single session or 3 months). 10 - 12 , 16 - 18 , 20 , 22 , 23 There were 7 reviews that included details about follow-up (eg, 1 week or 12 months). 10 , 13 , 17 , 18 , 21 , 23 , 25

Using USPSTF criteria to rate the quality of included reviews, 10 reviews were rated good 4 , 10 , 11 , 14 - 16 , 18 , 19 , 21 , 23 and 7 reviews were rated fair. 12 , 13 , 17 , 20 , 22 , 24 , 25 See eAppendix 3 in Supplement 1 for each review’s rating.

Figure 2 is a visual depiction of the following types of information about each included systematic review: condition, types of comparison treatments (shapes), types of massage therapy (color), number of articles included for each conclusion (bubble size), outcomes associated with massage therapy for pain (columns), and certainty of evidence rating (rows). There were 6 reviews mapped more than once, reflecting primary studies describing more than 1 health condition, 18 more than 1 type of massage therapy, 10 , 20 or outcomes associated with massage therapy compared with different comparators. 11 , 17 - 19 There were 7 conditions from reviews 14 , 16 - 19 , 21 , 22 that reported 1 conclusion rated as moderate-certainty evidence, all of which concluded that massage therapy was associated with beneficial outcomes for pain ( Table 1 ). However, most other conditions had conclusions rated as low- or very low–certainty evidence (12 reviews about 10 conditions 4 , 10 - 13 , 15 , 17 - 20 , 23 - 25 ). This rating means “Our confidence in the effect estimate is limited. The true effect may be substantially different from the estimate of effect,” or “We have very little confidence in the effect estimate.” See eAppendix 3 in Supplement 1 for conclusions in all reviews. This map included 4 conditions that did not appear in the 2018 map, 12 , 16 , 20 , 23 and there were 8 conditions in the 2018 map that did not have new reviews meeting eligibility criteria (mainly a formal grading of the certainty of evidence); 7 health conditions 10 , 11 , 13 - 15 , 17 , 18 , 21 , 22 , 24 , 25 were included in the 2018 map and the new map (see details in eAppendix 4 in Supplement 1 ).

Evidence about adverse events was collected by approximately half of included reviews, and no serious adverse events were reported. While 11 of 17 reviews 10 , 11 , 13 , 15 , 17 - 19 , 22 - 25 described adverse events, 2 reviews 18 , 23 included certainty of evidence conclusions for adverse events for 3 health conditions ( Table 2 ).

There is a large literature of original randomized clinical trials and systematic reviews of randomized clinical trials of massage therapy as a treatment for pain. Our systematic review found that despite this literature, there were only a few conditions for which authors of systematic reviews concluded that there was at least moderate-certainty evidence regarding health outcomes associated with massage therapy and pain. Most reviews reported low- or very low–certainty evidence. Although adverse events associated with massage therapy for pain were rare, the evidence was limited. For reviews that had conclusions about adverse events, authors were uncertain if there was a difference between groups or did not find a difference between groups and rated the evidence low to very low certainty of evidence.

Massage therapy is a broad term that is inclusive of many styles and techniques. We applied exclusion criteria determined a priori to help identify publications for inclusion in the evidence map. Despite that procedure, there was still a lack of clarity in determining what massage therapy is. For instance, acupressure was sometimes considered acupuncture and other times considered massage therapy, depending on author definition. In this case, we reviewed and included only publications that were explicitly labeled acupressure and did not review publications about acupuncture only. This highlights a fundamental issue with examining the evidence base of massage therapy for pain when there is ambiguity in defining what is considered massage therapy.

Unlike a pharmaceutical placebo, sham massage therapy may not be truly inactive. It is conceivable that even the light touch or touch with no clear criterion 26 used in sham massage therapy may be associated with some positive outcomes, meaning that patients who receive the massage therapy intervention and those who receive a sham massage therapy could both demonstrate some degree of symptom improvement. Limitations of sham comparators raise the question of whether sham or placebo treatment is an appropriate comparison group in massage therapy trials. It may be more informative to compare massage therapy with other treatments that are accessible and whose benefits are known so that any added beneficial outcomes associated with massage therapy could be better isolated and understood.

Compared with the 2018 map, our map included 4 new conditions not on the 2018 map, while 8 conditions from the 2018 map had no new reviews meeting eligibility criteria and 7 health conditions appeared in both maps. Despite identifying new conditions and conclusions with higher certainty of evidence in several reviews in our updated search, most included reviews reported low or very low certainty of evidence, suggesting that the most critical research need is for better evidence to increase certainty of evidence for massage therapy for pain. This is a challenge given that massage, like other complementary and integrative health interventions, does not have the historical research infrastructure that most health professions have. 27 Nevertheless, it is only when systematic reviews and meta-analyses are conducted with high-quality primary studies that the association or lack of association of massage therapy with pain will reach higher certainties of evidence. Studies comparing massage therapy with placebo or sham are probably not the priority; rather, the priority should be studies comparing massage therapy with other recommended, accepted, and active therapies for pain. Studies comparing massage therapy with other recommended therapies should also have a sufficiently long follow-up to allow any nonspecific outcomes (eg, those associated with receiving some new treatment) to dissipate. For example, this period has been proposed to be at least 6 months for studies of chronic pain.

There are 2 main limitations to this systematic review’s evidence map. The first, common to all systematic reviews, is that we may not have identified all potentially eligible evidence. If a systematic review was published in a journal not indexed in any of 5 databases we searched and we did not identify it as part of our search of references of included publications, then we would have missed it. Nevertheless, our search strategy identified more than 200 publications about massage therapy for pain published since July 2018, so we did not lack potential reviews to evaluate. The second limitation of evidence maps is that we did not independently evaluate the source evidence; in other words, we took conclusions of authors of the systematic review at face value. That is the nature of an evidence map. Particular to this application of the mapping process, we mapped the review we deemed most informative for the 2 health conditions that had more than 1 eligible review (back pain and labor pain). This necessarily requires judgment, and others could disagree with that judgment. We included the citation for reviews excluded from the map for this overlap reason in supplemental material, and interested readers can review it for additional information. As in all evidence-based products and particularly in 1 such as this covering a large and complex evidence base, it is possible that there are errors of data extraction and compilation. We used dual review to minimize the chance of such errors, but if we are notified of errors, we will correct them.

Although this systematic review found that the number of conclusions about the effectiveness of massage therapy that were judged to have at least moderate certainty of evidence was greater now than in 2018, it was still small relative to the need. More high-quality randomized clinical trials are needed to provide a stronger evidence base to assess the effect of massage therapy on pain. For painful conditions that do not have at least moderate-certainty evidence supporting use of massage therapy, new studies that address limitations of existing research are needed. The field of massage therapy would be best advanced by educating the wider research community with clearer definitions of massage therapy and whether it is appropriate to include multiple modalities in the same systematic review.

Accepted for Publication: May 15, 2024.

Published: July 15, 2024. doi:10.1001/jamanetworkopen.2024.22259

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Mak S et al. JAMA Network Open .

Corresponding Author: Selene Mak, PhD, MPH, Veterans Health Administration, Greater Los Angeles Healthcare System, 11301 Wilshire Blvd, Los Angeles, CA 90073 ( [email protected] ).

Author Contributions: Drs Mak and Shekelle had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Mak, Miake-Lye, Shekelle.

Acquisition, analysis, or interpretation of data: Mak, Allen, Begashaw, Beroes-Severin, De Vries, Lawson, Shekelle.

Drafting of the manuscript: Mak, Allen, Begashaw, Beroes-Severin, De Vries, Lawson, Shekelle.

Critical review of the manuscript for important intellectual content: Mak, Miake-Lye, Shekelle.

Statistical analysis: Allen.

Obtained funding: Shekelle.

Administrative, technical, or material support: Begashaw, Miake-Lye, Beroes-Severin, De Vries, Lawson.

Supervision: Mak, Shekelle.

Conflict of Interest Disclosures: None reported.

Funding/Support: Funding was provided by the Department of Veterans Affairs Health Services Research and Development.

Role of the Funder/Sponsor: The funders had no role in the collection, management, analysis, and interpretation of the data and preparation of the manuscript. The funders participated in the design and conduct of the study, the review and approval of the manuscript, and the decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 2 .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Open-source software analysis tools added by Endor Labs

Credit: Adobe Stock Images

During the Black Hat USA 2024 conference, Endor Labs unveiled new tools designed to enhance the security of open-source software within its software supply chain platform, reports DevOps .

The additional functions include analytics that assess the difficulty of upgrading an open-source software package and the potential risk of application disruption. The ability allows DevSecOps teams to make more informed decisions on whether to upgrade or patch a module. The platform also introduced Endor Magic Patches, which enable teams to apply patches from later releases to earlier versions of a module when upgrading is deemed too risky or complex.

The new tools address a critical gap in existing software composition analysis tools, which often identify vulnerabilities without offering actionable remediation advice, according to Jenn Gile, Endor Labs' director of product marketing. The enhancement is especially pertinent given the challenges posed by maintaining open-source software, as seen in the notorious Log4J vulnerability. The updates aim to help organizations better manage the risks associated with open-source dependencies and respond swiftly to emerging zero-day vulnerabilities.

Multiple organization targeting conducted in most cyberattacks

SC Staff August 21, 2024

Intrusions leveraging mass vulnerability scanning to compromise several organizations have accounted for 91% of all cyberattacks this year, compared with 69% in 2023.

Are your GitHub Action artifacts leaking tokens?

Laura French August 13, 2024

Palo Alto research found many open-source projects can be compromised through public artifacts.

Understanding and reducing supply chain risk and software vulnerability risks

SC Staff August 8, 2024

CyberRisk TV speaks with ThreatLocker's Danny Jenkins at the Black Hat 2024 conference in Las Vegas on their approach to protection rather than detection.

Related Events

How cisos manage supply chain risk.

On-Demand Event

Critical Connection: Strategies for Strengthening Supply Chain Defense

Apiiro’s integrated application security posture management (aspm) + software supply chain security (sscs) solution, get daily email updates.

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy .

Featured Article

Open source tools to boost your productivity

It’s good to have options if you want to escape the proprietary clutches of big tech.

For every yin, there’s a yang; for every action, a reaction; and for every piece of proprietary software, there’s an open source alternative. Or something like that.

The issue of “openness” in technology has rarely been so front and center in the public consciousness as it has these past couple of years. Twitter’s  steady demise has drawn millions to explore alternatives , many of which are open source . And the OpenAI power struggle last year also shone a spotlight on what “open source” might actually mean in the context of the burgeoning AI revolution .

The consumer software world has long offered “open” alternatives to the established incumbents, be that LibreOffice instead of Microsoft Office; GIMP over Photoshop; or Thunderbird in place of Outlook. There might be any number of reasons why an individual or business might prefer to journey down the open source route: Maybe it’s the added transparency and security compared to the proprietary players or the customizability it offers. Or some might just like to support a software development ethos that favors freedom and collaboration over walled gardens and vendor lock-in.

There are potential downsides to open source software, such as a lack of formal customer support, limited features, or technical hurdles around deployment. But it’s still good to know your options if you’re looking to bring a little more openness to your app stack — without compromising too much on your productivity.

With that in mind, TechCrunch has pulled together some open source alternatives to popular productivity apps. These might appeal to prosumers, freelancers, or small businesses looking to escape the clutches of the usual Big Tech players.

Penpot: Design and prototyping

Penpot is an open source web-based design tool, offering a range of plans targeting everyone from individuals to enterprises.

Although regulatory headwinds ultimately put an end to Adobe’s $20 billion bid for Figma last year, Penpot saw signups surge when news of the plans first emerged — and the startup went on to raise $8 million off the back of this .

Excalidraw is also a neat open source whiteboarding tool with collaborative features built in; it’s also worth checking out.

Cal.com: “Scheduling infrastructure”

Everyone loves Calendly, the scheduling platform (worth $3 billion) that helps people organize meetings without having to engage in multiple back-and-forth emails, messages and phone calls.

There’s also an open source challenger called Cal.com , touted as “scheduling infrastructure for absolutely everyone.” Cal.com can be self-hosted or hosted by the company itself as part of a SaaS offering, with multiple plans on offer . The company also raised a fairly chunky $25 million Series A round of investment in 2022.

Screenity: Screen recording

Loom emerged as one of the beneficiaries of the rapid transition to remote work, enabling asynchronous video communication through myriad screencasting, recording and sharing features. As with many startups, Loom struggled as the world returned to some semblance of normality, and the company exited to Atlassian last year for just shy of $1 billion .

Still, remote work isn’t going away, and anyone looking for an open source Loom alternative that doesn’t sit under the auspices of a billion-dollar corporation could do worse than checking out Screenity . However, it is limited to the Chrome browser for now.

Jitsi: Video conferencing

Zoom was one of the big winners of the rapid shift to remote work, but that doesn’t mean there isn’t room for a fully self-hostable and configurable video-conferencing alternative. That’s exactly what’s offered by Jitsi , an open source community-driven project started by founder Emil Ivov way back in 2003.

Users can head to meet.jitsi.com and instantly start a meeting. And while Jitsi is open source and free for anyone to deploy as they please, its parent company, 8×8, offers a paid service with additional features. It’s worth noting that 8×8 acquired Jitsi from Atlassian in 2018 .

Nextcloud: Cloud storage

Nextcloud is both a client-side and server-side solution for file storage. Other cloud storage providers, such as Shadow Drive , use Nextcloud’s infrastructure under the hood.

Users can self-host their Nextcloud instances or use a third-party provider suggested by Nextcloud, which helps users manage the setup and maintenance process.

Nextcloud can be used by hobbyists or enterprises looking to sidestep industry incumbents such as Dropbox, with a strong emphasis on privacy, security and data sovereignty.

Ghost: Publishing

Substack has built a business around providing tools for writers to create newsletters and similar content. However, Substack is a closed ecosystem, much like Medium.

Ghost is an open source publishing platform developed by former WordPress engineer John O’Nolan in 2013. While WordPress is also an open source solution, Ghost offers a cheaper alternative with a managed hosting instance and doesn’t take any fees from publishers on subscriptions.

Ghost raised $300,000 through crowdfunding during the project’s initial phase in 2013, with notable backers like Seth Godin, Leo Babauta and Microsoft.

TabbyML: Coding copilot

GitHub Copilot has emerged as the poster child of the AI-powered pair-progamming space, though Google and Amazon have introduced similar smarts .

Regardless, none of these incumbents’ coding assistants is open source and they can’t be self-hosted — something TabbyML hopes to address. Founded by two former Googlers last year, TabbyML raised $3.2 million in seed funding for an early iteration of what it calls an open source GitHub Copilot alternative, one that can be entirely self-hosted.

Chatwoot: Customer support

As one of the major players in the customer relationship space, Zendesk needs little introduction. But the private equity-owned facet of Zendesk might not be to everyone’s fancy. Plus Zendesk is, well, entirely proprietary.

Chatwoot , on the other hand, touts its open source chops that allow businesses to self-host the customer engagement platform, thus keeping all their data in-house.

PhotoPrism: Photo management

In 2020, Google Photos ended its free unlimited tier . The same year, a Berlin-based team operating under the name PhotoPrism emerged as an alternative of sorts, with the ability to run a self-hosted server on your desktop (Windows, Mac, or Linux) along with DigitalOcean, Raspberry Pi, FreeBSD, and many network-attached storage ( NAS ) devices.

PhotoPrism includes support for backing up photos, as well as tools for converting files, detecting duplicates, and recognizing friend-and-family faces in photos. The company offers a range of plans designed for individuals and organizations alike, with self-hosted and hosted options available.

Bitwarden: Password management

From LastPass to Dashlane and 1Password, there are no shortages of password-management tools out there that generate hard-to-guess passwords and store them in a secure digital vault. But Bitwarden has set itself apart by operating largely under an open source model, raising a chunky $100 million in funding in the process.

Bitwarden’s core components are open source, allowing anyone to view, modify and distribute the code. However, certain features are only available under a proprietary “source available” license, which still offers transparency, albeit with greater restrictions on what the end user can do with it.

AppFlowy: Task management

AppFlowy is an open source alternative to Notion , the $10 billion workplace productivity and collaboration tool.

Founded in 2021, AppFlowy pitches a self-hostable solution replete with tools for managing projects, taking notes, creating documents, and tracking the status of individual project items and deadlines. The company raised $6.4 million in funding last year from a who’s who of investors, including the founders of Automattic and YouTube.

Dub.co: Link management

For link shortening and management, Spectrum Equity owned Bitly is one of the popular tools, while Google recently put a final nail in its URL shortening service’s coffin . If you are looking for an open source alternative, Dub.co could be your answer.

Former Vercel employee Steven Tey started this as a side project in 2022 , transforming it into a company two years later. While Dub.co provides a lot of its own URL management services, including time-series data, personalization and a way to use branded links, it also allows you to self-host its solution .

More TechCrunch

Get the industry’s biggest tech news, techcrunch daily news.

Every weekday and Sunday, you can get the best of TechCrunch’s coverage.

Startups Weekly

Startups are the core of TechCrunch, so get our best coverage delivered weekly.

TechCrunch Fintech

The latest Fintech news and analysis, delivered every Tuesday.

TechCrunch Mobility

TechCrunch Mobility is your destination for transportation news and insight.

Bodies of Autonomy founder Mike Lynch and his daughter recovered from sunken Bayesian yacht

Grim news coming in from Sicily, Italy. Mike Lynch, the U.K. investor and founder of IT company Autonomy, has been identified as one of two bodies recovered from the Bayesian,…

Zomato buys Paytm’s entertainment ticket business for $244 million

Zomato, the Indian food delivery giant, has acquired the entertainment ticketing business of financial services firm Paytm for $244.1 million, signaling a strategic move to expand its “going out” offerings.…

Two incident management startups join forces as FireHydrant nabs Blameless

FireHydrant, an NYC incident management startup that launched in 2019, announced on Wednesday that it has acquired Blameless, a former competitor. The companies did not share the purchase price. Both…

Only 3 days left for massive savings on tickets to TechCrunch Disrupt 2024

Tick-tock! We’re in the last 3 days to save up to $600 on TechCrunch Disrupt 2024 tickets. Lock in these low prices before August 23 at 11:59 p.m. PT. Be…

Ring’s new battery doorbell has head-to-toe video

Ring announced on Wednesday the next generation of its Battery Doorbell. For $100, customers get extended battery life, color night vision, a head-to-toe view of visitors and a new push-pin…

Openmart wants to make it easier for enterprises to sell to local businesses

In 2020, Kathryn Wu launched a side hustle while she was working as a product engineer at Pinterest. Wu started a milk tea company, OhTea, with the hopes of connecting…

Grafana Labs raises $270M

Grafana Labs, the open source company that probably powers at least a few operational dashboards in your company, today announced the completion of a primary and secondary transaction worth about…

The new generation of heavy lift rockets is rising to challenge SpaceX

It can be hard to keep up with this fast-paced yet incredibly complex and highly regulated industry — but everything you need to stay up to date is right here, with links to deeper coverage.

Ford says a pickup truck will be the first EV built on its low-cost platform

The first vehicle that will be built on the low-cost EV platform Ford’s been developing in secret will be a mid-size pickup due out in 2027, the automaker announced Wednesday…

UK’s competition authority ends probes of Apple and Google but will use incoming powers to ‘resolve app store concerns’

Apple and Google can take a breath after the U.K. competition watchdog announced Wednesday it’s closing a pair of investigations into their respective mobile app ecosystems, citing administrative priorities. Since…

Story raises $80M at $2.25B valuation to build a blockchain for the business of content IP in the age of AI

AI giants like Anthropic, OpenAI and Stability AI have faced a lot of heat over how they’ve scraped data and rode rough-shod over others’ intellectual property when training and operating…

Skyfire lets AI agents spend your money

There’s a lot of hype about the promise of AI agents today, but payments are a huge limiting factor. Today, an AI agent might be able to plan a vacation…

WeTransfer’s expiry dates haven’t gone away, but you can at least extend them

WeTransfer’s expiry dates have been a major paint-point for users — especially creatives. The thing is, if you don’t download the files the moment you get the link or email…

India’s trade minister decries e-commerce growth, Amazon’s ‘predatory’ pricing

India’s commerce minister Piyush Goyal on Wednesday expressed concern over the rapid growth of e-commerce in the country, warning of potential disruption to small retailers. Speaking at the launch of…

SleekFlow snaps up $7M to tap the conversational AI opportunity across Asia  

SleekFlow, a Singapore- and Hong Kong-headquartered social commerce platform that has built a conversational AI suite for customer engagement targeted to Asian markets, said Wednesday it had secured a further…

The banks that loaned Musk $13B to buy Twitter might be having regrets

X, formerly known as Twitter, looks like a pretty bad investment right about now. As readers might recall, Elon Musk borrowed $13 billion from Morgan Stanley, Bank of America and…

Is your company AI washing? Rippling founder Parker Conrad thinks it might be.

Right now, there’s a mad scramble to capitalize on AI.

Move over, Apple: Meet the alternative app stores coming to the EU

A list of some of the alternative app stores iPhone users in the EU can try today. 

Google’s Nest Learning Thermostat is still the one to beat

The Nest Learning Thermostat remains a beautiful piece of industrial design, with some fresh twists that help it more seamlessly blend in.

Watch this robot quickly install roof shingles

Renovate says this model is faster and more accurate than its predecessor, despite being roughly half its weight.

Cisco employees face a month of silence ahead of second layoff in 2024

After tech giant Cisco announced plans for its second round of layoffs this year, employees tell TechCrunch that they will not know if they are affected for close to a…

OpenAI signs deal to train on Condé Nast content, surface stories in ChatGPT

OpenAI has inked a deal with Condé Nast — the publisher of storied outlets such as The New Yorker, Vogue, and Wired — to surface stories from its properties in…

PayPal could challenge Apple Wallet in the EU

Though PayPal has not yet confirmed it will now develop a competing wallet thanks to new APIs, the company has hinted in that such a plan is in the works.

Fintech shutdowns, Klarna’s move into banking and which companies are hiring

Welcome to TechCrunch Fintech! This week, we’re looking at the respective demise of Tally and Score, plus examining Klarna’s latest business move. We also have a list of fintech companies…

How Fabric plans to make advanced cryptography ubiquitous

Fabric has developed a custom RISC-V-based chip that is optimized to run the algorithms necessary to establish zero-knowledge proofs and enable fully homomorphic encryption.

Life sciences investor BEVC raising $25M climate fund

BEVC is the latest life sciences VC to add planetary health to its portfolio, suggesting that the trend is catching hold.

4 days left to save big on TechCrunch Disrupt 2024 tickets

The countdown continues! You have 4 days left to lock in your TechCrunch Disrupt 2024 tickets at the discounted rate. Don’t miss the chance to save up to $600 —…

YouTube takes on TikTok Shop with expanded Shopify partnership

As TikTok Shop gains traction, YouTube is expanding its partnership with Shopify to onboard more brands for its YouTube Shopping affiliate program, the company announced on Tuesday. The expanded partnership…

Waymo is now giving 100,000 robotaxi rides a week

Waymo disclosed Tuesday it’s now giving more than 100,000 paid robotaxi rides every week across its three main commercial markets in Los Angeles, San Francisco and Phoenix. Those figures were…

Discover how founders and investors are saving the planet at TechCrunch Disrupt 2024

Until recently, saving the world usually didn’t involve turning a profit. But as the world has warmed, a range of startups and investors have emerged that have squared the circle,…