latest research paper on recommender system

• Survey paper
• Open access
• Published: 03 May 2022

A systematic review and research perspective on recommender systems

• Deepjyoti Roy   ORCID: orcid.org/0000-0002-8020-7145 1 &
• Mala Dutta 1  

Journal of Big Data volume  9 , Article number:  59 ( 2022 ) Cite this article

78k Accesses

139 Citations

9 Altmetric

Metrics details

Recommender systems are efficient tools for filtering online information, which is widespread owing to the changing habits of computer users, personalization trends, and emerging access to the internet. Even though the recent recommender systems are eminent in giving precise recommendations, they suffer from various limitations and challenges like scalability, cold-start, sparsity, etc. Due to the existence of various techniques, the selection of techniques becomes a complex work while building application-focused recommender systems. In addition, each technique comes with its own set of features, advantages and disadvantages which raises even more questions, which should be addressed. This paper aims to undergo a systematic review on various recent contributions in the domain of recommender systems, focusing on diverse applications like books, movies, products, etc. Initially, the various applications of each recommender system are analysed. Then, the algorithmic analysis on various recommender systems is performed and a taxonomy is framed that accounts for various components required for developing an effective recommender system. In addition, the datasets gathered, simulation platform, and performance metrics focused on each contribution are evaluated and noted. Finally, this review provides a much-needed overview of the current state of research in this field and points out the existing gaps and challenges to help posterity in developing an efficient recommender system.

Introduction

The recent advancements in technology along with the prevalence of online services has offered more abilities for accessing a huge amount of online information in a faster manner. Users can post reviews, comments, and ratings for various types of services and products available online. However, the recent advancements in pervasive computing have resulted in an online data overload problem. This data overload complicates the process of finding relevant and useful content over the internet. The recent establishment of several procedures having lower computational requirements can however guide users to the relevant content in a much easy and fast manner. Because of this, the development of recommender systems has recently gained significant attention. In general, recommender systems act as information filtering tools, offering users suitable and personalized content or information. Recommender systems primarily aim to reduce the user’s effort and time required for searching relevant information over the internet.

Nowadays, recommender systems are being increasingly used for a large number of applications such as web [ 1 , 67 , 70 ], books [ 2 ], e-learning [ 4 , 16 , 61 ], tourism [ 5 , 8 , 78 ], movies [ 66 ], music [ 79 ], e-commerce, news, specialized research resources [ 65 ], television programs [ 72 , 81 ], etc. It is therefore important to build high-quality and exclusive recommender systems for providing personalized recommendations to the users in various applications. Despite the various advances in recommender systems, the present generation of recommender systems requires further improvements to provide more efficient recommendations applicable to a broader range of applications. More investigation of the existing latest works on recommender systems is required which focus on diverse applications.

There is hardly any review paper that has categorically synthesized and reviewed the literature of all the classification fields and application domains of recommender systems. The few existing literature reviews in the field cover just a fraction of the articles or focus only on selected aspects such as system evaluation. Thus, they do not provide an overview of the application field, algorithmic categorization, or identify the most promising approaches. Also, review papers often neglect to analyze the dataset description and the simulation platforms used. This paper aims to fulfil this significant gap by reviewing and comparing existing articles on recommender systems based on a defined classification framework, their algorithmic categorization, simulation platforms used, applications focused, their features and challenges, dataset description and system performance. Finally, we provide researchers and practitioners with insight into the most promising directions for further investigation in the field of recommender systems under various applications.

In essence, recommender systems deal with two entities—users and items, where each user gives a rating (or preference value) to an item (or product). User ratings are generally collected by using implicit or explicit methods. Implicit ratings are collected indirectly from the user through the user’s interaction with the items. Explicit ratings, on the other hand, are given directly by the user by picking a value on some finite scale of points or labelled interval values. For example, a website may obtain implicit ratings for different items based on clickstream data or from the amount of time a user spends on a webpage and so on. Most recommender systems gather user ratings through both explicit and implicit methods. These feedbacks or ratings provided by the user are arranged in a user-item matrix called the utility matrix as presented in Table 1 .

The utility matrix often contains many missing values. The problem of recommender systems is mainly focused on finding the values which are missing in the utility matrix. This task is often difficult as the initial matrix is usually very sparse because users generally tend to rate only a small number of items. It may also be noted that we are interested in only the high user ratings because only such items would be suggested back to the users. The efficiency of a recommender system greatly depends on the type of algorithm used and the nature of the data source—which may be contextual, textual, visual etc.

Types of recommender systems

Recommender systems are broadly categorized into three different types viz. content-based recommender systems, collaborative recommender systems and hybrid recommender systems. A diagrammatic representation of the different types of recommender systems is given in Fig.  1 .

Content-based recommender system

In content-based recommender systems, all the data items are collected into different item profiles based on their description or features. For example, in the case of a book, the features will be author, publisher, etc. In the case of a movie, the features will be the movie director, actor, etc. When a user gives a positive rating to an item, then the other items present in that item profile are aggregated together to build a user profile. This user profile combines all the item profiles, whose items are rated positively by the user. Items present in this user profile are then recommended to the user, as shown in Fig.  2 .

One drawback of this approach is that it demands in-depth knowledge of the item features for an accurate recommendation. This knowledge or information may not be always available for all items. Also, this approach has limited capacity to expand on the users' existing choices or interests. However, this approach has many advantages. As user preferences tend to change with time, this approach has the quick capability of dynamically adapting itself to the changing user preferences. Since one user profile is specific only to that user, this algorithm does not require the profile details of any other users because they provide no influence in the recommendation process. This ensures the security and privacy of user data. If new items have sufficient description, content-based techniques can overcome the cold-start problem i.e., this technique can recommend an item even when that item has not been previously rated by any user. Content-based filtering approaches are more common in systems like personalized news recommender systems, publications, web pages recommender systems, etc.

Collaborative filtering-based recommender system

Collaborative approaches make use of the measure of similarity between users. This technique starts with finding a group or collection of user X whose preferences, likes, and dislikes are similar to that of user A. X is called the neighbourhood of A. The new items which are liked by most of the users in X are then recommended to user A. The efficiency of a collaborative algorithm depends on how accurately the algorithm can find the neighbourhood of the target user. Traditionally collaborative filtering-based systems suffer from the cold-start problem and privacy concerns as there is a need to share user data. However, collaborative filtering approaches do not require any knowledge of item features for generating a recommendation. Also, this approach can help to expand on the user’s existing interests by discovering new items. Collaborative approaches are again divided into two types: memory-based approaches and model-based approaches.

Memory-based collaborative approaches recommend new items by taking into consideration the preferences of its neighbourhood. They make use of the utility matrix directly for prediction. In this approach, the first step is to build a model. The model is equal to a function that takes the utility matrix as input.

Model = f (utility matrix)

Then recommendations are made based on a function that takes the model and user profile as input. Here we can make recommendations only to users whose user profile belongs to the utility matrix. Therefore, to make recommendations for a new user, the user profile must be added to the utility matrix, and the similarity matrix should be recomputed, which makes this technique computation heavy.

Recommendation = f (defined model, user profile) where user profile  ∈  utility matrix

Memory-based collaborative approaches are again sub-divided into two types: user-based collaborative filtering and item-based collaborative filtering. In the user-based approach, the user rating of a new item is calculated by finding other users from the user neighbourhood who has previously rated that same item. If a new item receives positive ratings from the user neighbourhood, the new item is recommended to the user. Figure  3 depicts the user-based filtering approach.

User-based collaborative filtering

In the item-based approach, an item-neighbourhood is built consisting of all similar items which the user has rated previously. Then that user’s rating for a different new item is predicted by calculating the weighted average of all ratings present in a similar item-neighbourhood as shown in Fig.  4 .

Item-based collaborative filtering

Model-based systems use various data mining and machine learning algorithms to develop a model for predicting the user’s rating for an unrated item. They do not rely on the complete dataset when recommendations are computed but extract features from the dataset to compute a model. Hence the name, model-based technique. These techniques also need two steps for prediction—the first step is to build the model, and the second step is to predict ratings using a function (f) which takes the model defined in the first step and the user profile as input.

Recommendation = f (defined model, user profile) where user profile  ∉  utility matrix

Model-based techniques do not require adding the user profile of a new user into the utility matrix before making predictions. We can make recommendations even to users that are not present in the model. Model-based systems are more efficient for group recommendations. They can quickly recommend a group of items by using the pre-trained model. The accuracy of this technique largely relies on the efficiency of the underlying learning algorithm used to create the model. Model-based techniques are capable of solving some traditional problems of recommender systems such as sparsity and scalability by employing dimensionality reduction techniques [ 86 ] and model learning techniques.

Hybrid filtering

A hybrid technique is an aggregation of two or more techniques employed together for addressing the limitations of individual recommender techniques. The incorporation of different techniques can be performed in various ways. A hybrid algorithm may incorporate the results achieved from separate techniques, or it can use content-based filtering in a collaborative method or use a collaborative filtering technique in a content-based method. This hybrid incorporation of different techniques generally results in increased performance and increased accuracy in many recommender applications. Some of the hybridization approaches are meta-level, feature-augmentation, feature-combination, mixed hybridization, cascade hybridization, switching hybridization and weighted hybridization [ 86 ]. Table 2 describes these approaches.

Recommender system challenges

This section briefly describes the various challenges present in current recommender systems and offers different solutions to overcome these challenges.

Cold start problem

The cold start problem appears when the recommender system cannot draw any inference from the existing data, which is insufficient. Cold start refers to a condition when the system cannot produce efficient recommendations for the cold (or new) users who have not rated any item or have rated a very few items. It generally arises when a new user enters the system or new items (or products) are inserted into the database. Some solutions to this problem are as follows: (a) Ask new users to explicitly mention their item preference. (b) Ask a new user to rate some items at the beginning. (c) Collect demographic information (or meta-data) from the user and recommend items accordingly.

Shilling attack problem

This problem arises when a malicious user fakes his identity and enters the system to give false item ratings [ 87 ]. Such a situation occurs when the malicious user wants to either increase or decrease some item’s popularity by causing a bias on selected target items. Shilling attacks greatly reduce the reliability of the system. One solution to this problem is to detect the attackers quickly and remove the fake ratings and fake user profiles from the system.

Synonymy problem

This problem arises when similar or related items have different entries or names, or when the same item is represented by two or more names in the system [ 78 ]. For example, babywear and baby cloth. Many recommender systems fail to distinguish these differences, hence reducing their recommendation accuracy. To alleviate this problem many methods are used such as demographic filtering, automatic term expansion and Singular Value Decomposition [ 76 ].

Latency problem

The latency problem is specific to collaborative filtering approaches and occurs when new items are frequently inserted into the database. This problem is characterized by the system’s failure to recommend new items. This happens because new items must be reviewed before they can be recommended in a collaborative filtering environment. Using content-based filtering may resolve this issue, but it may introduce overspecialization and decrease the computing time and system performance. To increase performance, the calculations can be done in an offline environment and clustering-based techniques can be used [ 76 ].

Sparsity problem

Data sparsity is a common problem in large scale data analysis, which arises when certain expected values are missing in the dataset. In the case of recommender systems, this situation occurs when the active users rate very few items. This reduces the recommendation accuracy. To alleviate this problem several techniques can be used such as demographic filtering, singular value decomposition and using model-based collaborative techniques.

Grey sheep problem

The grey sheep problem is specific to pure collaborative filtering approaches where the feedback given by one user do not match any user neighbourhood. In this situation, the system fails to accurately predict relevant items for that user. This problem can be resolved by using pure content-based approaches where predictions are made based on the user’s profile and item properties.

Scalability problem

Recommender systems, especially those employing collaborative filtering techniques, require large amounts of training data, which cause scalability problems. The scalability problem arises when the amount of data used as input to a recommender system increases quickly. In this era of big data, more and more items and users are rapidly getting added to the system and this problem is becoming common in recommender systems. Two common approaches used to solve the scalability problem is dimensionality reduction and using clustering-based techniques to find users in tiny clusters instead of the complete database.

Methodology

The purpose of this study is to understand the research trends in the field of recommender systems. The nature of research in recommender systems is such that it is difficult to confine each paper to a specific discipline. This can be further understood by the fact that research papers on recommender systems are scattered across various journals such as computer science, management, marketing, information technology and information science. Hence, this literature review is conducted over a wide range of electronic journals and research databases such as ACM Portal, IEEE/IEE Library, Google Scholars and Science Direct [ 88 ].

The search process of online research articles was performed based on 6 descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*”, “Hybrid Recommend*”. The following research papers described below were excluded from our research:

News articles.

Master’s dissertations.

Non-English papers.

Unpublished papers.

Research papers published before 2011.

We have screened a total of 350 articles based on their abstracts and content. However, only research papers that described how recommender systems can be applied were chosen. Finally, 60 papers were selected from top international journals indexed in Scopus or E-SCI in 2021. We now present the PRISMA flowchart of the inclusion and exclusion process in Fig.  5 .

PRISMA flowchart of the inclusion and exclusion process. Abstract and content not suitable to the study: * The use or application of the recommender system is not specified: **

Each paper was carefully reviewed and classified into 6 categories in the application fields and 3 categories in the techniques used to develop the system. The classification framework is presented in Fig.  6 .

Classification framework

The number of relevant articles come from Expert Systems with Applications (23%), followed by IEEE (17%), Knowledge-Based System (17%) and Others (43%). Table 3 depicts the article distribution by journal title and Table 4 depicts the sector-wise article distribution.

Both forward and backward searching techniques were implemented to establish that the review of 60 chosen articles can represent the domain literature. Hence, this paper can demonstrate its validity and reliability as a literature review.

Review on state-of-the-art recommender systems

This section presents a state-of-art literature review followed by a chronological review of the various existing recommender systems.

Literature review

In 2011, Castellano et al. [ 1 ] developed a “NEuro-fuzzy WEb Recommendation (NEWER)” system for exploiting the possibility of combining computational intelligence and user preference for suggesting interesting web pages to the user in a dynamic environment. It considered a set of fuzzy rules to express the correlations between user relevance and categories of pages. Crespo et al. [ 2 ] presented a recommender system for distance education over internet. It aims to recommend e-books to students using data from user interaction. The system was developed using a collaborative approach and focused on solving the data overload problem in big digital content. Lin et al. [ 3 ] have put forward a recommender system for automatic vending machines using Genetic algorithm (GA), k-means, Decision Tree (DT) and Bayesian Network (BN). It aimed at recommending localized products by developing a hybrid model combining statistical methods, classification methods, clustering methods, and meta-heuristic methods. Wang and Wu [ 4 ] have implemented a ubiquitous learning system for providing personalized learning assistance to the learners by combining the recommendation algorithm with a context-aware technique. It employed the Association Rule Mining (ARM) technique and aimed to increase the effectiveness of the learner’s learning. García-Crespo et al. [ 5 ] presented a “semantic hotel” recommender system by considering the experiences of consumers using a fuzzy logic approach. The system considered both hotel and customer characteristics. Dong et al. [ 6 ] proposed a structure for a service-concept recommender system using a semantic similarity model by integrating the techniques from the view of an ontology structure-oriented metric and a concept content-oriented metric. The system was able to deliver optimal performance when compared with similar recommender systems. Li et al. [ 7 ] developed a Fuzzy linguistic modelling-based recommender system for assisting users to find experts in knowledge management systems. The developed system was applied to the aircraft industry where it demonstrated efficient and feasible performance. Lorenzi et al. [ 8 ] presented an “assumption-based multiagent” system to make travel package recommendations using user preferences in the tourism industry. It performed different tasks like discovering, filtering, and integrating specific information for building a travel package following the user requirement. Huang et al. [ 9 ] proposed a context-aware recommender system through the extraction, evaluation and incorporation of contextual information gathered using the collaborative filtering and rough set model.

In 2012, Chen et al. [ 10 ] presented a diabetes medication recommender model by using “Semantic Web Rule Language (SWRL) and Java Expert System Shell (JESS)” for aggregating suitable prescriptions for the patients. It aimed at selecting the most suitable drugs from the list of specific drugs. Mohanraj et al. [ 11 ] developed the “Ontology-driven bee’s foraging approach (ODBFA)” to accurately predict the online navigations most likely to be visited by a user. The self-adaptive system is intended to capture the various requirements of the online user by using a scoring technique and by performing a similarity comparison. Hsu et al. [ 12 ] proposed a “personalized auxiliary material” recommender system by considering the specific course topics, individual learning styles, complexity of the auxiliary materials using an artificial bee colony algorithm. Gemmell et al. [ 13 ] demonstrated a solution for the problem of resource recommendation in social annotation systems. The model was developed using a linear-weighted hybrid method which was capable of providing recommendations under different constraints. Choi et al. [ 14 ] proposed one “Hybrid Online-Product rEcommendation (HOPE) system” by the integration of collaborative filtering through sequential pattern analysis-based recommendations and implicit ratings. Garibaldi et al. [ 15 ] put forward a technique for incorporating the variability in a fuzzy inference model by using non-stationary fuzzy sets for replicating the variabilities of a human. This model was applied to a decision problem for treatment recommendations of post-operative breast cancer.

In 2013, Salehi and Kmalabadi [ 16 ] proposed an e-learning material recommender system by “modelling of materials in a multidimensional space of material’s attribute”. It employed both content and collaborative filtering. Aher and Lobo [ 17 ] introduced a course recommender system using data mining techniques such as simple K-means clustering and Association Rule Mining (ARM) algorithm. The proposed e-learning system was successfully demonstrated for “MOOC (Massively Open Online Courses)”. Kardan and Ebrahimi [ 18 ] developed a hybrid recommender system for recommending posts in asynchronous discussion groups. The system was built combining both collaborative filtering and content-based filtering. It considered implicit user data to compute the user similarity with various groups, for recommending suitable posts and contents to its users. Chang et al. [ 19 ] adopted a cloud computing technology for building a TV program recommender system. The system designed for digital TV programs was implemented using Hadoop Fair Scheduler (HFC), K-means clustering and k-nearest neighbour (KNN) algorithms. It was successful in processing huge amounts of real-time user data. Lucas et al. [ 20 ] implemented a recommender model for assisting a tourism application by using associative classification and fuzzy logic to predict the context. Niu et al. [ 21 ] introduced “Affivir: An Affect-based Internet Video Recommendation System” which was developed by calculating user preferences and by using spectral clustering. This model recommended videos with similar effects, which was processed to get optimal results with dynamic adjustments of recommendation constraints.

In 2014, Liu et al. [ 22 ] implemented a new route recommendation model for offering personalized and real-time route recommendations for self-driven tourists to minimize the queuing time and traffic jams infamous tourist places. Recommendations were carried out by considering the preferences of users. Bakshi et al. [ 23 ] proposed an unsupervised learning-based recommender model for solving the scalability problem of recommender systems. The algorithm used transitive similarities along with Particle Swarm Optimization (PSO) technique for discovering the global neighbours. Kim and Shim [ 24 ] proposed a recommender system based on “latent Dirichlet allocation using probabilistic modelling for Twitter” that could recommend the top-K tweets for a user to read, and the top-K users to follow. The model parameters were learned from an inference technique by using the differential Expectation–Maximization (EM) algorithm. Wang et al. [ 25 ] developed a hybrid-movie recommender model by aggregating a genetic algorithm (GA) with improved K-means and Principal Component Analysis (PCA) technique. It was able to offer intelligent movie recommendations with personalized suggestions. Kolomvatsos et al. [ 26 ] proposed a recommender system by considering an optimal stopping theory for delivering books or music recommendations to the users. Gottschlich et al. [ 27 ] proposed a decision support system for stock investment recommendations. It computed the output by considering the overall crowd’s recommendations. Torshizi et al. [ 28 ] have introduced a hybrid recommender system to determine the severity level of a medical condition. It could recommend suitable therapies for patients suffering from Benign Prostatic Hyperplasia.

In 2015, Zahálka et al. [ 29 ] proposed a venue recommender: “City Melange”. It was an interactive content-based model which used the convolutional deep-net features of the visual domain and the linear Support Vector Machine (SVM) model to capture the semantic information and extract latent topics. Sankar et al. [ 30 ] have proposed a stock recommender system based on the stock holding portfolio of trusted mutual funds. The system employed the collaborative filtering approach along with social network analysis for offering a decision support system to build a trust-based recommendation model. Chen et al. [ 31 ] have put forward a novel movie recommender system by applying the “artificial immune network to collaborative filtering” technique. It computed the affinity of an antigen and the affinity between an antibody and antigen. Based on this computation a similarity estimation formula was introduced which was used for the movie recommendation process. Wu et al. [ 32 ] have examined the technique of data fusion for increasing the efficiency of item recommender systems. It employed a hybrid linear combination model and used a collaborative tagging system. Yeh and Cheng [ 33 ] have proposed a recommender system for tourist attractions by constructing the “elicitation mechanism using the Delphi panel method and matrix construction mechanism using the repertory grids”, which was developed by considering the user preference and expert knowledge.

In 2016, Liao et al. [ 34 ] proposed a recommender model for online customers using a rough set association rule. The model computed the probable behavioural variations of online consumers and provided product category recommendations for e-commerce platforms. Li et al. [ 35 ] have suggested a movie recommender system based on user feedback collected from microblogs and social networks. It employed the sentiment-aware association rule mining algorithm for recommendations using the prior information of frequent program patterns, program metadata similarity and program view logs. Wu et al. [ 36 ] have developed a recommender system for social media platforms by aggregating the technique of Social Matrix Factorization (SMF) and Collaborative Topic Regression (CTR). The model was able to compute the ratings of users to items for making recommendations. For improving the recommendation quality, it gathered information from multiple sources such as item properties, social networks, feedback, etc. Adeniyi et al. [ 37 ] put forward a study of automated web-usage data mining and developed a recommender system that was tested in both real-time and online for identifying the visitor’s or client’s clickstream data.

In 2017, Rawat and Kankanhalli [ 38 ] have proposed a viewpoint recommender system called “ClickSmart” for assisting mobile users to capture high-quality photographs at famous tourist places. Yang et al. [ 39 ] proposed a gradient boosting-based job recommendation system for satisfying the cost-sensitive requirements of the users. The hybrid algorithm aimed to reduce the rate of unnecessary job recommendations. Lee et al. [ 40 ] proposed a music streaming recommender system based on smartphone activity usage. The proposed system benefitted by using feature selection approaches with machine learning techniques such as Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perception (MLP), Instance-based k -Nearest Neighbour (IBK), and Random Forest (RF) for performing the activity detection from the mobile signals. Wei et al. [ 41 ] have proposed a new stacked denoising autoencoder (SDAE) based recommender system for cold items. The algorithm employed deep learning and collaborative filtering method to predict the unknown ratings.

In 2018, Li et al. [ 42 ] have developed a recommendation algorithm using Weighted Linear Regression Models (WLRRS). The proposed system was put to experiment using the MovieLens dataset and it presented better classification and predictive accuracy. Mezei and Nikou [ 43 ] presented a mobile health and wellness recommender system based on fuzzy optimization. It could recommend a collection of actions to be taken by the user to improve the user’s health condition. Recommendations were made considering the user’s physical activities and preferences. Ayata et al. [ 44 ] proposed a music recommendation model based on the user emotions captured through wearable physiological sensors. The emotion detection algorithm employed different machine learning algorithms like SVM, RF, KNN and decision tree (DT) algorithms to predict the emotions from the changing electrical signals gathered from the wearable sensors. Zhao et al. [ 45 ] developed a multimodal learning-based, social-aware movie recommender system. The model was able to successfully resolve the sparsity problem of recommender systems. The algorithm developed a heterogeneous network by exploiting the movie-poster image and textual description of each movie based on the social relationships and user ratings.

In 2019, Hammou et al. [ 46 ] proposed a Big Data recommendation algorithm capable of handling large scale data. The system employed random forest and matrix factorization through a data partitioning scheme. It was then used for generating recommendations based on user rating and preference for each item. The proposed system outperformed existing systems in terms of accuracy and speed. Zhao et al. [ 47 ] have put forward a hybrid initialization method for social network recommender systems. The algorithm employed denoising autoencoder (DAE) neural network-based initialization method (ANNInit) and attribute mapping. Bhaskaran and Santhi [ 48 ] have developed a hybrid, trust-based e-learning recommender system using cloud computing. The proposed algorithm was capable of learning online user activities by using the Firefly Algorithm (FA) and K-means clustering. Afolabi and Toivanen [ 59 ] have suggested an integrated recommender model based on collaborative filtering. The proposed model “Connected Health for Effective Management of Chronic Diseases”, aimed for integrating recommender systems for better decision-making in the process of disease management. He et al. [ 60 ] proposed a movie recommender system called “HI2Rec” which explored the usage of collaborative filtering and heterogeneous information for making movie recommendations. The model used the knowledge representation learning approach to embed movie-related information gathered from different sources.

In 2020, Han et al. [ 49 ] have proposed one Internet of Things (IoT)-based cancer rehabilitation recommendation system using the Beetle Antennae Search (BAS) algorithm. It presented the patients with a solution for the problem of optimal nutrition program by considering the objective function as the recurrence time. Kang et al. [ 50 ] have presented a recommender system for personalized advertisements in Online Broadcasting based on a tree model. Recommendations were generated in real-time by considering the user preferences to minimize the overhead of preference prediction and using a HashMap along with the tree characteristics. Ullah et al. [ 51 ] have implemented an image-based service recommendation model for online shopping based random forest and Convolutional Neural Networks (CNN). The model used JPEG coefficients to achieve an accurate prediction rate. Cai et al. [ 52 ] proposed a new hybrid recommender model using a many-objective evolutionary algorithm (MaOEA). The proposed algorithm was successful in optimizing the novelty, diversity, and accuracy of recommendations. Esteban et al. [ 53 ] have implemented a hybrid multi-criteria recommendation system concerned with students’ academic performance, personal interests, and course selection. The system was developed using a Genetic Algorithm (GA) and aimed at helping university students. It combined both course information and student information for increasing system performance and the reliability of the recommendations. Mondal et al. [ 54 ] have built a multilayer, graph data model-based doctor recommendation system by exploiting the trust concept between a patient-doctor relationship. The proposed system showed good results in practical applications.

In 2021, Dhelim et al. [ 55 ] have developed a personality-based product recommending model using the techniques of meta path discovery and user interest mining. This model showed better results when compared to session-based and deep learning models. Bhalse et al. [ 56 ] proposed a web-based movie recommendation system based on collaborative filtering using Singular Value Decomposition (SVD), collaborative filtering and cosine similarity (CS) for addressing the sparsity problem of recommender systems. It suggested a recommendation list by considering the content information of movies. Similarly, to solve both sparsity and cold-start problems Ke et al. [ 57 ] proposed a dynamic goods recommendation system based on reinforcement learning. The proposed system was capable of learning from the reduced entropy loss error on real-time applications. Chen et al. [ 58 ] have presented a movie recommender model combining various techniques like user interest with category-level representation, neighbour-assisted representation, user interest with latent representation and item-level representation using Feed-forward Neural Network (FNN).

Comparative chronological review

A comparative chronological review to compare the total contributions on various recommender systems in the past 10 years is given in Fig.  7 .

Comparative chronological review of recommender systems under diverse applications

This review puts forward a comparison of the number of research works proposed in the domain of recommender systems from the year 2011 to 2021 using various deep learning and machine learning-based approaches. Research articles are categorized based on the recommender system classification framework as shown in Table 5 . The articles are ordered according to their year of publication. There are two key concepts: Application fields and techniques used. The application fields of recommender systems are divided into six different fields, viz. entertainment, health, tourism, web/e-commerce, education and social media/others.

Algorithmic categorization, simulation platforms and applications considered for various recommender systems

This section analyses different methods like deep learning, machine learning, clustering and meta-heuristic-based-approaches used in the development of recommender systems. The algorithmic categorization of different recommender systems is given in Fig.  8 .

Algorithmic categorization of different recommender systems

Categorization is done based on content-based, collaborative filtering-based, and optimization-based approaches. In [ 8 ], a content-based filtering technique was employed for increasing the ability to trust other agents and for improving the exchange of information by trust degree. In [ 16 ], it was applied to enhance the quality of recommendations using the account attributes of the material. It achieved better performance concerning with F1-score, recall and precision. In [ 18 ], this technique was able to capture the implicit user feedback, increasing the overall accuracy of the proposed model. The content-based filtering in [ 30 ] was able to increase the accuracy and performance of a stock recommender system by using the “trust factor” for making decisions.

Different collaborative filtering approaches are utilized in recent studies, which are categorized as follows:

Model-based techniques

Neuro-Fuzzy [ 1 ] based technique helps in discovering the association between user categories and item relevance. It is also simple to understand. K-Means Clustering [ 2 , 19 , 25 , 48 ] is efficient for large scale datasets. It is simple to implement and gives a fast convergence rate. It also offers automatic recovery from failures. The decision tree [ 2 , 44 ] technique is easy to interpret. It can be used for solving the classic regression and classification problems in recommender systems. Bayesian Network [ 3 ] is a probabilistic technique used to solve classification challenges. It is based on the theory of Bayes theorem and conditional probability. Association Rule Mining (ARM) techniques [ 4 , 17 , 35 ] extract rules for projecting the occurrence of an item by considering the existence of other items in a transaction. This method uses the association rules to create a more suitable representation of data and helps in increasing the model performance and storage efficiency. Fuzzy Logic [ 5 , 7 , 15 , 20 , 28 , 43 ] techniques use a set of flexible rules. It focuses on solving complex real-time problems having an inaccurate spectrum of data. This technique provides scalability and helps in increasing the overall model performance for recommender systems. The semantic similarity [ 6 ] technique is used for describing a topological similarity to define the distance among the concepts and terms through ontologies. It measures the similarity information for increasing the efficiency of recommender systems. Rough set [ 9 , 34 ] techniques use probability distributions for solving the challenges of existing recommender models. Semantic web rule language [ 10 ] can efficiently extract the dataset features and increase the model efficiency. Linear programming-based approaches [ 13 , 42 ] are employed for achieving quality decision making in recommender models. Sequential pattern analysis [ 14 ] is applied to find suitable patterns among data items. This helps in increasing model efficiency. The probabilistic model [ 24 ] is a famous tool to handle uncertainty in risk computations and performance assessment. It offers better decision-making capabilities. K-nearest neighbours (KNN) [ 19 , 37 , 44 ] technique provides faster computation time, simplicity and ease of interpretation. They are good for classification and regression-based problems and offers more accuracy. Spectral clustering [ 21 ] is also called graph clustering or similarity-based clustering, which mainly focuses on reducing the space dimensionality in identifying the dataset items. Stochastic learning algorithm [ 26 ] solves the real-time challenges of recommender systems. Linear SVM [ 29 , 44 ] efficiently solves the high dimensional problems related to recommender systems. It is a memory-efficient method and works well with a large number of samples having relative separation among the classes. This method has been shown to perform well even when new or unfamiliar data is added. Relational Functional Gradient Boosting [ 39 ] technique efficiently works on the relational dependency of data, which is useful for statical relational learning for collaborative-based recommender systems. Ensemble learning [ 40 ] combines the forecast of two or more models and aims to achieve better performance than any of the single contributing models. It also helps in reducing overfitting problems, which are common in recommender systems.

SDAE [ 41 ] is used for learning the non-linear transformations with different filters for finding suitable data. This aids in increasing the performance of recommender models. Multimodal network learning [ 45 ] is efficient for multi-modal data, representing a combined representation of diverse modalities. Random forest [ 46 , 51 ] is a commonly used approach in comparison with other classifiers. It has been shown to increase accuracy when handling big data. This technique is a collection of decision trees to minimize variance through training on diverse data samples. ANNInit [ 47 ] is a type of artificial neural network-based technique that has the capability of self-learning and generating efficient results. It is independent of the data type and can learn data patterns automatically. HashMap [ 50 ] gives faster access to elements owing to the hashing methodology, which decreases the data processing time and increases the performance of the system. CNN [ 51 ] technique can automatically fetch the significant features of a dataset without any supervision. It is a computationally efficient method and provides accurate recommendations. This technique is also simple and fast for implementation. Multilayer graph data model [ 54 ] is efficient for real-time applications and minimizes the access time through mapping the correlation as edges among nodes and provides superior performance. Singular Value Decomposition [ 56 ] can simplify the input data and increase the efficiency of recommendations by eliminating the noise present in data. Reinforcement learning [ 57 ] is efficient for practical scenarios of recommender systems having large data sizes. It is capable of boosting the model performance by increasing the model accuracy even for large scale datasets. FNN [ 58 ] is one of the artificial neural network techniques which can learn non-linear and complex relationships between items. It has demonstrated a good performance increase when employed in different recommender systems. Knowledge representation learning [ 60 ] systems aim to simplify the model development process by increasing the acquisition efficiency, inferential efficiency, inferential adequacy and representation adequacy. User-based approaches [ 2 , 55 , 59 ] specialize in detecting user-related meta-data which is employed to increase the overall model performance. This technique is more suitable for real-time applications where it can capture user feedback and use it to increase the user experience.

Optimization-based techniques

The Foraging Bees [ 11 ] technique enables both functional and combinational optimization for random searching in recommender models. Artificial bee colony [ 12 ] is a swarm-based meta-heuristic technique that provides features like faster convergence rate, the ability to handle the objective with stochastic nature, ease for incorporating with other algorithms, usage of fewer control parameters, strong robustness, high flexibility and simplicity. Particle Swarm Optimization [ 23 ] is a computation optimization technique that offers better computational efficiency, robustness in control parameters, and is easy and simple to implement in recommender systems. Portfolio optimization algorithm [ 27 ] is a subclass of optimization algorithms that find its application in stock investment recommender systems. It works well in real-time and helps in the diversification of the portfolio for maximum profit. The artificial immune system [ 31 ]a is computationally intelligent machine learning technique. This technique can learn new patterns in the data and optimize the overall system parameters. Expectation maximization (EM) [ 32 , 36 , 38 ] is an iterative algorithm that guarantees the likelihood of finding the maximum parameters when the input variables are unknown. Delphi panel and repertory grid [ 33 ] offers efficient decision making by solving the dimensionality problem and data sparsity issues of recommender systems. The Firefly algorithm (FA) [ 48 ] provides fast results and increases recommendation efficiency. It is capable of reducing the number of iterations required to solve specific recommender problems. It also provides both local and global sets of solutions. Beetle Antennae Search (BAS) [ 49 ] offers superior search accuracy and maintains less time complexity that promotes the performance of recommendations. Many-objective evolutionary algorithm (MaOEA) [ 52 ] is applicable for real-time, multi-objective, search-related recommender systems. The introduction of a local search operator increases the convergence rate and gets suitable results. Genetic Algorithm (GA) [ 2 , 22 , 25 , 53 ] based techniques are used to solve the multi-objective optimization problems of recommender systems. They employ probabilistic transition rules and have a simpler operation that provides better recommender performance.

Features and challenges

The features and challenges of the existing recommender models are given in Table 6 .

Simulation platforms

The various simulation platforms used for developing different recommender systems with different applications are given in Fig.  9 .

Simulation platforms used for developing different recommender systems

Here, the Java platform is used in 20% of the contributions, MATLAB is implemented in 7% of the contributions, different fold cross-validation are used in 8% of the contributions, 7% of the contributions are utilized by the python platform, 3% of the contributions employ R-programming and 1% of the contributions are developed by Tensorflow, Weka and Android environments respectively. Other simulation platforms like Facebook, web UI (User Interface), real-time environments, etc. are used in 50% of the contributions. Table 7 describes some simulation platforms commonly used for developing recommender systems.

Application focused and dataset description

This section provides an analysis of the different applications focused on a set of recent recommender systems and their dataset details.

Recent recommender systems were analysed and found that 11% of the contributions are focused on the domain of healthcare, 10% of the contributions are on movie recommender systems, 5% of the contributions come from music recommender systems, 6% of the contributions are focused on e-learning recommender systems, 8% of the contributions are used for online product recommender systems, 3% of the contributions are focused on book recommendations and 1% of the contributions are focused on Job and knowledge management recommender systems. 5% of the contributions concentrated on social network recommender systems, 10% of the contributions are focused on tourist and hotels recommender systems, 6% of the contributions are employed for stock recommender systems, and 3% of the contributions contributed for video recommender systems. The remaining 12% of contributions are miscellaneous recommender systems like Twitter, venue-based recommender systems, etc. Similarly, different datasets are gathered for recommender systems based on their application types. A detailed description is provided in Table 8 .

Performance analysis of state-of-art recommender systems

The performance evaluation metrics used for the analysis of different recommender systems is depicted in Table 9 . From the set of research works, 35% of the works use recall measure, 16% of the works employ Mean Absolute Error (MAE), 11% of the works take Root Mean Square Error (RMSE), 41% of the papers consider precision, 30% of the contributions analyse F1-measure, 31% of the works apply accuracy and 6% of the works employ coverage measure to validate the performance of the recommender systems. Moreover, some additional measures are also considered for validating the performance in a few applications.

Research gaps and challenges

In the recent decade, recommender systems have performed well in solving the problem of information overload and has become the more appropriate tool for multiple areas such as psychology, mathematics, computer science, etc. [ 80 ]. However, current recommender systems face a variety of challenges which are stated as follows, and discussed below:

Deployment challenges such as cold start, scalability, sparsity, etc. are already discussed in Sect. 3.

Challenges faced when employing different recommender algorithms for different applications.

Challenges in collecting implicit user data

Challenges in handling real-time user feedback.

Challenges faced in choosing the correct implementation techniques.

Challenges faced in measuring system performance.

Challenges in implementing recommender system for diverse applications.

Numerous recommender algorithms have been proposed on novel emerging dimensions which focus on addressing the existing limitations of recommender systems. A good recommender system must increase the recommendation quality based on user preferences. However, a specific recommender algorithm is not always guaranteed to perform equally for different applications. This encourages the possibility of employing different recommender algorithms for different applications, which brings along a lot of challenges. There is a need for more research to alleviate these challenges. Also, there is a large scope of research in recommender applications that incorporate information from different interactive online sites like Facebook, Twitter, shopping sites, etc. Some other areas for emerging research may be in the fields of knowledge-based recommender systems, methods for seamlessly processing implicit user data and handling real-time user feedback to recommend items in a dynamic environment.

Some of the other research areas like deep learning-based recommender systems, demographic filtering, group recommenders, cross-domain techniques for recommender systems, and dimensionality reduction techniques are also further required to be studied [ 83 ]. Deep learning-based recommender systems have recently gained much popularity. Future research areas in this field can integrate the well-performing deep learning models with new variants of hybrid meta-heuristic approaches.

During this review, it was observed that even though recent recommender systems have demonstrated good performance, there is no single standardized criteria or method which could be used to evaluate the performance of all recommender systems. System performance is generally measured by different evaluation matrices which makes it difficult to compare. The application of recommender systems in real-time applications is growing. User satisfaction and personalization play a very important role in the success of such recommender systems. There is a need for some new evaluation criteria which can evaluate the level of user satisfaction in real-time. New research should focus on capturing real-time user feedback and use the information to change the recommendation process accordingly. This will aid in increasing the quality of recommendations.

Conclusion and future scope

Recommender systems have attracted the attention of researchers and academicians. In this paper, we have identified and prudently reviewed research papers on recommender systems focusing on diverse applications, which were published between 2011 and 2021. This review has gathered diverse details like different application fields, techniques used, simulation tools used, diverse applications focused, performance metrics, datasets used, system features, and challenges of different recommender systems. Further, the research gaps and challenges were put forward to explore the future research perspective on recommender systems. Overall, this paper provides a comprehensive understanding of the trend of recommender systems-related research and to provides researchers with insight and future direction on recommender systems. The results of this study have several practical and significant implications:

Based on the recent-past publication rates, we feel that the research of recommender systems will significantly grow in the future.

A large number of research papers were identified in movie recommendations, whereas health, tourism and education-related recommender systems were identified in very few numbers. This is due to the availability of movie datasets in the public domain. Therefore, it is necessary to develop datasets in other fields also.

There is no standard measure to compute the performance of recommender systems. Among 60 papers, 21 used recall, 10 used MAE, 25 used precision, 18 used F1-measure, 19 used accuracy and only 7 used RMSE to calculate system performance. Very few systems were found to excel in two or more matrices.

Java and Python (with a combined contribution of 27%) are the most common programming languages used to develop recommender systems. This is due to the availability of a large number of standard java and python libraries which aid in the development process.

Recently a large number of hybrid and optimizations techniques are being proposed for recommender systems. The performance of a recommender system can be greatly improved by applying optimization techniques.

There is a large scope of research in using neural networks and deep learning-based methods for developing recommender systems. Systems developed using these methods are found to achieve high-performance accuracy.

This research will provide a guideline for future research in the domain of recommender systems. However, this research has some limitations. Firstly, due to the limited amount of manpower and time, we have only reviewed papers published in journals focusing on computer science, management and medicine. Secondly, we have reviewed only English papers. New research may extend this study to cover other journals and non-English papers. Finally, this review was conducted based on a search on only six descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*” and “Hybrid Recommend*”. Research papers that did not include these keywords were not considered. Future research can include adding some additional descriptors and keywords for searching. This will allow extending the research to cover more diverse articles on recommender systems.

Availability of data and materials

Not applicable.

Castellano G, Fanelli AM, Torsello MA. NEWER: A system for neuro-fuzzy web recommendation. Appl Soft Comput. 2011;11:793–806.

Article   Google Scholar  

Crespo RG, Martínez OS, Lovelle JMC, García-Bustelo BCP, Gayo JEL, Pablos PO. Recommendation system based on user interaction data applied to intelligent electronic books. Computers Hum Behavior. 2011;27:1445–9.

Lin FC, Yu HW, Hsu CH, Weng TC. Recommendation system for localized products in vending machines. Expert Syst Appl. 2011;38:9129–38.

Wang SL, Wu CY. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Appl. 2011;38:10831–8.

García-Crespo Á, López-Cuadrado JL, Colomo-Palacios R, González-Carrasco I, Ruiz-Mezcua B. Sem-Fit: A semantic based expert system to provide recommendations in the tourism domain. Expert Syst Appl. 2011;38:13310–9.

Dong H, Hussain FK, Chang E. A service concept recommendation system for enhancing the dependability of semantic service matchmakers in the service ecosystem environment. J Netw Comput Appl. 2011;34:619–31.

Li M, Liu L, Li CB. An approach to expert recommendation based on fuzzy linguistic method and fuzzy text classification in knowledge management systems. Expert Syst Appl. 2011;38:8586–96.

Lorenzi F, Bazzan ALC, Abel M, Ricci F. Improving recommendations through an assumption-based multiagent approach: An application in the tourism domain. Expert Syst Appl. 2011;38:14703–14.

Huang Z, Lu X, Duan H. Context-aware recommendation using rough set model and collaborative filtering. Artif Intell Rev. 2011;35:85–99.

Chen RC, Huang YH, Bau CT, Chen SM. A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Syst Appl. 2012;39:3995–4006.

Mohanraj V, Chandrasekaran M, Senthilkumar J, Arumugam S, Suresh Y. Ontology driven bee’s foraging approach based self-adaptive online recommendation system. J Syst Softw. 2012;85:2439–50.

Hsu CC, Chen HC, Huang KK, Huang YM. A personalized auxiliary material recommendation system based on learning style on facebook applying an artificial bee colony algorithm. Comput Math Appl. 2012;64:1506–13.

Gemmell J, Schimoler T, Mobasher B, Burke R. Resource recommendation in social annotation systems: A linear-weighted hybrid approach. J Comput Syst Sci. 2012;78:1160–74.

Article   MathSciNet   Google Scholar  

Choi K, Yoo D, Kim G, Suh Y. A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. Electron Commer Res Appl. 2012;11:309–17.

Garibaldi JM, Zhou SM, Wang XY, John RI, Ellis IO. Incorporation of expert variability into breast cancer treatment recommendation in designing clinical protocol guided fuzzy rule system models. J Biomed Inform. 2012;45:447–59.

Salehi M, Kmalabadi IN. A hybrid attribute–based recommender system for e–learning material recommendation. IERI Procedia. 2012;2:565–70.

Aher SB, Lobo LMRJ. Combination of machine learning algorithms for recommendation of courses in e-learning System based on historical data. Knowl-Based Syst. 2013;51:1–14.

Kardan AA, Ebrahimi M. A novel approach to hybrid recommendation systems based on association rules mining for content recommendation in asynchronous discussion groups. Inf Sci. 2013;219:93–110.

Chang JH, Lai CF, Wang MS, Wu TY. A cloud-based intelligent TV program recommendation system. Comput Electr Eng. 2013;39:2379–99.

Lucas JP, Luz N, Moreno MN, Anacleto R, Figueiredo AA, Martins C. A hybrid recommendation approach for a tourism system. Expert Syst Appl. 2013;40:3532–50.

Niu J, Zhu L, Zhao X, Li H. Affivir: An affect-based Internet video recommendation system. Neurocomputing. 2013;120:422–33.

Liu L, Xu J, Liao SS, Chen H. A real-time personalized route recommendation system for self-drive tourists based on vehicle to vehicle communication. Expert Syst Appl. 2014;41:3409–17.

Bakshi S, Jagadev AK, Dehuri S, Wang GN. Enhancing scalability and accuracy of recommendation systems using unsupervised learning and particle swarm optimization. Appl Soft Comput. 2014;15:21–9.

Kim Y, Shim K. TWILITE: A recommendation system for twitter using a probabilistic model based on latent Dirichlet allocation. Inf Syst. 2014;42:59–77.

Wang Z, Yu X, Feng N, Wang Z. An improved collaborative movie recommendation system using computational intelligence. J Vis Lang Comput. 2014;25:667–75.

Kolomvatsos K, Anagnostopoulos C, Hadjiefthymiades S. An efficient recommendation system based on the optimal stopping theory. Expert Syst Appl. 2014;41:6796–806.

Gottschlich J, Hinz O. A decision support system for stock investment recommendations using collective wisdom. Decis Support Syst. 2014;59:52–62.

Torshizi AD, Zarandi MHF, Torshizi GD, Eghbali K. A hybrid fuzzy-ontology based intelligent system to determine level of severity and treatment recommendation for benign prostatic hyperplasia. Comput Methods Programs Biomed. 2014;113:301–13.

Zahálka J, Rudinac S, Worring M. Interactive multimodal learning for venue recommendation. IEEE Trans Multimedia. 2015;17:2235–44.

Sankar CP, Vidyaraj R, Kumar KS. Trust based stock recommendation system – a social network analysis approach. Procedia Computer Sci. 2015;46:299–305.

Chen MH, Teng CH, Chang PC. Applying artificial immune systems to collaborative filtering for movie recommendation. Adv Eng Inform. 2015;29:830–9.

Wu H, Pei Y, Li B, Kang Z, Liu X, Li H. Item recommendation in collaborative tagging systems via heuristic data fusion. Knowl-Based Syst. 2015;75:124–40.

Yeh DY, Cheng CH. Recommendation system for popular tourist attractions in Taiwan using delphi panel and repertory grid techniques. Tour Manage. 2015;46:164–76.

Liao SH, Chang HK. A rough set-based association rule approach for a recommendation system for online consumers. Inf Process Manage. 2016;52:1142–60.

Li H, Cui J, Shen B, Ma J. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 2016;210:164–73.

Wu H, Yue K, Pei Y, Li B, Zhao Y, Dong F. Collaborative topic regression with social trust ensemble for recommendation in social media systems. Knowl-Based Syst. 2016;97:111–22.

Adeniyi DA, Wei Z, Yongquan Y. Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method. Appl Computing Inform. 2016;12:90–108.

Rawat YS, Kankanhalli MS. ClickSmart: A context-aware viewpoint recommendation system for mobile photography. IEEE Trans Circuits Syst Video Technol. 2017;27:149–58.

Yang S, Korayem M, Aljadda K, Grainger T, Natarajan S. Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive Statistical Relational Learning approach. Knowl-Based Syst. 2017;136:37–45.

Lee WP, Chen CT, Huang JY, Liang JY. A smartphone-based activity-aware system for music streaming recommendation. Knowl-Based Syst. 2017;131:70–82.

Wei J, He J, Chen K, Zhou Y, Tang Z. Collaborative filtering and deep learning based recommendation system for cold start items. Expert Syst Appl. 2017;69:29–39.

Li C, Wang Z, Cao S, He L. WLRRS: A new recommendation system based on weighted linear regression models. Comput Electr Eng. 2018;66:40–7.

Mezei J, Nikou S. Fuzzy optimization to improve mobile health and wellness recommendation systems. Knowl-Based Syst. 2018;142:108–16.

Ayata D, Yaslan Y, Kamasak ME. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron. 2018;64:196–203.

Zhao Z, Yang Q, Lu H, Weninger T. Social-aware movie recommendation via multimodal network learning. IEEE Trans Multimedia. 2018;20:430–40.

Hammou BA, Lahcen AA, Mouline S. An effective distributed predictive model with matrix factorization and random forest for big data recommendation systems. Expert Syst Appl. 2019;137:253–65.

Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Zhang Z, Fu Z. Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl-Based Syst. 2019;166:132–9.

Bhaskaran S, Santhi B. An efficient personalized trust based hybrid recommendation (TBHR) strategy for e-learning system in cloud computing. Clust Comput. 2019;22:1137–49.

Han Y, Han Z, Wu J, Yu Y, Gao S, Hua D, Yang A. Artificial intelligence recommendation system of cancer rehabilitation scheme based on IoT technology. IEEE Access. 2020;8:44924–35.

Kang S, Jeong C, Chung K. Tree-based real-time advertisement recommendation system in online broadcasting. IEEE Access. 2020;8:192693–702.

Ullah F, Zhang B, Khan RU. Image-based service recommendation system: A JPEG-coefficient RFs approach. IEEE Access. 2020;8:3308–18.

Cai X, Hu Z, Zhao P, Zhang W, Chen J. A hybrid recommendation system with many-objective evolutionary algorithm. Expert Syst Appl. 2020. https://doi.org/10.1016/j.eswa.2020.113648 .

Esteban A, Zafra A, Romero C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Syst. 2020;194:105385.

Mondal S, Basu A, Mukherjee N. Building a trust-based doctor recommendation system on top of multilayer graph database. J Biomed Inform. 2020;110:103549.

Dhelim S, Ning H, Aung N, Huang R, Ma J. Personality-aware product recommendation system based on user interests mining and metapath discovery. IEEE Trans Comput Soc Syst. 2021;8:86–98.

Bhalse N, Thakur R. Algorithm for movie recommendation system using collaborative filtering. Materials Today: Proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.235 .

Ke G, Du HL, Chen YC. Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks. Appl Soft Computing. 2021;104:107213.

Chen X, Liu D, Xiong Z, Zha ZJ. Learning and fusing multiple user interest representations for micro-video and movie recommendations. IEEE Trans Multimedia. 2021;23:484–96.

Afolabi AO, Toivanen P. Integration of recommendation systems into connected health for effective management of chronic diseases. IEEE Access. 2019;7:49201–11.

He M, Wang B, Du X. HI2Rec: Exploring knowledge in heterogeneous information for movie recommendation. IEEE Access. 2019;7:30276–84.

Bobadilla J, Serradilla F, Hernando A. Collaborative filtering adapted to recommender systems of e-learning. Knowl-Based Syst. 2009;22:261–5.

Russell S, Yoon V. Applications of wavelet data reduction in a recommender system. Expert Syst Appl. 2008;34:2316–25.

Campos LM, Fernández-Luna JM, Huete JF. A collaborative recommender system based on probabilistic inference from fuzzy observations. Fuzzy Sets Syst. 2008;159:1554–76.

Funk M, Rozinat A, Karapanos E, Medeiros AKA, Koca A. In situ evaluation of recommender systems: Framework and instrumentation. Int J Hum Comput Stud. 2010;68:525–47.

Porcel C, Moreno JM, Herrera-Viedma E. A multi-disciplinar recommender system to advice research resources in University Digital Libraries. Expert Syst Appl. 2009;36:12520–8.

Bobadilla J, Serradilla F, Bernal J. A new collaborative filtering metric that improves the behavior of recommender systems. Knowl-Based Syst. 2010;23:520–8.

Ochi P, Rao S, Takayama L, Nass C. Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int J Hum Comput Stud. 2010;68:472–82.

Olmo FH, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Syst Appl. 2008;35:790–804.

Zhen L, Huang GQ, Jiang Z. An inner-enterprise knowledge recommender system. Expert Syst Appl. 2010;37:1703–12.

Göksedef M, Gündüz-Öğüdücü S. Combination of web page recommender systems. Expert Syst Appl. 2010;37(4):2911–22.

Shao B, Wang D, Li T, Ogihara M. Music recommendation based on acoustic features and user access patterns. IEEE Trans Audio Speech Lang Process. 2009;17:1602–11.

Shin C, Woo W. Socially aware tv program recommender for multiple viewers. IEEE Trans Consum Electron. 2009;55:927–32.

Lopez-Carmona MA, Marsa-Maestre I, Perez JRV, Alcazar BA. Anegsys: An automated negotiation based recommender system for local e-marketplaces. IEEE Lat Am Trans. 2007;5:409–16.

Yap G, Tan A, Pang H. Discovering and exploiting causal dependencies for robust mobile context-aware recommenders. IEEE Trans Knowl Data Eng. 2007;19:977–92.

Meo PD, Quattrone G, Terracina G, Ursino D. An XML-based multiagent system for supporting online recruitment services. IEEE Trans Syst Man Cybern. 2007;37:464–80.

Khusro S, Ali Z, Ullah I. Recommender systems: Issues, challenges, and research opportunities. Inform Sci Appl. 2016. https://doi.org/10.1007/978-981-10-0557-2_112 .

Blanco-Fernandez Y, Pazos-Arias JJ, Gil-Solla A, Ramos-Cabrer M, Lopez-Nores M. Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Trans Consum Electron. 2008;54:727–35.

Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: Principles, methods and evaluation. Egyptian Inform J. 2015;16:261–73.

Yoshii K, Goto M, Komatani K, Ogata T, Okuno HG. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. IEEE Trans Audio Speech Lang Process. 2008;16:435–47.

Wei YZ, Moreau L, Jennings NR. Learning users’ interests by quality classification in market-based recommender systems. IEEE Trans Knowl Data Eng. 2005;17:1678–88.

Bjelica M. Towards TV recommender system: experiments with user modeling. IEEE Trans Consum Electron. 2010;56:1763–9.

Setten MV, Veenstra M, Nijholt A, Dijk BV. Goal-based structuring in recommender systems. Interact Comput. 2006;18:432–56.

Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng. 2005;17:734–49.

Symeonidis P, Nanopoulos A, Manolopoulos Y. Providing justifications in recommender systems. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. 2009;38:1262–72.

Zhan J, Hsieh C, Wang I, Hsu T, Liau C, Wang D. Privacy preserving collaborative recommender systems. IEEE Trans Syst Man Cybernet. 2010;40:472–6.

Burke R. Hybrid recommender systems: survey and experiments. User Model User-Adap Inter. 2002;12:331–70.

Article   MATH   Google Scholar  

Gunes I, Kaleli C, Bilge A, Polat H. Shilling attacks against recommender systems: a comprehensive survey. Artif Intell Rev. 2012;42:767–99.

Park DH, Kim HK, Choi IY, Kim JK. A literature review and classification of recommender systems research. Expert Syst Appl. 2012;39:10059–72.

Download references

Acknowledgements

We thank our colleagues from Assam Down Town University who provided insight and expertise that greatly assisted this research, although they may not agree with all the interpretations and conclusions of this paper.

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

Department of Computer Science & Engineering, Assam Down Town University, Panikhaiti, Guwahati, 781026, Assam, India

Deepjyoti Roy & Mala Dutta

You can also search for this author in PubMed   Google Scholar

Contributions

DR carried out the review study and analysis of the existing algorithms in the literature. MD has been involved in drafting the manuscript or revising it critically for important intellectual content. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Deepjyoti Roy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roy, D., Dutta, M. A systematic review and research perspective on recommender systems. J Big Data 9 , 59 (2022). https://doi.org/10.1186/s40537-022-00592-5

Download citation

Received : 04 October 2021

Accepted : 28 March 2022

Published : 03 May 2022

DOI : https://doi.org/10.1186/s40537-022-00592-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Machine learning
• Content-based filtering
• Collaborative filtering
• Deep learning

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A collaborative approach for research paper recommender system

Roles Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing – original draft, Writing – review & editing

Affiliations Department of Computer Science, Faculty of Computer Science and Information Technology, Bayero University, Kano, Nigeria, Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

Roles Supervision, Validation, Visualization, Writing – review & editing

* E-mail: [email protected]

Affiliation Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

Roles Funding acquisition, Resources

Affiliation Sekolah Tinggi Pariwisata Ambarrukmo, Yogyakarta, Indonesia

Affiliation Faculty of Information Technology and Business, Universitas Teknologi Yogyakarta, Yogyakarta, Indonesia

Roles Formal analysis, Investigation, Project administration, Resources, Validation, Visualization, Writing – review & editing

Affiliations Faculty of Information Technology and Business, Universitas Teknologi Yogyakarta, Yogyakarta, Indonesia, AMCS Research Center, Yogyakarta, Indonesia

• Khalid Haruna, 
• Maizatul Akmar Ismail, 
• Damiasih Damiasih, 
• Joko Sutopo, 
• Tutut Herawan

• Published: October 5, 2017
• https://doi.org/10.1371/journal.pone.0184516
• Reader Comments

Research paper recommenders emerged over the last decade to ease finding publications relating to researchers’ area of interest. The challenge was not just to provide researchers with very rich publications at any time, any place and in any form but to also offer the right publication to the right researcher in the right way. Several approaches exist in handling paper recommender systems. However, these approaches assumed the availability of the whole contents of the recommending papers to be freely accessible, which is not always true due to factors such as copyright restrictions. This paper presents a collaborative approach for research paper recommender system. By leveraging the advantages of collaborative filtering approach, we utilize the publicly available contextual metadata to infer the hidden associations that exist between research papers in order to personalize recommendations. The novelty of our proposed approach is that it provides personalized recommendations regardless of the research field and regardless of the user’s expertise. Using a publicly available dataset, our proposed approach has recorded a significant improvement over other baseline methods in measuring both the overall performance and the ability to return relevant and useful publications at the top of the recommendation list.

Citation: Haruna K, Akmar Ismail M, Damiasih D, Sutopo J, Herawan T (2017) A collaborative approach for research paper recommender system. PLoS ONE 12(10): e0184516. https://doi.org/10.1371/journal.pone.0184516

Editor: Feng Xia, Dalian University of Technology, CHINA

Received: June 10, 2017; Accepted: August 27, 2017; Published: October 5, 2017

Copyright: © 2017 Haruna et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All study files are available from: https://figshare.com/articles/Supporting_Information_Dataset_docx/5368408 .

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

The overabundance of information that is available over the internet makes information seeking a difficult task. Researchers find it difficult to access and keep track of the most relevant and promising research papers of their interest [ 1 ]. The easiest and the most common approach used in searching for related publications is to send a query message asking the web to provide you with specific information [ 2 ]. However, the results from this approach largely depend on how good the user is in fine-tuning the query message beside its inability to personalize the searching results.

Another classical approach used by most researchers is to follow the list of references from the documents they already possessed [ 3 ]. Even though this approach might be quite effective in some instances, it does not guarantee full coverage of recommending research papers and cannot trace papers published after the possessed paper. In addition, the list of references may not be publicly available and therefore hard for the researchers to access.

An alternative approach that has been proposed in the literature is the use of research paper recommender systems [ 4 , 5 ], to automatically suggest relevant papers to the researchers based on some initial information provided by the users that are more elaborate than a few keywords.

To provide more accurate and relevant recommendations, recommender systems incorporate the users’ contexts and the possible contextual information of the consumed contents [ 6 ]. Different researchers proposed the use of a different user provided information such as the use of a list of citations [ 7 ], the list of papers authored by an author [ 8 ], part of paper text [ 2 ], a single paper [ 9 ], and so on. In these approaches, a user profile is constructed from this initial information to represent the interests of the users and the system search for items or other profiles similar to the one provided to generate recommendations. The challenge was not just to provide a very rich recommendation to researchers at any time, any place and in any form but to also offer the right paper to the right researcher in the right way [ 10 – 12 ].

The major limitation of the existing approaches is their assumption of the availability of the whole content of the recommending papers to be freely accessible, which is not always true due to factors such as copyright restrictions. In an attempt to address this problem, Liu, et al . in [ 3 ] applied the concept of the collaborative approach to mine the hidden associations that exists between a target paper and its references to provide a unique and useful list of research papers as recommendations.

Motivated from [ 3 ], this paper presents a collaborative approach for research paper recommender system. In addition to mining the hidden associations between a target paper and its references, in this paper, we also put into cognizance the hidden associations between the target paper’s citations (see section 3). Similar to [ 3 ], our task is not to apply a direct relation between paper-citation relations because, in one way or the other, a researcher who is in possession of a research paper directly or indirectly has access to its limited references and also to its citations. Our aim is to identify the latent associations that exist between research papers based on the perspective of paper-citation relations. A candidate paper is qualified for consideration in [ 3 ] if it cited any of the target paper’s references. In our proposed approach, a candidate paper is qualified for consideration if and only if it cited any of the target paper’s references and there exist another paper which cited both the candidate and the target papers simultaneously. We then measure and weigh the extent of similarity between the target paper and the qualified candidate papers and recommend the top-N most similar papers based on the assumption that if there exist significant co-occurrence between the target paper and the qualified candidate papers, then there exist some extent of similarities between them. This strictness in qualifying a candidate paper helps in enhancing the overall performance of the approach and the ability to return relevant and useful recommendations at the top of the recommendation list.

The major contributions of our proposed approach are as follows;

• We utilized the advantages of publicly available contextual metadata to propose an independent research paper that does not require a priori user profile.
• Our approach provides personalized recommendations regardless of the research field and regardless of user expertise.

The outline of the rest of the paper is as follows. We first present some related works on recommending research papers. We then detailed our proposed approach. Next, we described our experiments, starting with the dataset and the baseline methods, followed by the evaluation procedures. We then discuss our findings and lastly conclude the paper with a brief concluding remark and future research directions.

2. Related work

Research paper recommenders that provide the best suggestions for all alternatives emerged over the last decade to help researchers on seemingly finding works of their interest over the Cyber Ocean of information. Collaborative filtering (CF) is one of the most successful techniques used in recommender systems [ 13 ]. It is a method which recommends items to target users based on what other similar users have previously preferred [ 14 – 16 ]. It has been used in various applications such as in recommending movies [ 17 ], audio CD [ 18 ], e-commerce [ 19 ], music [ 20 ], Usenet news [ 16 ], research papers [ 7 , 21 – 24 ] among others (see [ 25 ]). Some researchers [ 13 , 21 , 26 ], have criticized the use of this technique to recommend scholarly papers. Precisely the authors in [ 21 , 26 ], claimed that collaborative filtering is only effective in a domain where the number of users seeking recommendation is higher than the number of items to be recommended, such domains include movies [ 27 ], music [ 28 ], news [ 29 ] etc. While the argument in [ 13 ], is that researchers are not willing to spend their valuable time to provide explicit ratings to their consumed research papers, and therefore, leading to insufficient ratings by the researchers to the research papers. Furthermore, for a user to receive useful recommendations, a tangible number of ratings is required.

Nevertheless, despite these aforementioned problems, a significant amount of papers can be traced, which suggest relevant papers to researchers based on collaborative filtering by mining latent associations between scholarly papers. These associations are either directly obtained by taking into consideration paper citations as rating scores [ 7 ], or by monitoring the researchers’ actions implicitly [ 30 , 31 ]. Applying citation analysis such as bibliographical coupling [ 32 ] and co-citation analysis [ 33 ] has also been used to identify similar papers to a target paper [ 34 ].

The relationships among research papers have been categorized into direct and indirect relations in a survey conducted by [ 35 ]. In the paper, three approaches were identified for detecting the relationships between papers based on the perspective of paper sources. Namely, citation context, citation analysis, and content-based. The authors claimed that content-based approach becomes less appropriate in detecting relationships across research papers, due to its inability to accommodate some specific characteristics that exist in the research papers like author and citations. Therefore, it becomes suitable only for identifying similarity relations across regular documents. On the other hand, the use of citation analysis can generate more relations between research papers but cannot generate relations from semantic text. This weakness is addressed by using citation context based approach, which depicts more emphasis on determining some important features in the text classification process to increase classification performance.

A context-based collaborative framework (CCF) that uses only easily obtained citations relations as source data was proposed in [ 3 ]. The framework employs an association-mining technique to obtain a paper representation of the paper citation context. A pairwise comparison was then performed to compute the extent of similarities between papers. The use of collaborative filtering has also been explored in [ 7 ], by using citation-web between scholarly papers to create a rating matrix. The aim was to use the paper-citation relation to recommend some additional references to the input paper. In doing that, the authors investigated the use of six different algorithms for selecting citations. Using offline evaluation, they discovered large disparity in the returned accuracy by each of the six algorithms.

The authors in [ 6 ], hypothesized the author’s previous publications to constitute a clear signal of the latent interests of a researcher. The key part of their model was to enhance the user profile with the information coming directly from the references to the researcher’s previous works as well as the papers that cited them. However, the approach increases the well-known sparsity problem. To alleviate this problem, they extend their work in [ 8 ], to mine potential citations papers using imputed similarities through the use of collaborative filtering. They also refined the use of citing papers in characterizing a target candidate paper using fragments in the citation and potential citation papers. Whilst the approach works well for researchers with a single discipline, it generates poor results for the multidisciplinary researchers. To overcome this problem, an adaptive neighbor selection approach was proposed in [ 2 ], to overcome imputation-based collaborative filtering problem. Whereas authors in [ 2 , 6 , 8 ], recommend papers relevant to the researcher’s interest, they also addressed the serendipitous scholarly paper recommendation in [ 36 ].

On another development, the increasing number of research communities and social networking sites such as LiveJournal and MySpace have brought new opportunities for research paper recommendation systems. Researches show that users in online social networks tend to form knit groups [ 37 ], with strongly large connected components [ 38 ].

Several kinds of research have considered the social group formation and community membership in social networks and their use in recommender systems [ 39 – 46 ]. These researchers utilized the influence of social properties to suggest relevant information to individual or group of users based on social ties, which can either be strong or weak depending on the tie strength that represents the closeness and interaction frequency between the information source and recipient [ 47 , 48 ].

Recommendations from strong ties are believed to be more persuasive than those from weak ties [ 49 – 51 ]. This is because information transferred by strong ties is likely to be perceived as more relevant and reliable. To be specific, the authors of [ 45 , 52 ] proposed a novel algorithm called socially aware recommendation of scholarly papers (SARSP) that utilizes the aspect of social learning and networking for conference participants through the construction of relations in folksonomies and social ties. The algorithm recommends research papers issued by an active participant to other conference participants based on the computation of their social ties. This approach has been extended in [ 53 ], to include personality behavior in addition to social relations among smart conference attendees. A more detail survey on scholarly data is presented in [ 54 ] for more exploration.

The major challenge with the previous researches is that all the contextual information from the recommended, referenced and cited papers must be fully accessible to the recommenders, which are not always freely available due to factors such as copyright restrictions. Another major problem with the existing research paper recommender systems is their dependency on a priori user profile, which makes the system to work well only when it already has a number of registered users, a major hurdle for the construction of new recommender system. Furthermore, the recommendation coverage of most of the current paper recommenders are limited to a certain field of research, this is because recommending papers are stored prior and therefore the system cannot effectively scan the entire databases to find connections between papers. Moreover, most of the existing research paper frameworks are designed to work only on a single discipline, and therefore cannot be used to address the problems of multidisciplinary scholars. While the use of keyword-based query information retrieval technique through search engines is able to scan all document for relevant text, it also provides 100s of irrelevant documents, besides its inability to provide personalize results to the individual researchers.

Different from the existing works, in this paper, we propose a new approach based on collaborative filtering that utilizes only publicly available contextual metadata to personalize recommendations based on the hidden associations that exist between research papers. Our proposed approach does not only provide personalized recommendations regardless of the research field and regardless of user expertise but also handles multi-disciplinary problems.

3. Proposed collaborative research paper recommendation approach

Even though some researchers [ 6 , 13 , 21 , 26 ], claimed content based to be the most suitable approach when dealing with scholarly domain, other researchers [ 35 ] argued on its suitability because only become suitable in identifying similarity relations across regular documents but lacks some important features to effectively detect relationships across research papers.

In this paper, we are motivated to leverage the advantages of collaborative approach as it has proved to be effective in the domains of movies [ 27 ], music [ 28 ], news [ 29 ], e-commerce [ 19 ], etc. The unsuitability of the collaborative approach to research paper recommenders was referred to the lack of ratings to research papers by the researchers [ 13 ]. In bringing a solution to this problem, we mined rating score between researchers and research papers based on paper-citation relations. We use C ij to denote citation score between paper i and a cited-paper j from a paper-citation matrix C . If paper i cited a paper j , C ij = 1 otherwise C ij = 0.

We initiate our approach by first transforming all the recommending papers (in our dataset) into a paper-citation relations matrix in which, the rows and the columns respectively represent the recommending papers and their citations. Our approach aimed to deal with scenarios in which: (a) A researcher who finds an interesting paper after some initial searches, wants to get more other related papers similar to it. (b) A student received a paper by his supervisor to start a research in the topic area covered by it. (c) A reviewer wants to explore more based on a received paper that addresses a subject matter which he is not a specialist in. (d) A researcher who wants to explore more from his previous publication(s). In all these cases, we consider a situation where the references and citations of the possessed paper that indicate the user’s preferences are publicly available (which is usually the case in almost all the major academic databases).

Algorithm 1. Algorithm representing proposed approach.

Algorithm: Collaborative Research Paper Recommendation

Input: Target Paper

Output: Top-N Recommendation

Given a target paper p i as a query,

• For each of the references Rf j , extract all other papers p ci that also cited Rf j other than the target paper p i .
• For each of the citations Cf j , extract all other papers p ri that Cf j referenced other than the target paper p i .
• Qualify all the candidate papers p c from p ci that has been referenced by at least any of the p ri

• Recommend the top-N most similar papers to the user.

We accept the user’s query in order to identify the target-paper. Once the target paper is identified, we apply algorithm 1. The algorithm retrieves all the target paper’s references and citations. For each of the references, it extracts all other papers from the web (google scholar to be precise) that also cited any of those target paper’s references. In addition, for each of the target paper’s citations, it extracts all other papers from the web that referenced any of those target paper’s citations (in other words, all the references to the target paper’s citations) and we refer to these extracted papers as the target papers nearest neighbors. For each of the neighboring papers, we qualify candidate papers that are co-cited with the target paper and which has been referenced by at least any of the target papers references. We then measure the degree of similitude between these qualified candidate papers and the target paper by measuring their collaborative similarity using Jaccard similarity measure given by Eq (1) . We then recommend the top-N most comparable papers to the researcher.

Z 11 Represents the total number of attributes where X and Y both having a value of 1.

Z 01 Represents the total number of attributes where the attribute of X is 0 and the attribute of Y is 1.

Z 10 Represents the total number of attributes where the attribute of X is 1 and the attribute of Y is 0.

To illustrate our approach further, Fig 1 represents a target-paper ( p i ) with references ( Rf 1 to Rf N ) and citations ( Cit . 1 to Cit . N ). Each of the references of the target paper has other citations from any of Rec . 1 to Rec . N and/or Cit . 1 to Cit . N other than the target-paper ( p i ). Also, each of the citations to the target paper has other references from any of Rec . 1 to Rec . N and/or Rf 1 to Rf N other than the target-paper ( p i ). Our approach qualifies recommending papers ( Rec . 1 to Rec . N ) that are co-cited with the target paper and which has been referenced by at least any of the target papers references.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0184516.g001

For example, from Fig 1 , Rec . 1 and Rec . 2 are co-cited with the target paper by Cit . 1 . However, Ref . 2 does not have any connection to any of the target paper’s references and therefore disqualified by step 3 of our proposed algorithm. On the other hand, Rec . 1 does not only being co-cited with the target paper by Cit . 1 but also referenced one of the target papers references Ref . 1 . As can be observed from Fig 1 , only Rec . 1 and Rec . 3 are qualified candidate papers.

In the following section, we present the experiments setup.

4. Experiments setup

4.1 dataset.

We utilize the publicly available dataset presented in [ 2 ]. The dataset contained the publication list of 50 researchers whose research interests are from different fields of computer science that range from information retrieval, software engineering, user interface, security, graphics, databases, operating systems, embedded systems and programming languages. We retrieved every one of their references and citations and extracted from google scholar, every other paper that cited any of the references as well as all the references of each of the target paper’s citations. Some statistics of the utilized dataset is presented in Table 1 .

https://doi.org/10.1371/journal.pone.0184516.t001

4.2 Baseline methods

In assessing the effectiveness of our proposed framework, we compare the recommendation results with two baselines presented in [ 7 ] and [ 3 ]. The pattern introduced in [ 7 ] views citation relation matrix as a rating score and generates the recommendation based on common citations between the target paper and its neighboring papers. Given a target paper, the algorithm counts the number of times other citations were co-cited with it. The algorithm then recommends citations with the highest total co-citations summed over all recommending papers. The assumption was that, the more the co-citation in like manner between papers the higher their similarity. While [ 3 ], mined the hidden relationship between a target paper and all of its references. The task was to quantify the degree of closeness between the target paper and the other papers that also cited any of the target paper’s references. The rationale behind the approach was that, if two papers are significantly co-occurring with the same citing paper(s), then they should be similar to some extent.

4.3 Evaluation metrics

In order to evaluate the quality of our approach, for each of the target papers, we performed 5-fold cross validation to its references and citations by selecting 20% as a test set. We then assess the general performance using the three most commonly used evaluation metrics in retrieval systems: precision, recall and F1 measures. Precision given by Eq (2) , measures the capability of the system to reclaim as much relevant research papers as possible in response to the target paper request.

On the other hand, recall given by Eq (3) , measures the capability of the system to reclaim as few irrelevant research papers as possible in response to the target paper request.

Moreover, F1 measure given by Eq (4) is the harmonic mean between the precision and recall.

As users often scan only documents presented at the top ranked of the recommendation list, we feel imperative to also measure the system’s ability to provide useful recommendations at the top of the recommendation list using the two most widely used ranked information retrieval evaluation measures: Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).

5. Results and discussions

To be specific, the results of each evaluation metric in this section represent the overall averages over all the 50 researchers of our dataset. We start the comparison by assessing the general performance of our proposed approach in returning relevant research papers with the baseline methods based on the three most commonly used information retrieval evaluation metrics. Figs 2 – 4 , demonstrate the comparisons based on precision, recall and F1 evaluation measures respectively. As can be seen from Fig 2 , the precision results of our proposed approach has significantly outperformed the baseline methods (Context-Based Collaborative Filtering (CCF) proposed by [ 3 ] and Co-citation method proposed by [ 7 ]) in returning relevant research papers for all N recommendations values. This is because our approach is able to critically remove recommending papers that are less related to the target paper.

https://doi.org/10.1371/journal.pone.0184516.g002

https://doi.org/10.1371/journal.pone.0184516.g003

https://doi.org/10.1371/journal.pone.0184516.g004

Fig 3 depicts the comparison based on recall. As can be seen from the figure, the performance difference between our proposed approach and CCF is very much insignificant. In fact, the CCF method is even slightly better than our proposed approach when N = 5 and when N = 20. However, our proposed approach began to show the significant difference as the number of N increases, specifically when N is above 20. The low performance based on recall of our proposed approach is as a result of strict rules in qualifying a candidate paper. Thus, our approach is only after the most significant related recommending papers to the target paper and therefore leaving a lot of other less related papers unrecalled. Furthermore, Fig 4 depicts the harmonic mean between the precision and recall ( F 1 measure), and from the figure, the performance difference between our proposed approach and CCF is also insignificant for values of N less than or equals to 20. However, our approach began to show significant improvement over CCF when N is greater than 20. In all the three measures, the Co-citation method performs very low compared to our proposed approach. This is because the Co-citation method does not infer the hidden associations between paper-citation relations rather applies direct relations between a target paper and its neighboring papers.

Conclusively, the general performance of our proposed approach has outstandingly outperformed the baseline methods based on precision for all values of N . On the other hand, our proposed approach performs worse than CCF in a recommendation list of 5 based on recall and F1 performance measures. However, the major reason behind the low performance of our proposed approach based on recall is the strict rules in qualifying a candidate paper.

Our proposed approach is designed to favor precision which has more influence on user satisfaction than recall. This is because precision is the key element in the process of implementing a search solution [ 55 ]. Poor precision damages the reputation of a search system and discourages its use. High precision generally impresses search users [ 55 ]. That is why our proposed approach is only after the most significant related recommending papers to the target paper (the result of this can easily be seen from Fig 2 ), and therefore leaving a lot of other less related papers unrecalled. This is because recall is particularly important in applications where the user cannot afford to miss information such as issues related to security or compliance applications. The recall has less influence on user satisfaction than precision. Many searchers, especially on the Web, are satisfied by precise results, even where recall is low [ 56 ]. Notwithstanding, our proposed approach starts to show large disparities with the baseline methods when the number of N is above 5 for both recall and F 1 measures. Therefore, a very large N value is extremely important in order to recall as much qualitative and useful recommendations as possible.

Due to the fact that users usually scan only the top of the recommendation list, we also make the comparison based on how our approach is able to return relevant research papers at the top of the recommendation list. Figs 5 and 6 depict our comparisons based on Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) respectively. As can be seen from Fig 5 , that our proposed method has significantly outperformed the baseline methods based on mean average precision (MAP) in all cases in returning the relevant recommendations at the top of the recommendation list. Moreover, the comparison based on mean reciprocal rank (MRR) depicted by Fig 6 has also revealed that our proposed approach has outstandingly outperformed the baseline methods in all scenarios. It can easily be seen from the figure that our approach is able to return a relevant research paper at either rank 1 or rank 2 of the recommendation list for all queries.

https://doi.org/10.1371/journal.pone.0184516.g005

https://doi.org/10.1371/journal.pone.0184516.g006

As we have pointed out earlier, all these improvements are largely due to the strictness in qualifying a candidate paper which removed less relevant papers to the target paper. This, therefore, increases the system’s ability to return relevant and useful recommendations at the top of the recommendation list.

6. Conclusion and future work

In this paper, we utilized the publicly available contextual metadata to leverage the advantages of collaborative filtering approach in recommending a set of related papers to a researcher based on paper-citation relations. The approach mined the hidden associations between a research paper and its references and citations using paper-citation relations. The rationale behind the approach is that, if two papers are significantly co-occurring with the same citing paper(s), then they should be similar to some extent.

As demonstrated using a publicly available dataset, our proposed method outperforms the baseline methods in measuring both the overall performance and the ability to return relevant and useful research papers at the top of the recommendation list. Based on the three most commonly used information retrieval system metrics, our proposed approach have significantly improved the baseline methods based on precision, recall and F1 measures. Our proposed approach has also recorded significant improvements over the baseline methods in providing relevant and useful recommendations at the top of the recommendation list based on mean average precision (MAP) and mean reciprocal rank (MRR).

In addition to considering the collaborative relations among research papers, our next line of research is to also put into cognizance the public contextual contents, such as titles and abstracts of the recommending papers for better performances.

Supporting information

S1 dataset. the detail of the complete dataset can be accessed via https://figshare.com/articles/supporting_information_dataset_docx/5368408 ..

https://doi.org/10.1371/journal.pone.0184516.s001

• View Article
• Google Scholar
• 4. B. Gipp, J. Beel, and C. Hentschel, "Scienstein: A research paper recommender system," in Proceedings of the international conference on emerging trends in computing (icetic’09), 2009, pp. 309–315.
• 5. J. Beel, S. Langer, M. Genzmehr, and A. Nürnberger, "Introducing Docear's research paper recommender system," in Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, 2013, pp. 459–460.
• 6. K. Sugiyama and M.-Y. Kan, "Scholarly paper recommendation via user's recent research interests," in Proceedings of the 10th annual joint conference on Digital libraries, 2010, pp. 29–38.
• 7. S. M. McNee, I. Albert, D. Cosley, P. Gopalkrishnan, S. K. Lam, A. M. Rashid, et al., "On the recommending of citations for research papers," in Proceedings of the 2002 ACM conference on Computer supported cooperative work, 2002, pp. 116–125.
• 8. K. Sugiyama and M.-Y. Kan, "Exploiting potential citation papers in scholarly paper recommendation," in Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, 2013, pp. 153–162.
• 9. C. Nascimento, A. H. Laender, A. S. da Silva, and M. A. Gonçalves, "A source independent framework for research paper recommendation," in Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, 2011, pp. 297–306.
• 10. C. Prahalad, "Beyond CRM: CK Prahalad predicts customer context is the next big thing," American Management Association MwWorld, 2004.
• 11. K.-L. Skillen, L. Chen, C. D. Nugent, M. P. Donnelly, W. Burns, and I. Solheim, "Ontological user profile modeling for context-aware application personalization," in Ubiquitous Computing and Ambient Intelligence, ed: Springer, 2012, pp. 261–268.
• 12. Z. Yu, Y. Nakamura, S. Jang, S. Kajita, and K. Mase, "Ontology-based semantic recommendation for context-aware e-learning," in Ubiquitous Intelligence and Computing, ed: Springer, 2007, pp. 898–907.
• 13. R. Torres, S. M. McNee, M. Abel, J. A. Konstan, and J. Riedl, "Enhancing digital libraries with TechLens," in Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on, 2004, pp. 228–236.
• 16. P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl, "GroupLens: an open architecture for collaborative filtering of netnews," in Proceedings of the 1994 ACM conference on Computer supported cooperative work, 1994, pp. 175–186.
• 18. U. Shardanand and P. Maes, "Social information filtering: algorithms for automating “word of mouth”," in Proceedings of the SIGCHI conference on Human factors in computing systems, 1995, pp. 210–217.
• 20. N. Hariri, B. Mobasher, and R. Burke, "Context-aware music recommendation based on latenttopic sequential patterns," in Proceedings of the sixth ACM conference on Recommender systems, 2012, pp. 131–138.
• 21. N. Agarwal, E. Haque, H. Liu, and L. Parsons, "Research paper recommender systems: A subspace clustering approach," in International Conference on Web-Age Information Management, 2005, pp. 475–491.
• 22. K. Chandrasekaran, S. Gauch, P. Lakkaraju, and H. P. Luong, "Concept-based document recommendations for citeseer authors," in International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 2008, pp. 83–92.
• 23. M. Gori and A. Pucci, "Research paper recommender systems: A random-walk based approach," in Web Intelligence, 2006. WI 2006. IEEE/WIC/ACM International Conference on, 2006, pp. 778–781.
• 24. A. Kodakateri Pudhiyaveetil, S. Gauch, H. Luong, and J. Eno, "Conceptual recommender system for CiteSeerX," in Proceedings of the third ACM conference on Recommender systems, 2009, pp. 241–244.
• 25. K. Haruna, M. A. Ismail, and S. M. Shuhidan, "Domain of Application in Context-Aware Recommender Systems: A Review," Knowledge Management International Conference (KMICe) 2016, 29–30 August 2016, Chiang Mai, Thailand 2016.
• 27. A. Azaria, A. Hassidim, S. Kraus, A. Eshkol, O. Weintraub, and I. Netanely, "Movie recommender system for profit maximization," in Proceedings of the 7th ACM conference on Recommender systems, 2013, pp. 121–128.
• 28. Schedl M., Knees P., McFee B., Bogdanov D., and Kaminskas M., "Music recommender systems," in Recommender Systems Handbook , ed: Springer, 2015, pp. 453–492.
• 30. D. M. Pennock, E. Horvitz, S. Lawrence, and C. L. Giles, "Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach," in Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence, 2000, pp. 473–480.
• 32. Fano R., "Information theory and the retrieval of recorded information," Documentation in Action , Shera JH Kent A. Perry JW (Edts), New York: Reinhold Publ. Co, pp. 238–244, 1956.
• 33. I. V. Marshakova, "System of document connections based on references," Nauchno-Tekhnicheskaya Informatsiya Seriya 2-Informatsionnye Protsessy I Sistemy, pp. 3–8, 1973.
• 34. C. L. Giles, K. D. Bollacker, and S. Lawrence, "CiteSeer: An automatic citation indexing system," in Proceedings of the third ACM conference on Digital libraries, 1998, pp. 89–98.
• 35. Y. Sibaroni, D. H. Widyantoro, and M. L. Khodra, "Survey on research paper's relations," in Information Technology Systems and Innovation (ICITSI), 2015 International Conference on, 2015, pp. 1–6.
• 36. K. Sugiyama and M.-Y. Kan, "Serendipitous recommendation for scholarly papers considering relations among researchers," in Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, 2011, pp. 307–310.
• 38. Kumar R., Novak J., and Tomkins A., "Structure and evolution of online social networks," in Link mining : models , algorithms , and applications , ed: Springer, 2010, pp. 337–357.
• 39. F. C. T. Chua, H. W. Lauw, and E.-P. Lim, "Predicting item adoption using social correlation," in Proceedings of the 2011 SIAM International Conference on Data Mining, 2011, pp. 367–378.
• 40. I. Konstas, V. Stathopoulos, and J. M. Jose, "On social networks and collaborative recommendation," in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 2009, pp. 195–202.
• 41. H. Ma, I. King, and M. R. Lyu, "Learning to recommend with social trust ensemble," in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 2009, pp. 203–210.
• 42. H. Ma, D. Zhou, C. Liu, M. R. Lyu, and I. King, "Recommender systems with social regularization," in Proceedings of the fourth ACM international conference on Web search and data mining, 2011, pp. 287–296.
• 43. L. Backstrom, D. Huttenlocher, J. Kleinberg, and X. Lan, "Group formation in large social networks: membership, growth, and evolution," in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006, pp. 44–54.
• 52. F. Xia, N. Y. Asabere, H. Liu, N. Deonauth, and F. Li, "Folksonomy based socially-aware recommendation of scholarly papers for conference participants," in Proceedings of the 23rd International Conference on World Wide Web, 2014, pp. 781–786.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Systematic review of recommendation systems for course selection.

1. Introduction

2. motivation and rationale of the research, 3. research questions, 3.1. questions about the used algorithms.

• What preprocessing methods were applied?
• What recommendation system algorithms were used in the paper?
• What are the applied evaluation metrics?
• What are the performance results of applied evaluation metrics?

3.2. Questions about the Used Dataset

• Is the dataset published or accessible?
• How many records are there in the dataset?
• How many unique student records are there in the dataset?
• How many unique course records are there in the dataset?
• How many features are there in the dataset?
• How many features are used from the existing features?
• How many unique majors are there in the dataset?
• How did the authors split the training and testing set?

3.3. Questions about the Research

• What is the type of comparative produced in the study (algorithm level, preprocessing level, or data level)?
• What is the main aim of the study?
• What are the strong points of the research?
• What are the weak points of the research?

4. Research Methodology

4.1. title-level screening stage.

• The study addresses recommendation systems in the Education sector.
• The study must be primary.

4.2. Abstract-Level Screening Stage

4.3. full-text article scanning stage.

• The study was written in the English language.
• The study implies empirical experiments and provides the experiment’s results.

4.4. Full-Text Article Screening Stage

• Q1: Did the study conduct experiment in the course selection and courses recommendation system?
• Q2: Is there a comparison with other approaches in the conducted study?
• Q3: Were the performance measures fully defined?
• Q4: Was the method used in the study clearly described?
• Q5: Was the dataset and number of training and testing data identified?

4.5. Data Extraction Stage

5. research results, 5.1. the studies included in the slr, 5.1.1. collaborative filtering studies, 5.1.2. content-based filtering studies, 5.1.3. hybrid recommender system studies, 5.1.4. studies based on machine learning, 5.1.5. similarity-based study, 6. key studies analysis, 6.1. discussion of aims and contributions of the existing research works, 6.1.1. aim of studies that used collaborative filtering, 6.1.2. aim of studies that used content-based filtering, 6.1.3. aim of studies that used hybrid recommender systems, 6.1.4. aim of studies that used novel approaches, 6.1.5. aim of studies that used similarity-based filtering, 6.2. description of datasets used in the studies, 6.2.1. dataset description of studies that used collaborative filtering, 6.2.2. dataset description of studies that used content-based filtering, 6.2.3. dataset description of studies that used hybrid recommender systems, 6.2.4. dataset description of studies that used novel approaches.

• Train-test split.
• K-fold cross-validation.
• Nested time series splits.

6.2.5. Dataset Description of the Study That Used Similarity-Based Filtering

6.3. research evaluation, 6.3.1. research evaluation for studies that used collaborative filtering, 6.3.2. research evaluation for studies that used content-based filtering, 6.3.3. research evaluation for studies that used hybrid recommender systems, 6.3.4. research evaluation for studies that used novel approaches, 6.3.5. research evaluation for the study that used similarity-based filtering, 7. discussion of findings, 8. gaps, challenges, future directions and conclusions for (crs) selection, 8.2. challenges, 8.3. future directions, 9. conclusions.

• Making precise course recommendations that are tailored to each student’s interests, abilities, and long-term professional goals.
• Addressing the issue of “cold starts,” wherein brand-new students without prior course experience might not obtain useful, reliable, and precise advice.
• Ensuring that the system is flexible enough to accommodate various educational contexts, data accessibility, and the unique objectives of the advising system.
• Increasing suggestion recall and precision rates.
• Using preprocessing and data-splitting methods to enhance the predefined performance standards of the CRS overall as well as the predefined and measured quality of recommendations.

Author Contributions

Data availability statement, conflicts of interest.

• Iatrellis, O.; Kameas, A.; Fitsilis, P. Academic advising systems: A systematic literature review of empirical evidence. Educ. Sci. 2017 , 7 , 90. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Chang, P.C.; Lin, C.H.; Chen, M.H. A hybrid course recommendation system by integrating collaborative filtering and artificial immune systems. Algorithms 2016 , 9 , 47. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Xu, J.; Xing, T.; Van Der Schaar, M. Personalized course sequence recommendations. IEEE Trans. Signal Process. 2016 , 64 , 5340–5352. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Noaman, A.Y.; Ahmed, F.F. A new framework for e academic advising. Procedia Comput. Sci. 2015 , 65 , 358–367. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Pizzolato, J.E. Complex partnerships: Self-authorship and provocative academic-advising practices. NACADA J. 2006 , 26 , 32–45. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Unelsrød, H.F. Design and Evaluation of a Recommender System for Course Selection. Master’s Thesis, Institutt for Datateknikk og Informasjonsvitenskap, Trondheim, Norway, 2011. [ Google Scholar ]
• Kuh, G.D.; Kinzie, J.; Schuh, J.H.; Whitt, E.J. Student Success in College: Creating Conditions That Matter ; John Wiley & Sons: New York, NY, USA, 2011. [ Google Scholar ]
• Mostafa, L.; Oately, G.; Khalifa, N.; Rabie, W. A case based reasoning system for academic advising in Egyptian educational institutions. In Proceedings of the 2nd International Conference on Research in Science, Engineering and Technology (ICRSET’2014), Dubai, United Arab Emirates, 21–22 March 2014; pp. 21–22. [ Google Scholar ]
• Obeidat, R.; Duwairi, R.; Al-Aiad, A. A collaborative recommendation system for online courses recommendations. In Proceedings of the 2019 International Conference on Deep Learning and Machine Learning in Emerging Applications (Deep-ML), Istanbul, Turkey, 26–28 August 2019; pp. 49–54. [ Google Scholar ]
• Feng, J.; Xia, Z.; Feng, X.; Peng, J. RBPR: A hybrid model for the new user cold start problem in recommender systems. Knowl.-Based Syst. 2021 , 214 , 106732. [ Google Scholar ] [ CrossRef ]
• Kohl, C.; McIntosh, E.J.; Unger, S.; Haddaway, N.R.; Kecke, S.; Schiemann, J.; Wilhelm, R. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: A case study on CADIMA and review of existing tools. Environ. Evid. 2018 , 7 , 8. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Shminan, A.S.; Choi, L.J.; Barawi, M.H.; Hashim, W.N.W.; Andy, H. InVesa 1.0: The Conceptual Framework of Interactive Virtual Academic Advisor System based on Psychological Profiles. In Proceedings of the 2021 13th International Conference on Information & Communication Technology and System (ICTS), Surabaya, Indonesia, 20–21 October 2021; pp. 112–117. [ Google Scholar ]
• Wang, H.; Wei, Z. Research on Personalized Learning Route Model Based on Improved Collaborative Filtering Algorithm. In Proceedings of the 2021 2nd International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), Zhuhai, China, 24–26 September 2021; pp. 120–123. [ Google Scholar ]
• Shaptala, R.; Kyselova, A.; Kyselov, G. Exploring the vector space model for online courses. In Proceedings of the 2017 IEEE First Ukraine Conference on Electrical and Computer Engineering (UKRCON), Kyiv, Ukraine, 29 May–2 June 2017; pp. 861–864. [ Google Scholar ]
• Zhao, X.; Liu, B. Application of personalized recommendation technology in MOOC system. In Proceedings of the 2020 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), Vientiane, Laos, 11–12 January 2020; pp. 720–723. [ Google Scholar ]
• Wahyono, I.D.; Asfani, K.; Mohamad, M.M.; Saryono, D.; Putranto, H.; Haruzuan, M.N. Matching User in Online Learning using Artificial Intelligence for Recommendation of Competition. In Proceedings of the 2021 Fourth International Conference on Vocational Education and Electrical Engineering (ICVEE), Surabaya, Indonesia, 2–3 October 2021; pp. 1–4. [ Google Scholar ]
• Elghomary, K.; Bouzidi, D. Dynamic peer recommendation system based on trust model for sustainable social tutoring in MOOCs. In Proceedings of the 2019 1st International Conference on Smart Systems and Data Science (ICSSD), Rabat, Morocco, 3–4 October 2019; pp. 1–9. [ Google Scholar ]
• Mufizar, T.; Mulyani, E.D.S.; Wiyono, R.A.; Arifiana, W. A combination of Multi Factor Evaluation Process (MFEP) and the Distance to the Ideal Alternative (DIA) methods for majors selection and scholarship recipients in SMAN 2 Tasikmalaya. In Proceedings of the 2018 6th International Conference on Cyber and IT Service Management (CITSM), Parapat, Indonesia, 7–9 August 2018; pp. 1–7. [ Google Scholar ]
• Sutrisno, M.; Budiyanto, U. Intelligent System for Recommending Study Level in English Language Course Using CBR Method. In Proceedings of the 2019 6th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), Bandung, Indonesia, 18–20 September 2019; pp. 153–158. [ Google Scholar ]
• Gan, B.; Zhang, C. Research on the Application of Curriculum Knowledge Point Recommendation Algorithm Based on Learning Diagnosis Model. In Proceedings of the 2020 5th International Conference on Electromechanical Control Technology and Transportation (ICECTT), Nanchang, China, 5–17 May 2020; pp. 188–192. [ Google Scholar ]
• Ivanov, D.A.; Ivanova, I.V. Computer Self-Testing of Students as an Element of Distance Learning Technologies that Increase Interest in the Study of General Physics Course. In Proceedings of the 2018 IV International Conference on Information Technologies in Engineering Education (Inforino), Moscow, Russia, 22–26 October 2018; pp. 1–4. [ Google Scholar ]
• Anupama, V.; Elayidom, M.S. Course Recommendation System: Collaborative Filtering, Machine Learning and Topic Modelling. In Proceedings of the 2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, 25–26 March 2022; Volume 1, pp. 1459–1462. [ Google Scholar ]
• Sabnis, V.; Tejaswini, P.D.; Sharvani, G.S. Course recommendations in moocs: Techniques and evaluation. In Proceedings of the 2018 3rd International Conference on Computational Systems and Information Technology for Sustainable Solutions (CSITSS), Bengaluru, India, 20–22 December 2018; pp. 59–66. [ Google Scholar ]
• Britto, J.; Prabhu, S.; Gawali, A.; Jadhav, Y. A Machine Learning Based Approach for Recommending Courses at Graduate Level. In Proceedings of the 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, 27–29 November 2019; pp. 117–121. [ Google Scholar ]
• Peng, Y. A Survey on Modern Recommendation System based on Big Data. arXiv 2022 , arXiv:2206.02631. [ Google Scholar ]
• Bozyiğit, A.; Bozyiğit, F.; Kilinç, D.; Nasiboğlu, E. Collaborative filtering based course recommender using OWA operators. In Proceedings of the 2018 International Symposium on Computers in Education (SIIE), Jerez, Spain, 19–21 September 2018; pp. 1–5. [ Google Scholar ]
• Mondal, B.; Patra, O.; Mishra, S.; Patra, P. A course recommendation system based on grades. In Proceedings of the 2020 International Conference on Computer Science, Engineering and Applications (ICCSEA), Gunupur, India, 13–14 March 2020; pp. 1–5. [ Google Scholar ]
• Lee, E.L.; Kuo, T.T.; Lin, S.D. A collaborative filtering-based two stage model with item dependency for course recommendation. In Proceedings of the 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Tokyo, Japan, 19–21 October 2017; pp. 496–503. [ Google Scholar ]
• Malhotra, I.; Chandra, P.; Lavanya, R. Course Recommendation using Domain-based Cluster Knowledge and Matrix Factorization. In Proceedings of the 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, Indi, 23–25 March 2022; pp. 12–18. [ Google Scholar ]
• Huang, L.; Wang, C.D.; Chao, H.Y.; Lai, J.H.; Philip, S.Y. A score prediction approach for optional course recommendation via cross-user-domain collaborative filtering. IEEE Access 2019 , 7 , 19550–19563. [ Google Scholar ] [ CrossRef ]
• Zhao, L.; Pan, Z. Research on online course recommendation model based on improved collaborative filtering algorithm. In Proceedings of the 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), Chengdu, China, 24–26 April 2021; pp. 437–440. [ Google Scholar ]
• Ceyhan, M.; Okyay, S.; Kartal, Y.; Adar, N. The Prediction of Student Grades Using Collaborative Filtering in a Course Recommender System. In Proceedings of the 2021 5th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Ankara, Turkey, 21–23 October 2021; pp. 177–181. [ Google Scholar ]
• Dwivedi, S.; Roshni, V.K. Recommender system for big data in education. In Proceedings of the 2017 5th National Conference on E-Learning & E-Learning Technologies (ELELTECH), Hyderabad, India, 3–4 August 2017; pp. 1–4. [ Google Scholar ]
• Zhong, S.T.; Huang, L.; Wang, C.D.; Lai, J.H. Constrained matrix factorization for course score prediction. In Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, China, 8–11 November 2019; pp. 1510–1515. [ Google Scholar ]
• Chen, Z.; Song, W.; Liu, L. The application of association rules and interestingness in course selection system. In Proceedings of the 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA), Beijing, China, 10–12 March 2017; pp. 612–616. [ Google Scholar ]
• Chen, Z.; Liu, X.; Shang, L. Improved course recommendation algorithm based on collaborative filtering. In Proceedings of the 2020 International Conference on Big Data and Informatization Education (ICBDIE), Zhangjiajie, China, 23–25 April 2020; pp. 466–469. [ Google Scholar ]
• Ren, Z.; Ning, X.; Lan, A.S.; Rangwala, H. Grade prediction with neural collaborative filtering. In Proceedings of the 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Washington, DC, USA, 5–8 October 2019; pp. 1–10. [ Google Scholar ]
• Fernández-García, A.J.; Rodríguez-Echeverría, R.; Preciado, J.C.; Manzano, J.M.C.; Sánchez-Figueroa, F. Creating a recommender system to support higher education students in the subject enrollment decision. IEEE Access 2020 , 8 , 189069–189088. [ Google Scholar ] [ CrossRef ]
• Adilaksa, Y.; Musdholifah, A. Recommendation System for Elective Courses using Content-based Filtering and Weighted Cosine Similarity. In Proceedings of the 2021 4th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), Yogyakarta, Indonesia, 16–17 December 2021; pp. 51–55. [ Google Scholar ]
• Esteban, A.; Zafra, A.; Romero, C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowl.-Based Syst. 2020 , 194 , 105385. [ Google Scholar ] [ CrossRef ]
• Emon, M.I.; Shahiduzzaman, M.; Rakib, M.R.H.; Shathee, M.S.A.; Saha, S.; Kamran, M.N.; Fahim, J.H. Profile Based Course Recommendation System Using Association Rule Mining and Collaborative Filtering. In Proceedings of the 2021 International Conference on Science & Contemporary Technologies (ICSCT), Dhaka, Bangladesh, 5–7 August 2021; pp. 1–5. [ Google Scholar ]
• Alghamdi, S.; Sheta, O.; Adrees, M. A Framework of Prompting Intelligent System for Academic Advising Using Recommendation System Based on Association Rules. In Proceedings of the 2022 9th International Conference on Electrical and Electronics Engineering (ICEEE), Alanya, Turkey, 29–31 March 2022; pp. 392–398. [ Google Scholar ]
• Bharath, G.M.; Indumathy, M. Course Recommendation System in Social Learning Network (SLN) Using Hybrid Filtering. In Proceedings of the 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India, 2–4 December 2021; pp. 1078–1083. [ Google Scholar ]
• Nafea, S.M.; Siewe, F.; He, Y. On recommendation of learning objects using felder-silverman learning style model. IEEE Access 2019 , 7 , 163034–163048. [ Google Scholar ] [ CrossRef ]
• Huang, X.; Tang, Y.; Qu, R.; Li, C.; Yuan, C.; Sun, S.; Xu, B. Course recommendation model in academic social networks based on association rules and multi-similarity. In Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design (CSCWD), Nanjing, China, 9–11 May 2018; pp. 277–282. [ Google Scholar ]
• Baskota, A.; Ng, Y.K. A graduate school recommendation system using the multi-class support vector machine and KNN approaches. In Proceedings of the 2018 IEEE International Conference on Information Reuse and Integration (IRI), Salt Lake City, UT, USA, 6–9 July 2018; pp. 277–284. [ Google Scholar ]
• Jiang, W.; Pardos, Z.A.; Wei, Q. Goal-based course recommendation. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, AZ, USA, 4–8 March 2019; pp. 36–45. [ Google Scholar ]
• Liang, Y.; Duan, X.; Ding, Y.; Kou, X.; Huang, J. Data Mining of Students’ Course Selection Based on Currency Rules and Decision Tree. In Proceedings of the 2019 4th International Conference on Big Data and Computing, Guangzhou, China, 10–12 May 2019; pp. 247–252. [ Google Scholar ]
• Isma’il, M.; Haruna, U.; Aliyu, G.; Abdulmumin, I.; Adamu, S. An autonomous courses recommender system for undergraduate using machine learning techniques. In Proceedings of the 2020 International Conference in Mathematics, Computer Engineering and Computer Science (ICMCECS), Ayobo, Nigeria, 18–21 March 2020; pp. 1–6. [ Google Scholar ]
• Revathy, M.; Kamalakkannan, S.; Kavitha, P. Machine Learning based Prediction of Dropout Students from the Education University using SMOTE. In Proceedings of the 2022 4th International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, 20–22 January 2022; pp. 1750–1758. [ Google Scholar ]
• Oreshin, S.; Filchenkov, A.; Petrusha, P.; Krasheninnikov, E.; Panfilov, A.; Glukhov, I.; Kaliberda, Y.; Masalskiy, D.; Serdyukov, A.; Kazakovtsev, V.; et al. Implementing a Machine Learning Approach to Predicting Students’ Academic Outcomes. In Proceedings of the 2020 International Conference on Control, Robotics and Intelligent System, Xiamen, China, 27–29 October 2020; pp. 78–83. [ Google Scholar ]
• Verma, R. Applying Predictive Analytics in Elective Course Recommender System while preserving Student Course Preferences. In Proceedings of the 2018 IEEE 6th International Conference on MOOCs, Innovation and Technology in Education (MITE), Hyderabad, India, 29–30 November 2018; pp. 52–59. [ Google Scholar ]
• Bujang, S.D.A.; Selamat, A.; Ibrahim, R.; Krejcar, O.; Herrera-Viedma, E.; Fujita, H.; Ghani, N.A.M. Multiclass prediction model for student grade prediction using machine learning. IEEE Access 2021 , 9 , 95608–95621. [ Google Scholar ] [ CrossRef ]
• Srivastava, S.; Karigar, S.; Khanna, R.; Agarwal, R. Educational data mining: Classifier comparison for the course selection process. In Proceedings of the 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE), Shah Alam, Malaysia, 11–12 July 2018; pp. 1–5. [ Google Scholar ]
• Abed, T.; Ajoodha, R.; Jadhav, A. A prediction model to improve student placement at a south african higher education institution. In Proceedings of the 2020 International SAUPEC/RobMech/PRASA Conference, Cape Town, South Africa, 29–31 January 2020; pp. 1–6. [ Google Scholar ]
• Uskov, V.L.; Bakken, J.P.; Byerly, A.; Shah, A. Machine learning-based predictive analytics of student academic performance in STEM education. In Proceedings of the 2019 IEEE Global Engineering Education Conference (EDUCON), Dubai, United Arab Emirates, 8–11 April 2019; pp. 1370–1376. [ Google Scholar ]
• Sankhe, V.; Shah, J.; Paranjape, T.; Shankarmani, R. Skill Based Course Recommendation System. In Proceedings of the 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), Greater Noida, India, 2–4 October 2020; pp. 573–576. [ Google Scholar ]
• Kamila, V.Z.; Subastian, E. KNN and Naive Bayes for Optional Advanced Courses Recommendation. In Proceedings of the 2019 International Conference on Electrical, Electronics and Information Engineering (ICEEIE), Denpasar, Indonesia, 3–4 October 2019; Volume 6, pp. 306–309. [ Google Scholar ]
• Shah, D.; Shah, P.; Banerjee, A. Similarity based regularization for online matrix-factorization problem: An application to course recommender systems. In Proceedings of the TENCON 2017—2017 IEEE Region 10 Conference, Penang, Malaysia, 5–8 November 2017; pp. 1874–1879. [ Google Scholar ]

Click here to enlarge figure

Author	The Study Addresses Recommendation Systems in the Education Sector	Primary Study
Shminan et al. [ ]	No	Yes
Wang et al. [ ]	No	Yes
Shaptala et al. [ ]	No	Yes
Zhao et al. [ ]	No	Yes
ID Wahyono et al. [ ]	No	Yes

Author	The Study Addresses Recommendation Systems in the Education Sector	Primary Study
Elghomary et al. [ ]	No	Yes
Mufizar et al. [ ]	No	Yes
Sutrisno et al. [ ]	No	Yes
Gan et al. [ ]	No	Yes
Ivanov et al. [ ]	No	Yes

Author	Reason of Exclusion
Anupama et al. [ ]	Did not imply empirical experiments and did not provide experiments results
Sabnis et al. [ ]	The full text is not accessible

Author	Score					Total Score	Included
	Q1	Q2	Q3	Q4	Q5
Britto et al. [ ]	1	0	0.5	0.5	0.5	2.5	No
Obeidat et al. [ ]	0.5	1	0.5	0.5	0.5	3	Yes

Authors and Year	Algorithms Used	Comparative Type
A. Bozyiğit et al., 2018 [ ]	• Collaborative filtering;	Algorithm level
	• OWA (ordered weighted average).
B. Mondal et al., 2020 [ ]	• Collaborative filtering.	Algorithm level
E. L. Lee et al., 2017 [ ]	• Two-stage collaborative filtering;	Algorithm level
	• Personalized Ranking Matrix Factorization (BPR-MF);
	• Course dependency regularization;
	• Personalized PageRank;
	• Linear RankSVM.
I. Malhotra et al., 2022 [ ]	• Collaborative filtering;	Algorithm level
	• Domain-based cluster knowledge;
	• Cosine pairwise similarity evaluation;
	• Singular value decomposition ++;
	• Matrix factorization.
L. Huang et al., 2019 [ ]	• Cross-user-domain collaborative filtering.	Algorithm level
L. Zhao et al., 2021 [ ]	• Improved collaborative filtering;	Algorithm level
	• Historical preference fusion similarity.
M. Ceyhan et al., 2021 [ ]	• Collaborative filtering;	Algorithm level
	• Correlation-based similarities: (Pearson correlation coefficient, median-based robust correlation coefficient);
	• Distance-based similarities: Manhattan and Euclidian distance similarities.
R. Obeidat et al., 2019 [ ]	• Collaborative filtering;	Algorithm level
	• K-means clustering;
	• Association rules (Apriori algorithm, sequential, pattern discovery using equivalence classes algorithm).
S. Dwivedi et al., 2017 [ ]	• Collaborative filtering;	Algorithm level
	• Similarity log-likelihood.
S.-T. Zhong et al., 2019 [ ]	• Collaborative filtering;	Algorithm level
	• Constrained matrix factorization.
Z. Chen et al., 2017 [ ]	• Collaborative filtering;	Algorithm level
	• Association rules (Apriori).
Z. Chen et al., 2020 [ ]	• Collaborative filtering;	Algorithm level
	• Improved cosine similarity;
	• TF–IDF (term frequency–inverse document frequency).
Z. Ren et al., 2019 [ ]	• Neural Collaborative Filtering (NCF).	Algorithm level

Authors and Year	Used Algorithms	Comparative Type
A. J. Fernández-García et al., 2020 [ ]	• Content-based filtering.	Preprocessing level
Y. Adilaksa et al., 2021 [ ]	• Content-based filtering;	Algorithm level
	• Weighted cosine similarity; • TF–IDF.

Authors and Year	Used Algorithms	Comparative Type
Esteban, A. et al., 2020 [ ]	• Hybrid recommender system;	Algorithm level
	• Collaborative filtering;
	• Content-based filtering;
	• Genetic algorithm.
M. I. Emon et al., 2021 [ ]	• Hybrid recommender system;	Algorithm level
	• Collaborative filtering;
	• Association rules (Apriori algorithm).
S. Alghamdi et al., 2022 [ ]	• Hybrid recommender system;	Algorithm level
	• Content-based filtering;
	• Association rules (Apriori algorithm);
	• Jaccard coefficient.
S. G. G et al., 2021 [ ]	• Hybrid recommender system;	Algorithm level
	• Collaborative filtering;
	• Content-based filtering;
	• Lasso;
	• KNN;
	• Weighted average.
S. M. Nafea et al., 2019 [ ]	• Hybrid recommender system;	Algorithm level
	• Felder–Silverman learning styles model;
	• K-means clustering.
X. Huang et al., 2018 [ ]	• Hybrid recommender system;	Algorithm level
	• Association rules;
	• Improved multi-similarity.

Authors and Year	Used Algorithms	Comparative Type
A. Baskota et al., 2018 [ ]	• Forward feature selection;	Algorithm level
	• K-Nearest Neighbor (KNN);
	• Multi-class Support Vector Machines (MC-SVM).
Jiang, Weijie et al., 2019 [ ]	• Goal-based filtering;	Algorithm level
	• LSTM recurrent neural network.
Liang, Yu et al., 2019 [ ]	• Currency rules;	Preprocessing level
	• C4.5 decision tree.
M. Isma’il et al., 2020 [ ]	• Support Vector Machine (SVM).	Algorithm level
M. Revathy et al., 2022 [ ]	• KNN-SMOTE.	Algorithm level
Oreshin et al., 2020 [ ]	• Latent Dirichlet Allocation;	Algorithm level
	• FastTextSocialNetworkModel;
	• Catboost.
R. Verma et al., 2018 [ ]	• Support Vector Machines;	Algorithm level
	• Artificial Neural Networks (ANN).
S. D. A. Bujang et al., 2021 [ ]	• Random forests.	• Algorithm level
		• Preprocessing level
S. Srivastava et al., 2018 [ ]	• Support Vector Machines with radial basis kernel;	Algorithm level
	• KNN.
T. Abed et al., 2020 [ ]	• Naive Bayes.	Algorithm level
V. L. Uskov et al., 2019 [ ]	• Linear regression.	Algorithm level
V. Sankhe et al., 2020 [ ]	• Skill-based filtering;	Algorithm level
	• C-means fuzzy clustering;
	• Weighted mode.
V. Z. Kamila et al., 2019 [ ]	• KNN;	algorithm level
	• Naive Bayes.

Authors and Year	Used Algorithms	Comparative Type
D. Shah et al., 2017 [ ]	• Similarity-based regularization;	Algorithm level
	• Matrix factorization.

Authors and Year	Public	Records	Students	Courses	Majors	Features	Used Features	Preprocessing Steps	Data-Splitting Method
A. Bozyiğit et al., 2018 [ ]	No	N/A	221	76	N/A	N/A	N/A	N/A	Ten-fold cross-validation.
B. Mondal et al., 2020 [ ]	No	300	300	N/A	N/A	48	12	• Data cleaning: lowercase conversion, removing punctuation, striping white spaces.	N/A
E. L. Lee et al., 2017 [ ]	No	896,616	13,977	N/A	N/A	N/A	N/A	• Ignore the students whose 4-year registration records are incomplete.	Nested time-series split cross-validation (class 2008, class 2009 as a training set, and class 2010 as a testing set).
I. Malhotra et al., 2022 [ ]	No	N/A	1780	N/A	9	N/A	N/A	N/A	N/A
L. Huang et al., 2019 [ ]	No	52,311	1166	N/A	8	N/A	N/A	N/A	N/A
L. Zhao et al., 2021 [ ]	No	N/A	43,916	240	N/A	N/A	N/A	• Group data based on interest data points, • Eliminate noise by filtering the data noise constrained in 0,1, • Normalize all numerical features.	Five-fold cross-validation.
M. Ceyhan et al., 2021 [ ]	No	N/A	1506	1460	N/A	N/A	N/A	• The updated grade is taken into consideration if a student retakes any course.	• Nested time-series split cross-validation, • Train = 91.7% (from 2010/11-F to 2019/20-S), • Test = 8.3% (the whole 2020/21-F).
R. Obeidat et al., 2019 [ ]	Yes	22,144	10,000	16	N/A	N/A	N/A	• Remove incomplete records • Calculate the order of courses sequences events for each student, • Convert grades to a new grade scale, • Cluster students.	N/A
S. Dwivedi et al., 2017 [ ]	No	N/A	N/A	N/A	N/A	N/A	N/A	• Data cleaning, • Data discretization (converting low-level concept to high-level concept).	N/A
S. -T. Zhong et al., 2019 [ ]	No	N/A	N/A	N/A	8	N/A	N/A	N/A	N/A
Z. Chen et al., 2017 [ ]	No	N/A	N/A	N/A	N/A	N/A	N/A	Students’ score categorization (A, B, C).	N/A
Z. Chen et al., 2020 [ ]	No	18,457	2022	309	N/A	N/A	N/A	N/A	K-fold cross-validation.
Z. Ren et al., 2019 [ ]	No	N/A	43,099	N/A	151	N/A	N/A	Used different embedding dimensions for students, courses, and course instructors for different majors.	Nested time-series split cross-validation (data from Fall 2009 to Fall 2015 as a training set, and data from Spring 2016 as a testing set).

Authors and Year	Public	Records	Students	Courses	Majors	Features	Features Used	Preprocessing Steps	Data-Splitting Method
A. J. Fernández-García et al., 2020 [ ]	No	6948	323	N/A	N/A	10	10	• Feature deletion, • Class reduction, • One-hot encoding, • Creating new features, • Data scaling: MinMax Scaler, Standard Scaler, Robust Scaler, and Normalizer Scaler, • Data resampling: upsample, downsample, SMOTE.	• Train size = 80%, • Test size = 20%.
Y. Adilaksa et al., 2021 [ ]	No	N/A	N/A	N/A	N/A	N/A	N/A	• Case folding, • Word tokenization, • Punctuation removal, • Stop words removal.	N/A

Authors and Year	Pub	Recs	Students	Courses	Majors	Features	Used Features	Preprocessing Steps	Data-Splitting Method
Esteban, A. et al., 2020 [ ]	No	2500C	95	63	N/A	N/A	N/A	N/A	Five-fold cross-validation.
M. I. Emon et al., 2021 [ ]	No	N/A	250+	250+	20+	N/A	N/A	Feature extraction.	N/A
S. Alghamdi et al., 2022 [ ]	No	1820	38	48	N/A	N/A	7	Cluster sets for academic transcript datasets.	Five-fold cross-validation.
S. G. G et al., 2021 [ ]	No	N/A	~6000	~4000	18	N/A	N/A	N/A	N/A
S. M. Nafea et al., 2019 [ ]	No	N/A	80	N/A	N/A	N/A	N/A	N/A	Student dataset was split into cold-start students, cold-start learning objects, and all students.
X. Huang et al., 2018 [ ]	Yes	N/A	56,600	860	N/A	N/A	N/A	N/A	• Train size = 80%, • Test size = 20%.

Authors and Year	Public	Records	Students	Courses	Majors	Features	Features Used	Preprocessing Steps	Data-Splitting Method
A. Baskota et al., 2018 [ ]	No	16,000	N/A	N/A	N/A	N/A	N/A	• Data cleaning, • Data scaling.	• Train size = 14,000, • Test size = 2000.
Jiang, Weijie et al., 2019 [ ]	No	4,800,000	164,196	10,430	17	N/A	N/A	N/A	Nested time-series split cross-validation (data from F’08 to F’15 as a training set, data in Sp’16 as validation set & data in Sp’17 as test set)
Liang, Yu et al., 2019 [ ]	No	35,000	N/A	N/A	N/A	N/A	N/A	Data cleaning.	N/A
M. Isma’il et al., 2020 [ ]	No	8700	N/A	9	N/A	N/A	4	• Data cleaning, • Data encoding.	N/A
M. Revathy et al., 2022 [ ]	No	N/A	1243	N/A	N/A	N/A	33	• One-hot encoding for categorical features, • Principal Component Analysis (PCA).	• Train size = 804, • Test size = 359.
Oreshin et al., 2020 [ ]	No	N/A	>20,000	N/A	N/A	N/A	112	• One-hot encoding, • Removed samples with unknown values.	Nested time-series split cross-validation.
R. Verma et al., 2018 [ ]	No	658	658	N/A	N/A	13	11	Data categorization.	Ten-fold cross-validation.
S. D. A. Bujang et al., 2021 [ ]	No	1282	641	2	N/A	13	N/A	• Ranked and grouped the students into five categories of grades, • Applied oversampling SMOTE (Synthetic Minority Over-sampling Technique), • Applied two feature selection methods: Wrapper and filter-based.	Ten-fold cross-validation.
S. Srivastava et al., 2018 [ ]	No	1988	2890	N/A	N/A	N/A	14	Registration number transformation.	• Train = 1312, • Test = 676.
T. Abed et al., 2020 [ ]	No	N/A	N/A	N/A	N/A	N/A	18	Balanced the dataset using under sampling.	Ten-fold cross-validation.
V. L. Uskov et al., 2019 [ ]	No	90+	N/A	N/A	N/A	16	N/A	Data cleaning	• Train = 80%, • Test = 20%.
V. Sankhe et al., 2020 [ ]	No	N/A	2000	15	7	N/A	N/A	N/A	N/A
V. Z. Kamila et al., 2019 [ ]	No	N/A	N/A	N/A	N/A	N/A	N/A	N/A	• Train size = 75%, • Test size= 25%.

Authors and Year	Public	Records	Students	Courses	Majors	Features	Features Used	Preprocessing Steps	Data-Splitting Method
D. Shah et al., 2017 [ ]	No	N/A	• Dataset 1 = 300 students • Dataset 2 = 84 students	• Dataset 1 = 10 • Dataset 2 = 26	N/A	N/A	• Student features = 3 • Course features = 30	N/A	• Train size = 90% • Test size = 10%

Authors and Year	Evaluation Metrics and Values	Strengths	Weaknesses
A. Bozyiğit et al., 2018 [ ]	MAE = 0.063.	• Compared the performance of the proposed OWA approach with the performance of other popular approaches.	• The number of features and features used in the dataset is not provided, • The dataset description is not detailed, • Did not use RMSE for evaluation, considered the standard as it’s more accurate, • Mentioned that some preprocessing had been carried out but did not give any details regarding it.
B. Mondal et al., 2020 [ ]	• MSE = 3.609, • MAE = 1.133, • RMSE = 1.8998089, • Precision, • Recall.	• Used many metrics for evaluation, • The implementation of algorithms is comprehensively explained.	• Did not mention whether they split data for testing or used the training data for testing, • Did not provide the exact measures of precision and recall.
E. L. Lee et al., 2017 [ ]	AUC = 0.9709.	• Compared the performance of the proposed approach with the performance of other approaches, • Used a very large dataset, • Achieved a very high AUC, • The implementation of algorithms is comprehensively explained.	• Did not provide the percentage of the train-test split, • The number of courses in the dataset is not mentioned (it only mentions course registration records).
I. Malhotra et al., 2022 [ ]	• MAE = 0.468, • RMSE = 0.781.	• The implementation of algorithms is comprehensively explained with examples, • Used RMSE and MAE for evaluation.	• The dataset description is not detailed, • The method of splitting the training and testing dataset is not provided, • Did not mention whether they have done any preprocessing on the dataset or if it was used as it is, • The proposed approach is not compared to any other approaches in the evaluation section.
L. Huang et al., 2019 [ ]	• AverHitRate between 0.6538, 1, • AverACC between 0.8347, 1.	• The literature is meticulously discussed, • The implementation is comprehensively explained in detail.	• The method of splitting the training and testing dataset is not provided, • Did not mention whether they have conducted any preprocessing on the dataset or if it was used as it is.
L. Zhao et al., 2021 [ ]	• Precision, • Recall.	• The implementation is comprehensively explained.	• The exact numbers for the evaluation metrics used in the paper are not provided, • The numbers of features and features used in the dataset are not provided.
M. Ceyhan et al., 2021 [ ]	• Coverage, • F1-measure, • Precision, • Sensitivity, • Specificity, • MAE, • RMSE, • Binary MAE, • Binary RMSE.	• Used many metrics for evaluation.	• The implemented algorithm and similarities explanation were very brief
R. Obeidat et al., 2019 [ ]	• Coverage measure (using SPADES | with clustering) = 0.376, 0.28, 0.594, 0.546, • Coverage measure (using Apriori | with clustering) = 0.46, 0.348, 0.582, 0.534.	• Confirmed by experiment that clustering significantly improves the generation and coverage of two association rules: SPADES and Apriori	• The dataset description is not detailed, • The method of splitting the training and testing dataset is not provided, • The implementation is not discussed in detail.
S. Dwivedi et al., 2017 [ ]	• RMSE = 0.46.	• The proposed system is efficient as it proved to work well with big data, • The implementation of algorithms is comprehensively explained.	• Did not provide any information about the dataset, • The literature review section was very brief.
S.-T. Zhong et al., 2019 [ ]	• MAE (CS major) = 6.6764 ± 0.0029, • RMSE (CS major) = 4.5320 ± 0.0022.	• Used eight datasets for model training and evaluation, • Dataset description is detailed, • Compared the performance of the proposed approach with the performance of other popular approaches.	• The percentage of train-test splitting is not consistent among the eight datasets.
Z. Chen et al., 2017 [ ]	• Confidence, • Support.	• The implementation of algorithms is comprehensively explained with examples.	• Did not provide any information about the used dataset, • Did not include any information about the preprocessing of the dataset, • Did not provide useful metrics for evaluation, • The performance of the proposed approach is not compared to other similar approaches.
Z. Chen et al., 2020 [ ]	• Precision, • Recall, • F1-score.	• Compared the performance of the proposed approach with the performance of other popular approaches: cosine similarity and improved cosine similarity.	• The exact numbers for the evaluation metrics used in the paper are not provided, • The numbers of features and features used in the dataset are not provided.
Z. Ren et al., 2019 [ ]	• PTA, • MAE.	• Compared the performance of the proposed approach with the performance of other approaches, • The implementation of algorithms is comprehensively explained, • The number of students in the dataset is big.	• The dataset description is not detailed.

Authors and Year	Evaluation Metrics and Values	Strengths	Weaknesses
A. J. Fernández-García et al., 2020 [ ]	• Accuracy, • Precision, • Recall, • F1-score.	• Included a section that contains the implementation code, • The literature is meticulously discussed and followed by a table for a summary, • Compared the effect of various preprocessing steps on the final measures of different machine-learning approaches and provided full details about these metrics, • The implementation of each preprocessing step is explained in detail.	• N/A
Y. Adilaksa et al., 2021 [ ]	• The percentage of recommendation diversity = 81.67%, • Accuracy = 64%.	• The preprocessing steps are discussed in detail, • The implementation is comprehensively explained, • Confirmed by the experiment that using the weighted cosine similarity instead of the traditional cosine similarity significantly increased the accuracy of the course recommendations system.	• Did not provide any information about the used dataset, • The method of splitting the training and testing dataset is not provided, • The accuracy measurement is not specified.

Authors and Year	Evaluation Metrics and Values	Strengths	Weaknesses
Esteban, A. et al., 2020 [ ]	• RMSE = 0.971, • Normalized discount cumulative gain (nDCG) = 0.682, • Reach = 100%, • Time = 3.022s.	• The literature is meticulously discussed and followed by a table for a summary, • The implementation of algorithms is comprehensively explained with examples, • Compared the performance of the proposed hybrid approach with other similar approaches, • Used many useful metrics for evaluation.	• Mentioned that some preprocessing had been carried out but did not give any details regarding it, • The number of students in the dataset is relatively low.
M. I. Emon et al., 2021 [ ]	• Accuracy, • Precision, • Recall, • F1-score.	• Compared the performance of the proposed hybrid approach with the used standalone algorithms.	• The exact numbers for the evaluation metrics used in the paper are not provided, • The dataset description is not detailed, • The method of splitting the training and testing dataset is not provided.
S. Alghamdi et al., 2022 [ ]	• MAE = 0.772, • RMSE = 1.215.	• The dataset description is detailed, • The implementation of algorithms is clearly explained.	• Other similar approaches are not stated in the literature, • The number of students in the dataset is relatively low.
S. G. G et al., 2021 [ ]	RMSE = 0.931.	• EDA of the dataset is included in the paper, • Compared the performance of different approaches against the proposed approach, • The implementation is comprehensively discussed and explained.	• The dataset description is not detailed, • The method of splitting the training and testing dataset is not provided, • Similar approaches are not stated in the literature, • Did not mention whether they conducted any preprocessing on the dataset or if it was used as it is.
S. M. Nafea et al., 2019 [ ]	• MAE for cold students = 0.162, • RMSE for cold students = 0.26, • MAE for cold Learning Objects (Los) = 0.162, • RMSE for cold LOs = 0.3.	• Achieved higher accuracy than standalone traditional approaches mentioned in the paper: collaborative filtering and content-based recommendations, • The implementation is comprehensively explained with examples.	• Mentioned that some preprocessing had been carried out but did not give any details regarding it, • The dataset description is not detailed, • The number of students in the dataset is relatively low.
X. Huang et al., 2018 [ ]	• Precision, • Recall, • F1-score.	• The implementation of the proposed approach is comprehensively explained with examples, • Compared the performance of the proposed hybrid approach with other similar approaches through testing.	• The dataset description is not detailed, • Did not mention whether they have done any preprocessing on the dataset or if it was used as it is, • The exact numbers for the evaluation metrics used in the paper are not provided.

Authors and Year	Evaluation Metrics and Values	Strengths	Weaknesses
A. Baskota et al., 2018 [ ]	• Accuracy = 61.6%, • Precision = 61.2%, • Recall = 62.6%, • F1-score = 61.5%.	• Compared the performance of the proposed approach with the performance of other popular approaches, • Used many evaluation metrics and provided the exact numbers for each metric for the evaluation result.	• The dataset description is not detailed.
Jiang, Weijie et al., 2019 [ ]	• The A model: accuracy = 75.23%, F-score = 60.24%, • The B model: accuracy = 88.05%, F-score = 42.01%.	• The implementation of algorithms is comprehensively explained with examples, • Included various sets of hyperparameters and carried out extensive testing.	• Did not mention whether they have done any preprocessing on the dataset or if it was used as it is, • Did not mention the number of features in the dataset, • The performance of the proposed approach is not compared to other similar approaches, • Did not mention the exact percentages for splitting data.
Liang, Yu et al., 2019 [ ]	• Support rate.	• The implementation of algorithms is comprehensively explained.	• The dataset description is not detailed, • A literature review has not been discussed, • The performance of the proposed approach is not compared to other similar approaches, • Did not provide many useful metrics for evaluation and explained that was due to the large number of data sets selected for the experiment.
M. Isma’il et al., 2020 [ ]	• Accuracy = 99.94%.	• Compared the performance of the proposed machine-learning algorithm with the performance of other algorithms through testing.	• Did not mention the training and test set sizes, • The machine learning algorithms used are not explained, • Only used the accuracy measure for evaluation, • The dataset description is not detailed.
M. Revathy et al., 2022 [ ]	• Accuracy = 97.59%, • Precision = 97.52%, • Recall = 98.74%, • Sensitivity = 98.74%, • Specificity = 95.56%.	• Used many evaluation metrics and provided the exact numbers for each metric for the evaluation result, • Provided detailed information about the preprocessing steps, • Compared the performance of the proposed approach with the performance of other approaches, • Provided the exact numbers for each metric for the evaluation result.	N/A
Oreshin et al., 2020 [ ]	• Accuracy = 0.91 ± 0.02, • ROC-AUC = 0.97 ± 0.01, • Recall = 0.83 ± 0.02, • Precision = 0.86 ± 0.03.	• Used many evaluation metrics and provided the exact numbers for each metric for the evaluation result, • Provided detailed information about the preprocessing steps.	• Contains many English grammar and vocabulary errors, • The dataset description is not detailed, • The machine learning algorithms used are not explained, • Did not specify the parameters for the nested time-series split cross-validation.
R. Verma et al., 2018 [ ]	• Accuracy (SVM) = 88.5%, • Precision, • Recall, • F1-score.	• The implementation of algorithms is comprehensively explained, • Compared the performance of several machine-learning algorithms with the performance of other algorithms through testing and concluded that the best two were SVM and ANN.	• The exact numbers for the evaluation metrics used in the paper are not provided except for the achieved accuracy of SVM.
S. D. A. Bujang et al., 2021 [ ]	• Accuracy = 99.5%, • Precision 99.5%, • Recall = 99.5%, • F1-score = 99.5%.	• Included all the exact numbers for the evaluation metrics used in the evaluation, • Compared the performance of six machine learning algorithms and concluded that random forests performed the best based on the evaluation metrics, • EDA of the dataset is included in the paper, • The literature is meticulously discussed and followed by a table for a summary, • Provided detailed information about the used dataset.	• The number of courses is very low (only 2).
S. Srivastava et al., 2018 [ ]	• Accuracy (from 1 cluster to 100) = 99.40%:87.72%.	• Compared the performance of the proposed approach with the performance of other popular approaches, • Provided a confusion matrix for all the used approaches.	• Accuracy is the only metric used for evaluation, • The dataset description is not detailed.
T. Abed et al., 2020 [ ]	• Accuracy = 69.18%.	• Compared the performance of the proposed approach with the performance of other popular approaches: Random Forest, J48, Naive Bayes, Logistic Regression, Sequential Minimal Optimization, and a Multilayer Perceptron.	• The dataset description is not detailed, • Only used the accuracy measure for evaluation, • Did not include an explanation for the implemented algorithms and why they were initially chosen.
V. L. Uskov et al., 2019 [ ]	• Average error = 3.70%.	• Through extensive testing of various ML algorithms, they concluded that linear regression was the best candidate for the problem as the data was linear; • The implementation of algorithms is comprehensively explained.	• The dataset description is not detailed, • Only used the accuracy measure for evaluation, • Did not use RMSE for the evaluation of linear regression.
V. Sankhe et al., 2020 [ ]	• Accuracy = 81.3%	• The implementation of algorithms is comprehensively explained.	• The dataset description is not detailed, • The method of splitting the training and testing dataset is not provided, • Did not mention whether they have conducted any preprocessing on the dataset or if it was used as it is.
V. Z. Kamila et al., 2019 [ ]	• Accuracy of KNN K = 1:100.00% • Accuracy of Naive Bayes algorithm = 100.00%	• Provided the exact numbers for each metric for the evaluation result.	• The implemented algorithms explanation was very brief, • The performance of the proposed approach is not compared to other similar approaches, • Did not provide any information about the dataset used, • Did not mention whether they have conducted any preprocessing on the dataset or if it was used as it is.

Authors and Year	Evaluation Metrics	Strengths	Weaknesses
D. Shah et al., 2017 [ ]	• Normalized mean absolute error (NMAE) = 0.0023, • Computational Time Comparison.	• The implementation of the two compared algorithms is comprehensively explained, • Compared the accuracy of recommendations from both algorithms as well as the speed.	• Did not mention whether they have conducted any preprocessing on the dataset or if it was used as it is, • Similar approaches are not stated in the literature, in addition, the literature was very brief, • Did not use RMSE for evaluation, which is considered the standard as its more accurate.

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Algarni, S.; Sheldon, F. Systematic Review of Recommendation Systems for Course Selection. Mach. Learn. Knowl. Extr. 2023 , 5 , 560-596. https://doi.org/10.3390/make5020033

Algarni S, Sheldon F. Systematic Review of Recommendation Systems for Course Selection. Machine Learning and Knowledge Extraction . 2023; 5(2):560-596. https://doi.org/10.3390/make5020033

Algarni, Shrooq, and Frederick Sheldon. 2023. "Systematic Review of Recommendation Systems for Course Selection" Machine Learning and Knowledge Extraction 5, no. 2: 560-596. https://doi.org/10.3390/make5020033

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

A Curated List of Must-read Papers on Recommender System.

hongleizhang/RSPapers

Folders and files.

Name		Name
276 Commits

Repository files navigation

Must-read papers on recommender system.

This repository provides a curated list of papers and tutorials about Recommender Systems (RS) including systematic tutorials, comprehensive surveys, general recommender system, social recommender system, deep learing-based recommender system, cold start problem in recommender system, efficient recommender system, exploration and exploitation problem in recommender system, explainability in recommender system as well as click through rate prediction for recommender system, knowledge graph for recommeder system, review based recommender system, conversational recommender system, industrial/practical recommender system and privacy preserving recommender system.

[ New! ] Add the new part of LLM for RS .

[ New! ] Add the new part of RS-Tutorials .

[ New! ] Add the new part of Privacy&Security RS .

===========================================================================

00-Tutorials: contain so many tutorials on recommendation systems given by prominent researchers at many top-tier conferences

01-Surveys: a set of comprehensive surveys about recommender system, such as hybrid recommender systems, social recommender systems, poi recommender systems, deep-learning based recommonder systems and so on.

02-General RS: a set of famous recommendation papers which make predictions with some classic models and practical theory.

03-Social RS: several papers which utilize trust/social information in order to alleviate the sparsity of ratings data.

04-Deep Learning-based RS: a set of papers to build a recommender system with deep learning techniques.

05-Cold Start Problem in RS: some papers specifically dealing with the cold start problems inherent in collaborative filtering.

06-POI RS: it focus on helping users explore attractive locations with the information of location-based social networks.

07-Efficient RS: some techniques for efficient recommender system in order to training and making recommendation efficiently.

08-EE Problem in RS: some articles about exploration and exploitation problems in recommendation.

09-Explainability on RS: it focus on addressing the problem of 'why', they not only provide the user with the recommendations, but also make the user aware why such items are recommended by generating recommendation explanations.

10-CTR Prediction for RS: as one part of recommendation, click-through rate prediction focuses on the elaboration of candidate sets for recommendation.

11-Knowledge Graph for RS: knowledge graph, as the side information of behavior interaction matrix in recent years, which can effectively alleviate the problem of data sparsity and cold start, and can provide a reliable explanation for recommendation results.

12-Review based RS: some articles about review or text based recommendations.

13-Conversational RS: some papers made use of natural language processing technology to interactively provide recommendations.

14-Industrial RS: some papers on best practices published in industry.

15-Privacy&Security RS: some papers about privacy preserving and security in recommder systems.

16-LLM for RS: some papers about large language models in recommder systems.

* Please help to contribute this list by adding pull request with the template below.

Deepak et al. Recommender Problems for Web Application. ICML, 2011.

Bart et al. Explaining the user experience of recommender systems. Recsys, 2012.

Ester et al. Recommendation in Social Networks. Recsys, 2013.

Ivan et al. Cross-Domain Recommender Systems. Recsys, 2014.

Steck et al. Interactive Recommender Systems. Recsys, 2015.

Frank et al. Real-Time Recommendation of Streamed Data. Recsys, 2015.

Boratto et al. Group Recommender Systems. Recsys, 2016.

Alex et al. Deep Learning for Recommender Systems. Recsys, 2017.

Bart et al. -Privacy for Recommender Systems. Recsys, 2017.

Xu et al. Deep learning for matching in search and recommendation. SIGIR, 2018.

Massimo et al. Sequence-Aware Recommenders. Recsys, 2018.

Wang et al. Learning and Reasoning on Graph for Recommendation. CIKM, 2019.

Sonie et al. Concept to Code: Deep Learning for Multitask Recommendation. Recsys, 2019.

Michael at al. Fairness & Discrimination in Recommendation & Retrieval. Recsys, 2019.

Yang et al. Deep Transfer Learning for Search and Recommendation. WWW, 2020.

Deldjoo et al. Adversarial Machine Learning in Recommender Systems. WSDM, 2020.

Fan et al. Graph Neural Networks for Recommendations. IJCAI, 2021.

Lei et al. Conversational Recommendation: Formulation, Methods, and Evaluation. Recsys, 2021.

Yu et al. Self-Supervised Learning in Recommender Systems. WWW, 2022.

Zhao et al. Automated Machine Learning for Recommendations: Fundamentals and Advances. WWW, 2022.

Arpita et al. ** Exploring the Impact of Large Language Models on Recommender Systems: An Extensive Review ** ArXiv, 2024.

Burke et al. Hybrid Recommender Systems: Survey and Experiments. USER MODEL USER-ADAP, 2002.

Adomavicius et al. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE TKDE, 2005.

Su et al. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.

Asela et al. A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. J. Mach. Learn. Res, 2009.

Cacheda et al. Comparison of collaborative filtering algorithms: Limitations of current techniques and proposals for scalable, high-performance recommender systems. ACM TWEB, 2011.

Zhang et al. Tag-aware recommender systems: a state-of-the-art survey. J COMPUT SCI TECHNOL, 2011.

Tang et al. Social recommendation: a review. SNAM, 2013.

Yang et al. A survey of collaborative filtering based social recommender systems. COMPUT COMMUN, 2014.

Shi et al. Collaborative filtering beyond the user-item matrix: A survey of the state of the art and future challenges. ACM COMPUT SURV, 2014.

Gunes et al. Shilling attacks against recommender systems: a comprehensive survey. Artificial Intelligence Review, 2014.

Chen et al. Recommender systems based on user reviews: the state of the art. USER MODEL USER-ADAP, 2015.

Xu et al. Social networking meets recommender systems: survey. Int.J.Social Network Mining, 2015.

Yu et al. A survey of point-of-interest recommendation in location-based social networks. In Workshops at AAAI, 2015.

Efthalia et al. Parallel and Distributed Collaborative Filtering: A Survey. Comput. Surv., 2016.

Singhal et al. Use of Deep Learning in Modern Recommendation System: A Summary of Recent Works. arXiv, 2017.

Muhammad et al. Cross Domain Recommender Systems: A Systematic Literature Review. ACM Comput. Surv, 2017.

Massimo et al. Sequence-Aware Recommender Systems. ACM Comput. Surv, 2018.

Zhang et al. Deep learning based recommender system: A survey and new perspectives. ACM Comput.Surv, 2018.

Batmaz et al. A review on deep learning for recommender systems: challenges and remedies. Artificial Intelligence Review, 2018.

Zhang et al. Explainable Recommendation: A Survey and New Perspectives. arXiv, 2018.

Liu et al. Survey of matrix factorization based recommendation methods by integrating social information. Journal of Software, 2018.

Shoujin et al. A Survey on Session-based Recommender Systems. arXiv, 2019.

Shoujin et al. Sequential Recommender Systems: Challenges, Progress and Prospects. IJCAI, 2019.

Zhu et al. Research Commentary on Recommendations with Side Information: A Survey and Research Directions. Electron. Commer. Res. Appl., 2019.

Lina et al. Recommendations on the Internet of Things: Requirements, Challenges, and Directions. IEEE Internet Comput., 2019.

Sriharsha et al. A Survey on Group Recommender Systems. J. Intell. Inf. Syst., 2019.

Dietmar et al. A Survey on Conversational Recommender Systems. arXiv, 2020.

Qingyu et al. A Survey on Knowledge Graph-Based Recommender Systems. arXiv, 2020.

Yang et al. Deep Learning on Knowledge Graph for Recommender System: A Survey. arXiv, 2020.

Wang et al. Graph Learning Approaches to Recommender Systems: A Review. arXiv, 2020.

Yashar et al. Adversarial Machine Learning in Recommender Systems-State of the art and Challenges. arXiv, 2020.

May et al. Recommender Systems for the Internet of Things: A Survey. arXiv, 2020.

Wu et al. Graph Neural Networks in Recommender Systems: A Survey. arXiv, 2020.

Chen et al. Bias and Debias in Recommender System: A Survey and Future Directions. arXiv, 2020.

Zhu et al. Cross-Domain Recommendation-Challenges, Progress, and Prospects. arxiv, 2021.

Zhang et al. Deep Learning for Click-Through Rate Estimation. IJCAI, 2021.

Wu et al. A Survey on Neural Recommendation: From Collaborative Filtering to Content and Context Enriched Recommendation. TKDE, 2021.

Chen et al. A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions. arxiv, 2021.

Lin et al. A Survey on Reinforcement Learning for Recommender Systems. arXiv, 2022.

Zheng et al. AutoML for Deep Recommender Systems: A Survey. arXiv, 2022.

Chen et al. Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation. , arXiv, 2022.

Yu et al. Self-Supervised Learning for Recommender Systems: A Survey. arXiv, 2022.

Wang et al. A Survey on the Fairness of Recommender Systems. TOIS, 2022.

Wang et al. Deep Meta-learning in Recommendation Systems: A Survey. arXiv, 2022.

Zhang et al. A Survey on Privacy-preserving Federated Recommender Systems (In Chinese). AAS, 2022. [pdf]

Wu et al. A Survey on Large Language Models for Recommendation. arXiv, 2023.

Lin et al. How Can Recommender Systems Benefit from Large Language Models: A Survey. arXiv, 2023.

General Recommender System

Goldberg et al. Using collaborative filtering to weave an information tapestry. COMMUN ACM, 1992.

Resnick et al. GroupLens: an open architecture for collaborative filtering of netnews. CSCW, 1994.

Sarwar et al. Application of dimensionality reduction in recommender system-a case study. 2000.

Sarwar et al. Item-based collaborative filtering recommendation algorithms. WWW, 2001.

Linden et al. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE INTERNET COMPUT, 2003.

Lemire et al. Slope one predictors for online rating-based collaborative filtering. SDM, 2005.

Zhou et al. Bipartite network projection and personal recommendation. Physical Review E, 2007.

Mnih et al. Probabilistic matrix factorization. NIPS, 2008.

Koren et al. Factorization meets the neighborhood: a multifaceted collaborative filtering model. SIGKDD, 2008.

Pan et al. One-class collaborative filtering. ICDM, 2008.

Hu et al. Collaborative filtering for implicit feedback datasets. ICDM, 2008.

Weimer et al. Improving maximum margin matrix factorization. Machine Learning, 2008.

Koren et al. Matrix factorization techniques for recommender systems. Computer, 2009.

Agarwal et al. Regression-based latent factor models. SIGKDD, 2009.

Koren et al. The bellkor solution to the netflix grand prize. Netflix prize documentation, 2009.

Rendle et al. BPR: Bayesian personalized ranking from implicit feedback. UAI, 2009.

Koren et al. Collaborative filtering with temporal dynamics. COMMUN ACM, 2010.

Khoshneshin et al. Collaborative filtering via euclidean embedding. RecSys, 2010.

Liu et al. Online evolutionary collaborative filtering Recsys. RecSys, 2010.

Koren et al. Factor in the neighbors: Scalable and accurate collaborative filtering. TKDD, 2010.

Ning et al. SLIM: Sparse Linear Methods for Top-N Recommender Systems. ICDM, 2011.

Chen et al. Feature-based matrix factorization. arXiv, 2011.

Rendle et al. Factorization Machines with libFM. TIST, 2012.

Rendle. Learning recommender systems with adaptive regularization. WSDM, 2012.

Zhong et al. Contextual collaborative filtering via hierarchical matrix factorization. SDM, 2012.

Lee et al. Local low-rank matrix approximation. ICML, 2013.

Kabbur et al. Fism: factored item similarity models for top-n recommender systems. KDD, 2013.

Johnson et al. Logistic Matrix Factorization for Implicit Feedback Data. NIPS Workshop, 2014.

Hu et al. Your neighbors affect your ratings: on geographical neighborhood influence to rating prediction. SIGIR, 2014.

Hernández-Lobato et al. Probabilistic matrix factorization with non-random missing data. ICML, 2014.

Yang et al. TopicMF: Simultaneously Exploiting Ratings and Reviews for Recommendation. AAAI, 2014.

Shi et al. Semantic path based personalized recommendation on weighted heterogeneous information networks. CIKM, 2015.

Grbovic et al. E-commerce in your inbox: Product recommendations at scale. KDD, 2015.

Barkan et al. Item2vec: neural item embedding for collaborative filtering. Machine Learning for Signal Processing, 2016.

Liang et al. Factorization Meets the Item Embedding: Regularizing Matrix Factorization with Item Co-occurrence. RecSys, 2016

Liang et al. Modeling user exposure in recommendation. WWW, 2016.

He et al. Fast matrix factorization for online recommendation with implicit feedback. SIGIR, 2016.

Hsieh et al. Collaborative metric learning. WWW, 2017.

He et al. Translation-based Recommendation. RecSys, 2017.

Bayeret al. A generic coordinate descent framework for learning from implicit feedback. WWW, 2017.

Ruining et al. Translation-based Recommendation. RecSys, 2017.

Wu et al. Large-scale Collaborative Ranking in Near-Linear Time. KDD, 2017.

Rajiv et al. Translation-based factorization machines for sequential recommendation. RecSys 2018.

Gao et al. BiNE: Bipartite Network Embedding. SIGIR, 2018.

Wu et al. Sql-rank: A listwise approach to collaborative ranking. ICML, 2018.

Xiangnan et al. Adversarial Personalized Ranking for Recommendation. SIGIR, 2018.

Zhang et al. Metric Factorization: Recommendation beyond Matrix Factorization. 2018.

Lei et al. Spectral Collaborative Filtering. RecSys, 2018.

Feng et al. Adversarial Collaborative Neural Network for Robust Recommendation. SIGIR, 2019.

Chen et al. Collaborative Similarity Embedding for Recommender Systems. arXiv, 2019.

Chuan et al. Heterogeneous Information Network Embedding for Recommendation. TKDE, 2019.

Huafeng et al. Deep Generative Ranking for Personalized Recommendation. Recsys, 2019.

Xiang et al. Neural Graph Collaborative Filtering. SIGIR, 2019.

Wenjie et al. Denoising Implicit Feedback for Recommendation. arXiv, 2020.

Rendle et al. Neural Collaborative Filtering vs. Matrix Factorization Revisited. arXiv, 2020.

Social Recommender System

Ma, Hao, et al. Sorec: social recommendation using probabilistic matrix factorization. CIKM, 2008.

Jamali et al. Trustwalker: a random walk model for combining trust-based and item-based recommendation. SIGKDD, 2009.

Ma et al. Learning to recommend with trust and distrust relationships. RecSys, 2009.

Ma et al. Learning to recommend with social trust ensemble. SIGIR, 2009.

Jamali et al. A matrix factorization technique with trust propagation for recommendation in social networks. RecSys, 2010.

Ma, Hao, et al. Recommender systems with social regularization. WSDM, 2011.

Ma, Hao et al. Learning to recommend with explicit and implicit social relations. ACM T INTEL SYST TEC, 2011.

Ma, Hao. An experimental study on implicit social recommendation. SIGIR, 2013.

Yang et al. Social collaborative filtering by trust. IJCAI, 2013.

Jiliang et al. Exploiting Local and Global Social Context for Recommendation. IJCAI, 2013.

Zhao et al. Leveraging social connections to improve personalized ranking for collaborative filtering. CIKM, 2014.

Chen et al. Context-aware collaborative topic regression with social matrix factorization for recommender systems. AAAI, 2014.

Guo et al. TrustSVD: Collaborative Filtering with Both the Explicit and Implicit Influence of User Trust and of Item Ratings. AAAI, 2015.

Wang et al. Social recommendation with strong and weak ties. CIKM, 2016.

Jiliang et al. Recommendation with Social Dimensions. AAAI, 2016.

Li et al. Social recommendation using Euclidean embedding. IJCNN, 2017.

Zhang et al. Collaborative User Network Embedding for Social Recommender Systems. SDM, 2017.

Yang et al. Social collaborative filtering by trust. IEEE T PATTERN ANAL, 2017.

Park et al. UniWalk: Explainable and Accurate Recommendation for Rating and Network Data. arXiv, 2017.

Rafailidis et al. Learning to Rank with Trust and Distrust in Recommender Systems. RecSys, 2017.

Xixi et al. Additive Co-Clustering with Social Influence for Recommendation. RecSys, 2017.

Zhao et al. Collaborative Filtering with Social Local Models. ICDM, 2017.

Wang et al. Collaborative Filtering with Social Exposure: A Modular Approach to Social Recommendation. AAAI, 2018.

Wenqi et al. Deep Modeling of Social Relations for Recommendation. AAAI, 2018

Xuying et al. Personalized Privacy-Preserving Social Recommendation. AAAI,2018.

Liu et al. Social Recommendation with an Essential Preference Space. AAAI,2018.

Wen et al. Network embedding based recommendation method in social networks. WWW Poster, 2018.

Lin et al. Recommender Systems with Characterized Social Regularization. CIKM Short Paper, 2018.

Yu et al. Adaptive implicit friends identification over heterogeneous network for social recommendation. CIKM, 2018.

Honglei et al. Social Collaborative Filtering Ensemble. PRICAI, 2018.

Chen et al. Integrating Dual User Network Embedding With Matrix Factorization For Social Recommender Systems. IJCNN, 2019.

Wenqi et al. Graph Neural Networks for Social Recommendation. WWW, 2019.

Song et al. Session-based Social Recommendation via Dynamic Graph Attention Networks. WSDM, 2019.

Wenqi et al. Deep Social Collaborative Filtering. RecSys, 2019.

Wenqi et al. Deep Adversarial Social Recommendation. IJCAI, 2019.

Qitian et al. Feature Evolution Based Multi-Task Learning for Collaborative Filtering with Social Trust. IJCAI, 2019.

Wu et al. SocialGCN: An Efficient Graph Convolutional Network based Model for Social Recommendation. AAAI, 2019.

Wu et al. Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender System. WWW, 2019.

Chong Chen et al. An Efficient Adaptive Transfer Neural Network for Social-aware Recommendation. SIGIR, 2019.

Wu et al. A Neural Influence Diffusion Model for Social Recommendation. SIGIR, 2019.

Cheng et al. An Efficient Adaptive Transfer Neural Network for Social-aware Recommendation. SIGIR, 2019.

Chong et al. Social Attentional Memory Network: Modeling Aspect- and Friend-Level Differences in Recommendation. WSDM, 2019.

Yang et al. Modelling High-Order Social Relations for Item Recommendation. arXiv, 2020.

Junliang et al. Enhance Social Recommendation with Adversarial Graph Convolutional Networks. TKDE, 2020.

Chaochao et al. Secure Social Recommendation based on Secret Sharing. arXiv, 2020.

Yuanfu et al. Social Influence Attentive Neural Network for Friend-Enhanced Recommendation. ECML-PKDD, 2020.

Liqiang Song et al. DREAM: A Dynamic Relational-Aware Model for Social Recommendation. CIKM, 2020.

Bo Jin et al. Partial Relationship Aware Influence Diffusion via a Multi-channel Encoding Scheme for Social Recommendation. CIKM, 2020.

Liangwei Yang et al. ConsisRec: Enhancing GNN for Social Recommendation via Consistent Neighbor Aggregation. SIGIR, 2021.

Bairan Fu et al. Dual Side Deep Context-aware Modulation for Social Recommendation. WWW, 2021.

Du et al. Socially-aware Dual Contrastive Learning for Cold-Start Recommendation. SIGIR, 2022.

Yuan et al. Semantic and Structural View Fusion Modeling for Social Recommendation. TKDE, 2022.

Deep Learning based Recommender System

Salakhutdinov et al. Restricted Boltzmann machines for collaborative filtering. ICML, 2007.

Aäron et al. Deep content-based music recommendation. NIPS, 2013.

Huang et al. Learning deep structured semantic models for web search using clickthrough data. CIKM, 2013.

Wang et al. Collaborative deep learning for recommender systems. KDD, 2015.

Sedhain et al. Autorec: Autoencoders meet collaborative filtering. WWW, 2015.

Ali et al. A Multi-View Deep Learning Approach for Cross Domain User Modeling in Recommendation Systems. WWW, 2015

Li et al. Deep collaborative filtering via marginalized denoising auto-encoder. CIKM, 2015.

Ruining et al. VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback. AAAI, 2016.

Hidasi et al. Session-based recommendations with recurrent neural networks. ICLR, 2016.

Covington et al. Deep neural networks for youtube recommendations. RecSys, 2016.

Cheng et al. Wide & deep learning for recommender systems. Workshop on RecSys, 2016.

Zheng et al. A neural autoregressive approach to collaborative filtering. ICML, 2016.

Wu et al. Collaborative denoising auto-encoders for top-n recommender systems. WSDM, 2016.

Tan et al. Improved recurrent neural networks for session-based recommendations. Workshop on Deep Learning for Recommender Systems, 2016.

Kang et al. Visually-Aware Fashion Recommendation and Design with Generative Image Models. ICDM, 2017.

Wu et al. Recurrent Recommender Networks. WSDM, 2017.

Lian et al. CCCFNet: a content-boosted collaborative filtering neural network for cross domain recommender systems. WWW, 2017.

He et al. Neural collaborative filtering. WWW, 2017.

Zhao et al. Leveraging Long and Short-term Information in Content-aware Movie Recommendation. arXiv, 2017.

Li et al. Deep Collaborative Autoencoder for Recommender Systems: A Unified Framework for Explicit and Implicit Feedback. arXiv, 2017.

Xue et al. Deep Matrix Factorization Models for Recommender Systems. IJCAI, 2017. code

He et al. Outer Product-based Neural Collaborative Filtering. IJCAI, 2018.

Dong et al. CFGAN: A Generic Collaborative Filtering Framework based on Generative Adversarial Networks. CIKM, 2018.

Zhao et al. Learning and Transferring IDs Representation in E-commerce. KDD, 2018.

Liang et al. Variational Autoencoders for Collaborative Filtering. WWW, 2018.

Ebesu et al. Collaborative Memory Network for Recommendation Systems. SIGIR, 2018.

Lian et al. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. KDD, 2018.

Zhang et al. Next Item Recommendation with Self-Attention. 2018.

Li et al. Learning from History and Present: Next-item Recommendation via Discriminatively Exploiting User Behaviors. KDD, 2018.

Grbovic et al. Real-time Personalization using Embeddings for Search Ranking at Airbnb. KDD, 2018.

Ying et al. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. KDD, 2018.

Hu et al. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. KDD, 2018.

Christakopoulou et al. Local Latent Space Models for Top-N Recommendation. KDD, 2018.

Bhagat et al. Buy It Again: Modeling Repeat Purchase Recommendations. KDD, 2018.

Wang et al. Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba. KDD, 2018.

Tran et al. Regularizing Matrix Factorization with User and Item Embeddings for Recommendation. CIKM, 2018.

Zhou et al. Micro behaviors: A new perspective in e-commerce recommender systems. WSDM, 2018.

Chen et al. Sequential recommendation with user memory networks. WSDM, 2018.

Beutel et al. Latent Cross: Making Use of Context in Recurrent Recommender Systems. WSDM, 2018.

Tang et al. Personalized top-n sequential recommendation via convolutional sequence embedding. WSDM, 2018.

Chae et al. CFGAN: A Generic Collaborative Filtering Framework based on Generative Adversarial Networks. CIKM, 2018.

Wu et al. Session-based Recommendation with Graph Neural Networks. AAAI, 2019.

Zhi-Hong et al. DeepCF: A Unified Framework of Representation Learning and Matching Function Learning in Recommender System. AAAI, 2019.

Zeping et al. Adaptive User Modeling with Long and Short-Term Preferences for Personalized Recommendation. IJCAI, 2019.

Dong Xi et al. BPAM: Recommendation Based on BP Neural Network with Attention Mechanism. IJCAI, 2019.

Xin et al. CFM: Convolutional Factorization Machines for Context-Aware Recommendation. IJCAI, 2019.

Xiao Zhou et al. Collaborative Metric Learning with Memory Network for Multi-Relational Recommender Systems. IJCAI, 2019.

Junyang et al. Convolutional Gaussian Embeddings for Personalized Recommendation with Uncertainty. IJCAI, 2019.

Feng Yuan et al. DARec: Deep Domain Adaptation for Cross-Domain Recommendation via Transferring Rating Patterns. IJCAI, 2019.

Yanan et al. Learning Shared Vertex Representation in Heterogeneous Graphs with Convolutional Networks for Recommendation. IJCAI, 2019.

Jiani et al. STAR-GCN: Stacked and Reconstructed Graph Convolutional Networks for Recommender Systems. IJCAI, 2019.

An et al. CosRec: 2D Convolutional Neural Networks for Sequential Recommendation. CIKM, 2019.

Sun et al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. CIKM, 2019.

Hongwei et al. Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation. WWW, 2019.

Maurizio et al. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys, 2019.

Maurizio et al. A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research. arXiv, 2019.

Xin et al. CFM: Convolutional factorization machines for context-aware recommendation. IJCAI, 2019.

Wu et al. SSE-PT: Sequential Recommendation Via Personalized Transformer. RecSys, 2020.

Huafeng et al. Deep Global and Local Generative Model for Recommendation. WWW, 2020.

Choi et al. LT-OCF: Learnable-Time ODE-based Collaborative Filtering. CIKM, 2021.

Yun et al. Multi Datasource LTV User Representation (MDLUR). KDD, 2023.

Liu et al. CT4Rec: Simple yet Effective Consistency Training for Sequential Recommendation. KDD, 2023.

Xi et al. On-Device Integrated Re-Ranking with Heterogeneous Behavior Modeling. KDD, 2023.

Cold Start Problem in Recommender System

Schein et al. Methods and metrics for cold-start recommendations. SIGIR, 2002.

Seung-Taek et al. Pairwise Preference Regression for Cold-start Recommendation. RecSys, 2009.

Gantner et al. Learning attribute-to-feature mappings for cold-start recommendations. ICDM, 2010.

Sedhain et al. Social collaborative filtering for cold-start recommendations. RecSys, 2014.

Zhang et al. Addressing cold start in recommender systems: A semi-supervised co-training algorithm. SIGIR, 2014.

Kula. Metadata embeddings for user and item cold-start recommendations. arXiv, 2015.

Sedhain et al. Low-Rank Linear Cold-Start Recommendation from Social Data. AAAI. 2017.

Man et al. Cross-domain recommendation: an embedding and mapping approach. IJCAI, 2017.

Cohen et al. Expediting Exploration by Attribute-to-Feature Mapping for Cold-Start Recommendations. RecSys, 2017.

Volkovs et al. DropoutNet: Addressing Cold Start in Recommender Systems. NIPS, 2017

Dureddy et al. Handling Cold-Start Collaborative Filtering with Reinforcement Learning. arXiv, 2018.

Fu et al. Deeply Fusing Reviews and Contents for Cold Start Users in Cross-Domain Recommendation Systems. AAAI, 2019.

Li. From Zero-Shot Learning to Cold-Start Recommendation. AAAI, 2019

Hoyeop. Estimating Personalized Preferences Through Meta-Learning for User Cold-Start Recommendation. KDD, 2019.

Ruobing et al. Internal and Contextual Attention Network for Cold-start Multi-channel Matching in Recommendation. IJCAI, 2020.

Sun et al. LARA: Attribute-to-feature Adversarial Learning for New-item Recommendation. WSDM, 2020.

Lu et al. Meta-learning on heterogeneous information networks for cold-start recommendation. KDD, 2020.

POI Recommender System

Mao et al. Exploiting geographical influence for collaborative point-of-interest recommendation. SIGIR, 2011.

Chen et al. Fused matrix factorization with geographical and social influence in location-based social networks. AAAI, 2012.

Jia et al. iGSLR: personalized geo-social location recommen dation: a kernel density estimation approach. SIGSPA, 2013.

Jia et al. Lore: exploiting sequential influence for location recommendations. SIGSPATIAL, 2014

Jia et al. Geosoca: Exploiting geographical, social and cat egorical correlations for point-of-interest recommendations. SIGIR, 2015.

Huayu et al. Point-of-Interest Recommendations:Learning Potential Check-ins from Friends. KDD, 2016.

Jing et al. Category-aware next point-of-interest recommendation via listwise Bayesian personalized ranking. IJCAI, 2017.

Jarana et al. A Personalised Ranking Framework with Multiple Sampling Criteria for Venue Recommendation. CIKM, 2017.

Huayu et al. Learning user's intrinsic and extrinsic interests for point-of-interest recommendation: a unified approach. IJCAI, 2017.

Feng et al. POI2Vec: Geographical Latent Representation for Predicting Future Visitors. AAAI, 2017.

Wei Liu et al. Geo-ALM: POI Recommendation by Fusing Geographical Information and Adversarial Learning Mechanism. IJCAI, 2019.

et al. Contextualized Point-of-Interest Recommendation. IJCAI, 2020.

Efficient RS

Karatzoglou et al. Collaborative filtering on a budget. AISTAT, 2010.

Zhou et al. Learning binary codes for collaborative filtering. SIGKDD, 2012.

Zhang et al. Preference preserving hashing for efficient recommendation. SIGIR, 2014.

Zhang et al. Discrete collaborative filtering. SIGIR, 2016.

Lian et al. Discrete Content-aware Matrix Factorization. SIGKDD, 2017.

Zhang et al. Discrete Personalized Ranking for Fast Collaborative Filtering from Implicit Feedback. AAAI, 2017.

Zhang et al. Discrete Deep Learning for Fast Content-Aware Recommendation. WSDM, 2018.

Han et al. Discrete Factorization Machines for Fast Feature-based Recommendatio. IJCAI, 2018.

Lian et al. Discrete Matrix Factorization and Extension for Fast Item Recommendation. TKDE, 2019.

Chenghao et al. Discrete Social Recommendation. AAAI, 2019.

Guibing et al. Discrete Trust-aware Matrix Factorization for Fast Recommendation. IJCAI, 2019.

Defu et al. LightRec: a Memory and Search-Efficient Recommender System. WWW, 2020.

Yang et al. A Generic Network Compression Framework for Sequential Recommender Systems. SIGIR, 2020.

Xiangnan et al. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. SIGIR, 2020.

Hansen et al. Content-aware Neural Hashing for Cold-start Recommendation. SIGIR, 2020.

Zhang et al. Deep Pairwise Hashing for Cold-start Recommendation. TKDE, 2020.

Liu et al. Discrete Listwise Collaborative Filtering for Fast Recommendation SDM, 2021

Auer et al. Using confidence bounds for exploitation-exploration trade-offs. JMLR, 2002.

Li et al. A contextual-bandit approach to personalized news article recommendation. WWW, 2010.

Li et al. Exploitation and exploration in a performance based contextual advertising system. SIGKDD, 2010.

Chapelle et al. An empirical evaluation of thompson sampling. NIPS, 2011.

Féraud et al. Random forest for the contextual bandit problem. Artificial Intelligence and Statistics. 2016.

Li et al. Collaborative filtering bandits. SIGIR, 2016.

Wang et al. Factorization Bandits for Interactive Recommendation. AAAI, 2017.

Explainability on RS

Huang et al. Improving Sequential Recommendation with Knowledge-Enhanced Memory Networks. SIGIR, 2018.

Wang et al. Tem: Tree-enhanced embedding model for explainable recommendation. WWW, 2018.

Lu et al. Why I like it: multi-task learning for recommendation and explanation. RecSys, 2018.

Wang et al. Explainable Reasoning over Knowledge Graphs for Recommendation. AAAI, 2019.

Cao et al. Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences. WWW, 2019.

Zhongxia et al. Co-Attentive Multi-Task Learning for Explainable Recommendation. IJCAI, 2019.

Min et al. Explainable Fashion Recommendation: A Semantic Attribute Region Guided Approach. IJCAI, 2019.

Peijie et al. Dual Learning for Explainable Recommendation: Towards Unifying User Preference Prediction and Review Generation. WWW, 2020.

CTR Prediction for RS

Richardson et al. Predicting Clicks - Estimating the Click-Through Rate for New Ads. WWW, 2007.

Steffen et al. Fast Context-aware Recommendations with Factorization Machines. SIGIR, 2011.

H. Brendan McMahan et al. Ad Click Prediction a View from the Trenches. KDD, 2013.

Aaron et al. Deep content-based music recommendation. NIPS, 2013.

Xinran He et al. Practical Lessons from Predicting Clicks on Ads at Facebook. ADKDD, 2014.

Ying Shan et al. Deep Crossing - Web-Scale Modeling without Manually Crafted Combinatorial Features. KDD, 2016.

Weinan Zhang et al. Deep Learning over Multi-field Categorical Data. arXiv, 2016.

• Yuchin et al. Field-aware Factorization Machines for CTR Prediction. RecSys, 2016.

Cheng et al. Wide & Deep Learning for Recommender Systems. arXiv, 2016.

Qu et al. Product-based neural networks for user response prediction. ICDM, 2016.

Chao et al. Recurrent recommender networks. WSDM, 2017.

Guo et al. A Factorization-Machine based Neural Network for CTR Prediction. arXiv, 2017.

Xiao et al. Attentional Factorization Machines - Learning the Weight of Feature Interactions via Attention Networks. arXiv, 2017.

Guo et al. Deepfm: A factorization-machine based neural network for ctr prediction. IJCAI, 2017

Gai et al. Learning Piece-wise Linear Models from Large Scale Data for Ad Click Prediction. arXiv, 2017.

Xiangnan He et al. Neural Factorization Machines for Sparse Predictive Analytics. arXiv, 2017.

Ruoxi et al. Deep & Cross Network for Ad Click Predictions. ADKDD, 2017

Zhou et al. Deep Interest Network for Click-Through Rate Prediction. KDD 2018.

Lian et al. xDeepFM - Combining Explicit and Implicit Feature Interactions for Recommender Systems. arXiv, 2018.

Zhou et al Deep Session Interest Network for Click-Through Rate Prediction. IJCAI, 2019.

Zhou et al. Deep Interest Evolution Network for Click-Through Rate Prediction. AAAI, 2019

Yang et al. Operation-aware Neural Networks for User Response Prediction. 2019.

Liu et al. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. 2019.

Wentao et al. Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction. KDD, 2019.

Qi et al. Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction. KDD, 2019.

Fuzheng et al. Collaborative Knowledge Base Embedding for Recommender Systems. KDD, 2016.

Shu et al. TFNet: Multi-Semantic Feature Interaction for CTR Prediction. SIGIR, 2020.

Weinan et al. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction. SIGIR, 2020

Wang et al. BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-Textual Features for CTR Prediction. KDD, 2023

Min et al. Scenario-Adaptive Feature Interaction for Click-Through Rate Prediction. KDD, 2023

Knowledge Graph for RS

Hongwei et al. DKN: Deep Knowledge-Aware Network for News Recommendation. WWW, 2018.

Hongwei et al. Ripplenet-Propagating user preferences on the knowledge graph for recommender systems. CIKM, 2018.

• Hongwei et al. Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems. KDD, 2019.

Xiang et al. Reinforced Negative Sampling over Knowledge Graph for Recommendation. WWW, 2020.

Wang et al. CKAN: Collaborative Knowledge-aware Attentive Network for Recommender Systems. SIGIR, 2020.

Wu et al. Graph dna: Deep neighborhood aware graph encoding for collaborative filtering. AISTAT, 2020.

Review based RS

Chong et al. Collaborative Topic Modeling for Recommending Scientific Articles. KDD, 2011.

McAuley et al. Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text. RecSys, 2013.

Guang et al. Ratings Meet Reviews, a Combined Approach to Recommend. RecSys, 2014

Wei et al. Collaborative Multi-Level Embedding Learning from Reviews for Rating Prediction. IJCAI, 2016.

Kim et al. Convolutional Matrix Factorization for Document Context-Aware Recommendation. RecSys, 2016.

Yunzhi et al. Rating-boosted latent topics Understanding users and items with ratings and reviews. IJCAI, 2016.

Seo et al. Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction. RecSys, 2016.

Lei et al. Joint Deep Modeling of Users and Items Using Reviews for Recommendation. WSDM, 2017.

Zhiyong et al. A3NCF An Adaptive Aspect Attention Model for Rating Prediction. IJCAI, 2018.

Jinyao et al. ANR: Aspect-based Neural Recommender CIKM, 2018.

Yichao et al. Coevolutionary recommendation model Mutual learning between ratings and reviews. WWW, 2018.

Yi et al. Multi-pointer co-attention networks for recommendation. KDD, 2018.

Chong et al. Neural attentional rating regression with review-level explanations. WWW, 2018.

Libing et al. Parl: Let strangers speak out what you like. CIKM, 2018.

Libing et al. A context-aware user-item representation learning for item recommendation. TOIS, 2019.

Donghua et al. DAML: Dual Attention Mutual Learning between Ratings and Reviews for Item Recommendation. KDD, 2019.

Liu et al. NRPA Neural Recommendation with Personalized Attention. SIGIR, 2019.

Noveen et al. How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements. SIGIR, 2020.

Conversational RS

Zhao et al. Interactive collaborative filtering. CIKM, 2013.

Negar et al. Context adaptation in interactive recommender systems. RecSys, 2014.

Yasser et al. History-guided conversational recommendation. WWW, 2014.

Konstantina et al. Towards Conversational Recommender Systems. KDD, 2016.

Konstantina et al. Q&R: A Two-Stage Approach toward Interactive Recommendation. KDD, 2018.

Sun et al. Conversational Recommender System. SIGIR, 2018.

Yongfeng et al. Towards Conversational Search and Recommendation: System Ask, User Respond. CIKM, 2018.

Raymond et al. Towards Deep Conversational Recommendations. NeurIPS, 2018.

Tong et al. A Visual Dialog Augmented Interactive Recommender System. KDD, 2019.

Qibin et al. Towards Knowledge-Based Recommender Dialog System. EMNLP, 2019.

Yuanjiang et al. Adversarial Attacks and Detection on Reinforcement Learning-Based Interactive Recommender Systems. SIGIR, 2020.

Wenqiang et al. Conversational Recommendation: Formulation, Methods, and Evaluation. SIGIR, 2020.

Xingshan et al. Dynamic Online Conversation Recommendation. ACL, 2020.

Wenqiang et al. Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. WSDM, 2020.

Kun et al. Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion. KDD, 2020.

Wenqiang et al. Interactive Path Reasoning on Graph for Conversational Recommendation. KDD, 2020.

Sijin et al. Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning. SIGIR, 2020.

Kai et al. Latent Linear Critiquing for Conversational Recommender Systems. WWW, 2020.

Lixin et al. Neural Interactive Collaborative Filtering. SIGIR, 2020.

Lixin et al. Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation. WSDM, 2020.

Shijun et al. Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users. arXiv, 2020.

Zeming et al. Towards Conversational Recommendation over Multi-Type Dialogs. ACL, 2020.

Zhongxia et al. Towards Explainable Conversational Recommendation. IJCAI, 2020.

Jie et al. Towards Question-based Recommender Systems. SIGIR, 2020.

Hu et al. User Memory Reasoning for Conversational Recommendation. arXiv, 2020.

Industrial RS

• Chaudhari et al. Distributional Off-Policy Evaluation for Slate Recommendations AAAI, 2018.
• Mihajlo et al. Real-time Personalization using Embeddings for Search Ranking at Airbnb. KDD.2018.

Kun et al. Learning Piece-wise Linear Models from Large Scale Data for Ad Click Prediction. arXiv, 2017.

Zhibo et al. Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba. KDD, 2018.

Guorui et al. Deep Interest Evolution Network for Click-Through Rate Prediction. AAAI, 2019.

Guorui et al. Deep Interest Network for Click-Through Rate Prediction. KDD, 2018.

Xiao et al. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. SIGIR, 2018.

Han et al. Learning Tree-based Deep Model for Recommender Systems. KDD, 2018.

Lin et al. Visualizing and Understanding Deep Neural Networks in CTR Prediction. SIGIR, 2018.

Qiwei et al. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba. KDD,2019.

Wentao et al. Click-Through Rate Prediction with the User Memory Network. KDD, 2019

Yufei et al. Deep Session Interest Network for Click-Through Rate Prediction. arXiv, 2019.

Han et al. Joint Optimization of Tree-based Index and Deep Model for Recommender Systems. NIPS, 2019.

Chao et al. Multi-Interest Network with Dynamic Routing for Recommendation at Tmall. CIKM, 2019.

Wentao et al. Representation Learning-Assisted Click-Through Rate Prediction. arXiv, 2019.

Fuyu et al. SDM: Sequential Deep Matching Model for Online Large-scale Recommender System. CIKM, 2019.

Wentao et al. MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction. CIKM, 2020.

Zhe et al. COLD: Towards the Next Generation of Pre-Ranking System. KDD, 2020.

Weinan et al. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction. SIGIR, 2020.

Ze et al. Deep Match to Rank Model for Personalized Click-Through Rate Prediction. AAAI, 2020.

Shu et al. Deep Time-Stream Framework for Click-Through Rate Prediction by Tracking Interest Evolution. AAAI, 2020.

Changhua et al. Personalized Re-ranking for Recommendation. RecSys, 2019.

Liyi et al. A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction. CIKM, 2020.

Yu et al. EdgeRec: Recommender System on Edge in Mobile Taobao. CIKM, 2020.

Yufei et al. MTBRN: Multiplex Target-Behavior Relation Enhanced Network for Click-Through Rate Prediction. CIKM, 2020.

Xu et al. Multi-Channel Integrated Recommendation with Exposure Constraints. KDD, 2023.

• Xiangyu et al. Whole-Chain Recommendations. CIKM, 2020.

[ByteDance]

• Chao et al. Graph-Based Model-Agnostic Data Subsampling for Recommendation Systems. KDD, 2023.

Xinran et al. Practical Lessons from Predicting Clicks on Ads at Facebook. KDD, 2014.

Maxim et al. Deep Learning Recommendation Model for Personalization and Recommendation Systems. arXiv, 2019.

Li et al. AdaTT: Adaptive Task-to-Task Fusion Network for Multitask Learning in Recommendations. KDD, 2023.

James et al. The YouTube Video Recommendation System. RecSys, 2010.

Jason et al. Label Partitioning For Sublinear Ranking. JMLR, 2013.

Paul et al. Deep Neural Networks for YouTube Recommendations.** RecSys, 2016.

Heng et al. Wide & Deep Learning for Recommender Systems. DLRS, 2016.

Ruoxi et al. Deep & Cross Network for Ad Click Predictions. KDD, 2017.

Alex et al. Latent Cross: Making Use of Context in Recurrent Recommender Systems. WSDM, 2018.

Alex et al. Fairness in Recommendation Ranking through Pairwise Comparisons. KDD, 2019.

Xinyang et al. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. RecSys, 2019.

Wang et al. Fresh Content Needs More Attention: Multi-Funnel Fresh Content Recommendation. KDD, 2023.

Zhang et al. Empowering Long-Tail Item Recommendation through Cross Decoupling Network. KDD, 2023.

Tang et al. Improving Training Stability for Multitask Ranking Models in Recommender Systems. KDD, 2023.

Huifeng et al. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. IJCAI, 2017.

Bin et al. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. WWW, 2019.

Huifeng et al. PAL: A Position-bias Aware Learning Framework for CTR Prediction in Live Recommender Systems. RecSys, 2019.

Kai et al. Automatic Feature Engineering From Very High Dimensional Event Logs Using Deep Neural Networks. KDD, 2019.

Yishi et al. GraphSAIL Graph Structure Aware Incremental Learning for Recommender Systems. CIKM, 2020.

Huifeng et al. DeepFM: An End-to-End Wide & Deep Learning Framework for CTR Prediction. arXiv, 2018.

Meizi et al. Micro Behaviors: A New Perspective in E-commerce Recommender Systems. WSDM, 2018.

Xiangyu et al. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. KDD, 2018.

Xiangyu et al. Deep Reinforcement Learning for List-wise Recommendations. arXiv, 2019.

[Microsoft]

Po-Sen et al. Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. CIKM, 2013.

Ali Elkahky et al. A Multi-View Deep Learning Approach for Cross Domain User Modeling in Recommendation Systems. WWW, 2015.

Oren et al. ITEM2VEC: NEURAL ITEM EMBEDDING FOR COLLABORATIVE FILTERING. ICML, 2016.

Guanjie et al. DRN: A Deep Reinforcement Learning Framework for News Recommendation. WWW, 2018.

John et al. Modeling and Simultaneously Removing Bias via Adversarial Neural Networks. arXiv, 2018.

Chen et al. Privileged Features Distillation at Taobao Recommendations. KDD, 2020.

Hongwei et al. RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems. CIKM, 2018.

Jianxun et al. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. KDD, 2018.

Chuhan et al. Neural News Recommendation with Attentive Multi-View Learning. IJCAI, 2019.

Le et al. Personalized Multimedia Item and Key Frame Recommendation. IJCAI, 2019.

Shu et al. Session-Based Recommendation with Graph Neural Networks. AAAI, 2019.

Le et al. SocialGCN: An Efficient Graph Convolutional Network based Model for Social Recommendation. arXiv, 2019.

• Balazs et al. Session-based recommendations with recurrent neural networks. ICLR, 2016.

[Pinterest]

Ying et al. PinSage: Graph Convolutional Neural Networks for Web-Scale Recommender Systems. KDD, 2018

Pal et al. PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest. KDD, 2020

Yang et al. MultiSage: Empowering GCN with Contextualized Multi-Embeddings on Web-Scale Multipartite Networks. KDD, 2020

• Junlin et al. FAT-DeepFFM: Field Attentive Deep Field-aware Factorization Machine. arXiv, 2019.

Qitian et al. Dual graph attention networks for deep latent representation of multifaceted social effects in recommender systems. WWW, 2019.

Wen et al. Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction. WWW, 2020.

Ruobing et al. Deep Feedback Network for Recommendation. IJCAI, 2020

Tongwen et al. GateNet:Gating-Enhanced Deep Network for Click-Through Rate Prediction. arXiv, 2020.

[X (Twitter)]

• Zhang et al. TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter. KDD, 2023.

Junwei et al. Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising. WWW, 2018.

Shaunak et al. Learning to Create Better Ads Generation and Ranking Approaches for Ad Creative Refinement. CIKM, 2020.

Privacy in RS

John et al. Collaborative filtering with privacy via factor analysis. SIGIR, 2002.

Arvind et al. Robust De-anonymization of Large Sparse Datasets. S&P, 2008.

Udi et al. BlurMe: inferring and obfuscating user gender based on ratings. RecSys, 2012.

Hua et al. Differentially Private Matrix Factorization. IJCAI, 2015.

Li et al. Data poisoning attacks on factorization-based collaborative filtering. NIPS, 2016.

Erez et al. Secure Multi-Party Protocols for Item-Based Collaborative Filtering. Recsys, 2017.

Meng et al. Personalized Privacy-Preserving Social Recommendation. AAAI, 2018.

Shin et al. Privacy Enhanced Matrix Factorization for Recommendation with Local Differential Privacy. TKDE, 2018.

Chen et al. Privacy Preserving Point-of-interest Recommendation Using Decentralized Matrix Factorization. AAAI, 2018.

Istvan et al. Decentralized Recommendation Based on Matrix Factorization-A Comparison of Gossip and Federated Learning. PKDD, 2019.

Aidmar et al. Efficient Privacy-Preserving Recommendations based on Social Graphs. RecSys, 2019.

Chen et al. Attacking Recommender Systems with Augmented User Profiles. CIKM, 2020.

Zhang et al. GCN: Based User Representation Learning for Unifying Robust Recommendation and Fraudster Identification. SIGIR, 2020.

Wang et al. Global and Local Differential Privacy for Collaborative Bandits. RecSys, 2020.

Fang et al. Influence Function based Data Poisoning Attacks to Top-N Recommender Systems. WWW, 2020.

Zhang et al. Practical Data Poisoning Attack against Next-Item Recommendation. arXiv, 2020.

Beigi et al. Privacy-Aware Recommendation with Private-Attribute Protection using Adversarial Learning. WSDM, 2020.

Hu et al. PrivNet: Safeguarding Private Attributes in Transfer Learning for Recommendation. EMNLP, 2020.

Chen et al. Secure Social Recommendation based on Secret Sharing. ECAI, 2020.

Cohen et al. A Black-Box Attack Model for Visually-Aware Recommenders. WSDM, 2021.

Huang et al. Data Poisoning Attacks to Deep Learning Based Recommender Systems. NDSS, 2021.

Zhang et al. Graph Embedding for Recommendation against Attribute Inference Attacks. arXiv, 2021.

[Federated Learning]

Muhammad et al. Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System. arXiv, 2019.

Li et al. Federated Recommendation System via Differential Privacy. ISIT, 2020.

Lin et al. FedRec: Federated Recommendation with Explicit Feedback. IEEE IS, 2020.

Qi et al. FedRec: Privacy-Preserving News Recommendation with Federated Learning. arXiv, 2020.

Lin et al. Meta Matrix Factorization for Federated Rating Predictions. SIGIR, 2020.

Chai et al. Secure Federated Matrix Factorization. IEEE IS, 2020.

Flanagan et al. Federated Multi-view Matrix Factorization for Personalized Recommendations. PKDD, 2020.

Muhammad et al. FedFast: Going Beyond Average for Faster Training of Federated Recommender Systems. KDD, 2020.

Wu et al. FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendation. arXiv, 2021.

Vito et al. FedeRank: User Controlled Feedback with Federated Recommender Systems. ECIR, 2021.

Liang et al. FedRec++: Lossless Federated Recommendation with Explicit Feedback. AAAI, 2021.

Wu et al. Hierarchical Personalized Federated Learning for User Modeling. WWW, 2021.

Kalloori et al. Horizontal Cross-Silo Federated Recommender Systems. RecSys, 2021.

Farwa et al. A Payload Optimization Method for Federated Recommender Systems. RecSys, 2021.

Lin et al. FR-FMSS: Federated Recommendation via Fake Marks and Secret Sharing. RecSys, 2021.

Minto et al. Stronger Privacy for Federated Collaborative Filtering With Implicit Feedback. RecSys, 2021.

Liu et al. FedCT: Federated Collaborative Transfer for Recommendation. SIGIR, 2021.

Liu et al. Federated Social Recommendation with Graph Neural Network. TIST, 2021.

Lin et al. Learning Federated Representations and Recommendations with Limited Negatives. arXiv, 2021.

Li et al. Federated Matrix Factorization with Privacy Guarantee. VLDB Endowment, 2022.

Zhang et al. PipAttack: Poisoning Federated Recommender Systems for Manipulating Item Promotion. WSDM, 2022.

Rong et al. Poisoning Deep Learning based Recommender Model in Federated Learning Scenarios. IJCAI, 2022.

Zhang et al. LightFR: Lightweight Federated Recommendation with Privacy-preserving Matrix Factorization. ACM TOIS, 2023.

Geng et al. Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). arXiv, 2023.

Cui et al. M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems. arXiv, 2022.

Sileo et al. Zero-Shot Recommendation as Language Modeling. ECIR, 2022.

Gao et al. Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. arXiv, 2023.

Li et al. GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation. arXiv, 2003.

Dai et al. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv, 2023.

Liu et al. Is ChatGPT a Good Recommender? A Preliminary Study. arXiv, 2023.

Zhang et al. Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation. arXiv, 2023.

Hou et al. Large Language Models are Zero-Shot Rankers for Recommender Systems. arXiv, 2023.

Wang et al. Generative Recommendation: Towards Next-generation Recommender Paradigm. arXiv, 2023.

Bao et al. TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. arXiv, 2023.

Wang et al. Zero-Shot Next-Item Recommendation using Large Pretrained Language Models. arXiv, 2023.

Luke et al. Leveraging Large Language Models in Conversational Recommender Systems. arXiv, 2023.

Yang et al. PALR: Personalization Aware LLMs for Recommendation. arXiv, 2023.

Li et al. CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendation. arXiv, 2024.

Star History

RSAlgorithms

Recently, we have launched an open source project RSAlgorithms , which provides an integrated training and testing framework. In this framework, we implement a set of classical traditional recommendation methods which make predictions only using rating data and social recommendation methods which utilize trust/social information in order to alleviate the sparsity of ratings data. Besides, we have collected some classical methods implemented by others for your convenience.

Acknowledgements

Specially summerize the papers about Recommender Systems for you, and if you have any questions, please contact me generously. Last but not least, the ability of myself is limited so I sincerely look forward to working with you to contribute it.

Thank @ ShawnSu for collecting papers about POI Recommender Systems.

Thank @ Wang Zhe for his advice about EE in RS.

Highly thank @ Yujia Zhang for her summary on Hashing for RS.

Thank @ Zixuan Yang and @ vicki1109 for his collecting papers about CTR Prediction for RS.

Thank @ ShomyLiu for collecting papers about Review based RS.

Thank @ Fanshaoliu for collecting papers about Industrial RS.

Contributors 15

Subscribe to the PwC Newsletter

Join the community, add a new evaluation result row, recommendation systems.

1616 papers with code • 54 benchmarks • 56 datasets

The Recommendation Systems task is to produce a list of recommendations for a user. The most common methods used in recommender systems are factor models (Koren et al., 2009; Weimer et al., 2007; Hidasi & Tikk, 2012) and neighborhood methods (Sarwar et al., 2001; Koren, 2008). Factor models work by decomposing the sparse user-item interactions matrix to a set of d dimensional vectors one for each item and user in the dataset. Factor models are hard to apply in session-based recommendations due to the absence of a user profile. On the other hand, neighborhood methods, which rely on computing similarities between items (or users) are based on co-occurrences of items in sessions (or user profiles). Neighborhood methods have been used extensively in session-based recommendations.

( Image credit: CuMF_SGD )

Benchmarks Add a Result

--> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> -->

Trend	Dataset	Best Model	Paper	Code	Compare
		GLocal-K
		HyperML
		GHRS
		Bayesian timeSVD++ flipped
		SSCF
		ConvNCF
		H+Vamp Gated
		NESCL
		GLocal-K
		I-CFN
		EASE
		IGMC
		C2CRS
		CARCA Abs + Con
		ProxyRCA
		MG-GAT
		GRALS
		DANSER
		ProxyRCA
		GRALS
		Fashion GAE
		Ekar*
		CARCA Learnt + Con
		INN
		KTUP (soft)
		DGRec
		TransCF
		HetroFair
		DANSER
		TLSAN
		NeuMF
		TransCF
		TransCF
		TransCF
		TransCF
		RATE-CSE
		RATE-CSE
		RANK-CSE
		RANK-CSE
		RANK-CSE
		DGRec
		KGNN-LS
		HGN
		HGN
		SASRec
		Multi-Gradient Descent
		HAKG
		LT-OCF
		CFM
		SASRec
		HetroFair
		HetroFair
		HetroFair
		HetroFair

Most implemented papers

Neural collaborative filtering.

hexiangnan/neural_collaborative_filtering • WWW 2017

When it comes to model the key factor in collaborative filtering -- the interaction between user and item features, they still resorted to matrix factorization and applied an inner product on the latent features of users and items.

Wide & Deep Learning for Recommender Systems

Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort.

FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction

In this paper, a new model named FiBiNET as an abbreviation for Feature Importance and Bilinear feature Interaction NETwork is proposed to dynamically learn the feature importance and fine-grained feature interactions.

Session-based Recommendations with Recurrent Neural Networks

We apply recurrent neural networks (RNN) on a new domain, namely recommender systems.

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Learning sophisticated feature interactions behind user behaviors is critical in maximizing CTR for recommender systems.

Neural Graph Collaborative Filtering

Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF.

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

On one hand, the xDeepFM is able to learn certain bounded-degree feature interactions explicitly; on the other hand, it can learn arbitrary low- and high-order feature interactions implicitly.

Variational Autoencoders for Collaborative Filtering

dawenl/vae_cf • 16 Feb 2018

This non-linear probabilistic model enables us to go beyond the limited modeling capacity of linear factor models which still largely dominate collaborative filtering research. We introduce a generative model with multinomial likelihood and use Bayesian inference for parameter estimation.

Deep Learning Recommendation Model for Personalization and Recommendation Systems

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks.

MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask

We also turn the feed-forward layer in DNN model into a mixture of addictive and multiplicative feature interactions by proposing MaskBlock in this paper.

A survey on large language models for recommendation

• Published: 22 August 2024
• Volume 27 , article number  60 , ( 2024 )

Cite this article

• Likang Wu 1 , 2 ,
• Zhi Zheng 1 , 2   na1 ,
• Zhaopeng Qiu 2   na1 ,
• Hao Wang 1 ,
• Hongchao Gu 1 ,
• Tingjia Shen 1 ,
• Chuan Qin 2 ,
• Chen Zhu 2 ,
• Hengshu Zhu 2 ,
• Hui Xiong 3 &
• Enhong Chen 1  

221 Accesses

Explore all metrics

Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) and have recently gained significant attention in the domain of Recommendation Systems (RS). These models, trained on massive amounts of data using self-supervised learning, have demonstrated remarkable success in learning universal representations and have the potential to enhance various aspects of recommendation systems by some effective transfer techniques such as fine-tuning, prompt tuning, etc. The crucial aspect of harnessing the power of language models in enhancing recommendation quality is the utilization of their high-quality representations of textual features and their extensive coverage of external knowledge to establish correlations between items and users. To provide a comprehensive understanding of the existing LLM-based recommendation systems, this survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec), with the latter being systematically sorted out for the first time. Furthermore, we systematically review and analyze existing LLM-based recommendation systems within each paradigm, providing insights into their methodologies, techniques, and performance. Additionally, we identify key challenges and several valuable findings to provide researchers and practitioners with inspiration. We have also created a GitHub repository to index relevant papers and resources on LLMs for recommendation ( https://github.com/WLiK/LLM4Rec-Awesome-Papers ).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Zero-Shot Recommendation as Language Modeling

Third international workshop on algorithmic bias in search and recommendation (bias@ecir2022).

Handling Sparsity in Cross-Domain Recommendation Systems: Review

Explore related subjects.

• Artificial Intelligence

Data Availability

No datasets were generated or analysed during the current study.

https://en.wikipedia.org/wiki/Large_language_model

Liu, P., Zhang, L., Gulla, J.A.: Pre-train, prompt and recommendation: a comprehensive survey of language modelling paradigm adaptations in recommender systems. arXiv:2302.03735 (2023)

Geng, S., Liu, S., Fu, Z., Ge, Y., Zhang, Y.: Recommendation as language processing (RLP): a unified pretrain, personalized prompt & predict paradigm (P5). In: RecSys, pp. 299–315 (2022)

Da’u, A., Salim, N.: Recommendation system based on deep learning methods: a systematic review and new directions. Artificial Intelligence Review. 53 (4), 2709–2748 (2020)

Sileo, D., Vossen, W., Raymaekers, R.: Zero-shot recommendation as language modeling. In: ECIR (2). Lecture Notes in Computer Science, vol. 13186, pp. 223–230 (2022)

Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv:2303.18223 (2023)

Gao, Y., Sheng, T., Xiang, Y., Xiong, Y., Wang, H., Zhang, J.: Chat-rec: towards interactive and explainable llms-augmented recommender system. arXiv:2303.14524 (2023)

Li, L., Zhang, Y., Chen, L.: Personalized prompt learning for explainable recommendation. ACM Transactions on Information Systems. 41 (4), 1–26 (2023)

Google Scholar  

Zeng, Z., Xiao, C., Yao, Y., Xie, R., Liu, Z., Lin, F., Lin, L., Sun, M.: Knowledge transfer via pre-training for recommendation: a review and prospect. Frontiers in Big Data. 4 , 602071 (2021)

Article   Google Scholar  

Lin, J., Dai, X., Xi, Y., Liu, W., Chen, B., Zhang, H., Liu, Y., Wu, C., Li, X., Zhu, C., Guo, H., Yu, Y., Tang, R., Zhang, W.: How Can Recommender Systems Benefit from Large Language Models: A Survey (2024)

Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., Li, Q.: Recommender Systems in the Era of Large Language Models (LLMs) (2024)

Li, L., Zhang, Y., Liu, D., Chen, L.: Large Language Models for Generative Recommendation: A Survey and Visionary Discussions (2024)

Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., Lian, D., Chen, E.: When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities (2023)

Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (1), pp. 4171–4186 (2019)

Qiu, Z., Wu, X., Gao, J., Fan, W.: U-BERT: pre-training user representations for improved recommendation. In: AAAI, pp. 4320–4327 (2021)

Wu, C., Wu, F., Yu, Y., Qi, T., Huang, Y., Xie, X.: Userbert: Contrastive user model pre-training. arXiv:2109.01274 (2021)

Yang, Y., Qiao, Y., Shao, J., Yan, X., Yang, T.: Lightweight composite re-ranking for efficient keyword search with BERT. In: WSDM, pp. 1234–1244 (2022)

Wu, X., Magnani, A., Chaidaroon, S., Puthenputhussery, A., Liao, C., Fang, Y.: A multi-task learning framework for product ranking with BERT. In: WWW, pp. 493–501 (2022)

Rahmani, S., Naghshzan, A., Guerrouj, L.: Improving code example recommendations on informal documentation using bert and query-aware lsh: a comparative study. arXiv:2305.03017 (2023)

Zhang, S., Zheng, N., Wang, D.: GBERT: pre-training user representations for ephemeral group recommendation. In: CIKM, pp. 2631–2639 (2022)

Yao, S., Tan, J., Chen, X., Zhang, J., Zeng, X., Yang, K.: Reprbert: distilling BERT to an efficient representation-based relevance model for e-commerce. In: KDD, pp. 4363–4371 (2022)

Muhamed, A., Keivanloo, I., Perera, S., Mracek, J., Xu, Y., Cui, Q., Rajagopalan, S., Zeng, B., Chilimbi, T.: Ctr-bert: cost-effective knowledge distillation for billion-parameter teacher models. In: NeurIPS Efficient Natural Language and Speech Processing Workshop (2021)

Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., Jiang, P.: Bert4rec: sequential recommendation with bidirectional encoder representations from transformer. In: CIKM, pp. 1441–1450 (2019)

Zhao, Q.: Resetbert4rec: a pre-training model integrating time and user historical behavior for sequential recommendation. In: SIGIR, pp. 1812–1816 (2022)

Fu, J., Yuan, F., Song, Y., Yuan, Z., Cheng, M., Cheng, S., Zhang, J., Wang, J., Pan, Y.: Exploring adapter-based transfer learning for recommender systems: empirical studies and practical insights. arXiv:2305.15036 (2023)

Hu, J., Xia, W., Zhang, X., Fu, C., Wu, W., Huan, Z., Li, A., Tang, Z., Zhou, J.: Enhancing sequential recommendation via llm-based semantic embedding learning. In: Companion Proceedings of the ACM on Web Conference 2024, pp. 103–111 (2024)

Hou, Y., Mu, S., Zhao, W.X., Li, Y., Ding, B., Wen, J.: Towards universal sequence representation learning for recommender systems. In: KDD, pp. 585–593 (2022)

Hou, Y., He, Z., McAuley, J., Zhao, W.X.: Learning vector-quantized item representation for transferable sequential recommenders. In: Proceedings of the ACM Web Conference 2023. WWW ’23, pp. 1162–1171, New York, USA (2023)

Wu, C., Wu, F., Qi, T., Huang, Y.: Empowering news recommendation with pre-trained language models. In: SIGIR, pp. 1652–1656 (2021)

Yu, Y., Wu, F., Wu, C., Yi, J., Liu, Q.: Tiny-newsrec: effective and efficient plm-based news recommendation. In: EMNLP, pp. 5478–5489 (2022)

Liu, Q., Zhu, J., Dai, Q., Wu, X.: Boosting deep CTR prediction with a plug-and-play pre-trainer for news recommendation. In: COLING, pp. 2823–2833 (2022)

Yuan, Z., Yuan, F., Song, Y., Li, Y., Fu, J., Yang, F., Pan, Y., Ni, Y.: Where to go next for recommender systems? id-vs. modality-based recommender models revisited. arXiv:2303.13835 (2023)

Li, R., Deng, W., Cheng, Y., Yuan, Z., Zhang, J., Yuan, F.: Exploring the upper limits of text-based collaborative filtering using large language models: discoveries and insights. arXiv:2305.11700 (2023)

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: open pre-trained transformer language models. arXiv:2205.01068 (2022)

Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059 (2021)

Penha, G., Hauff, C.: What does BERT know about books, movies and music? probing BERT for conversational recommendation. In: RecSys, pp. 388–397 (2020)

Yang, B., Han, C., Li, Y., Zuo, L., Yu, Z.: Improving conversational recommendation systems’ quality with context-aware item meta-information. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 38–48 (2022)

Shen, T., Li, J., Bouadjenek, M.R., Mai, Z., Sanner, S.: Towards understanding and mitigating unintended biases in language model-driven conversational recommendation. Information Processing & Management. 60 (1), 103139 (2023)

Zhang, Z., Wang, B.: Prompt learning for news recommendation. arXiv:2304.05263 (2023)

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: NeurIPS (2020)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. In: NeurIPS (2022)

Liu, J., Liu, C., Lv, R., Zhou, K., Zhang, Y.: Is chatgpt a good recommender? A preliminary study. CoRR. arXiv:2304.10149 (2023)

Dai, S., Shao, N., Zhao, H., Yu, W., Si, Z., Xu, C., Sun, Z., Zhang, X., Xu, J.: Uncovering chatgpt’s capabilities in recommender systems. arXiv:2305.02182 (2023)

Lin, G., Zhang, Y.: Sparks of artificial general recommender (AGR): early experiments with chatgpt. CoRR. arXiv:2305.04518 (2023)

Sanner, S., Balog, K., Radlinski, F., Wedin, B., Dixon, L.: Large language models are competitive near cold-start recommenders for language-and item-based preferences. In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 890–896 (2023)

Mysore, S., McCallum, A., Zamani, H.: Large language model augmented narrative driven recommendations. arXiv:2306.02250 (2023)

Xi, Y., Liu, W., Lin, J., Zhu, J., Chen, B., Tang, R., Zhang, W., Zhang, R., Yu, Y.: Towards open-world recommendation with knowledge augmentation from large language models. ArXiv. arXiv:2306.10933 (2023)

Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J.J., Zhao, W.X.: Large language models are zero-shot rankers for recommender systems. arXiv:2305.08845 (2023)

Sun, W., Yan, L., Ma, X., Ren, P., Yin, D., Ren, Z.: Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv:2304.09542 (2023)

Yang, Z., Wu, J., Luo, Y., Zhang, J., Yuan, Y., Zhang, A., Wang, X., He, X.: Large language model can interpret latent space of sequential recommender. arXiv:2310.20487 (2023)

Guo, F., Li, W., Zhuang, H., Luo, Y., Li, Y., Yan, L., Zhang, Y.: Generating diverse criteria on-the-fly to improve point-wise LLM rankers. arXiv:2404.11960 (2024)

He, Z., Xie, Z., Jha, R., Steck, H., Liang, D., Feng, Y., Majumder, B.P., Kallus, N., McAuley, J.J.: Large language models as zero-shot conversational recommenders. In: CIKM, pp. 720–730 (2023)

Acharya, A., Singh, B., Onoe, N.: LLM based generation of item-description for recommendation system. In: RecSys, pp. 1204–1207 (2023)

Gao, S., Fang, J., Tu, Q., Yao, Z., Chen, Z., Ren, P., Ren, Z.: Generative news recommendation. CoRR. arXiv:2403.03424 (2024)

Yang, S., Ma, W., Sun, P., Ai, Q., Liu, Y., Cai, M., Zhang, M.: Sequential recommendation with latent relations based on large language model. arXiv:2403.18348 (2024)

Lyu, H., Jiang, S., Zeng, H., Xia, Y., Luo, J.: Llm-rec: personalized recommendation via prompting large language models. arXiv:2307.15780 (2023)

Wang, L., Lim, E.: Zero-shot next-item recommendation using large pretrained language models. arXiv:2304.03153 (2023)

Wang, Y., Chu, Z., Ouyang, X., Wang, S., Hao, H., Shen, Y., Gu, J., Xue, S., Zhang, J.Y., Cui, Q., Li, L., Zhou, J., Li, S.: Enhancing recommender systems with large language model reasoning graphs. arXiv:2308.10835 (2023)

Liu, Q., Chen, N., Sakai, T., Wu, X.: Once: boosting content-based recommendation with both open- and closed-source large language models. arXiv:2305.06566 (2023)

Wei, W., Ren, X., Tang, J., Wang, Q., Su, L., Cheng, S., Wang, J., Yin, D., Huang, C.: Llmrec: large language models with graph augmentation for recommendation. arXiv:2311.00423 (2023)

Ren, X., Wei, W., Xia, L., Su, L., Cheng, S., Wang, J., Yin, D., Huang, C.: Representation learning with large language models for recommendation. arXiv:2310.15950 (2023)

Shu, Y., Gu, H., Zhang, P., Zhang, H., Lu, T., Li, D., Gu, N.: Rah! recsys-assistant-human: a human-central recommendation framework with large language models. arXiv:2308.09904 (2023)

Shi, W., He, X., Zhang, Y., Gao, C., Li, X., Zhang, J., Wang, Q., Feng, F.: Large language models are learnable planners for long-term recommendation. (2024). https://api.semanticscholar.org/CorpusID:268230856

Huang, X., Lian, J., Lei, Y., Yao, J., Lian, D., Xie, X.: Recommender AI agent: integrating large language models for interactive recommendations. arXiv:2308.16505 (2023)

Jin, J., Chen, X., Ye, F., Yang, M., Feng, Y., Zhang, W., Yu, Y., Wang, J.: Lending interaction wings to recommender systems with conversational agents. Advances in Neural Information Processing Systems. 36 (2024)

Huang, D., Markovitch, D.G., Stough, R.A.: Can chatbot customer service match human service agents on customer satisfaction? an investigation in the role of trust. Journal of Retailing and Consumer Services. 76 , 103600 (2024)

Zhang, J., Bao, K., Wang, W., Zhang, Y., Shi, W., Xu, W., Feng, F., Chua, T.: Prospect personalized recommendation on large language model-based agent platform. arXiv:2402.18240 (2024)

Zhang, J., Hou, Y., Xie, R., Sun, W., McAuley, J., Zhao, W.X., Lin, L., Wen, J.-R.: Agentcf: collaborative learning with autonomous language agents for recommender systems. In: Proceedings of the ACM on Web Conference 2024, pp. 3679–3689 (2024)

Wang, W., Lin, X., Feng, F., He, X., Chua, T.: Generative recommendation: towards next-generation recommender paradigm. arXiv:2304.03516 (2023)

Yoon, S., He, Z., Echterhoff, J.M., McAuley, J.J.: Evaluating large language models as generative user simulators for conversational recommendation. arXiv:2403.09738 (2024)

Wang, L., Zhang, J., Chen, X., Lin, Y., Song, R., Zhao, W.X., Wen, J.-R.: Recagent: a novel simulation paradigm for recommender systems. arXiv:2306.02552 (2023)

Zhang, A., Sheng, L., Chen, Y., Li, H., Deng, Y., Wang, X., Chua, T.: On generative agents in recommendation. arXiv:2310.10108 (2023)

Zhang, W., Wu, C., Li, X., Wang, Y., Dong, K., Wang, Y., Dai, X., Zhao, X., Guo, H., Tang, R.: Tired of plugins? large language models can be end-to-end recommenders. arXiv:2404.00702 (2024)

Dai, D., Sun, Y., Dong, L., Hao, Y., Sui, Z., Wei, F.: Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. arXiv:2212.10559 (2022)

Wang, L., Lim, E.: The whole is better than the sum: using aggregated demonstrations in in-context learning for sequential recommendation. arXiv:2403.10135 (2024)

Wang, X., Tang, X., Zhao, W.X., Wang, J., Wen, J.-R.: Rethinking the evaluation for conversational recommendation in the era of large language models. arXiv:2305.13112 (2023)

Zhang, C., Wu, S., Zhang, H., Xu, T., Gao, Y., Hu, Y., Chen, E.: Notellm: a retrievable large language model for note recommendation. In: Companion Proceedings of the ACM on Web Conference 2024, pp. 170–179 (2024)

Petrov, A.V., Macdonald, C.: Generative sequential recommendation with gptrec. arXiv:2306.11114 (2023)

Kang, W., Ni, J., Mehta, N., Sathiamoorthy, M., Hong, L., Chi, E.H., Cheng, D.Z.: Do llms understand user preferences? evaluating llms on user rating prediction. arXiv:2305.06474 (2023)

Zhai, J., Liao, L., Liu, X., Wang, Y., Li, R., Cao, X., Gao, L., Gong, Z., Gu, F., He, J., et al.: Actions speak louder than words: trillion-parameter sequential transducers for generative recommendations. In: Forty-first International Conference on Machine Learning

Krishna, S., Agarwal, C., Lakkaraju, H.: Understanding the effects of iterative prompting on truthfulness. arXiv:2402.06625 (2024)

Bao, K., Zhang, J., Zhang, Y., Wang, W., Feng, F., He, X.: Tallrec: an effective and efficient tuning framework to align large language model with recommendation. arXiv:2305.00447 (2023)

Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford Alpaca: An Instruction-following LLaMA model. GitHub (2023)

Ji, J., Li, Z., Xu, S., Hua, W., Ge, Y., Tan, J., Zhang, Y.: Genrec: large language model for generative recommendation. arXiv e-prints, 2307 (2023)

Chen, Z.: Palr: Personalization aware llms for recommendation. arXiv:2305.07622 (2023)

Zheng, Z., Chao, W., Qiu, Z., Zhu, H., Xiong, H.: Harnessing large language models for text-rich sequential recommendation. In: Proceedings of the ACM on Web Conference 2024, pp. 3207–3216 (2024)

Chu, Z., Hao, H., Ouyang, X., Wang, S., Wang, Y., Shen, Y., Gu, J., Cui, Q., Li, L., Xue, S., et al.: Leveraging large language models for pre-trained recommender systems. arXiv:2308.10837 (2023)

Jin, W., Mao, H., Li, Z., Jiang, H., Luo, C., Wen, H., Han, H., Lu, H., Wang, Z., Li, R., et al.: Amazon-m2: a multilingual multi-locale shopping session dataset for recommendation and text generation. arXiv preprint arXiv:2307.09688 (2023)

Friedman, L., Ahuja, S., Allen, D., Tan, T., Sidahmed, H., Long, C., Xie, J., Schubiner, G., Patel, A., Lara, H., et al.: Leveraging large language models in conversational recommender systems. arXiv:2305.07961 (2023)

Li, X., Zhang, Y., Malthouse, E.C.: Pbnr: prompt-based news recommender system. arXiv:2304.07862 (2023)

Li, J., Zhang, W., Wang, T., Xiong, G., Lu, A., Medioni, G.: Gpt4rec: a generative framework for personalized recommendation and user interests interpretation. arXiv:2304.03879 (2023)

Yue, Z., Rabhi, S., Moreira, G.d.S.P., Wang, D., Oldridge, E.: Llamarec: two-stage recommendation using large language models for ranking. arXiv:2311.02089 (2023)

Wang, X., Zhou, K., Wen, J., Zhao, W.X.: Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In: KDD, pp. 1929–1937 (2022)

Lin, X., Wang, W., Li, Y., Feng, F., Ng, S.-K., Chua, T.-S.: A multi-facet paradigm to bridge large language model and recommendation. arXiv:2310.06491 (2023)

Bao, K., Zhang, J., Wang, W., Zhang, Y., Yang, Z., Luo, Y., Feng, F., He, X., Tian, Q.: A bi-step grounding paradigm for large language models in recommendation systems. arXiv:2308.08434 (2023)

Zhang, Y., Feng, F., Zhang, J., Bao, K., Wang, Q., He, X.: Collm: integrating collaborative embeddings into large language models for recommendation. arXiv:2310.19488 (2023)

Zhang, W., Liu, H., Du, Y., Zhu, C., Song, Y., Zhu, H., Wu, Z.: Bridging the information gap between domain-specific model and general llm for personalized recommendation. arXiv:2311.03778 (2023)

Zhu, Y., Wu, L., Guo, Q., Hong, L., Li, J.: Collaborative large language model for recommender systems. arXiv:2311.01343 (2023)

Liao, J., Li, S., Yang, Z., Wu, J., Yuan, Y., Wang, X., He, X.: Llara: aligning large language models with sequential recommenders. arXiv:2312.02445 (2023)

Li, X., Chen, C., Zhao, X., Zhang, Y., Xing, C.: E4srec: an elegant effective efficient extensible solution of large language models for sequential recommendation. arXiv:2312.02443 (2023)

Guo, N., Cheng, H., Liang, Q., Chen, L., Han, B.: Integrating large language models with graphical session-based recommendation. arXiv:2402.16539 (2024)

Guan, Z., Wu, L., Zhao, H., He, M., Fan, J.: Enhancing collaborative semantics of language model-driven recommendations via graph-aware learning. arXiv:2406.13235 (2024)

Liu, Z., Wu, L., He, M., Guan, Z., Zhao, H., Feng, N.: Dr. e bridges graphs with large language models through words. arXiv:2406.15504 (2024)

Qu, Z., Xie, R., Xiao, C., Sun, X., Kang, Z.: The elephant in the room: rethinking the usage of pre-trained language model in sequential recommendation. arXiv:2404.08796 (2024)

Rajput, S., Mehta, N., Singh, A., Hulikal Keshavan, R., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V., Samost, J., et al.: Recommender systems with generative retrieval. Advances in Neural Information Processing Systems. 36 (2024)

Zheng, Z., Qiu, Z., Hu, X., Wu, L., Zhu, H., Xiong, H.: Generative job recommendations with large language model. arXiv:2307.02157 (2023)

Wu, L., Qiu, Z., Zheng, Z., Zhu, H., Chen, E.: Exploring large language model for graph data understanding in online job recommendations. arXiv:2307.05722 (2023)

Du, Y., Luo, D., Yan, R., Liu, H., Song, Y., Zhu, H., Zhang, J.: Enhancing job recommendation through llm-based generative adversarial networks. arXiv:2307.10747 (2023)

Zhao, H., Zheng, S., Wu, L., Yu, B., Wang, J.: Lane: Logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. arXiv:2407.02833 (2024)

Zhang, J., Xie, R., Hou, Y., Zhao, W.X., Lin, L., Wen, J.: Recommendation as instruction following: a large language model empowered recommendation approach. arXiv:2305.07001 (2023)

Cui, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: M6-rec: generative pretrained language models are open-ended recommender systems. arXiv:2205.08084 (2022)

Yin, B., Xie, J., Qin, Y., Ding, Z., Feng, Z., Li, X., Lin, W.: Heterogeneous knowledge fusion: a novel approach for personalized recommendation via llm. In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 599–601 (2023)

Li, L., Zhang, Y., Chen, L.: Prompt distillation for efficient llm-based recommendation. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 1348–1357 (2023)

Lu, W., Lian, J., Zhang, W., Li, G., Zhou, M., Liao, H., Xie, X.: Aligning large language models for controllable recommendations. arXiv:2403.05063 (2024)

Xiao, S., Liu, Z., Shao, Y., Di, T., Middha, B., Wu, F., Xie, X.: Training large-scale news recommenders with pretrained language models in the loop. In: KDD, pp. 4215–4225 (2022)

Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., et al.: Large language models are effective text rankers with pairwise ranking prompting. arXiv:2306.17563 (2023)

Mao, Z., Wang, H., Du, Y., Wong, K.-F.: Unitrec: a unified text-to-text transformer and joint contrastive learning framework for text-based recommendation. In: Annual Meeting of the Association for Computational Linguistics (2023). https://api.semanticscholar.org/CorpusID:258888030

Li, X., Chen, B., Hou, L., Tang, R.: Ctrl: connect tabular and language model for ctr prediction. arXiv:2306.02841 (2023)

He, R., McAuley, J.: Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering. In: Proceedings of the 25th International Conference on World Wide Web, pp. 507–517 (2016)

Hou, Y., Li, J., He, Z., Yan, A., Chen, X., McAuley, J.J.: Bridging language and items for retrieval and recommendation. arXiv:2403.03952 (2024)

Wan, M., McAuley, J.: Item recommendation on monotonic behavior chains. In: Proceedings of the 12th ACM Conference on Recommender Systems, pp. 86–94 (2018)

Wu, Y., Wu, W., Xing, C., Zhou, M., Li, Z.: Sequential matching network: a new architecture for multi-turn response selection in retrieval-based chatbots. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 496–505 (2017)

Wu, F., Qiao, Y., Chen, J.-H., Wu, C., Qi, T., Lian, J., Liu, D., Xie, X., Gao, J., Wu, W., et al.: Mind: a large-scale dataset for news recommendation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3597–3606 (2020)

Liu, Y., Zhang, W., Dong, B., Fan, Y., Wang, H., Feng, F., Chen, Y., Zhuang, Z., Cui, H., Li, Y., et al.: U-need: a fine-grained dataset for user needs-centric e-commerce conversational recommendation. arXiv:2305.04774 (2023)

Sun, Z., Si, Z., Zang, X., Leng, D., Niu, Y., Song, Y., Zhang, X., Xu, J.: Kuaisar: A unified search and recommendation dataset. (2023) https://doi.org/10.1145/3583780.3615123

Yuan, G., Yuan, F., Li, Y., Kong, B., Li, S., Chen, L., Yang, M., Yu, C., Hu, B., Li, Z., et al.: Tenrec: a large-scale multipurpose benchmark dataset for recommender systems. arXiv:2210.10629 (2022)

Cheng, Y., Pan, Y., Zhang, J., Ni, Y., Sun, A., Yuan, F.: An image dataset for benchmarking recommender systems with raw pixels. arXiv:2309.06789 (2023)

Harte, J., Zorgdrager, W., Louridas, P., Katsifodimos, A., Jannach, D., Fragkoulis, M.: Leveraging large language models for sequential recommendation. In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1096–1102 (2023)

Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 8086–8098 (2022)

Zhang, J., Bao, K., Zhang, Y., Wang, W., Feng, F., He, X.: Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. arXiv:2305.07609 (2023)

Deldjoo, Y., Noia, T.D.: Cfairllm: consumer fairness evaluation in large-language model recommender system. arXiv:2403.05668 (2024)

Ferrara, E.: Should chatgpt be biased? challenges and risks of bias in large language models. arXiv:2304.03738 (2023)

Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., Miłoś, P.: Focused transformer: contrastive training for context scaling. arXiv:2307.03170 (2023)

Silva, Í., Marinho, L., Said, A., Willemsen, M.C.: Leveraging chatgpt for automated human-centered explanations in recommender systems. In: Proceedings of the 29th International Conference on Intelligent User Interfaces, pp. 597–608 (2024)

Wang, Y., Tian, C., Hu, B., Yu, Y., Liu, Z., Zhang, Z., Zhou, J., Pang, L., Wang, X.: Can small language models be good reasoners for sequential recommendation? In: Proceedings of the ACM on Web Conference 2024, pp. 3876–3887 (2024)

Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S.J., Seo, M.: Towards continual knowledge learning of language models. In: ICLR (2022)

Download references

This research was partially supported by grants from National Key Research and Development Program of China (Grant No. 2021YFF0901003).

Author information

Zhi Zheng and Zhaopeng Qiu are contributed equally to this work

Authors and Affiliations

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, JinZhai Road, Hefei, 230026, Anhui, China

Likang Wu, Zhi Zheng, Hao Wang, Hongchao Gu, Tingjia Shen, Qi Liu & Enhong Chen

Career Science Lab, BOSS Zhipin, Beijing, 100020, China

Likang Wu, Zhi Zheng, Zhaopeng Qiu, Chuan Qin, Chen Zhu & Hengshu Zhu

Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 510000, Guangdong, China

You can also search for this author in PubMed   Google Scholar

Contributions

Likang Wu, Zhi Zheng, and Zhaopeng Qiu wrote the main manuscript text. Hongchao Gu and Tingjia Shen developed the GitHub project. Chuan Qin, Chen Zhu, and Qi Liu prepared the paper revision. Hao Wang, Hengshu Zhu, Hui Xiong, and Enhong Chen organized this project. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Hao Wang , Hengshu Zhu , Hui Xiong or Enhong Chen .

Ethics declarations

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Competing Interests

The authors declare no competing interests.

Ethical Approval

Not applicable.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Wu, L., Zheng, Z., Qiu, Z. et al. A survey on large language models for recommendation. World Wide Web 27 , 60 (2024). https://doi.org/10.1007/s11280-024-01291-2

Download citation

Received : 17 June 2024

Revised : 09 July 2024

Accepted : 17 July 2024

Published : 22 August 2024

DOI : https://doi.org/10.1007/s11280-024-01291-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Large language models
• Recommendation system
• Find a journal
• Publish with us
• Track your research

• Publications
• News and Events
• Education and Outreach

Software Engineering Institute

Sei digital library, latest publications, embracing ai: unlocking scalability and transformation through generative text, imagery, and synthetic audio, august 28, 2024 • webcast, by tyler brooks , shannon gallagher , dominic a. ross.

In this webcast, Tyler Brooks, Shannon Gallagher, and Dominic Ross aim to demystify AI and illustrate its transformative power in achieving scalability, adapting to changing landscapes, and driving digital innovation.

Counter AI: What Is It and What Can You Do About It?

August 27, 2024 • white paper, by nathan m. vanhoudnos , carol j. smith , matt churilla , shing-hon lau , lauren mcilvenny , greg touhill.

This paper describes counter artificial intelligence (AI) and provides recommendations on what can be done about it.

Using Quality Attribute Scenarios for ML Model Test Case Generation

August 27, 2024 • conference paper, by rachel brower-sinning , grace lewis , sebastián echeverría , ipek ozkaya.

This paper presents an approach based on quality attribute (QA) scenarios to elicit and define system- and model-relevant test cases for ML models.

3 API Security Risks (and How to Protect Against Them)

August 27, 2024 • podcast, by mckinley sconiers-hasan.

McKinley Sconiers-Hasan discusses three API risks and how to address them through the lens of zero trust.

Lessons Learned in Coordinated Disclosure for Artificial Intelligence and Machine Learning Systems

August 20, 2024 • white paper, by allen d. householder , vijay s. sarvepalli , jeff havrilla , matt churilla , lena pons , shing-hon lau , nathan m. vanhoudnos , andrew kompanek , lauren mcilvenny.

In this paper, the authors describe lessons learned from coordinating AI and ML vulnerabilities at the SEI's CERT/CC.

On the Design, Development, and Testing of Modern APIs

July 30, 2024 • white paper, by alejandro gomez , alex vesey.

This white paper discusses the design, desired qualities, development, testing, support, and security of modern application programming interfaces (APIs).

Evaluating Large Language Models for Cybersecurity Tasks: Challenges and Best Practices

July 26, 2024 • podcast, by jeff gennari , samuel j. perl.

Jeff Gennari and Sam Perl discuss applications for LLMs in cybersecurity, potential challenges, and recommendations for evaluating LLMs.

Capability-based Planning for Early-Stage Software Development

July 24, 2024 • podcast, by anandi hira , bill nichols.

This SEI podcast introduces capability-based planning (CBP) and its use and application in software acquisition.

A Model Problem for Assurance Research: An Autonomous Humanitarian Mission Scenario

July 23, 2024 • technical note, by gabriel moreno , anton hristozov , john e. robert , mark h. klein.

This report describes a model problem to support research in large-scale assurance.

Safeguarding Against Recent Vulnerabilities Related to Rust

June 28, 2024 • podcast, by david svoboda.

David Svoboda discusses two vulnerabilities related to Rust, their sources, and how to mitigate them.

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• News & Events
• FDA Newsroom
• Press Announcements

FDA Approves and Authorizes Updated mRNA COVID-19 Vaccines to Better Protect Against Currently Circulating Variants

FDA News Release

Today, the U.S. Food and Drug Administration approved and granted emergency use authorization (EUA) for updated mRNA COVID-19 vaccines (2024-2025 formula) to include a monovalent (single) component that corresponds to the Omicron variant KP.2 strain of SARS-CoV-2. The mRNA COVID-19 vaccines have been updated with this formula to more closely target currently circulating variants and provide better protection against serious consequences of COVID-19, including hospitalization and death. Today’s actions relate to updated mRNA COVID-19 vaccines manufactured by ModernaTX Inc. and Pfizer Inc.

In early June, the FDA advised manufacturers of licensed and authorized COVID-19 vaccines that the COVID-19 vaccines (2024-2025 formula) should be monovalent JN.1 vaccines. Based on the further evolution of SARS-CoV-2 and a rise in cases of COVID-19, the agency subsequently determined and advised manufacturers that the preferred JN.1-lineage for the COVID-19 vaccines (2024-2025 formula) is the KP.2 strain, if feasible.

“Vaccination continues to be the cornerstone of COVID-19 prevention,” said Peter Marks, M.D., Ph.D., director of the FDA’s Center for Biologics Evaluation and Research. “These updated vaccines meet the agency’s rigorous, scientific standards for safety, effectiveness, and manufacturing quality. Given waning immunity of the population from previous exposure to the virus and from prior vaccination, we strongly encourage those who are eligible to consider receiving an updated COVID-19 vaccine to provide better protection against currently circulating variants.”

The updated mRNA COVID-19 vaccines include Comirnaty and Spikevax, both of which are approved for individuals 12 years of age and older, and the Moderna COVID-19 Vaccine and Pfizer-BioNTech COVID-19 Vaccine, both of which are authorized for emergency use for individuals 6 months through 11 years of age.

What You Need to Know

• Unvaccinated individuals 6 months through 4 years of age are eligible to receive three doses of the updated, authorized Pfizer-BioNTech COVID-19 Vaccine or two doses of the updated, authorized Moderna COVID-19 Vaccine.
• Individuals 6 months through 4 years of age who have previously been vaccinated against COVID-19 are eligible to receive one or two doses of the updated, authorized Moderna or Pfizer-BioNTech COVID-19 vaccines (timing and number of doses to administer depends on the previous COVID-19 vaccine received).
• Individuals 5 years through 11 years of age regardless of previous vaccination are eligible to receive a single dose of the updated, authorized Moderna or Pfizer-BioNTech COVID-19 vaccines; if previously vaccinated, the dose is administered at least 2 months after the last dose of any COVID-19 vaccine.
• Individuals 12 years of age and older are eligible to receive a single dose of the updated, approved Comirnaty or the updated, approved Spikevax; if previously vaccinated, the dose is administered at least 2 months since the last dose of any COVID-19 vaccine.
• Additional doses are authorized for certain immunocompromised individuals ages 6 months through 11 years of age as described in the Moderna COVID-19 Vaccine and Pfizer-BioNTech COVID-19 Vaccine fact sheets.

Individuals who receive an updated mRNA COVID-19 vaccine may experience similar side effects as those reported by individuals who previously received mRNA COVID-19 vaccines and as described in the respective prescribing information or fact sheets. The updated vaccines are expected to provide protection against COVID-19 caused by the currently circulating variants. Barring the emergence of a markedly more infectious variant of SARS-CoV-2, the FDA anticipates that the composition of COVID-19 vaccines will need to be assessed annually, as occurs for seasonal influenza vaccines.

For today’s approvals and authorizations of the mRNA COVID-19 vaccines, the FDA assessed manufacturing and nonclinical data to support the change to include the 2024-2025 formula in the mRNA COVID-19 vaccines. The updated mRNA vaccines are manufactured using a similar process as previous formulas of these vaccines. The mRNA COVID-19 vaccines have been administered to hundreds of millions of people in the U.S., and the benefits of these vaccines continue to outweigh their risks.

On an ongoing basis, the FDA will review any additional COVID-19 vaccine applications submitted to the agency and take appropriate regulatory action.

The approval of Comirnaty (COVID-19 Vaccine, mRNA) (2024-2025 Formula) was granted to BioNTech Manufacturing GmbH. The EUA amendment for the Pfizer-BioNTech COVID-19 Vaccine (2024-2025 Formula) was issued to Pfizer Inc.

The approval of Spikevax (COVID-19 Vaccine, mRNA) (2024-2025 Formula) was granted to ModernaTX Inc. and the EUA amendment for the Moderna COVID-19 Vaccine (2024-2025 Formula) was issued to ModernaTX Inc.

Related Information

• Comirnaty (COVID-19 Vaccine, mRNA) (2024-2025 Formula)
• Spikevax (COVID-19 Vaccine, mRNA) (2024-2025 Formula)
• Moderna COVID-19 Vaccine (2024-2025 Formula)
• Pfizer-BioNTech COVID-19 Vaccine (2024-2025 Formula)
• FDA Resources for the Fall Respiratory Illness Season
• Updated COVID-19 Vaccines for Use in the United States Beginning in Fall 2024
• June 5, 2024, Meeting of the Vaccines and Related Biological Products Advisory Committee

The FDA, an agency within the U.S. Department of Health and Human Services, protects the public health by assuring the safety, effectiveness, and security of human and veterinary drugs, vaccines and other biological products for human use, and medical devices. The agency also is responsible for the safety and security of our nation’s food supply, cosmetics, dietary supplements, radiation-emitting electronic products, and for regulating tobacco products.

Evaluating Recommender Systems: Survey and Framework

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, 1 introduction, 2 conceptual basis, 2.1 recommender systems.

2.2 Evaluation of Recommender Systems

3 recommender systems evaluation: a review.

3.1 Evaluation Objectives

Consumer	[ , , , , , ]
Consumer Groups	[ ]
Platform Provider	[ , , ]
Item Provider	[ ]
Multiple Stakeholders	[ , , ]

3.2 Evaluation Design Space: Evaluation Principles

3.3 evaluation design space: experiment type.

Offline	Method: simulation of user behavior based on past interactions
	Task: defined by the researcher, purely algorithmic
	Repeatability: evaluation of an arbitrary number of experiments (e.g., algorithmic settings, models) possible at low cost
	Scale: large dataset, large number of users
	Insights: quantitative, narrow (focused on the predictive performance of algorithms)
User Study	Method: user observation in live or laboratory setting
	Task: defined by the researcher, carried out by the user
	Repeatability: expensive (recruitment of users)
	Scale: small cohort of users
	Insights: quantitative and/or qualitative (live user data, logging of user actions, eye tracking, questionnaires before/during/after task)
Online	Method: real-world user observation, online field experiment
	Task: self-selected by the user, carried out by the user
	Repeatability: expensive (requires full system and users)
	Scale: size of the cohort of users depending on evaluation system and user base
	Insights: quantitative and/or qualitative (live user data, logging of user actions, questionnaires before/during/after exposure to the system)

3.3.1 Offline Evaluation.

3.3.2 user study., 3.3.3 online evaluation., 3.4 evaluation design space: evaluation aspects, 3.4.1 types of data..

Accuracy	High	Low
Abundance	Low	High
Expressivity of user preference	Positive and negative	Positive
Measurement reference	Absolute	Relative

3.4.2 Data Collection.

MovieLens20M [ ]	Movie ratings	20,000,263 ratings; range [0.5,5]
MovieLens1M [ ]	Movie ratings	1,000,209 ratings; range [1,5]
BookCrossing [ ]	Book ratings	1,157,112 ratings; range [1,10]
Yelp	Business ratings	8,021,122 ratings; range [0,5]
MovieTweetings [ ]	Movie ratings	871,272 ratings; range [0,10]

3.4.3 Data Quality and Biases.

3.4.4 evaluation metrics..

Prediction accuracy	Mean absolute error (MAE)	[ , ]
	(Root) Mean squared error ((R)MSE)	[ , ]
Usage prediction	Recall, precision, F-score	[ , ]
	Receiver operating characteristic curve (ROC)	[ ]
	Area under ROC curve (AUC)	[ ]
Ranking	Normalized discounted cumulative gain (NDCG)	[ ]
	Mean reciprocal rank (MRR)	[ ]
Novelty	Item novelty	[ ]
	Global long-tail novelty	[ , ]
Diversity	intra-list similarity (ILS)	[ ]
Coverage	Item coverage	[ , ]
	User space coverage	[ , ]
	Gini index	[ ]
Serendipity	Unexpectedness	[ ]
	Serendipity	[ , ]
Fairness across users	Value unfairness	[ ]
	Absolute unfairness	[ ]
	Over/underestimation of fairness	[ ]
Fairness across items	Pairwise fairness	[ ]
	Disparate treatment ratio (DTR)	[ ]
	Equal expected exposure	[ ]
	Equity of amortized attention	[ ]
	Disparate impact ratio (DIR)	[ ]
	Viable- \(\Lambda\) test	[ ]
Business-oriented	Click-through rate (CTR)	[ , , ]
	Adoption and conversion rate	[ , ]
	Sales and revenue	[ , ]
Articles providing an overview of metrics: [ , , , , , , ].

3.4.5 Evaluation System.

4 mapping a fictitious case to fevr, 4.1 mapping to fevr.

Overall Goal	To evaluate whether users are able to find likable music in the recommendations computed by the novel RecAlg algorithm
Stakeholders	Users of the system (algorithm) Artists may also benefit from an increased item diversity as a more diverse set of artists may be represented
Properties	Item diversity in the recommendations; catalog coverage
Hypothesis/Research Question	\(H_1\) : RecAlg provides users (on average) with more diverse recommendations with respect to the intra-list diversity while maintaining prediction accuracy compared to the baseline algorithm.
Control Variables	Follow accountability framework by Bellogín and Said [ ] (for randomization in dataset splitting to prevent selection bias)
Generalization Power	Limited due to lack of user involvement and dataset biases
Reliability	Follow accountability framework by Bellogín and Said [ ]
	Offline Evaluation with A/B-testing
Types of Data	Implicit ratings (listening events), side information for music tracks
Data Collection	LFM-2b dataset [ ]
Data Quality and Biases	Platform bias, popularity bias, skewed gender distribution, imbalanced country distribution
Evaluation Metrics	Prediction accuracy with RMSE; intra-list similarity in terms of different unique artists
Evaluation System	Existing evaluation framework Elliot [ ]

4.2 Limitations and Discussion

5 discussion, conclusion, and future directions.

• Behare N Jeet D (2024) The Art and Science of User Engagement The Rise of Over-the-Top (OTT) Media and Implications for Media Consumption and Production 10.4018/979-8-3693-0116-6.ch009 (130-159) Online publication date: 26-Jan-2024 https://doi.org/10.4018/979-8-3693-0116-6.ch009
• de Miguel J Velázquez-Iturbide J (2024) What Is an Algorithm? Encyclopedia of Information Science and Technology, Sixth Edition 10.4018/978-1-6684-7366-5.ch072 (1-17) Online publication date: 1-Jul-2024 https://doi.org/10.4018/978-1-6684-7366-5.ch072
• Slade E Rennick-Egglestone S Ng F Kotera Y Llewellyn-Beardsley J Newby C Glover T Keppens J Slade M (2024) The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and Performance JMIR Mental Health 10.2196/45754 11 (e45754) Online publication date: 29-Mar-2024 https://doi.org/10.2196/45754
• Show More Cited By

Index Terms

Human-centered computing

Human computer interaction (HCI)

HCI design and evaluation methods

Information systems

Information retrieval

Evaluation of retrieval results

Retrieval tasks and goals

Recommender systems

Recommendations

A survey of serendipity in recommender systems.

We summarize most efforts on serendipity in recommender systems.We compare definitions of serendipity in recommender systems.We classify the state-of-the-art serendipity-oriented recommendation algorithms.We review methods to assess serendipity in ...

Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols

Exploiting temporal context has been proved to be an effective approach to improve recommendation performance, as shown, e.g. in the Netflix Prize competition. Time-aware recommender systems (TARS) are indeed receiving increasing attention. A wide range ...

Evaluating collaborative filtering recommender systems

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being ...

Information

Published in.

University of Sydney, Australia

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• Framework for EValuating Recommender systems

Funding Sources

• Austrian Science Fund (FWF)

Contributors

Other metrics, bibliometrics, article metrics.

• 57 Total Citations View Citations
• 18,820 Total Downloads
• Downloads (Last 12 months) 13,511
• Downloads (Last 6 weeks) 1,433
• Bertino A (2024) Serendipität und Open Data. Richtlinien für die Implementierung von Empfehlungssystemen auf Entdeckungsplattformen Bibliothek Forschung und Praxis 10.1515/bfp-2023-0059 Online publication date: 1-Jul-2024 https://doi.org/10.1515/bfp-2023-0059
• Fan Y Ji Y Zhang J Sun A (2024) Our Model Achieves Excellent Performance on MovieLens: What Does It Mean? ACM Transactions on Information Systems 10.1145/3675163 Online publication date: 1-Jul-2024 https://dl.acm.org/doi/10.1145/3675163
• Zeng Z Wang Y Zhao Y Shi W (2024) A Survey of Music Recommendation Systems Proceedings of the 5th International Conference on Computer Information and Big Data Applications 10.1145/3671151.3671243 (507-519) Online publication date: 26-Apr-2024 https://dl.acm.org/doi/10.1145/3671151.3671243
• Deffayet R Thonet T Hwang D Lehoux V Renders J de Rijke M (2024) SARDINE: Simulator for Automated Recommendation in Dynamic and Interactive Environments ACM Transactions on Recommender Systems 10.1145/3656481 2 :3 (1-34) Online publication date: 5-Jun-2024 https://dl.acm.org/doi/10.1145/3656481
• Jeunen O Potapov I Ustimenko A Baeza-Yates R Bonchi F (2024) On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 10.1145/3637528.3671687 (1222-1233) Online publication date: 25-Aug-2024 https://dl.acm.org/doi/10.1145/3637528.3671687
• Vrijenhoek S Daniil S Sandel J Hollink L (2024) Diversity of What? On the Different Conceptualizations of Diversity in Recommender Systems Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency 10.1145/3630106.3658926 (573-584) Online publication date: 3-Jun-2024 https://dl.acm.org/doi/10.1145/3630106.3658926
• Bauer C Zangerle E Said A (2024) Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives ACM Transactions on Recommender Systems 10.1145/3629170 2 :1 (1-31) Online publication date: 7-Mar-2024 https://dl.acm.org/doi/10.1145/3629170

View options

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

NTRS - NASA Technical Reports Server

Available downloads, related records.

Blog The Education Hub

https://educationhub.blog.gov.uk/2024/08/20/gcse-results-day-2024-number-grading-system/

GCSE results day 2024: Everything you need to know including the number grading system

Thousands of students across the country will soon be finding out their GCSE results and thinking about the next steps in their education.   

Here we explain everything you need to know about the big day, from when results day is, to the current 9-1 grading scale, to what your options are if your results aren’t what you’re expecting.  

When is GCSE results day 2024?  

GCSE results day will be taking place on Thursday the 22 August.     

The results will be made available to schools on Wednesday and available to pick up from your school by 8am on Thursday morning.  

Schools will issue their own instructions on how and when to collect your results.   

When did we change to a number grading scale?  

The shift to the numerical grading system was introduced in England in 2017 firstly in English language, English literature, and maths.  

By 2020 all subjects were shifted to number grades. This means anyone with GCSE results from 2017-2020 will have a combination of both letters and numbers.  

The numerical grading system was to signal more challenging GCSEs and to better differentiate between students’ abilities - particularly at higher grades between the A *-C grades. There only used to be 4 grades between A* and C, now with the numerical grading scale there are 6.  

What do the number grades mean?  

The grades are ranked from 1, the lowest, to 9, the highest.  

The grades don’t exactly translate, but the two grading scales meet at three points as illustrated below.  

The bottom of grade 7 is aligned with the bottom of grade A, while the bottom of grade 4 is aligned to the bottom of grade C.    

Meanwhile, the bottom of grade 1 is aligned to the bottom of grade G.  

What to do if your results weren’t what you were expecting?  

If your results weren’t what you were expecting, firstly don’t panic. You have options.  

First things first, speak to your school or college – they could be flexible on entry requirements if you’ve just missed your grades.   

They’ll also be able to give you the best tailored advice on whether re-sitting while studying for your next qualifications is a possibility.   

If you’re really unhappy with your results you can enter to resit all GCSE subjects in summer 2025. You can also take autumn exams in GCSE English language and maths.  

Speak to your sixth form or college to decide when it’s the best time for you to resit a GCSE exam.  

Look for other courses with different grade requirements     

Entry requirements vary depending on the college and course. Ask your school for advice, and call your college or another one in your area to see if there’s a space on a course you’re interested in.    

Consider an apprenticeship    

Apprenticeships combine a practical training job with study too. They’re open to you if you’re 16 or over, living in England, and not in full time education.  

As an apprentice you’ll be a paid employee, have the opportunity to work alongside experienced staff, gain job-specific skills, and get time set aside for training and study related to your role.   

You can find out more about how to apply here .  

Talk to a National Careers Service (NCS) adviser    

The National Career Service is a free resource that can help you with your career planning. Give them a call to discuss potential routes into higher education, further education, or the workplace.   

Whatever your results, if you want to find out more about all your education and training options, as well as get practical advice about your exam results, visit the  National Careers Service page  and Skills for Careers to explore your study and work choices.   

You may also be interested in:

• Results day 2024: What's next after picking up your A level, T level and VTQ results?
• When is results day 2024? GCSEs, A levels, T Levels and VTQs

Tags: GCSE grade equivalent , gcse number grades , GCSE results , gcse results day 2024 , gsce grades old and new , new gcse grades

Sharing and comments

Share this page, related content and links, about the education hub.

The Education Hub is a site for parents, pupils, education professionals and the media that captures all you need to know about the education system. You’ll find accessible, straightforward information on popular topics, Q&As, interviews, case studies, and more.

Please note that for media enquiries, journalists should call our central Newsdesk on 020 7783 8300. This media-only line operates from Monday to Friday, 8am to 7pm. Outside of these hours the number will divert to the duty media officer.

Members of the public should call our general enquiries line on 0370 000 2288.

Sign up and manage updates

Follow us on social media, search by date.

August 2024

M	T	W	T	F	S	S
			1	2	3	4
5		7	8	9	10	11
	13	14	15	16	17	18
		21	22	23	24	25
26	27		29		31

Comments and moderation policy

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 28 August 2024

Carbon emissions from the 2023 Canadian wildfires

• Brendan Byrne   ORCID: orcid.org/0000-0003-0619-3045 1 ,
• Junjie Liu   ORCID: orcid.org/0000-0002-7184-6594 1 , 2 ,
• Kevin W. Bowman 1 , 3 ,
• Madeleine Pascolini-Campbell 1 ,
• Abhishek Chatterjee   ORCID: orcid.org/0000-0002-3680-0160 1 ,
• Sudhanshu Pandey 1 ,
• Kazuyuki Miyazaki   ORCID: orcid.org/0000-0002-1466-4655 1 ,
• Guido R. van der Werf   ORCID: orcid.org/0000-0001-9042-8630 4 ,
• Debra Wunch 5 ,
• Paul O. Wennberg   ORCID: orcid.org/0000-0002-6126-3854 2 , 6 ,
• Coleen M. Roehl   ORCID: orcid.org/0000-0001-5383-8462 2 &
• Saptarshi Sinha   ORCID: orcid.org/0009-0009-3308-0311 7  

Nature ( 2024 ) Cite this article

453 Altmetric

Metrics details

• Carbon cycle

The 2023 Canadian forest fires have been extreme in scale and intensity with more than seven times the average annual area burned compared to the previous four decades 1 . Here, we quantify the carbon emissions from these fires from May to September 2023 on the basis of inverse modelling of satellite carbon monoxide observations. We find that the magnitude of the carbon emissions is 647 TgC (570–727 TgC), comparable to the annual fossil fuel emissions of large nations, with only India, China and the USA releasing more carbon per year 2 . We find that widespread hot–dry weather was a principal driver of fire spread, with 2023 being the warmest and driest year since at least 1980 3 . Although temperatures were extreme relative to the historical record, climate projections indicate that these temperatures are likely to be typical during the 2050s, even under a moderate climate mitigation scenario (shared socioeconomic pathway, SSP 2–4.5) 4 . Such conditions are likely to drive increased fire activity and suppress carbon uptake by Canadian forests, adding to concerns about the long-term durability of these forests as a carbon sink 5 , 6 , 7 , 8 .

Similar content being viewed by others

Multi-decadal increase of forest burned area in Australia is linked to climate change

Projected increases in western US forest fire despite growing fuel constraints

Fuel availability not fire weather controls boreal wildfire severity and carbon emissions

Canadian forests cover a vast area of nearly 362 million ha (ref. 9 ), amounting to 8.5% of the global forested area 10 . These forests are an important sink of carbon, absorbing fossil carbon dioxide (CO 2 ) from the atmosphere and slowing the pace of climate warming 11 , 12 . However, climate change is increasing forest fire activity, acting to suppress the carbon uptake capacity of these forests 13 . Although more frequent fires have been widespread, 2023 has seen forest fires on an extreme scale. With 15 million ha of Canadian forests burned (about 4% of forest area) 1 , 2023 saw more than seven times (8  σ ) the average burned area over the preceding 40 years (1983–2022 mean, 2.2 million ha; range, 0.2–7.1 million ha) 1 . The adverse societal impacts of these fires are clear: 232,000 evacuations and poor air quality affecting millions 14 . However, the carbon emissions from the fire events remain uncertain. In this study, we quantify these emissions through inverse modelling of satellite observations of carbon monoxide (CO). Then, we examine concurrent climate anomalies and projected changes in the prevalence of hot–dry weather under climate change. Finally, we discuss the implications of fires for the Canadian carbon budget.

Fire emissions

Fire carbon emissions can be tracked from space using bottom-up and top-down approaches. Bottom-up approaches use satellite observations to track fire activity, such as burned area 15 or fire radiative power 16 . Emissions of CO 2 , CO and other trace gases are then estimated by combining the estimates of fire activity with quantities such as fuel loads and emission factors. Although these bottom-up estimates are continually improving, inventories can vary significantly in global and regional trace gas and aerosol emission estimates 15 , 17 . Top-down approaches provide a method for refining bottom-up trace gas emission estimates by optimally scaling emission estimates to be consistent with the observed concentrations of trace gases in fire plumes. A strength of this approach is that it integrates emissions from both flaming and smouldering combustion to capture net emissions.

In this study, we perform top-down estimates of CO emissions from the 2023 Canadian fires based on observational constraints from the TROPOspheric monitoring instrument (TROPOMI) space-based CO retrievals (Fig. 1a,b ). These estimates are performed using three different bottom-up fire emission inventories: the global fire emissions database (GFED4.1s) 15 , the global fire assimilation system v.1.2 (GFAS) 16 and the quick fire emissions dataset v.2.6r1 (QFED) 18 . For each inversion, the combined carbon emissions released as CO and CO 2 (CO 2  + CO) are then estimated using the CO 2 /CO emission factors from the same bottom-up database. The CO 2 /CO emission ratios can be highly variable, adding uncertainty to our analysis. We incorporate some of this uncertainty here as each bottom-up database has different mean emission ratios for Canadian forests (range, 7.7–10.8 gC of CO 2 per gC of CO 2 ). Details for these inversions are provided in the methods and a description of the inversion results and evaluation of the performance of the top-down estimates are provided in Supplementary Information sections 1 and 2 ). We find the top-down estimates are relatively insensitive to choices about inversion configuration but do show sensitivity to prescribed hydroxyl radical (OH) abundances 19 , which determine the atmospheric lifetime of the CO emitted (Supplementary Information section 1 and Supplementary Fig. 1 ).

a – c , May–September TROPOMI dry-air mole fractions of CO ( X CO ) averaged over 2019–2022 ( a ) and for 2023 ( b ) aggregated to a 2° × 2.5° grid. c , Canadian forest fire carbon emissions (from CO and CO 2 ) for the 2023 May–September fire season, compared with fire emissions during 2010–2022 (distribution shown by box-and-whisker plots). Top-down emissions over 2010–2022 are estimated from MOPITT (2010–2021) and TROPOMI (2019–2022) CO retrievals. d , A comparison of May–September Canadian fire emissions with 2022 territorial fossil carbon emissions for the ten largest emitting countries, obtained from Global Carbon Budget 2022 2 .

Figure 1c shows the bottom-up and top-down CO 2  + CO carbon emissions from fires during May–September 2023. The bottom-up datasets show large differences, ranging from 234 to 735 TgC (mean of 469 TgC). This range is reduced by 69% in the top-down estimates (570–727 TgC), which also give a larger mean estimate of 647 TgC. Emissions during 2023 far exceed typical Canadian forest fire emissions, with 2010–2022 average emissions of 29–82 TgC for the bottom-up inventories and 121 TgC for top-down estimates (Supplementary Fig. 2 ). To contextualize these numbers, we compare the top-down estimates to annual national fossil fuel emissions for the ten largest emitters (Fig. 1d ). The 5 month 2023 emissions are more than four times larger than Canadian annual fossil fuel emissions (149 TgC yr −1 ) and comparable to India’s annual emissions (740 TgC yr −1 ).

Fire activity is affected by several complex drivers, including fuel traits 20 and ignition probability 21 . However, fire weather—hot and dry conditions—has been shown to be extremely important in driving fire behaviour 22 . Climate data show an exceptionally hot and dry fire season for Canadian forests during 2023 (Fig. 2 ). This was the driest January–September period for Canadian forests since at least 1980, with about 86% of forested area having below-average precipitation and about 52% being more than 1 s.d. below the 2003–2022 average (Supplementary Fig. 4 ). May–September 2023 was the warmest since at least 1980, with about 100% of the forest area above average and about 90% being more than 1 s.d. above the 2003–2022 average. Similarly, the vapour pressure deficit (VPD), which is closely associated with fire activity 22 , 23 , 24 , was the third highest since 1980, including 85% of the forest area being above average and about 54% being more than 1  s.d. above the 2003–2022 average.

a – d , Maps (left) and time series (right) of CPC global unified gauge-based cumulative precipitation (∑P) ( a ), MERRA-2 2 m temperature (with 2 week running mean) ( b ), MERRA-2 VPD (with 2 week running mean) ( c ) and fire CO 2  + CO emissions from the GFED4.1s database ( d ). All maps are shown at a spatial resolution of 0.5° × 0.625° and Z -scores are for the area-mean of Canadian forests. Note that GFED4.1s is shown instead of the inversion results because those are at a coarser spatial resolution and cover a shorter time period, maps of prior and posterior mean fire emissions are shown in Supplementary Fig. 14 . Months are shown from January (J) to December (D).

Although hot–dry conditions were widespread across Canadian forests, there are two notable regional patterns. Western Quebec (49°–55° N, 72°–80° W), which is typically relatively wet (Supplementary Fig. 5a ), had exceptionally dry conditions during 2023, with precipitation through September being 23.7 cm (49%) below average. Coupled with extreme heat and VPD during June–July, fire emissions in this region contributed about 15% of the national total (Supplementary Fig. 6 ). The other notable region was northwestern Canada near the Great Slave Lake (57°–62° N, 110°−125° W). This region is drier than western Quebec on average, with about half the annual precipitation. However, 2023 was exceptional, with both a large precipitation deficit of 8.1 cm (27% of January–September total) and exceptionally warm conditions throughout May–September (+2.6 °C) (Supplementary Fig. 6 ). This region contributed about 61% of the total Canadian forest fire emissions.

Fires and climate

The relationship between climate variability and fire emissions for Canadian forests is examined in Fig. 3 , which shows fire emissions as a function of temperature and precipitation Z -scores over 2003–2023 for the 0.5° × 0.625° grid cells, in which Z -scores are the anomalies divided by the standard deviation. May–September emissions are lowest for combined cool–wet conditions (5.2 gC m −2 ), whereas emissions increase when either temperature is above average (19.5 gC m −2 ) or precipitation is below average (9.2 gC m −2 ). However, emissions are largest for combined warm–dry conditions (35.7 gC m −2 ). In particular, fire emissions are much increased during exceptionally hot and dry conditions (99.6 gC m −2 , temperature Z  > 1 and precipitation Z  < −1). These hot–dry conditions were much more prevalent in 2023 than in preceding years, with a mean May–September T2M Z -score of 2.3 and a precipitation Z -score of −1.1 across grid cells, explaining why fire emissions were extreme during 2023. Notably, the number of individual fires during 2023 was not unusual, with 6,623 relative to a 10 yr average of 5,597 (ref. 25 ). Yet, probably primarily driven by these hot–dry conditions 24 , many of these fires grew to enormous sizes with hundreds of megafires (greater than 10,000 ha) recorded.

Mean May–September GFED4.1s CO 2  + CO fire emissions as a function of May–September T2M Z -score and January–September precipitation Z -score for each 0.5° × 0.625° forested grid cell during 2003–2023 (using 2003–2022 as a baseline). For individual years, the mean Z -scores across forested grid cells are shown with ‘X’. The projected decadal-mean temperature and precipitation Z -scores for the median CMIP6 model under SSP 2–4.5 are shown by the circles. The CMIP6 Z -scores are calculated using the 2000–2019 period as a baseline but use the reanalysis 2003–2022 standard deviations (see section on ‘Climate data’). The historical and projected T2M and precipitation over the Canadian boreal forests simulated by the CMIP6 ensemble are shown in Supplementary Fig. 12 .

Next, we examine future climate conditions in the region and how they compare to the concurrent climate conditions that led to the massive fires. Figure 3 shows the decadal mean temperature and precipitation Z -scores for the median of 27 models from the coupled model intercomparison project phase 6 (CMIP6) 26 under the moderate-warming shared socioeconomic pathway (SSP) 2–4.5 (ref. 4 ). Large projected temperature increases are found to occur, with average temperatures in the 2050s similar to 2023. More modest increases in precipitation are projected, indicating a ‘speeding up’ of the water cycle, in which both evaporation and precipitation rates increase (Supplementary Fig. 12 shows ensemble distribution). Studies indicate that the combined effect will result in regional increases in moisture deficits for Canadian forests through the end of the twenty-first century 6 , 27 , 28 . Beyond the 2050s, average temperature and precipitation conditions are projected to exceed the historical range. These changes will impact the boreal carbon cycle in many ways, such as changing fuel loads and species composition, which complicates projections of future fire activity. However, increases in boreal fire emissions linked to warming have been reported over recent decades 13 , 27 , 29 , 30 and several studies have projected further increases in Canadian fire activity with future warming 5 , 6 , 7 , 8 . Thus, we find that warming, coupled with regionally increasing moisture deficits, is likely to drive increased fire carbon emissions from Canadian forests.

Canadian carbon budget implications

As a party to the Paris Agreement, Canada is obligated to track economy-wide greenhouse gas (GHG) emissions and removals in a national GHG inventory (NGHGI). This includes tracking emissions and removals from ‘managed’ lands, for which human interventions and practices have been applied to perform production, ecological or social functions 31 . However, the 2006 Intergovernmental Panel on Climate Change (IPCC) guidelines for national GHG inventories 31 and Canadian NGHGI 32 differ in how emissions and removals over managed lands are categorized. The IPCC guidelines treat all emissions and removals on managed land as anthropogenic, whereas the Canadian NGHGI treats ‘natural disturbances’ as non-anthropogenic. This difference in categorization leads to large differences between the Canadian NGHGI and an estimate using the IPCC guideline definitions.

Figure 4 shows that NGHGI removals on managed forest land are almost exactly compensated by emissions from harvested wood products, such that the total CO 2 emissions for Canada are dominated by the energy sector (more than 90% of net emissions). However, we see that natural disturbances are shown to be of considerable magnitude, amounting to nearly 60% of total CO 2 emissions in 2021. The 2023 CO + CO 2 fire emissions across managed Canadian forests (see section on ‘Managed land’) are estimated to be 421 (388–461) TgC, amounting to 2.5–3 years of economy-wide CO 2 emissions.

Lines show the annual net emissions or removals from managed forest land (green), harvested wood products (brown), natural disturbances that are not counted towards Canada’s emissions (red) and the economy-wide net CO 2 emissions (grey). The top-down estimates of the 2023 CO 2  + CO fires emissions over managed land are shown in black. Total CO 2 emissions, harvested wood products and forest land emissions and removals were obtained from Table A11-1 of the NGHGI 32 , whereas natural disturbances were obtained from Table 6-5 of the NGHGI. All quantities presented are in units of teragrams of carbon (1 TgC = 1 MtC = 1,012 gC), which can be converted to units of megatonnes of CO 2 (MtCO 2 ) by multiplying by a factor of 3.664.

Regardless of their characterization, fire carbon emissions will affect the growth rate atmospheric CO 2 . As such, monitoring changes in the carbon budget across both managed and unmanaged land is important. Including all land in the Canadian carbon budget, top-down estimates find that Canadian ecosystems are a sink of CO 2 when constrained by either in situ or space-based CO 2 observations. Using both data types, an ensemble of atmospheric CO 2 inversion systems report that Canadian carbon stocks increased 366 ± 88.6 TgC yr −1 over 2015–2020 11 , contributing about 30% of the net land carbon sink. Similarly, space-based biomass estimates find carbon accumulation in Canadian boreal forests, although smaller in magnitude 13 , 33 , 34 . Thus, Canadian forests play an important role in mitigating anthropogenic emissions, slowing the rise of atmospheric CO 2 . The large carbon release resulting from the 2023 Canadian fires puts into question the durability of this sink. Others 13 showed that fires have acted to suppress the carbon uptake potential of Canadian forests over the past 30 years. Although Canadian forests have historically experienced large stand-replacing fires at infrequent intervals of 30 to more than 100 years 35 , 36 , 37 , increases in fire frequency will probably reduce biomass recovery and affect species composition 37 , 38 , 39 , 40 . It has also been argued that fire, insects and droughts may already be driving Canadian forests into a carbon source 41 , 42 . In the extreme case that expansive fires, such as that of 2023, become the norm (burning 4% of Canadian forest area), all Canadian forests could burn every 25 years. So, although the magnitude is uncertain, it is likely that increasing fire activity in Canadian forests will reduce the capacity of these Canadian forests to continue to act as a carbon sink.

The role of Canada’s fire management strategy in managing fire carbon emissions also deserves some discussion. Fire management strategies require balancing several considerations, including socioeconomic costs, ecological impacts and carbon emissions. Canada’s present strategy adopts a risk-based approach, for which decisions on whether or not to suppress fires are made on a fire-by-fire basis 43 , with differing priorities across provinces and territories. Understanding how fire regimes will change with climate change is thus of high importance, for future decision criteria and costing.

Conclusions

The 2023 fire season was the warmest and driest for Canadian forests since at least 1980, resulting in vast carbon emissions from forest fires. Using TROPOMI CO retrievals, we estimate the total May–September CO 2  + CO emissions from these fires to be 647 TgC (range 570–727 TgC), comparable in magnitude to India’s annual fossil fuel CO 2 emissions. The 2023 warmth was exceptional based on the last 44 years but CMIP6 climate models project that the temperatures of 2023 will become normal by the 2050s. Such changes are likely to increase fire activity 5 , 6 , 7 , 8 , risking the carbon uptake potential of Canadian forests. This will impact allowable emissions for reaching warming targets, as reduced carbon sequestration by ecosystems must be compensated for by adjusting anthropogenic emissions reductions.

Climate data

Precipitation estimates were derived from Climate Prediction Center (CPC) global unified gauge-based analysis of daily precipitation data provided by the National Oceanic and Atmospheric Administration from their website at https://psl.noaa.gov (refs. 44 , 45 ). MERRA-2 2 m temperature (T2M) and dew point temperature at 2 m (T2MDEW) were obtained from the single-level diagnostics file 3 . VPD was calculated from these quantities using:

where es is the saturation vapour pressure and ea is the vapour pressure, calculated from T2MDEW and T2M, respectively, using the formulation of ref. 46 . The Z -scores for precipitation, T2M and VPD were calculated relative to the 20-year baseline of 2003–2022; for example, for T2M this is calculated as:

where T2M year is the May–September mean T2M for a given year and T2M 2003–2022 is the 20-element ensemble of May–September mean T2Ms during 2003–2022.

CMIP6 data were downloaded from the Canadian Climate Data and Scenarios website ( https://climate-scenarios.canada.ca/?page=cmip6-scenarios ). We examine the ensemble median of 27 models provided on a 1° × 1° grid (technical documentation at https://climate-scenarios.canada.ca/?page=pred-cmip6-notes ). The models included are based on data availability and are tabulated at https://climate-scenarios.canada.ca/?page=cmip6-model-list . T2M and precipitation are analysed for the historical and future scenarios. We combine the historical simulation with SSP 2–4.5 (ref. 4 ), shown in the main text or SSP 5–8.5 as shown in Supplementary Fig. 7 .

The Z -scores are calculated from the median of the CMIP6 ensemble by calculating the temporal mean of the median model mean over 2000–2019 for a given grid cell, whereas the reanalysis data are used to estimate internal variability. Therefore, the T2M Z -score is calculated as:

Sources and sinks

Fossil co emissions.

Anthropogenic CO emissions are obtained from the community emissions data system (CEDS) for historical emissions 47 ; specifically we use version CEDS-2021-04-21 (ref. 48 ).

Prior fire CO 2 and CO emissions

Fire CO 2 and CO emissions are obtained from the GFED, GFAS and QFED databases. GFED4.1s 15 provides estimates of biomass burning using a biogeochemical model ingesting MODIS 500 m burned area 49 in combination with 1 km thermal anomalies and 500 m surface reflectance observations to estimate burned area associated with small fires using a statistical model 50 . These data were downloaded from https://www.globalfiredata.org/ . GFAS v.1.2 provides estimates of daily biomass burning emissions by assimilating MODIS fire radiative power observations 16 . These data were downloaded from the atmosphere data store ( https://ads.atmosphere.copernicus.eu ). We use v.2.6 of the QFED gridded emission estimates 18 . These data were downloaded from https://portal.nccs.nasa.gov/datashare/iesa/aerosol/emissions/QFED/v2.6r1/0.25/QFED/ . For all biomass burning datasets, we release fire emissions at the model surface but incorporate a 3 hourly diurnal cycle based on ref. 51 . Year-specific emissions are used for the prior in the atmospheric CO inversions.

Biogenic emissions, atmospheric CO production and OH data

Biogenic emissions, atmospheric CO production and OH data were all derived from the outputs of the MOMO-Chem chemical data assimilation 52 . An updated version of the tropospheric chemistry reanalysis v.2 (TCR-2) 53 produced using MOMO-Chem is used to evaluate the atmospheric production and loss of CO. The reanalysis is produced through the assimilation of several satellite measurements of ozone, CO, NO 2 , HNO 3 and SO 2 . The chemical loss of CO was estimated using the reanalysis OH fields. Because of the multiconstituent data assimilation, the reanalysis OH shows improved agreements in global distributions over remote oceans in comparison with the ATom aircraft measurements from the surface to the upper troposphere 53 . Constraints obtained for OH profiles have a large potential to influence the chemistry of the entire troposphere, including oxidation of non-methane hydrocarbons (NMHCs) to estimate the chemical production of CO. The biogenic emissions at the surface were obtained from the model of emissions of gases and aerosols from Nature v.2.1 (MEGAN2.1) 54 . Year-specific fields were only available through 2018 and estimates for that year are repeated for more recent years. We also perform a supplemental sensitivity analysis for the impact of prescribed OH abundances on inferred emissions using the fields of ref. 55 , which are commonly used for GEOS-Chem methane inversions 56 .

CO retrievals

TROPOMI is a grating spectrometer aboard the ESA Sentinel-5 Precursor (S5P) satellite which measures Earth-reflected radiances 57 . CO total column densities are retrieved in the shortwave infrared (around 2.3 μm) using the shortwave infrared CO retrieval algorithm 58 , 59 . TROPOMI CO retrievals 60 were downloaded from the Copernicus data space ecosystem ( https://dataspace.copernicus.eu/ ). We use S5P RPRO L2 CO (processor v.2.4.0) through 25 July 2022, then switch to S5P OFFL L2 CO for more recent data (processor v.2.5.0 or 2.5.0). Retrieved CO total column densities are then converted to dry-air mole fractions of CO ( X CO ) using the dry-air surface pressure and hypsometric equation. The column averaging kernel is similarly converted to mole-fraction space. Individual retrievals (quality flag ≥ 0.5) from each orbit are aggregated into super-observations using the model grid (2° × 2.5°).

The retrieval uncertainty on super-observations is taken to be the mean uncertainty on all retrievals in a given super-observation. This approach is used because systematic errors may exist between retrievals, such that assuming random errors would underestimate the true retrieval error. For assimilation into NASA carbon monitoring system-flux (CMS-Flux), we calculate observational errors that incorporate error in the atmospheric transport model. For this, we follow the approach of ref. 61 . First, we perform a forward model simulation with the prior fluxes for 2019–2023. Then we take the observational uncertainty to be the standard deviation between the simulated and real TROPOMI super-observations over a moving window of 30° latitude, 30° longitude and 30 days (across all years). The uncertainties estimated using this approach range over 3.5–14.3 ppb (5–95 percentiles), whereas retrieval errors range over 1.4–4.9 ppb. Thus, the observational errors are dominated by representativeness errors.

We use the MOPITT (measurements of pollution in the troposphere) satellite thermal-infrared–near-infrared (TIR–NIR) CO retrieval. Version 9 (L2V19.9.3) 62 is used from 2009 to 31 October 2022, whereas L2V19.10.3.beta is used from 1 November 2022 onwards. These data were downloaded from the EarthData ASDC ( https://asdc.larc.nasa.gov/data/MOPITT/ ). As with TROPOMI, profile retrievals were converted into dry-air mole fractions of CO ( X CO ) for assimilation; however, unlike TROPOMI, we do not generate super-observations but instead assimilate individual observations. This is because the footprint of MOPITT retrievals (22 × 22 km 2 ) is much coarser than TROPOMI retrievals (3 × 7 km 2 ).

The total carbon column observing network (TCCON) consists of ground-based Fourier transform spectrometers which retrieve X CO , X CO 2 and other species from observations of solar radiation 63 . In this study, we examine GGG2020 (ref. 64 ) TCCON data from Park Falls 65 and East Trout Lake 66 . These data were obtained from the TCCON Data Archive hosted by CaltechDATA at https://tccondata.org . Super-observations are created for each site as hourly averages; we only include hours with five or more observations.

Atmospheric CO inversions

We perform a series of CO inversion analyses using the CMS-Flux atmospheric inversion system. This inversion model is descended from the GEOS-Chem adjoint model 67 and has been used for CO 2 (refs. 68 , 69 ) and CO inversion analyses 70 . The inversions in this study are all performed globally at 2° × 2.5° spatial resolution using MERRA-2 reanalysis. CEDS anthropogenic emissions, biogenic atmospheric CO production, direct biogenic CO emissions and fire emissions (from GFED4.1s, GFAS or QFED) and atmospheric OH fields are all prescribed in the forward simulations (see section on ‘Sources and sinks’). Four-dimensional variational data assimilation (4D-Var) is used to optimize scaling factors on the net surface flux for each grid cell (combined anthropogenic, fire and direct biogenic CO flux). The posterior CO fluxes are then decomposed into anthropogenic, fire and biogenic fluxes using the fractional contribution of the prior (an approach widely used for CO inversions).

A series of MOPITT X CO inversions are performed over 2010–2021. Weekly fluxes are optimized over the period 7 November of the preceding year (YYYY − 1) to 1 February of the next year (YYYY + 1), the optimized fluxes in the desired year are retained (YYYY) and the fluxes outside this period are discarded as spin-up or spin-down. These inversions are performed using the GFED4.1s fire inventory. Prior uncertainties on emissions are assumed to be proportional to the emissions, with a scale factor uncertainty of 200%.

TROPOMI X CO inversions are performed over 2019–2023. These inversions are performed over a truncated period of 1 April to 30 September, with April then being discarded as spin-up. Several different inversion configurations are used to quantify the uncertainty in posterior fluxes due to both Bayesian posterior uncertainties and systematic choices about error specification and inversion configuration, both of which have been shown to contribute significantly to inversion error estimates 11 .

Three ensembles of inversions are performed on the basis of the three different prior fire inventories: GFED4.1s, GFAS or QFED (Extended Data Fig. 1a ). Each prior inventory was subjected to four different experimental configurations (Extended Data Fig. 1b ). In one case, the X CO super-observations error is taken to be the mean retrieval uncertainty across all retrievals included in a given super-observation. This approach typically gives an uncertainty of 1.3–4.9 ppb. The other case uses an observational error estimate that incorporates representativeness errors (see section on ‘TROPOMI’), which are typically between 3.5 and 14.3 ppb. The experimental configurations also differ by the treatment of prior uncertainties on the fluxes. These uncertainties are not well known a priori, thus we use two very different approaches. In the first approach, we assume that the errors on fluxes are equal to 200% of the prior flux estimate. In the second approach, we assume that flux uncertainties are near constant in flux units (scale factor uncertainty times control flux is constant, this is truncated to scale factors uncertainties between 0.25 and 1,000). Finally, we also vary the temporal optimization to either 3 or 7 days. As with the prior flux uncertainties, there are many possible choices for temporal optimization, so we choose two reasonable estimates to quantify the sensitivity to this choice. The spread in maximum a posteriori estimates across these different set-ups gives an indication of the uncertainty in estimated fluxes due to the set-up decisions.

We also estimate the Bayesian posterior uncertainty (Extended Data Fig. 1c ), which derives from uncertainties in the prior fluxes and observations. This uncertainty is estimated using the Monte Carlo method introduced by ref. 71 and formalized by ref. 72 We perform the experiment during 2023 for each prior inventory and use 40 inversion ensemble members using the inversion configuration with TROPOMI X CO representativeness errors and 7 day optimization.

Finally, for each prior inventory, we calculate the posterior best estimates and uncertainties from the experiments described above (Extended Data Fig. 1d ). The best estimate is taken to be the mean across the four different inversion configurations. The uncertainty on this estimate is taken as the square-root of the sum of the variances resulting from the different inversion configurations and Monte Carlo posterior covariance estimate. The overall best estimate is taken to be the average across the best estimates for the prior inventory ensembles and the overall uncertainty is taken to be the range of 1  σ uncertainties across the three prior inventory ensembles.

We estimate posterior CO 2 fluxes from the posterior CO emissions using the CO 2 /CO emission ratios provided by the prior GFED4.1s, GFAS and QFED inventories. Each inventory has different CO 2 /CO emission; thus, we use the emission ratio to estimate the posterior CO 2 from the same inventory that was used as the prior inventory. This incorporates some uncertainty CO 2 /CO emission ratio into the CO 2 emission estimates.

Regional masks

Forest area.

Forest area is defined using v.6.1 of the MODIS MCD12C1 product 73 . On the basis of the type 1 majority land cover, we define forests to include the categories evergreen needleleaf forests, evergreen broadleaf forests, deciduous needleleaf forests, deciduous broadleaf forests, mixed forests, woody savannas and savannas.

Managed land

The map of managed lands 74 was accessed through personal communication with M. Hafer and A. Dyk (the map was only created for cartographic communication purposes). The extent of land considered managed forest in Canada for the purposes of GHG reporting to the United Nations Framework Convention on Climate Change cannot be mapped in detail. That information comes from provincial/territorial forest inventories that are not spatially explicit and cannot be mapped. Supplementary Fig. 13 shows the managed land map and the fractional managed/unmanaged for 2° × 2.5° grid cells.

Data availability

The dataset produced for this study can be accessed at JPL Open Repository, https://doi.org/10.48577/jpl.V5GR9F .

Code availability

The Python and Bash codes used in this study are available at Zenodo ( https://doi.org/10.5281/zenodo.12709398 ) 75 .

Jain, P. et al. Canada under fire—drivers and impacts of the record-breaking 2023 wildfire season. ESS Open Archive https://doi.org/10.22541/essoar.170914412.27504349/v1 (2024).

Friedlingstein, P. et al. Global carbon budget 2022. Earth Syst. Sci. Data 14 , 4811–4900 (2022).

Article   ADS   Google Scholar  

MERRA-2 tavg1 2d slv Nx: 2d,1-Hourly, Time-Averaged, Single-Level, Assimilation, Single-Level Diagnostics V5.12.4 (GSFC DAAC, accessed 24 October 2023); https://doi.org/10.5067/VJAFPLI1CSIV .

Meinshausen, M. et al. The shared socio-economic pathway (SSP) greenhouse gas concentrations and their extensions to 2500. Geosci. Model Dev. 13 , 3571–3605 (2020).

Article   ADS   CAS   Google Scholar  

Wang, X. et al. Projected changes in fire size from daily spread potential in Canada over the 21st century. Environ. Res. Lett. 15 , 104048 (2020).

Kitzberger, T., Falk, D. A., Westerling, A. L. & Swetnam, T. W. Direct and indirect climate controls predict heterogeneous early-mid 21st century wildfire burned area across western and boreal North America. PLoS ONE 12 , 0188486 (2017).

Article   Google Scholar  

Nitschke, C. R. & Innes, J. L. Climatic change and fire potential in south-central British Columbia, Canada. Glob. Change Biol. 14 , 841–855 (2008).

Phillips, C. A. et al. Escalating carbon emissions from North American boreal forest wildfires and the climate mitigation potential of fire management. Sci. Adv. 8 , 7161 (2022).

The State of Canada’s Forests Annual Report 2022 (Government of Canada, 2022); https://natural-resources.canada.ca/our-natural-resources/forests/state-canadas-forests-report/16496 .

Global Forest Resources Assessment 2020: Main Report (FAO, 2020); https://doi.org/10.4060/ca9825en .

Byrne, B. et al. National CO 2 budgets (2015–2020) inferred from atmospheric CO 2 observations in support of the global stocktake. Earth Syst. Sci. Data 15 , 963–1004 (2023).

Deng, Z. et al. Comparing national greenhouse gas budgets reported in UNFCCC inventories against atmospheric inversions. Earth Syst. Sci. Data 14 , 1639–1675 (2022).

Wang, J. A., Baccini, A., Farina, M., Randerson, J. T. & Friedl, M. A. Disturbance suppresses the aboveground carbon sink in North American boreal forests. Nat. Clim. Change 11 , 435–441 (2021).

O’Neill, N. & Otis, D. Military deploys 350 soldiers to Northwest Territories, 68 per cent of population evacuated. CTV News (22 August 2023).

Werf, G. R. et al. Global fire emissions estimates during 1997–2016. Earth Syst. Sci. Data 9 , 697–720 (2017).

Kaiser, J. W. et al. Biomass burning emissions estimated with a global fire assimilation system based on observed fire radiative power. Biogeosciences 9 , 527–554 (2012).

Pan, X. et al. Six global biomass burning emission datasets: intercomparison and application in one global aerosol model. Atmos. Chem. Phys. 20 , 969–994 (2020).

Darmenov, A. S. & Silva, A. M. The Quick Fire Emissions Dataset (QFED): Documentation of Versions 2.1, 2.2 and 2.4 (NASA Global Modeling and Assimilation Office, accessed 10 November 2023); https://gmao.gsfc.nasa.gov/pubs/docs/Darmenov796.pdf .

Voulgarakis, A. et al. Analysis of present day and future OH and methane lifetime in the ACCMIP simulations. Atmos. Chem. Phys. 13 , 2563–2587 (2013).

Varner, J. M., Kane, J. M., Kreye, J. K. & Engber, E. The flammability of forest and woodland litter: a synthesis. Curr. For. Rep. 1 , 91–99 (2015).

Veraverbeke, S. et al. Lightning as a major driver of recent large fire years in North American boreal forests. Nat. Clim. Change 7 , 529–534 (2017).

Williams, A. P. et al. Observed impacts of anthropogenic climate change on wildfire in California. Earth’s Future 7 , 892–910 (2019).

Clarke, H. et al. Forest fire threatens global carbon sinks and population centres under rising atmospheric water demand. Nat. Commun. 13 , 7161 (2022).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Juang, C. S. et al. Rapid growth of large forest fires drives the exponential response of annual forestfire area to aridity in the western United States. Geophys. Res. Lett. 49 , e2021GL097131 (2022).

Fire Statistics (Canadian Interagency Forest Fire Centre, accessed 3 November 2023); https://ciffc.net/statistics .

Eyring, V. et al. Overview of the coupled model intercomparison project phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9 , 1937–1958 (2016).

Parisien, M.-A. et al. Abrupt, climate-induced increase in wildfires in British Columbia since the mid-2000s. Commun. Earth Environ. 4 , 309 (2023).

Haughian, S. R., Burton, P. J., Taylor, S. W. & Curry, C. Expected effects of climate change on forest disturbance regimes in British Columbia. J. Ecosyst. Manag. https://doi.org/10.22230/jem.2012v13n1a152 (2012).

Zheng, B. et al. Record-high CO 2 emissions from boreal fires in 2021. Science 379 , 912–917 (2023).

Article   ADS   CAS   PubMed   Google Scholar  

Scholten, R. C., Coumou, D., Luo, F. & Veraverbeke, S. Early snowmelt and polar jet dynamics co-influence recent extreme Siberian fire seasons. Science 378 , 1005–1009 (2022).

2006 IPCC Guidelines for National Greenhouse Gas Inventories (IGES, 2006).

National Inventory Report: Greenhouse Gas Sources and Sinks in Canada (Environment Canada, 2023); https://publications.gc.ca/site/eng/9.506002/publication.html .

Xu, L. et al. Changes in global terrestrial live biomass over the 21st century. Sci. Adv. 7 , 9829 (2021).

Tagesson, T. et al. Recent divergence in the contributions of tropical and boreal forests to the terrestrial carbon sink. Nat. Ecol. Evol. 4 , 202–209 (2020).

Article   PubMed   Google Scholar  

Larsen, C. & MacDonald, G. An 840-year record of fire and vegetation in a boreal white spruce forest. Ecology 79 , 106–118 (1998).

Stocks, B. et al. Large forest fires in Canada, 1959–1997. J. Geophys. Res. Atmos. 107 , 5 (2002).

Johnstone, J. F. et al. Fire, climate change, and forest resilience in interior Alaska. Can. J. For. Res. 40 , 1302–1312 (2010).

Whitman, E., Parisien, M.-A., Thompson, D. K. & Flannigan, M. D. Short-interval wildfire and drought overwhelm boreal forest resilience. Sci. Rep. 9 , 18796 (2019).

Walker, X. J. et al. Increasing wildfires threaten historic carbon sink of boreal forest soils. Nature 572 , 520–523 (2019).

Wang, J. A. et al. Extensive land cover change across arctic–boreal northwestern North America from disturbance and climate forcing. Glob. Change Biol. 26 , 807–822 (2020).

Kurz, W. A. et al. Mountain pine beetle and forest carbon feedback to climate change. Nature 452 , 987–990 (2008).

Kurz, W. A., Stinson, G., Rampley, G. J., Dymond, C. C. & Neilson, E. T. Risk of natural disturbances makes future contribution of Canada’s forests to the global carbon cycle highly uncertain. Proc. Natl Acad. Sci. USA 105 , 1551–1555 (2008).

Tymstra, C., Stocks, B. J., Cai, X. & Flannigan, M. D. Wildfire management in Canada: review, challenges and opportunities. Prog. Disaster Sci. 5 , 100045 (2020).

Xie, P. et al. A gauge-based analysis of daily precipitation over East Asia. J. Hydrometeorol. 8 , 607–626 (2007).

Chen, M. et al. Assessing objective techniques for gauge-based analyses of global daily precipitation. J. Geophys. Res. Atmos. https://doi.org/10.1029/2007JD009132 (2008).

Buck, A. Buck Research CR-1A User’s Manual (Appendix 1) (Buck Research Instruments, 1996).

Hoesly, R. M. et al. Historical (1750–2014) anthropogenic emissions of reactive gases and aerosols from the community emissions data system. Geosci. Model Dev. 11 , 369–408 (2018).

Smith, S. J., Ahsan, H. & Mott, A. CEDS v 2021 04 21 gridded emissions data. PNNL Datahub https://doi.org/10.25584/PNNLDataHub/1779095 (2021).

Giglio, L., Randerson, J. T. & Van Der Werf, G. R. Analysis of daily, monthly, and annual burned area using the fourth-generation global fire emissions database (GFED4). J. Geophys. Res. Biogeosci. 118 , 317–328 (2013).

Randerson, J. T., Chen, Y., Van Der Werf, G., Rogers, B. & Morton, D. Global burned area and biomass burning emissions from small fires. J. Geophys. Res. Biogeosci. https://doi.org/10.1029/2012JG002128 (2012).

Mu, M. et al. Daily and 3-hourly variability in global fire emissions and consequences for atmospheric model predictions of carbon monoxide. J. Geophys. Res. Atmos. https://doi.org/10.1029/2011JD016245 (2011).

Miyazaki, K., Bowman, K. W., Yumimoto, K., Walker, T. & Sudo, K. Evaluation of a multi-model, multi-constituent assimilation framework for tropospheric chemical reanalysis. Atmos. Chem. Phys. 20 , 931–967 (2020).

Miyazaki, K. et al. Updated tropospheric chemistry reanalysis and emission estimates, TCR-2, for 2005–2018. Earth Syst. Sci. Data 12 , 2223–2259 (2020).

Guenther, A. B. et al. The model of emissions of gases and aerosols from Nature version 2.1 (MEGAN2.1): an extended and updated framework for modeling biogenic emissions. Geosci. Model Dev. 5 , 1471–1492 (2012).

Park, R. J., Jacob, D. J., Field, B. D., Yantosca, R. M. & Chin, M. Natural and transboundary pollution influences on sulfate-nitrate-ammonium aerosols in the United States: implications for policy. J. Geophys. Res. Atmos. https://doi.org/10.1029/2003JD004473 (2004).

Wecht, K. J., Jacob, D. J., Frankenberg, C., Jiang, Z. & Blake, D. R. Mapping of North American methane emissions with high spatial resolution by inversion of SCIAMACHY satellite data. J. Geophys. Res. Atmos. 119 , 7741–7756 (2014).

Veefkind, J. et al. TROPOMI on the ESA Sentinel-5 Precursor: a GMES mission for global observations of the atmospheric composition for climate, air quality and ozone layer applications. Remote Sens. Environ. 120 , 70–83 (2012).

Landgraf, J. et al. Carbon monoxide total column retrievals from TROPOMI shortwave infrared measurements. Atmos. Meas. Tech. 9 , 4955–4975 (2016).

Article   CAS   Google Scholar  

Borsdorff, T. et al. Improving the TROPOMI CO data product: update of the spectroscopic database and destriping of single orbits. Atmos. Meas. Tech. 12 , 5443–5455 (2019).

Copernicus Sentinel-5P (Processed by ESA): TROPOMI Level 2 Carbon Monoxide Total Column Products. Version 02 (European Space Agency, 2021); https://doi.org/10.5270/S5P-bj3nry0 .

Heald, C. L. et al. Comparative inverse analysis of satellite (MOPITT) and aircraft (TRACE-P) observations to estimate Asian sources of carbon monoxide. J. Geophys. Res. Atmos. https://doi.org/10.1029/2004JD005185 (2004).

Deeter, M. et al. The MOPITT version 9 CO product: sampling enhancements and validation. Atmos. Meas. Tech. 15 , 2325–2344 (2022).

Wunch, D. et al. The total carbon column observing network.  Phil. Trans. R. Soc. A 369 , 2087–2112 (2011).

Laughner, J. L. et al. The total carbon column observing network’s GGG2020 data version. Earth Syst. Sci. Data   16 , 2197–2260 (2024).

Wennberg, P. O. et al. TCCON Data From Park Falls (US), Release GGG2020.R1 (CaltechDATA, 2022); https://doi.org/10.14291/tccon.ggg2020.parkfalls01.R1 .

Wunch, D. et al. TCCON Data From East Trout Lake, SK (CA), Release GGG2020.R0 (CaltechDATA, 2022).

Henze, D. K., Hakami, A. & Seinfeld, J. H. Development of the adjoint of GEOS-CHEM. Atmos. Chem. Phys. 7 , 2413–2433 (2007).

Liu, J. et al. Carbon monitoring system flux estimation and attribution: impact of ACOS GOSAT XCO2 sampling on the inference of terrestrial biospheric sources and sinks. Tellus B Chem. Phys. Meteorol. 66 , 22486 (2014).

Liu, J. et al. Carbon monitoring system flux net biosphere exchange 2020. Earth Syst. Sci. Data 13 , 299–330 (2021).

Byrne, B. et al. The carbon cycle of southeast Australia during 2019–2020: drought, fires, and subsequent recovery. AGU Adv. 2 , e2021AV000469 (2021).

Chevallier, F., BrÅLeon, F.-M. & Rayner, P. J. Contribution of the orbiting carbon observatory to the estimation of CO 2 sources and sinks: theoretical study in a variational data assimilation framework. J. Geophys. Res. Atmos. https://doi.org/10.1029/2006JD007375 (2007).

Stanley, M., Kuusela, M., Byrne, B. & Liu, J. Technical note: posterior uncertainty estimation via a monte carlo procedure specialized for data assimilation. EGUsphere 2024 , 1–20 (2024).

Google Scholar  

Friedl, M. & Sulla-Menashe, D. MODIS/Terra+Aqua Land Cover Type Yearly L3 Global 0.05Deg CMG V061 [Data set]. (NASA EOSDIS Land Processes Distributed Active Archive Center, accessed 11 November 2023); https://doi.org/10.5067/MODIS/MCD12C1.061 .

Inventory and Land-use Change (Natural Resources Canada, accessed 2 April 2024; https://natural-resources.canada.ca/climate-change-adapting-impacts-and-reducing-emissions/climate-change-impacts-forests/carbon-accounting/inventory-and-land-use-change/13111 .

Byrne, B. bkabyrne/2023CanadaFires: v1.0.0. Zenodo https://doi.org/10.5281/zenodo.12709398 (2024).

Download references

Acknowledgements

The research carried out at the Jet Propulsion Laboratory, California Institute of Technology, was under a contract with the National Aeronautics and Space Administration. Resources supporting this work were provided by the NASA High-End Computing programme through the NASA Advanced Supercomputing Division at Ames Research Center. Authors B.B., A.C., J.L. and K.B. acknowledge the support from NASA Orbiting Carbon Observatory Science Team Program and the Carbon Monitoring System Program (grant no. NNH20ZDA001N-CMS). We acknowledge the World Climate Research Programme, which, through its Working Group on Coupled Modelling, coordinated and promoted CMIP6. We thank the climate modelling groups for producing and making available their model output, the Earth System Grid Federation (ESGF) for archiving the data and providing access and the many funding agencies who support CMIP6 and ESGF. GFAS is generated using Copernicus Atmosphere Monitoring Service Information 2020; neither the European Commission nor ECMWF is responsible for any use that may be made of the information it contains. The East Trout Lake TCCON station is funded through an infrastructure grant from the Canada Foundation for Innovation (grant no. 35278) and the Ontario Research Fund (grant no. 35278). The Park Falls TCCON site was supported by NASA (grant no. 80NSSC22K1066). We thank J. L. Laughner for guidance with the TCCON data. We thank M. Hafer and A. Dyk for providing information on Canada’s managed land. And we thank L. Baskaran for help in rasterizing these data.

Author information

Authors and affiliations.

Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA

Brendan Byrne, Junjie Liu, Kevin W. Bowman, Madeleine Pascolini-Campbell, Abhishek Chatterjee, Sudhanshu Pandey & Kazuyuki Miyazaki

Division of Geological and Planetary Sciences, California Institute of Technology, Pasadena, CA, USA

Junjie Liu, Paul O. Wennberg & Coleen M. Roehl

Joint Institute for Regional Earth System Science and Engineering, University of California, Los Angeles, CA, USA

Kevin W. Bowman

Meteorology & Air Quality Group, Wageningen University and Research, Wageningen, The Netherlands

Guido R. van der Werf

Department of Physics, University of Toronto, Toronto, Ontario, Canada

Debra Wunch

Division of Engineering and Applied Science, California Institute of Technology, Pasadena, CA, USA

Paul O. Wennberg

Department of Energy, Environmental, and Chemical Engineering, Washington University in St. Louis, St. Louis, MO, USA

Saptarshi Sinha

You can also search for this author in PubMed   Google Scholar

Contributions

B.B., J.L., K.W.B., M. P.-C., A.C. and S.P. conceptualized and designed the study. K.M. provided atmospheric CO production and OH estimates. G.R.v.d.W extended the GFED4.1s dataset for this experiment. D.W., P.O.W. and C.M.R. provided TCCON data. S.S. provided MERRA-2 reanalysis for the model. B.B. conducted the analysis and wrote the paper, with input from all authors.

Corresponding author

Correspondence to Brendan Byrne .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature thanks Mike Flannigan, Johannes Kaiser and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended data fig. 1 schematic diagram of the tropomi xco inversion procedure..

(a) Ensembles of inversions are performed based on three different flux inventories. (b) To quantify the sensitivity to systematic error sources, four inversions are performed that differ in observational error constraints, prior error constraints, and temporal optimization frequency. (c) Bayesian posterior error estimates are estimate for 2023 by following the Monte Carlo approach for 4D-Var of Chevalier et al. 71 . (d) The posterior best estimates are taken as the average maximum a posteriori estimate across inversion configurations while the uncertainty is taken to be the sum-of-squares of the error components estimated in (b) and (c).

Supplementary information

Supplementary information.

Supplementary sections 1 and 2, Figs. 1–14, Tables 1–3 and references.

Peer Review File

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Byrne, B., Liu, J., Bowman, K.W. et al. Carbon emissions from the 2023 Canadian wildfires. Nature (2024). https://doi.org/10.1038/s41586-024-07878-z

Download citation

Received : 29 November 2023

Accepted : 25 July 2024

Published : 28 August 2024

DOI : https://doi.org/10.1038/s41586-024-07878-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

arXiv's Accessibility Forum starts next month!

Help | Advanced Search

Computer Science > Artificial Intelligence

Title: automated design of agentic systems.

Abstract: Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We formulate a new research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, control flows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

Comments:	Website:
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	[cs.AI]
	(or [cs.AI] for this version)
	Focus to learn more arXiv-issued DOI via DataCite

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .