![\"\"](\"http:\/\/www.alpha-quantum.com\/blog\/wp-content\/uploads\/2020\/05\/img_5ed4354677fb7.png\")
<\/p>\n
Figure 1: Types of recommender systems<\/p>\n
What are content-based recommender systems?<\/h1>\n
Content-based recommender systems generate recommendations by relying on attributes of items and\/or users. User attributes can include age, sex, job type and other personal information. Item attributes on the other hand, are descriptive information that distinguishes individual items from each other. In case of movies, this could include title, cast, description, genre and others.<\/p>\n
By relying on features, those of users and items, content-based recommender systems are more like a traditional machine learning problem than is the case for collaborative filtering. Content-based method uses item-based or user-based features to predict an action of the user for a given item. User\u2019s action can be a specific rating, a buy decision, like or dislike, a decision to view a movie and similar.<\/p>\n
One of the advantages of content-based recommendation is user independence \u2013 to make recommendations to a user, it does not require information about other users, unlike collaborative filtering. This makes content-based approach easier to scale. Another benefit is that the recommendations are more transparent, as the recommender can more clearly explain recommendation in terms of the features used.<\/p>\n
Content-based approach also has its drawbacks, one is over-specialization \u2013 if the user is only interested in specific categories, recommender will have difficulty recommending items outside of this scope, leading to user remaining in its current circle of items\/interests. Content-based approaches also often require domain knowledge to produce relevant item and user features.<\/p>\n
We will now build an implementation of content-based recommender in python, using the MovieLens dataset<\/a>.<\/p>\n Our recommender system will be able to recommend movies to us, based on movie plots and based on combination of features, such as top actors, director, keywords, producer and screenplay writers of the movies.<\/p>\n First, we load the models:<\/p>\n <\/p>\n Next, we import data from https:\/\/www.kaggle.com\/rounakbanik\/the-movies-dataset<\/a> and https:\/\/grouplens.org\/datasets\/movielens\/latest\/<\/a>:<\/p>\n df_data\u00a0=\u00a0pd.read_csv(‘movies_metadata.csv’,\u00a0low_memory=False)<\/p>\n One of the pre-processing steps for our recommender involves removing movies which have low number of votes:<\/p>\n #\u00a0determine\u00a0the\u00a0minimum\u00a0number\u00a0of\u00a0votes\u00a0that\u00a0the\u00a0movie\u00a0must\u00a0have\u00a0to\u00a0be\u00a0included\u00a0<\/em><\/p>\n #\u00a0exclude\u00a0movies\u00a0that\u00a0do\u00a0not\u00a0have\u00a0minimum\u00a0number\u00a0of\u00a0votes<\/em><\/p>\n Our first content-based recommender will have a goal of recommending movies which have a similar plot to a selected movie.<\/p>\n We will use “overview” feature from our dataset:<\/p>\n To compare movie plots, we first need to compute their numerical representation. There are various approaches we can use, from bag of words, word embeddings to TF-IDF, we will select the latter.<\/p>\n TF-IDF of a word in a document which is part of a larger corpus of documents is a combination of two values. One is term frequency (TF), which measures how frequently the word occurs in the document.<\/p>\n However, some of the words, such as \u201cthe\u201d and \u201cis\u201d, occur frequently in all documents and we want to downscale the importance of such words. This is accomplished by multiplying TF with the inverse document frequency.<\/p>\n This ensures that only those words are considered important for the document that are frequent in this document but more rarely present in the rest of the corpus.<\/p>\n To build the TF-IDF representation of movie plots we will use the TfidfVectorizer from scikit-learn. We first fit TfidfVectorizer on train data set of movie plot descriptions and then transform the movie plots into TF-IDF numerical representation:<\/p>\n <\/p>\n Now that we have numerical vectors, representing each movie plot description, we can compute similarity of movies by calculating their pair-wise cosine similarities and storing them in cosine similarity matrix:<\/p>\n With cosine similarity matrix computed, we can define the function “recommendations” that will return top recommendations for a given movie.<\/p>\n The function first determines the index of the input movie, retrieves the similarities of movies with selected movie, sorts them and returns the titles of movies with the highest similarity to the selected movie.<\/p>\n <\/p>\n We can now produce our recommendation for a given film, e.g. \u2018Batman\u2019:<\/p>\n recommendations(‘Batman’,\u00a0df,\u00a0cosine_similarity_matrix,\u00a010)<\/p>\n 3693\u00a0\u00a0\u00a0 Batman Beyond: Return of the Joker\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 5962\u00a0\u00a0\u00a0 The Dark Knight Rises\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 7379\u00a0\u00a0\u00a0 Batman vs Dracula\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 5476\u00a0\u00a0\u00a0 Batman: Under the Red Hood\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 6654\u00a0\u00a0\u00a0 Batman: Mystery of the Batwoman\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 3911\u00a0\u00a0\u00a0 Batman Begins\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 6334\u00a0\u00a0\u00a0 Batman: The Dark Knight Returns, Part <\/em><\/p>\n 1<\/em>770\u00a0\u00a0\u00a0\u00a0 Batman & Robin\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 4725\u00a0\u00a0\u00a0 The Dark Knight\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 709\u00a0\u00a0\u00a0\u00a0 Batman Returns\u00a0\u00a0\u00a0 <\/em><\/p>\n <\/p>\n The recommender based on overview is of a limited quality as it considers only the movie plot.<\/p>\n We will now explore a different recommender, which gives more focus to other metadata (keywords, actors, director, producer, genres and screenplay authors) when recommending movies.<\/p>\n To use additional metadata, we first need to extract it from separate files, keywords.csv and credits.csv, and merge it with the main pandas dataframe:<\/p>\n <\/p>\n Again, we will keep only the movies with highest vote counts using similar code as previously.<\/p>\n We next create a new feature for each movie, which consists of top 4 actors in the movie. We also concatenate and lowercase the name and surname of actors. We namely want e.g. Tom of Tom Hanks to be distinct from Tom in Tom Selleck, replacing both names with tomhanks and tomselleck, respectively:<\/p>\n We will now create similar features for director, screenplay and producer of movies. To simplify, we only use first person detected per job type.<\/p>\n After generating individual metadata, we merge them in a single feature with the ability to individually weight different features. This allows us to build highly flexible recommenders, as we will see later on.<\/p>\n <\/p>\n <\/p>\n After generating the feature, we again need to vectorize it. We will not use TF-IDF, as it reduces the importance of words which occur in many documents, in our case this e.g. also includes actors, directors, screenplay writer, producers.<\/p>\n We will thus use CountVectorizer for this purpose.<\/p>\n Example recommendations:<\/p>\n recommendations(‘Toy Story’,\u00a0df_movies,\u00a0cosine_similarity_matrix_count_based,\u00a010)<\/p>\n 4252\u00a0\u00a0\u00a0 Toy Story 3\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>5823\u00a0\u00a0\u00a0 Toy Story That Time Forgot <\/em>785\u00a0\u00a0\u00a0\u00a0 Small Soldiers\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>5702\u00a0\u00a0\u00a0 Hawaiian Vacation\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>1358\u00a0\u00a0\u00a0 Toy Story 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>273\u00a0\u00a0\u00a0\u00a0 Pinocchio\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>1680\u00a0\u00a0\u00a0 The Transformers: The Movie<\/em>833\u00a0\u00a0\u00a0\u00a0 Child’s Play\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>966\u00a0\u00a0\u00a0\u00a0 Toys\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em>4836\u00a0\u00a0\u00a0 Ted\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n The usage of relative weights to control importance of different metadata allows us to quickly build a new recommender focused on other aspects of movies.<\/p>\n We can e.g. increase weight for the director to recommend movies that are highly likely directed by the same director as the input movie:<\/p>\n 4837\u00a0\u00a0\u00a0 Tin Toy\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 4860\u00a0\u00a0\u00a0 Knick Knack\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/em><\/p>\n 3182\u00a0\u00a0\u00a0 Luxo Jr.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 1358\u00a0\u00a0\u00a0 Toy Story 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 1012\u00a0\u00a0\u00a0 A Bug’s Life\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 4532\u00a0\u00a0\u00a0 Cars 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n 5423\u00a0\u00a0\u00a0 Mater and the Ghostlight<\/em><\/p>\n 3268\u00a0\u00a0\u00a0 Cars\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/em><\/p>\n A quick search shows that all of the movies recommender were directed by the director of Toy Story \u2013 John Lasseter.<\/p>\n In this article, we have introduced several content-based recommender systems in python, using MovieLens data set.<\/p>\n Recommender systems utilize big data about our interactions with items and try to find patterns which show what items are most popular with users that are similar to us or find items that are most similar to items that we have purchased in the past.<\/p>\n Besides content-based method used in this article, recommenders are also often using collaborative filtering approach or a combination of both, known as hybrid methods, which try to combine both main approaches in a way which minimizes the drawbacks of any of the individual methods. Hybrid recommenders are the most common type of recommenders, found in online platforms today.<\/p>\n <\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Recommender systems are methods that predict users\u2019 interests and make meaningful recommendations to them for different items, such as songs to play…<\/p>\n","protected":false},"author":1,"featured_media":435,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[46],"tags":[],"_links":{"self":[{"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/posts\/431"}],"collection":[{"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/comments?post=431"}],"version-history":[{"count":2,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/posts\/431\/revisions"}],"predecessor-version":[{"id":434,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/posts\/431\/revisions\/434"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/media\/435"}],"wp:attachment":[{"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/media?parent=431"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/categories?post=431"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alpha-quantum.com\/blog\/wp-json\/wp\/v2\/tags?post=431"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Content-based recommender system for recommendation of movies<\/h1>\n
import pandas as pd\r\n\r\nimport ast\r\n\r\nfrom sklearn.feature_extraction.text import CountVectorizer\r\n\r\nfrom sklearn.feature_extraction.text import TfidfVectorizer\r\n\r\nfrom sklearn.metrics.pairwise import cosine_similarity\r\n\r\nimport seaborn as sns\r\n\r\nimport numpy as np\r\n\r\nimport matplotlib.pyplot as plt<\/pre>\n
df_data = df_data[df_data['vote_count'].notna()]\r\n\r\nplt.figure(figsize=(20,5))\r\n\r\nsns.distplot(df_data['vote_count'])\r\n\r\nplt.title(\"Histogram of vote counts\")<\/pre>\n
min_votes = np.percentile(df_data['vote_count'].values, 85)<\/pre>\n
df = df_data.copy(deep=True).loc[df_data['vote_count'] > min_votes]<\/pre>\n
Content-based recommender that recommends movies based on similarity of movie plots<\/h2>\n
# removing rows with missing overview\r\n\r\ndf = df[df['overview'].notna()]\r\n\r\ndf.reset_index(inplace=True)\r\n\r\n\r\n# processing of overviews\r\n\r\ndef process_text(text):\r\n\r\n # replace multiple spaces with one\r\n\r\n text = ' '.join(text.split())\r\n\r\n # lowercase\r\n\r\n text = text.lower()\r\n\r\n return text\r\n\r\ndf['overview'] = df.apply(lambda x: process_text(x.overview),axis=1)<\/pre>\n
TF-IDF approach<\/h3>\n
tf_idf = TfidfVectorizer(stop_words='english')\r\n\r\ntf_idf_matrix = tf_idf.fit_transform(df['overview']);<\/pre>\n
# calculating cosine similarity between movies\r\n\r\ncosine_similarity_matrix = cosine_similarity(tf_idf_matrix, tf_idf_matrix)<\/pre>\n
def index_from_title(df,title):\r\n\r\nreturn df[df['original_title']==title].index.values[0]\r\n\r\n\r\n# function that returns the title of the movie from its index\r\n\r\ndef title_from_index(df,index):\r\n\r\nreturn df[df.index==index].original_title.values[0]\r\n\r\n\r\n# generating recommendations for given title\r\n\r\ndef recommendations( original_title, df,cosine_similarity_matrix,number_of_recommendations):\r\n\r\nindex = index_from_title(df,original_title)\r\n\r\nsimilarity_scores = list(enumerate(cosine_similarity_matrix[index]))\r\n\r\nsimilarity_scores_sorted = sorted(similarity_scores, key=lambda x: x[1], reverse=True)\r\n\r\nrecommendations_indices = [t[0] for t in similarity_scores_sorted[1:(number_of_recommendations+1)]]\r\n\r\nreturn df['original_title'].iloc[recommendations_indices]<\/pre>\n
Content-based recommender based on keywords, actors, screenplay, director, producer and genres features<\/h2>\n
df_keywords = pd.read_csv('keywords.csv')\r\n\r\ndf_credits = pd.read_csv('credits.csv')\r\n\r\n# Some ids have irregular format, so we will remove them\r\n\r\ndf_cb = df_data.copy(deep=True)[df_data.id.apply(lambda x: x.isnumeric())]\r\n\r\ndf_cb['id'] = df_cb['id'].astype(int)\r\n\r\ndf_keywords['id'] = df_keywords['id'].astype(int)\r\n\r\ndf_credits['id'] = df_credits['id'].astype(int)\r\n\r\n# Merging keywords, credits of movies with main data set\r\n\r\ndf_movies_data = pd.merge(df_cb, df_keywords, on='id')\r\n\r\ndf_movies_data = pd.merge(df_movies_data, df_credits, on='id')<\/pre>\nmax_number_of_actors = 4\r\n\r\ndef return_actors(cast):\r\n\r\nactors = []\r\n\r\ncount = 0\r\n\r\nfor row in ast.literal_eval(cast) :\r\n\r\nif count<max_number_of_actors:\r\n\r\nactors.append(row['name'].lower().replace(\" \",\"\"))\r\n\r\nelse:\r\n\r\nbreak\r\n\r\ncount+=1\r\n\r\nreturn ' '.join(actors)\r\n\r\n\r\ndf_movies['actors']=df_movies.apply(lambda x: return_actors(x.cast),axis=1)<\/pre>\n
def return_producer_screenplay_director(crew,crew_type):\r\n\r\npersons = []\r\n\r\nfor row in ast.literal_eval(crew) :\r\n\r\nif row['job'].lower()==crew_type:\r\n\r\npersons.append(row['name'].lower().replace(\" \",\"\"))\r\n\r\nreturn ' '.join(persons)\r\n\r\n\r\ndf_movies['director']=df_movies.apply(lambda x: return_producer_screenplay_director(x.crew,'director'),axis=1)\r\n\r\ndf_movies['screenplay']=df_movies.apply(lambda x: return_producer_screenplay_director(x.crew,'screenplay'),axis=1)\r\n\r\ndf_movies['producer']=df_movies.apply(lambda x: return_producer_screenplay_director(x.crew,'producer'),axis=1)<\/pre>\n
# relative importance of different features\r\n\r\nw_genres = 2\r\n\r\nw_keywords = 3\r\n\r\nw_actors = 3\r\n\r\nw_director = 1\r\n\r\nw_producer = 1\r\n\r\nw_screenplay = 1\r\n\r\n# function for merging features\r\n\r\ndef concatenate_features(df_row):\r\n\r\ngenres = []\r\n\r\nfor genre in ast.literal_eval(df_row['genres']) :\r\n\r\ngenres.append(genre['name'].lower())\r\n\r\ngenres = ' '.join(genres)\r\n\r\nkeywords = []\r\n\r\nfor keyword in ast.literal_eval(df_row['keywords']) :\r\n\r\nkeywords.append(keyword['name'])\r\n\r\nkeywords = ' '.join(keywords)\r\n\r\nreturn ' '.join([genres]*w_genres)+' '+' '.join([keywords]*w_keywords)+' '+' '.join([df_row['actors']]*w_actors)+' '+' '.join([df_row['director']]*w_director)+' '+' '.join([df_row['producer']]*w_producer)+' '+' '.join([df_row['screenplay']]*w_screenplay)<\/pre>\n
df_movies['features'] = df_movies.apply(concatenate_features,axis=1)\r\n\r\n# pre-processing text of features\r\n\r\ndef process_text(text):\r\n\r\n# replace multiple spaces with one\r\n\r\ntext = ' '.join(text.split())\r\n\r\n# lowercase\r\n\r\ntext=text.lower()\r\n\r\nreturn text\r\n\r\ndf_movies['features'] = df_movies.apply(lambda x: process_text(x.features),axis=1)<\/pre>\n
vect = CountVectorizer(stop_words='english')\r\n\r\nvect_matrix = vect.fit_transform(df_movies['features'])\r\n\r\ncosine_similarity_matrix_count_based = cosine_similarity(vect_matrix, vect_matrix)<\/pre>\n
w_director = 100\r\n\r\ndf_movies['features'] = df_movies.apply(concatenate_features,axis=1)\r\n\r\nvect = CountVectorizer(stop_words='english')\r\n\r\nvect_matrix = vect.fit_transform(df_movies['features'])\r\n\r\ncosine_similarity_matrix_count_based = cosine_similarity(vect_matrix, vect_matrix)\r\n\r\nrecommendations('Toy Story', df_movies, cosine_similarity_matrix_count_based, 8)<\/pre>\nConclusion<\/h1>\n
Notice: Trying to access array offset on value of type null in /var/www/alpha-quantum.com/public_html/blog/wp-content/plugins/woocommerce/includes/class-woocommerce.php on line 202