Data Sciece from Scratch 之 Recommender System

在线课程笔记整理

以上是在线学习课程的笔记，下面就是《Data Science from Scratch》的笔记部分。

Recommending What’s Popular

users_interests = [
["Hadoop", "Big Data", "HBase", "Java", "Spark", "Storm", "Cassandra"],
["NoSQL", "MongoDB", "Cassandra", "HBase", "Postgres"],
["Python", "scikit-learn", "scipy", "numpy", "statsmodels", "pandas"],
["R", "Python", "statistics", "regression", "probability"],
["machine learning", "regression", "decision trees", "libsvm"],
["Python", "R", "Java", "C++", "Haskell", "programming languages"],
["statistics", "probability", "mathematics", "theory"],
["machine learning", "scikit-learn", "Mahout", "neural networks"],
["neural networks", "deep learning", "Big Data", "artificial intelligence"],
["Hadoop", "Java", "MapReduce", "Big Data"],
["statistics", "R", "statsmodels"],
["C++", "deep learning", "artificial intelligence", "probability"],
["pandas", "R", "Python"],
["databases", "HBase", "Postgres", "MySQL", "MongoDB"],
["libsvm", "regression", "support vector machines"]
]

我们使用这样的一个数据集作为示例，看看我们的推荐系统是怎么运作的。

首先看看哪一个语言是特别流行的？

popular_interests = Counter(interest
                            for user_interests in users_interests
                            for interest in user_interests).most_common()

##[('Python', 4), ('R', 4),....

得到这个，我们只需要推荐给一个用户最流行的但他还没有感兴趣的语言：

def most_popular_new_interests(user_interests,max_results=5):
                                suggestions = [(interest,frequency)
                                for interest,frequency in popular_interests
                                if interest not in user_interests]
    return suggestions[:max_results]

print most_popular_new_interests(users_interests[1])
#[('Python', 4), ('R', 4), ('Java', 3), ('regression', 3), ('statistics', 3)]

在这种情况下，“很多人喜欢python意味着你也可能喜欢”并不是一个很好的解决方案，如果一个新用户来了，我们不知道他们喜欢什么，这是一种很好的办法。让我们来看看如何根据用户的兴趣来推荐？

User-Based Collaborative Filtering

一种考虑用户的兴趣，看他与哪个用户相似，然后将与之相似的用户的东西推荐给他。

余弦相似

一种计算两个向量相似度的计算公式，也就是余弦公式。

def dot(v, w):
    """v_1 * w_1 + ... + v_n * w_n"""
    return sum(v_i * w_i for v_i, w_i in zip(v, w))

为了计算余弦，我们需要构建向量：

unique_interests = sorted(list({interest
                        for user_interests in users_interests
                        for interest in user_interests}))

下面我们为每一个用户建立兴趣向量:

def make_user_interest_vector(user_interests):
    return [1 if interest in user_interests else 0
            for interest in unique_interests]
print make_user_interest_vector(users_interests[0])
#[1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
#建立矩阵
user_interest_matrix = map(make_user_interest_vector,users_interests)

现在，如果user_interest_matrix[i][j]等于1表示用户i对j有兴趣，由于我们的数据集特别小，所以我们可以计算每个用户与其他用户间的相似程度。

user_similarities = [[cosine_similarity(interest_vector_i,interest_vector_j)
                for interest_vector_i in user_interest_matrix]
                for interest_vector_j in user_interest_matrix]

这样user_similarities[i][j]表示用户i和用户j之间的相似度。但是user_similarities[i]只是用户i与其他用户之间的相似度。我们可以利用这个来编写一个函数来计算给定一个用户计算与之最为相似的其他用户。

def most_similar_users_to(user_id):
    pairs = [(other_user_id,similarity)
        for other_user_id,similarity in
        enumerate(user_similarities[user_id])
        if user_id!=other_user_id and similarity>0]
    return sorted(pairs,key=lambda (_,similarity):similarity,reverse=True)

print most_similar_users_to(0)
###########################
[(9, 0.5669467095138409),
(1, 0.3380617018914066),
(8, 0.1889822365046136),
(13,0.1690308509457033),
(5, 0.1543033499620919)]

那么我们怎么将一个新的推荐给一个用户吗？对于每一个东西，我们将与之相似的其他用户相似度加起来即可。

def user_based_suggestions(user_id,include_current_interests=False):
    suggestions = defaultdict(float)
    for other_user_id,similarity in most_similar_users_to(user_id):
    for interest in users_interests[other_user_id]:
    suggestions[interest] += similarity

    suggestions = sorted(suggestions.items(),key=lambda (_,weight):weight,reverse=True)
    if include_current_interests:
        return suggestions
    else:
        return [(suggestions,weight) for suggestions,weight in suggestions
        if suggestions not in users_interests[user_id]]

print user_based_suggestions(0)
#[('MapReduce', 0.5669467095138409), ('MongoDB', 0.50709255283711), ('Postgres',0.50709255283711), ('NoSQL', 0.3380617018914066), ('neural networks', 0.18898223。。。。。。

到现在基于用户的推荐系统已经完成，当数据集特别大的时候，这种算法就不合适了，我们考虑另一种基于物品的算法。

Item-Based Collaborative Filtering

另一种算法就是直接计算物品之间的相似度，然后推荐将相似的其他物品推荐给他。为了做到这一点，我们首先将user-interest矩阵翻转，以达到列对应兴趣行对应用户。

interest_user_matrix = [[user_interest_vector[j] #取第一列
                    for user_interest_vector in user_interest_matrix]
                    for j,_ in enumerate(unique_interests)]

到这里，[1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]表示用户0，8，和9对Big Data感兴趣。

接下来计算物品的相似度：

def most_similar_interests_to(interest_id):
    similarities = interest_similarities[interest_id]
    pairs = [(unique_interests[other_interest_id], similarity)
         for other_interest_id, similarity in enumerate(similarities)
         if interest_id != other_interest_id and similarity > 0]
    return sorted(pairs,
              key=lambda (_, similarity): similarity,
              reverse=True)

# print most_similar_interests_to(0)

def item_based_suggestions(user_id, include_current_interests=False):
    suggestions = defaultdict(float)
    user_interest_vector = user_interest_matrix[user_id]
    for interest_id, is_interested in enumerate(user_interest_vector):
    if is_interested == 1:
    similar_interests = most_similar_interests_to(interest_id)
    for interest, similarity in similar_interests:
        suggestions[interest] += similarity

    suggestions = sorted(suggestions.items(),
        key=lambda (_, similarity): similarity,
        reverse=True)

    if include_current_interests:
        return suggestions
    else:
        return [(suggestion, weight)
        for suggestion, weight in suggestions
        if suggestion not in users_interests[user_id]]

到这，基于物品的协同过滤已经讲完，还有一部分关于奇异值分解的，我就没有看，等有时间再看，最近学校申事情多！