In today’s world, a recommendation system is an integral and expected component in any online service. This is especially true in a large scale entertainment streaming service such as Pandora. Users of such services have grown accustomed to a personalized experience not only fitting their expressed interest, but also their situations, through recommendations. In constructing these recommendation algorithms and systems, practitioners often need to consider a complex array of constraints such as different item formats, diverse preferences for degree of exploration, query originations (typed search in mobile, voice command), user tiers (free vs. premium) and a variety of clients. In addition, recommendations are expected to come with low latency and scale to millions of users. Our lives as practitioners are made simultaneously more interesting and more hectic by these constraints. We use four use case scenarios to illustrate the complexities faced by industrial scale recommendation systems, largely generalized from experience in building the largest streaming audio recommender systems at Pandora. Since 2005, Pandora’s music recommendation platform has powered the well known machine learning driven personalized radio product and premium on demand product, complete with support to a host of clients from web, mobile, and voice platforms for 65+ million monthly listeners in the U.S. Firstly, we share the content understanding aspect — genome tagging, keyword tagging, and combination of expert tagging and automatic tagging to form the most important knowledge building blocks of recommendations and explanations. Secondly, we share user modeling, from scalability issues from detailed user actions, to user feature extraction for ML systems. Thirdly, we discuss algorithms, ranging from core recommender model and their deployment, to ensemble re-ranking systems that could take shape of a contextual multi-armed bandit algorithm and possibly reinforcement learning. Finally, we discuss new challenges brought on by recently popular voice interaction platforms and our approaches. In this tutorial, we emphasize on the comprehensive view of a practical and scalable system, complete with offline and online evaluations. Let us learn, discuss and explore ideas together!
Short Bio
Dr. Tao Ye is a Director of Science and Principal Scientist at Pandora. She is a founding member of the Pandora science team, and has been working on personalized recommendation systems, measurements, and user modeling since 2010. Most recently, she has been leading the Personalization, Search and Voice science team that advances the machine learning and data driven innovations in many personalized music recommendation and search functionalities in Pandora. In the larger RecSys research community, she co-founded and co-chaired the Large Scale Recommender Systems workshop for 5 years (2013-2017), and gave many invited talks at conferences and RecSys workshops. She has two decades of experience in the software industry, holding research scientist and lead engineer positions in social media, networking and mobile systems. She holds 14 granted patents and has published 12 peer reviewed papers. She received her PhD from University of Melbourne in Electrical and Electronic Engineering, her MS from UC Berkeley in EECS and dual BS degrees from Stony Brook University in CS and Engineering Chemistry.