Large-scale multi-task learning recommender systems on Ray

When building a production recommender system, it is important to take all sorts of feedback into consideration, such as user visits to the store, purchases, online clicks, customer service, etc. In this scenario, developing a system to learn multiple tasks at the same time will greatly improve the overall performance compared with constructing a number of task-specific models separately. However, building such a system can be very complex and often requires the model to be trained on a combined large-scale dataset. At Verizon, we greatly benefited from Ray Train and Ray Tune in accelerating distributed model training and hyperparameter tuning workloads. In this talk, we will present an end-to-end multi-task learning recommender system Verizon AI Center built using PyTorch, Ray, and BigDL on big data platforms.