PAI-Rec is a platform for end-to-end, in-depth customization and development of recommendation systems. With this platform, enterprise developers can independently build, develop, iterate, and maintain their recommendation systems.
Overview
Recommendation systems require complex systematic engineering. In most cases, a recommendation system integrates offline processing modules, online processing modules, and real-time data forwarding. A recommendation engine consists of recall, ranking, filtering, and re-ranking modules. The recommendation modules and pipelines of PAI-Rec work based on the big data architecture of Alibaba Cloud Apsara. Developers can select service types based on their enterprise technology stacks and development habits, and can customize the code of recommendation pipelines. In addition, PAI-Rec provides a variety of tools, such as the data diagnostics and analysis tool, recommendation result debugging tool, and engine release management tool. The A/B testing service and the experiment report platform help you greatly improve the iteration efficiency of your recommendation systems.
PAI-Rec covers the whole recommendation system development process. You can perform data analysis based on the logs collected by frontend event tracking tasks, and then customize the code of feature engineering and recall and ranking algorithms, as well as engine configuration files, experiment report metrics, and statistics-related code based on business requirements. This helps you easily build recommendation systems for various recommendation scenarios, and shortens the period for recommendation system construction and optimization. You can use PAI-Rec to build a new recommendation system or optimize an existing recommendation system.
PAI-Rec provides a white-box development model that brings developers a more transparent, controllable, and flexible development experience. In addition, if the recommendation algorithm team and engineering team of your enterprise are not mature, we recommend that you start the service by using the industry-specific algorithm models customized by the Alibaba Cloud algorithm team at the initial stage. This not only assists your enterprise in deploying a complete recommendation system within a short period of time but also helps developers quickly get started with independent model training and effect evaluation. If you need further cooperation with Alibaba Cloud, such as wanting Alibaba Cloud engineers to provide in-depth tuning customization and share tuning experience, you can contact Alibaba Cloud customer service.
PAI-Rec also provides a variety of capabilities such as cold start, recommendation control, and online learning. The related solutions are complex. If you want to use these capabilities, you can contact your account manager or technical team of Alibaba Cloud for further communication.
For more details, go to the official website of PAI-Rec.
Development process
Benefits
PAI-Rec provides the following benefits:
Highly transparent white-box development
PAI-Rec provides a large amount of source code, which allows users to understand the details of recommendation algorithms and customize code to meet various business requirements.
The source code includes the following types: source code for feature engineering and sample processing, script code for calling recall and ranking models, source code of EasyRec recall and ranking models, and service source code of the PAI-Rec engine.
Convenient recommendation solution customization process
Users need to only configure user tables, item tables, and behavior tables to generate recall and ranking scripts and configuration files. This simplifies recommendation solution customization.
Comprehensive engine and experiment management system
A comprehensive engine management and experiment management platform is provided. Users can use this platform to easily manage recall and ranking components and update engine parameters.
Fine-grained metric monitoring and reporting
A metric and report management platform is provided for users to customize metrics and obtain experiment results by day or by hour. This ensures accurate monitoring of recommendation results and timely feedback.
Guarantee of consistency between offline and online features
A tool is provided for comparing consistency between online and offline features. This prevents experiment errors caused by consistency issues.
Intelligent data diagnostics and analytics
The intelligent data diagnostics feature is provided to help developers quickly understand data and select features and time windows for feature engineering based on diagnostics results.
Tools for intuitive observation of recommendation results
A variety of diagnostic tools are provided to help users observe recommendation results and recall data in a visualized manner.
Powerful FeatureStore
PAI-Rec is integrated with FeatureStore to help users manage features in a better way, which improves experiment efficiency.
All-round technical support
Various technical support services are provided to answer questions of users, helping them make good use of solutions.
Dependent cloud services
EasyRec of Platform for AI (PAI) is used to train recall and ranking models, and the PAI-Rec engine in Go is used to build recommendation systems. DataWorks or Machine Learning Designer of PAI is used to edit and schedule the code for feature engineering as well as sample and model training. Graph Compute and Hologres are used to store user features and support item-to-item (I2I) queries and vector queries. Elastic Algorithm Service (EAS) of PAI is used to provide scalable scoring services. The following section describes the related services:
PAI is a machine learning and deep learning engineering platform intended for developers and enterprises. PAI provides end-to-end AI development services including data labeling, model building, model training, model deployment, and inference optimization.
EasyRec is an algorithm framework that incorporates industry-leading deep learning models. It supports TensorFlow 1.12 and later versions, TensorFlow 2.4 and earlier versions, and PAI-TensorFlow. It meets end-to-end recommendation requirements, including recall, coarse ranking, fine ranking, re-ranking, multi-objective ranking, and cold start. EasyRec helps developers accelerate feature iterations to meet end-to-end recommendation requirements.
DataWorks and MaxCompute are two cloud-native big data services. The two services can be used together. They provide easy-to-use development tools and stable data environments for feature processing, sample generation, profile management, model scheduling, data update, and other phases in recommendation systems.
NoteWe recommend that you use DataWorks and MaxCompute because PAI-Rec does not support other big data services. If you really need to use other big data services, you may need to modify related engine code. In this case, you must communicate with architects in advance.
Hologres is a unified real-time data warehousing service developed by Alibaba Cloud. You can use Hologres to write, update, process, and analyze large amounts of data in real time. Hologres supports standard SQL syntax, is compatible with PostgreSQL, and supports most PostgreSQL functions. Hologres supports online analytical processing (OLAP) and ad hoc analysis for up to petabytes of data, and provides high-concurrency and low-latency online data services. Hologres supports fine-grained isolation of multiple workloads and enterprise-level security capabilities. Hologres is deeply integrated with MaxCompute, Realtime Compute for Apache Flink, and DataWorks to provide full-stack online and offline data warehousing solutions for enterprises.
You can use Hologres to store real-time user behavior sequences, user features, and recall data, and also use the vector recall feature provided by Hologres.
Graph Compute is a high-performance distributed graph computing service developed by Alibaba Cloud. It provides an end-to-end graph computing service that supports trillions of data records. Graph Compute allows you to efficiently store, query, and compute complex graph relationship data by using graph algorithms and models. Graph Compute can be used in a wide range of scenarios such as recommended advertisements for searches, real-time risk control, knowledge graph, and social networks.