Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog MaxCompute-Based Artificial Intelligence Recommendation Solution

MaxCompute-Based Artificial Intelligence Recommendation Solution

This article explains an Artificial Intelligence Recommendation (AIRec) solution based on MaxCompute.

By Wu Shilong, Alibaba Cloud Intelligent Senior Product Expert

I. Background Introduction

Different ways of transforming new users and improving the stickiness of old users have become crucial when the dividends of the Internet industry have passed. The cost of obtaining customers is increasing, the online time of users has remained virtually the same, and the information explosion has become crucial. Intelligent personalized recommendation is undoubtedly one of the proven and important means that boasts its presence in mobile apps or enterprises everywhere every day.

Industry Trends

"In Tmall 2018, Double 11 generated 45.3 billion AI personalized recommendations." Jiang Fan, Vice President of Alibaba and President of Taobao, said, "Taobao may be the world's largest application of artificial intelligence." He continued, "We can also see that the traffic based on personalized recommendations has exceeded the traffic brought by search and other methods. This is a very, very big change."

The term information explosion first appeared in the 1980s. All kinds of information grew exponentially. Learning different ways to deal with overloaded information has become an important issue. This means consumers, information publishers, and the platform will face huge challenges now and in the future. The essence of a personalized recommendation system is to connect information and users, improve user satisfaction, and obtain a reasonable user group for information publishers efficiently. The aim is to maximize the transformation of platform value.

MaxCompute Product Background

MaxCompute is an analytics-oriented, enterprise-class SaaS-mode cloud data warehouse that provides fast, fully managed online data warehouse services in a serverless architecture. It removes the limitations of traditional data platforms in terms of resource extensibility and elasticity, minimizes user O&M investment, and lets you analyze and process large amounts of data cost-effectively. Tens of thousands of enterprises are performing data computing and analysis based on MaxCompute to transform data into business insights efficiently.

Artificial Intelligence Recommendation Product Background

Artificial Intelligence Recommendation, based on Alibaba's leading big data and artificial intelligence technologies, solves the problem of association between user needs and content display according to users' interest preferences. It provides cloud recommendation services and Machine Learning Platform for AI for global enterprises and developers by combining Alibaba's accumulation in multiple industries, such as e-commerce, content, news, video livestreaming, and social networking.

Artificial Intelligence Recommendation is commonplace in our life and work. For example, an industrial customer has a lot of information for employees to view in the enterprise. Artificial Intelligence Recommendation is not only for ToC enterprises. ToB enterprises in the enterprise, including some well-known enterprises, use a lot of information internally, which is convenient, fast, and efficient for employees to view. This demand is also a common phenomenon in ToB enterprises. The demand for ToC enterprises is more obvious. If you pay attention to the industry report, you will find that the dividend of the Internet has basically ended, and the penetration rate of users is already very high. When the industry development and user increment reached the bottleneck, the online time of users increased from 6.1 hours in 2020 to 6.3 hours in 2021, and the online time of users was stagnating. Then, the enterprise will also face two problems. One problem is that the cost of acquiring customers for enterprise users and the difficulty of increment are getting higher. Second, with the increasing cost of acquiring customers, how can existing customers increase their online duration? These two questions show that determining how to fully and efficiently transform users is very important for enterprises (regardless of new or existing customers.)

II. Artificial Intelligence Recommendation Business Scenarios and Values

Which Industries Need Artificial Intelligence Recommendation?

Whether it is the e-commerce industry, the content industry, the news industry, or the industries mentioned above, various industries will use Artificial Intelligence Recommendation. Everyone will think of ToC's industries when they hear Artificial Intelligence Recommendation but ToB's industries also need Artificial Intelligence Recommendation because there is a lot of information and articles in an enterprise. The recommendation itself has penetrated all aspects of the products we usually use. Its presence is apparent in the perspective of product form, e-commerce, content, and news. The content that we are more interested in is predicted through big data algorithms, which improves the user experience significantly.

1

Pain Point

E-Commerce/Retail Industry

  • High customer acquisition cost and high churn rate
  • Low transaction conversion rate and low repurchase rate
  • Low efficiency of manual rule recommendation and poor effect

Content/Information/Video Industry

  • High customer acquisition cost and high churn rate
  • Low user stickiness/popularity
  • Low efficiency of manual rule recommendation and poor effect

Scenario

E-Commerce/Retail Industry

  • App Homepage Commodity
  • Store Homepage Commodity
  • Item Details Page
  • Store Activity Page
  • Other

Content/Information/Video Industry

  • Homepage Commodity
  • Content/Information/Video Details Page
  • Thematic/Thematic Commodity

The Effect after Customers Have Used Artificial Intelligence Recommendation

As shown in the table below, after the enterprise has finished using Artificial Intelligence Recommendation, the various effect indicators have been improved significantly:

2

Artificial Intelligence Recommendation Architecture Based on Data Warehouse

Artificial Intelligence Recommendation is a typical big data application scenario that is strongly dependent on the data warehouse. In terms of data connection, Artificial Intelligence Recommendation leverages MaxCompute. It can analyze and manage enterprise data better by applying some computing capabilities within MaxCompute, thus realizing Artificial Intelligence Recommendation business scenarios. If your data volume is small, you can also use the SDK to push data to Artificial Intelligence Recommendation to implement your business scenarios. Artificial Intelligence Recommendation supports algorithm customization and business customization, giving enterprises full flexibility, autonomy, and control.

3

III. Self-Built Pain Points and Product Advantages

Self-Built Pain Points (Artificial Intelligence Recommendation)

Costs

  • High Requirements on Staff: Self-built recommendation systems have high requirements on staff. System development, data processing, recall, and development and optimization of sorting models all require the long-term participation of senior developers and algorithm engineers.
  • Long Cycle for Launching: The recommended system has a complex architecture and needs to be optimized consistently until it meets the requirements for launching. The development time required is more than three months.
  • High O&M Costs: High maintenance costs in the later stages of upgrades and iterations and self-built systems

Recommendation Effects

  • Difficulty in Effect Tuning: Applying mainstream algorithms may not have good results. You must also consider multi-dimensional recommendation effects, such as application domain data, relevance, novelty, and timeliness.
  • Core Metrics are Difficult to Unify: If you want to improve CTR while increasing user stay duration, you cannot take multiple core metrics into account.
  • Long Iteration Cycle and Fast Pace of Business: The pace of business is usually very fast, but the internal workforce is limited. It is necessary to have a fast pace, high service stability, and quick results with a smaller workforce.

Subsequent Maintenance

  • Difficult Adaptation: Any set of recommendation engines cannot fully adapt to the business demands of enterprises. Alibaba Cloud Artificial Intelligence Recommendation provides black and white box integration.
  • Operational Ease of Use: The recommendation system is complex, and it is difficult for operators that do not have algorithm knowledge but need intervention.
  • Service Stability: Recommended scenarios are usually used on high-traffic pages and require high system performance, stability, and elasticity.

Challenges for Self-Built Data Warehouses

To drive business growth with data, enterprises need to build and manage data warehouses. When building and managing data warehouses, enterprises face the following challenges:

  1. High start-up cost, long construction cycle, and value is difficult to verify quickly
  2. How can you deal with diverse data, embrace new technologies, and fully tap the value of data?
  3. It is difficult to share enterprise data assets, and the cost of data innovation is high.
  4. Complex platform architecture and high operating costs
  5. Extensibility and elasticity to meet business needs

4

Business Growth

It is no problem to build an Artificial Intelligence Recommendation system through self-built methods, but how can we ensure the effect of building recommendations? The effect of Artificial Intelligence Recommendation is defined differently in different industries. While improving the effect, a lot of work needs to be done in the recommendation system.

Multi-Scenario and Business Adaptation

  • Different enterprises (different business objectives and effects)
  • Different stages of the same enterprise
  • Different scenarios at the same stage
  • Different demands in the same scenario

Experimental Platform

  • Selection
  • Recall
  • Sort
  • Business Demands

Alibaba Cloud Artificial Intelligence Recommendation = leading algorithm capability + stable and efficient engineering system + complete and flexible product capability

5

Advantages of Artificial Intelligence Recommendation

Out-of-the-Box

  • Highly productized and industrialized (e-commerce/content/news)
  • Coverage of the entire link (supports Umeng SDK behavior collection)

Recommendation Accuracy

  • Industry and scenario oriented optimization
  • The industry and Alibaba-developed mainstream algorithm encapsulation
  • Guess what you like and related recommendations
  • Multi-objective model training

Fully Managed

  • Ensure online service stability
  • Flexible lifting and distribution services
  • Rich data quality diagnostic features, online service monitoring, and alerting

Flexible Adaptation

  • Operations Assistant: Products and operations can intervene in recommendations quickly
  • Development and Algorithms: Integrate powerful offline and online link development capabilities

Benefits

Simple and Easy to Use

  • Data Warehouse

    • Optimized high-performance storage and computing for data warehouses
    • Multi-service pre-integration, standard SQL, simple development
    • Enterprise-class services-built-in with comprehensive management and security capabilities

  • As a Service

    • Serverless, O&M-free
    • Pay by volume (no money)
    • Automatic upgrade

6

Matching the Elastic Expansion of Business Development

  • With dynamic scaling, there is no need to advance capacity planning to meet sudden business growth.
  • Storage-Compute independent scaling with no extensibility limits
  • Business growth performance is not degraded.

7
8

Support for Multiple Analysis Scenarios

Near-real-time, interactive analysis, AI analysis, and data lake analysis are enhanced to support more business scenarios.

  • Machine Learning Platform for AI (PAI)

    • PAI-Native Integration
    • Built-In Spark ML
    • Mars Scientific Computing

  • Traditional Data Warehouses

    • ETL: SQL +UDF
    • BI: Query Acceleration and MC-Hologres

  • Data Lake Analytics

    • SQL External Table
    • Data Federation
    • Processing of Unstructured Data

  • Real-Time Data Warehouse

    • Near Real-Time Writing
    • Near Real-Time Analysis

Open Platform

Open interfaces and ecosystems are supported on fully managed services, providing flexibility for data, application migration, and secondary development.

  • Open Management Port

    • Java/Python SDKs
    • Standard JDBC Interface

  • Data Access

    • Open interface for importing and exporting data (tunnel upload and download)
    • LOAD/UNLOAD: Free, high performance import /export orc with parquet open format to the data lake

  • Compatible with mainstream syntax

    • MaxCompute SQL is compatible with Hive SQL syntax.
    • Spark on MaxCompute supports native Spark Streaming.

  • Open Ecosystem

    • Flink/Kafka/Presto Connector
    • Airflow/Azkaban/Kettle scheduling
    • Supports Tableau, FineBI, and common JDBC BI tools

9

IV. Configure and Start Services

Basic Process of Product Usage

Cost Optimization: You can use the Starter Edition instance test in the POC phase. After the test is completed, you can upgrade the service to the Standard Edition with a few clicks and switch workloads.

It is mainly divided into four steps:

  1. Data Preparation
  2. Create an Instance
  3. Policy Configuration
  4. Test and Release

It takes only 3-5 days for engineers to build basic services.

10

Start an Instance Using Historical Data (MaxCompute)

11

Configuration in the Console

12
13
14
15

Use Server SDK to Start an Instance

16

Start an Instance Using Umeng + Server SDK

17

V. Interpretation of Key Functions

Guess What You Like + Related Recommendations

For the e-commerce industry, AIRec provides two features: "Guess What You Like" and related recommendations. The "Guess What You Like" feature is used on the homepage and commodity tab pages. Related recommendations are used on commodity display pages and details pages. The recommendation community can be built based on the AIRec for the content industry.

18

Real-Time Recommendation

Real-time interaction is an essential basic function to promote consumer immersive browsing. To implement real-time interactive recommendations, AIRec learns a user's interests and interest changes in real-time and updates the next set of recommendation results to be pushed to the user.

19

Negative Feedback on an Item

In the process of interaction between the recommendation system and users, recommendation results that do not meet the user's expectations may occur. Here, negative feedback becomes an important entry point for recommendation and user dialogue. The Artificial Intelligence Recommendation supports the negative feedback function of a single product dimension, a product category dimension, and other product feature dimensions.

20

Business Policy Configuration

Real-time online and offline ensures the recommended quality of high-quality products. You can also set deduplication rules to ensure that the same product or content is not recommended to users repeatedly within the specified time interval. After setting category diversity rules, the diversity of recommendations is guaranteed, and product singularity is avoided.

21
22

Diverse Scenarios

Scenarios are used as traffic portals for personalized recommendations and can be customized differently on different pages and different user groups. For example, the recommendation of the first page, the recommendation of the channel page, the personal center page, the search empty result page, the product details page, the shopping cart page, etc. Recommendation PLUS supports customized differentiated scenario selection rules.

23
24

Configure Rules in the Console

Choose Business Customization > Scene Management on the left-side navigation pane:

25

A/B Testing Platform

The process is listed below:

26
27

Configure Rules in the Console:

28
29
30

Machine Learning Platform for AI

100+ algorithm components, complete business development framework, drag-and-drop development platform

31

Recommended business logic encapsulation and algorithm models out of the box:

32

Visualize and analyze model metrics:

33

Supports offline and online model deployment methods:

34

VI. MaxCompute + Artificial Intelligence Recommendation Learning

MaxCompute Learning Path

The official website of MaxCompute, a cloud data warehouse in SaaS mode, provides a large number of learning materials to help you start your cloud data warehouse journey.

Artificial Intelligence Recommendation Learning

Artificial Intelligence Recommendation (AIRec) is based on Alibaba's leading big data and artificial intelligence technology, combined with accumulation in multiple industry fields, such as e-commerce, content, news, ApsaraVideo Live, and social networking. It also provides personalized recommendation services for global enterprises and developers.

MaxCompute Big Data Artificial Intelligence Deep Learning E-Commerce O&M Machine Learning Platform For AI Search Recommendation Development Tools AIRec Real-time Recommendation
0 0 0
Share on

Read previous post:

Diagnosing Slow Jobs in MaxCompute with Logview

Read next post:

MaxCompute + Open Search-Based Search and Development Practices in E-Commerce and Retail Industries

Alibaba Cloud MaxCompute

137 posts | 20 followers

You may also like

Comments

Alibaba Cloud MaxCompute

137 posts | 20 followers

Related Products

More Posts by Alibaba Cloud MaxCompute

See All