By Zhenzi
I want to summarize my past experience and make a plan for the long-term development, explaining what frontend intelligence is more clearly.
In 2017, the concept of "frontend intelligence" emerged. At that time, I was in charge of UC International Browser and had some attempts from the frontend scenario: user experience measurement, UI automated testing, and code generation based on a design draft. In 2019, www.imgcook.com was released on the D2 frontend forum. Then, the concept of frontend intelligence was officially established.
The first step was to improve frontend R&D efficiency with the help of intelligence. We wanted to take imgcook's design draft generation code as a starting point and solve partial R&D efficiency issues, such as compiling frontend cut-in UI code facing frontend engineers.
If design draft generation code is the foundation, then, from the perspective of the users we serve, frontend engineers, the accuracy, and the availability and maintainability of the generated code are the keys to success. Through the machine vision identification capability, the understanding capability of deep neural networks, and the frontend UI code expressive ability of the code output rule engine, the machine can take the place of the frontend engineers to "see" design drafts, "cut drawings," and "generate UI code." Thus, with the improvement of identification, understanding, and expression ability, it brings about the promotion of accuracy, usability, and maintainability.
Next, we need to expand and enrich the usage scenarios to improve delivery efficiency. We apply intelligence to smart UI to resolve the development work of massive modules and components brought by personalized UI. Then, we apply intelligence to server glue layer code generation in replacement of the server and Node FaaS to write specific business logic based on the business capabilities and apply intelligence to automated UI testing. The test time and the labor cost of the test investment are reduced significantly.
Finally, we need to retrieve business value from frontend intelligence. We apply intelligence to marketing campaigns and channel shopping guides and use the intelligent UI recommendation algorithms to generate users undertaking scenarios automatically to improve core business metrics. Secondly, we apply intelligence to the requirement generation code platform BizCook and build indicator-driven a multi-role collaboration business development platform. This enables online collaboration and automatic coding, improving the business trial and iteration efficiency significantly. Thirdly, we apply intelligence to material compliance review to solve the merchant material compliance review problems of Tmall Try-Before-You-Buy and advertising material compliance review problems of the Tmall Co-Branding Project. It also helps the Group establish a general and configurable material compliance review platform.
These processes show our efforts towards the extreme production of R&D efficiency, delivery efficiency, business trial, and iteration efficiency by working extra hours and with the help of more than 30 frontend services in 52 businesses.
2-3-1-5:
Five Key Links:
The phrases "business efficiency, delivery efficiency, and R&D efficiency" seem easy to understand. However, in Alibaba's current scenarios, there are a lot of difficult problems and challenges behind these phrases.
Taking UICook's intelligent scenario and intelligent design as an example, one page may contain more than ten modules; one module may contain about five components; one component will have more than 20 styles of design; one module involves 100 times of coding, debugging, testing, and release; one page has at least 10 x 100=1000 times of page setup, configuration, data binding, and release scheduling. What's more, there are more than 1000 pages for one big promotion. The work above needs to be done 1000 x 1000=1 million times for Alibaba Double 11, Double 12, Special Purchase Festival for the Spring Festival, and a series of marketing campaign scenarios. It is difficult to complete optimization in some local fields with only the help of tools. Therefore, in a scenario that is one thousand times more complex than other common scenarios, it may bring about new problems in a new field since intelligence creates new frontend productivity. This is an opportunity for Alibaba's frontend and frontend engineers alike.
In the preceding picture, the process in which intelligent UI is generated into applications is split up, which will be merged at the end. It is crushed first and then digested. The digested product is called the "top-level design of frontend intelligence," which has been in development and advancing steadily for three years.
We are committed to building intelligent UI assets that can support the entire business system of Alibaba. This is the core of the whole design. We need support from the intelligent generation and Intelligent UI recommendation algorithms, the upgrade of intelligent design production technologies, and BizCook and UICook (Jingmi)* as the iterative evolution for the technical products of direct service business behind the UI processing and production. Meanwhile, we also need to convert intelligent UI generation, UI recommendation algorithm, and intelligent design system to general capabilities and basic technologies to serve more businesses. As such, the intelligent UI assets can be applied in various business scenarios. In addition, the usage metrics and usage conditions can be delivered reversely with data as a carrier to form a closed loop, driving the frontend intelligence system forward to become more mature and perfect.
Among them, the production economy and convenient consumption are the issues concerning us the most. With the help of D2C and S2C generated intelligently, we built machine production UI, interactive logic, and business logic capability to reduce the production cost of UI, which is referred to as "production economy." Today, different Alibaba businesses are developing at different stages and encounter different problems. Therefore, the convenience depends. Just like gears, only when the occlusion is better can we move forward together. For any business under market exploration, the key to "convenient consumption" should be the out-of-the-box feature to minimize the access and use costs and improve the access and use effect. For the regular business under market expansion, the key to "convenient consumption" should be customization and openness to enable on-demand customization of business while meeting the complex users and businesses' needs. Meanwhile, technical personnel should be allowed to customize their business platforms and technical engineering systems.
Customers and business parties often ask me a very philosophical question, what is "business intelligence," and what is "intelligent business?" On this issue, facts speak louder than words. The answer and experience are gained through practices. In short, the purpose of this task is to put these products and technologies through a series of product matrices and technology matrices to a business system, rather than simply defining some technical products from a technical perspective.
Similar to development, there are basic philosophies behind a framework and a perfect theoretical system behind the philosophies. Without the theoretical system, we can't complete abstraction and cropping in complex scenarios in the future or some changeable scenarios. Frontend intelligence technologies and business systems must be supported by a set of theories, including business indicator driver (BizCook), product definition (BizCook & Jingmi), technology generation (P2C, D2C, and S2C), and operation and use (Xiaoer workbench).
First, what is business metric-driven (BizCook)? Today, Alibaba has many businesses, such as Taobao, Tmall, and Juhuasuan (a group purchase service on Taobao), with various definitions of business metrics. Even if the metrics are uniform, the business scenarios are different, and the definitions of user activity are also different. For example, the Tmall Try-Before-You-Buy service requires users to pick samples, whereas the shopping activity of Taobao requires users to browse and follow content. Therefore, the definition of service metrics must be made available in each service scenario. However, the driver is not the same, which can be used uniformly in focus (definition), transparency (transmission), and iteration (execution) mode. Only when the goals are scenario-specific and drives are unified can effective business R&D model upgrades, chemical fission, or derived possibilities be achieved.
The second one is about product definition (BizCook & Jingmi). Product definitions are for core product elements. Intelligent UI asset abstracts the definition and understanding of the element in each scenario to a common divisor, allowing you to deploy a custom user experience based on standards and flexibility in each scenario. Intelligent UI asset is an important product infrastructure for digital business. Regarding consumption, a product may have more than 20 scenarios. One scenario may have dozens of temporal and spatial dimensions, and one dimension may correspond to users in hundreds of layers. Only with the help of intelligent UI assets can products be defined. This way, users can experience the value of digital commerce under their own interests, preferences, and habits. Therefore, technicians and designers can work together to integrate algorithms and data assets continuously to restore our understanding of consumers. This is done on the premise of security, privacy, and compliance and standardized to the unified product design and definition field in the frontend intelligent terminal intelligence field. Besides, the vector data of abstract user features and scenario features are transmitted while protecting users' privacy.
The third one is the technology generation (P2C, D2C, S2C). Algorithms and data work together to create intelligent UI assets for Service. Technology with no services provided cannot reflect the value of technology, which is a waste of cost. The supply of intelligent UI assets is driven by services and a convergence between the upper-layer consumption of intelligent UI. The bottom-layer supply of intelligent UI assets is generated with the help of technical teams of different businesses for scenario-based and customized implementation. At the same time, collect and roll black data from the consumer side, use the indicator-driven technology generation system to push for iteration and maturity in a targeted and efficient manner.
The last one is about operations (Xiaoer workbench). Driven by business metrics, product (productivity), design (design capability), and technology (business capability) work together and only deliver the definition of the product. This is like object-oriented programming, where we define a cup:
class Cup { init ( material, height, diameter... [The path used by the operation is here]) { material handling ([The results of operational use are here!]) }
func material processing (material){ if (material = = crystal){ 1. Check crystal inventory. 2. Estimate crystal cost. 3... } } }
[take operation product selection as an example] new cup (crystal, 10 cm, 5 cm...) [The starting point for operational use is here]
In marketing operations, the product definition for the delivery of products, designs, and technologies centering on business metrics is the possibility of product. For example, the material may be glass or crystal, and the operation decides the specific parameters for the operation based on business metrics, inventory, cost, and other conditions.
In theory, the second step is to establish a set of tools undertaking theory. However, theories without a good set of tools for continuous evolution cannot be implemented. So, a set of tools is necessary.
The organization has changed, the business has changed, and the personnel has switched to other positions. Only through the productization of tools and technical capabilities can our system continue to move forward on the shoulders of predecessors. In Alibaba's scenarios, we eventually generate a set of basic frontend intelligence tool matrices by practice, as shown in the following figure:
We will use these tools to serve various internal businesses of Alibaba. At the same time, we will also serve our merchants and users, injecting vitality into merchants and ISV ecosystems to reduce costs, providing predictable, satisfactory, and easy-to-use products.
For big promotion scenarios, such as the Double 11 Global Shopping Festival, marketing products, and channel services, the intelligent UI assets can "make big promotion function as normal," reduce the planning, design, and implementation costs of the big promotion, improve the undertaking efficiency and business efficiency with rich scenarios, people-merchandise-venue mode, and personalized user products. This makes the daily shopping guide business like a "market." We will also launch services targeting the global app matrix of the Alibaba Group and deeply integrate with traffic scenarios to increase the frequency and effectiveness of the second- and third-ring media's engagement with users. With the help of intelligent UI assets and related tool product systems, we can build internal capabilities for agile business innovation and find our direction and pace with the business to deal with market changes and threats from competing products flexibly.
We need to combine these tools with organizations and Alibaba's existing environment. Let's use a simple example, the "zero R&D" campaign and the application of Jingmi in a smart venue are conducted before the Double 11 Global Shopping Festival every year. Based on the Double 11 scenario, we hope to make the products, design, generation, and application of digital operations capabilities available to users so they can better understand and consume intelligent UI assets. Through continuous learning and application, we wish to make it a first choice for you to solve problems with intelligence, efficiency, high quality, and personalization.
In addition, in daily business, the algorithm model capability is used to map and translate the concepts of automation, reduce the introduction of new concepts, and enable the business, PD, designer, operation staff, technical staff, and quality staff to be jointly responsible for business metrics in their respective familiar concepts.
In addition to combining methodology, tools, and organizations, we need a system and mechanism, regardless of whether it is a business system, product system, or design system. The most valuable parts often occur in the intersection. Supported by methodology, tools, and organizations, this intersection is mutually restrictive. Any local innovation can be constrained globally by other systems, and they can also empower each other. Any local innovation can deliver values to other systems globally and breed chemical reactions. This leads to cross-border, cross-domain, and cross-system global innovation.
A customer's perspective is what we need first to provide service. Only by knowing who we are serving can we know what problem to solve. The core customers of frontend intelligence development are closely related to our development stage. In the initial stage, for technical staff, we focus on solving the problem of customer-side frontend business R&D efficiency to build the capability of design draft generation code. In the mid-to-late stage, for business staff, we must apply the technical capability to the business and solve the problem of business lack of consumer products to build the capability of transparent, fast, and direct business intervention. For PD, we solve the problems of agile business innovation and fast trial and error to build the capability of "the WYSIWYG requirement on the code generated in the design draft, which can be delivered online after being marked." For designers, we solve the problems of difficult execution of design specifications, high cost of design innovation and R&D, and difficult implementation to build the capability of design-production unification.
Operational use has not been defined as a core customer in our system because the Xiaoer workbench led by Yuanxin solves a series of operations problems of investment promotion, product selection, and delivery with a complete data-driven operation technology system established and helps the business customization based on their needs by means of PU and SOP arrangement. "Noah," led by Shuwen, with the help of the experience accumulated for years by "Ark" in big promotion activities, reduces the use cost of the Xiaoer workbench. We only need to introduce "Noah" or other PU and SOP capabilities into the frontend intelligence-enabled business process to achieve smooth operation.
Quality assurance has not been defined as a core customer in our system because the quality assurance system led by Qingling has been cooperating with us in-depth in the early stage of frontend intelligence. The Quality of the ATS automated test module and automated testing of TMQ based on machine vision and algorithms have been achieved perfectly. Moreover, the underlying technology system and upper-layer openness are sufficient to support our "out-of-the-box" feature, ensure the consistency of the features and experience between the two sets of systems, and support the quality assurance engineers to perform quality acceptance and control.
For Alibaba, technical staff constitute a huge group and customer services involve complex scenarios open to end users. Cross-platform, requirement change, and personalization are the core issues in these complex and changeable scenarios. A technique for eliminating repetitive labor is required. Otherwise, only the technicians themselves can fill in this big pit. For technicians, cross-platform, requirement change, and personalization do not have much challenge and innovation for technology but only a problem of time cost, adapting to different platforms, realizing the changing requirements, and developing different personalized products one by one are the "repetitive work" that needs to be eliminated urgently.
Only by studying cross-platform business R&D issues can we define the functions of RunCook and solve the problem of adaptation and degradation of host business capabilities in the business R&D process accurately. Only by deeply studying the change of requirements can we define the functions of BizCook, imgCook, and LogicCook accurately and understand how the metrics defined by the business are described as requirements by PD through the NLP algorithm model. How can designers use design language to express PD's needs? How can design drafts use imgCook to generate UI and interaction logic, changes in UI content, and status? How can LogicCook be obtained from PaaS and instantiated as glue layer code of Node FaaS and mounted to the UI and interaction logic? Only when we deeply study consumers can we define the functions of UICook and Jingmi and reduce the cost for frontend business R&D personnel accurately to develop product functions for users of different levels through the intelligent design-production unification and the ability of code generation, bringing the ultimate product experience for consumers.
When introducing the data middle ground platform, Xiaopeng said, "Take observatory as an example. Xiaoyaozi (Daniel Zhang) can see the business governance of the whole group in a macro view through the observatory, which is the top node. The lower sub-node involves the matrix between different business executives and businesses. Each business has its own business data logic and data decision-making system. The lower-sub-node involves each activity or some nodes, which basically form a decision-making system. I am deeply convinced of this view. Only systematic decision-making can make the business work, which, however, may still be hindered by the execution."
There is a vivid metaphor in the technical products: P9's strategy, P8's planning, P7's design, P5's execution, and finally, a good idea and theory fail to convert to a good technical product. Why can't it be P9's strategy, P8's planning, design, and execution? This is because information may be lost during transmission. Simply put, Aunt Zhang said you went home with a female classmate after school, Aunt Li said you were in love with a female classmate, and Uncle Wang said you got married to a female classmate. Based on the study information loss of information theory, signals will be disturbed when they are transmitted in the medium, and the fluctuation of electrical signals in the value range does not meet the requirements of signal voltage, resulting in the loss of some codes in the information. The prejudice of Aunt Li and Uncle Wang is the interference that leads to information loss. Next, let's talk about how the business personnel can prevent interference with the help of some knowledge in information theory.
When it comes to information theory, coding and information transmission are inevitable. Information needs to be encoded in the process of transmitting information and how to use the minimum amount of codes to represent the information to be transmitted is the goal of our research. Let's assume the communication between two areas involves four types of messages, A, B, C, and D. Which coding method should be used to minimize the use of resources? If the probability for the occurrence of these four types is the same, which is 1/4 , then the equivalent encoding method should be used, which is:
This way, the optimal encoding effect can be achieved. The average encoding length is:
However, what if the probability is not the same? For example, the probabilities of messages A, B, C, and D are 1/2, 1/4, 1/8, and 1/8.. How should it be coded? To keep the average length of encoding as small as possible, messages A with a high probability of occurrence should use a short encoding method, and messages C and D with a low probability of occurrence should use the long encoding method. This encoding method is called variable length encoding, which achieves better effectiveness than equivalent length encoding. The following table shows the encoding method, which is the optimal encoding method.
The average encoding length is: 
, which is better than equivalent length encoding.
A lot of ambiguity and redundancy issues exist in the data metrics that the business personnel use to make decisions. This requires a coding mechanism to encode the business metrics effectively, which ensures more accurate and effective transmission of decision information. The reason is that the more information redundancy, the greater the probability of loss, the lower the effectiveness of coding, and the more redundant information coding. We have rebuilt the metric system and the encoding method for customer businesses on BizCook. The first step in the effective transmission of decision-making information is to encode the information contained in the metrics.
If there is still a probability distribution of message sequences that satisfies the q distribution but uses the optimal coding method of p distribution, then its average coding length is:
Where, Hp(q) , known as the Cross Entropy of q distribution relative to p distribution. It measures the average encoding length when the q distribution uses the optimal encoding method of the p distribution. Cross entropy is not symmetric, which is Hp(q) ≠ Hq(p). The significance of cross entropy is that it provides us with a way to measure the different degrees of two distributions. The greater the difference between p and q, the greater the cross entropy Hp(q) compared to H(p), as shown in the following figure:
Their different sizes are Dp(q), which is called Kullback-Leibler Divergence (KL Divergence) in information theory. It follows the following formula:
The KL divergence can be regarded as the distance between two distributions or a measure of the divergence degree between two distributions.
For business personnel, business metrics that are disassembled layer by layer around their own two or three core goals also have q and p distribution. We can measure the deviation degree of products, design, and technology in undertaking these metrics on BizCook according to cross entropy or determine the information loss of decision-making metrics in the undertaking process on BizCook based on KL divergence.
It is the same as single variables; if there are two variables X and Y, we can calculate their Joint Entropy:
When we already know the distribution of Y in advance, we can calculate Conditional Entropy:
Some information may be shared between variables X and Y. Therefore, the information entropy can be considered as a piece of information, as shown in the following figure:
The information entropy of single variable H(X) or (H(Y)) is generally less than the multivariable H(X,Y). If conditional entropy is also included, the relationship between them can be seen more clearly from the information bar.
It can be seen that
We define the overlapping part of H(X) and H(Y) as Mutual Entropy of X and Y and record it as I(X,Y) followed by I(X,Y) = H(X) + H(Y) – H(X,Y).
The mutual entropy represents the information contained in X about Y or vice versa. The X and Y information that does not overlap is defined as Variation of Information and recorded as V(X,Y). Then, V(X,Y) = H(X,Y) – I(X,Y) can measure the gap between variables X and Y. If V(X,Y) is 0, it means one variable can check all the information about the other one. With the increase of V(X,Y), X and Y are less correlated. The representation can be seen in the figure below:
For business personnel, it is easy to measure the correlation between metrics and their goals based on the mutual entropy when the product, design, and technical departments undertake their own decisions. BizCook can measure data using mutual entropy and variation of information to help business personnel solve the deviation from the implementation of the target.
Since business decisions are encoded as business metrics, business production metrics are encoded as product metrics, design metrics, and technical metrics. With the help of the frontend intelligent generation technology system, decision-making and execution will be unified unprecedentedly, other than being separated. BizCook solved the information loss problem arising from the example we mentioned above.
Unlike the business service, in customer service, the design in most cases directly determines the vision and interaction of the UI, which is also a core part of the customer service. Therefore, in most cases, products and designs jointly define the functions of the products and can be put together for discussion.
The tools for product definition products are requirement documents, and the tools for designing and defining products are design drafts and interaction drafts. The two drafts discuss: What kind of users? In what scenarios? What can be seen? What can be done? The requirement document is more abstract in the visual aspect and more specific in the functional aspect, while the design draft and interaction draft are more specific in the visual aspect and more abstract in the functional aspect. As a result, PD and the designer decide to integrate the requirement document and drafts. Therefore, most of the design drafts we usually obtain in the production process are like this:
What a wonderful combination of vision and function! Integrating the design and product functions makes the design draft clear for us to see. Since the PD and designers prefer this product definition and description method, why don't we apply the method to the frontend intelligence field? Therefore, with the ability of "WYSIWYG product definition based on the code generated from the design draft" born on BizCook and based on the code generated from the design draft of imgCook, the business logic code generation and recommendation of LogicCook are expanded. This way, PD and designers can complete MVP in the process of product definition and conduct a small batch of product verification online smoothly and gradually.
As you can see from the figure above, the content in the preview area on the left side is all generated by imgCook through the design draft. Based on this, the PD can select a block to define business for this block to undertake, related business metrics to complete, functional constraints, and personalized service. From business to sub-businesses, pages, and blocks, we can connect the product definition process with business decisions and business metrics seamlessly.
Therefore, when BizCook is serving the two core clients (product and design), the first principle is to conform to the working habits of PD and designers and use the familiar concepts of PD and designers to describe the definition of the product and the function of the business, without introducing new concepts and obscure technical concepts. This way, the product can be defined as "highly authentic," and the costs for PD and designers to understand and use the product can be reduced significantly.
From serving frontend R&D personnel at the beginning, serving product and design personnel later, and serving business personnel in the future, there is a main idea behind this. Efficiency-oriented innovation in the technical field must be transitioned to production and the business field for upgrades. What slows the progress of most technicians is that the production only stalls in the technical field while being separated from the business.
We must realize that the transition from technology to production or business is a complicated and long process. Entering the production field requires a global view and architecture capabilities, and entering the business field requires a global view and business ability. Without change and improvement, development is meaningless.
Most of my colleagues still feel confused after a series of practices. In the beginning, they seem to know exactly what to do, but when they are doing it, they feel at a loss. Generally, I think we need to look inside and observe the whole. The so-called look-inside requires us to find out the reasons within ourselves. What is really valuable is the thinking behind the plan and achievements, which cannot be sensed, perceived, or understood by the façade. Take two tuning forks as an example. The reason why knocking one can cause the other to resonate is that the two tuning forks have the same texture. If we cannot resonate with other people in their views, our texture is not consistent with the texture of others. So, we simply practice ourselves. The so-called observe-the-whole requires us to find the reason why we are doing this. Don't be greedy or aim too high. Then, understand each step we take: Is it too light? Too heavy? Too soon? Too slow? It is also necessary to look back and examine every step made in the past.
Nowadays, frontend intelligence has made some technical achievements while coupled with many problems. The attempts to expand frontend intelligence to the production and business fields have received some small successes but suffered from the incapability in the process and at the source. This requires us to look at the big picture, such as major trends and major strategies, as well as small closed loops, small problems, and small details. We should try to make it easy to learn and use intelligent UI assets and frontend intelligence, solving practical problems for first-line business personnel in the R&D process.
To this end, we visualize the future development into some general directions, major trends, and major strategies. At the same time, we also disassemble it into "out-of-the-box algorithms," "business results," and a series of small closed loops, small problems, small details. Under the guidance of Professor Zhanyan, we established the theme of "user experience upgrade" in the new fiscal year. We focus on the problems we want to solve and implement frontend intelligence:
Therefore, we set our OKR (under discussion) to constrain our development direction and goals and give a clear and specific path:
I hope we can build data, labels, algorithms, materials, and more in the new year. Intelligent UI assets can continue to help us upgrade our production capacity. Meanwhile, with the help of PipCook, we can lower the threshold for frontend applications and make the frontend intelligence universally open to each frontend of the Alibaba Group. Also, I hope every colleague that participates in frontend intelligence will find more technical, engineering, and business possibilities brought about by machine learning and algorithms in their business scenarios and create more business value with their wisdom! Let's do this together!
Imgcook: Intelligent Code Generation from Design Drafts with a 100% Accuracy Rate
Intelligent Design: Detailed Interpretation of the Color System
66 posts | 3 followers
FollowAlibaba F(x) Team - February 25, 2021
Alibaba Clouder - October 21, 2020
Alibaba Clouder - April 15, 2021
XianYu Tech - December 13, 2021
Proxima - April 30, 2021
Alibaba F(x) Team - December 14, 2020
66 posts | 3 followers
Follow
Alibaba Cloud Linux
Alibaba Cloud Linux is a free-to-use, native operating system that provides a stable, reliable, and high-performance environment for your applications.
Learn More
Offline Visual Intelligence Software Packages
Offline SDKs for visual production, such as image segmentation, video segmentation, and character recognition, based on deep learning technologies developed by Alibaba Cloud.
Learn More
Platform For AI
A platform that provides enterprise-level data modeling services based on machine learning algorithms to quickly meet your needs for data-driven operations.
Learn More
Machine Translation
Relying on Alibaba's leading natural language processing and deep learning technology.
Learn MoreMore Posts by Alibaba F(x) Team
