Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Alibaba Cloud Realtime Compute for Apache Flink Provides Solutions for the Financial Industry

Alibaba Cloud Realtime Compute for Apache Flink Provides Solutions for the Financial Industry

This article discusses the challenges of today's finance industry and explores the solutions Realtime Compute for Apache Flink offers.

Step up the digitalization of your business with Alibaba Cloud 2020 Double 11 Big Sale! Get new user coupons and explore over 16 free trials, 30+ bestselling products, and 6+ solutions for all your needs!

Industry Challenges

In recent years, with the rapid development of the Internet finance industry, various kinds of business within traditional financial institutions, such as exchanges, securities companies, and banks, have been increasingly integrated with the Internet. For example, major financial institutions have launched mobile app clients to support users to handle various financial services, such as mobile payment, financial management, online lending, and the purchase of financial products. The new business methods also lay down new requirements for the financial industry.

  • Long-Chain and Fast-Changing Business ¨C Based on business requirements, the analysis of user behavior traces and transaction records is needed to deal with rapid business changes.
  • Growing Demands for Real-Time Data ¨C More real-time data is needed to support business decision-making. For example, you may need to adjust different resource levels based on traffic monitoring and sales. Besides, you may need real-time data to enhance interaction with users in some activities. When it comes to the marketing of financial products, it is necessary to adjust strategies according to real-time user behaviors.
  • Higher Requirements for Data Quality ¨C Data directly affects the effectiveness of business decisions and online operations. Data quality determines whether the computing results can be used as a basis for decision making.
  • More Requirements for Real-Time Risk Control ¨C Traditionally, risk control systems rely on expert rules to play their role. Now, they must be transformed in real-time to provide intelligent risk control in typical financial scenarios, such as credit default, account security, and loan fraud.
  • Fraud Detection ¨C Real-time monitoring and identification should be conducted for user accounts, transactions, and other relevant matters in the process of business handling. Quick identification of abnormal transactions can reduce the possibility of property loss.
  • Credit Evaluation ¨C In the audit process of the bank business, such as loans, quick identification of false information, information fraud, and other malicious fraud, high-risk group identification and overall credit risk assessment should be completed to shorten the audit process.

Solutions

Solutions based on Realtime Compute for Apache Flink allows financial institutions to easily address the preceding challenges. We use Apache Flink to build real-time data warehouses and real-time anti-fraud systems to help financial institutions quickly build a real-time risk control system. The data warehouse architecture is shown below:

1

The data processing of real-time data warehouses involves the following key steps:

  1. Data Generation: Typically, data comes from two sources

    • User Behavior Logs: User operations on applications generate a series of logs, including clicks, jumps, browsing, duration, models, IP, and other information.
    • Related Information in Databases: Business activities, such as ordering, are recorded in databases.
  2. Data Collection: Logs and database content need to be uploaded to the message queue to enable the entire data link to flow. For example, the data in logs can be reported to the message queue in real-time with tools, such as log collection. Database data can be collected to the message queue using Alibaba Cloud DataWorks, Data Transmission Service (DTS), or other open-source components.
  3. Data Processing: The raw data of message queues, such as DataHub and Kafka, often does not have the same format or content. The data must be cleaned (ETL) to be fully utilized by downstream businesses. The whole ETL process is a very important part of the architecture design of real-time data warehouses. This process requires low latency, low cost, high scalability, and accurate calculations of business metrics.

When selecting the system type, we need to select Flink for data processing. Flink features powerful data processing capabilities, low latency, and high throughput, which ensures business output. At the same time, Alibaba Cloud launched Alibaba Cloud Realtime Compute for Apache Flink to provide users with an all-in-one highly available Flink service.

When designing the data architecture, you can build the ODS, DWD, and ADS layers based on the basic methodology of the data warehouse to reduce data redundancy and data storage costs and make the data structure more scalable.

  1. Data Analysis: Part of the data processed by Flink (ETL) can be directly used by customers. This kind of data is related to real-time metrics, such as daily app activation, page view, and unique visitors. Other data requires multi-dimensional analysis before being used by customers. This requires OLAP systems, such as Hologres. After the data is written into OLAP systems, you can obtain the relevant data by merging and querying historical data.
  2. Data Mining: Predicting the future from history has always been something people strive to do. For companies, making a correct judgment on future trends can help them last forever. Machine learning is a method to predict the future based on historical data and maximize the value of data.
  3. Business System: The processed data can directly serve relevant customers, such as operators, decision-makers, and application developers. For example, operators can adjust operation strategies based on the data in real-time reports to improve conversion rates. The real-time risk control system is used to report risky events in real-time to avoid business loss.

Flink-Based Real-Time Data Warehouse

2

Flink is mainly used for computing the ETL and BI metrics in real-time data warehouses. It is also integrated with various upstream and downstream systems.

Successful Cases

ZhongAn Insurance

Based on Alibaba Cloud Realtime Compute for Apache Flink, ZhongAn Insurance has built a real-time data warehouse. Its application scenarios include marketing activities, real-time monitoring, and anti-fraud.

  • In marketing activities, the data warehouse is mainly used to monitor traffic and transactions. Real-time computing allows ZhongAn Insurance to view the traffic at each entry of an activity, page view, unique visitors, and the transaction volume. Through dynamic observation of user behaviors, more accurate marketing strategies can be formulated to improve conversion.
  • Real-time monitoring is mainly used to monitor KPIs in real-time, such as daily insurance premiums, the daily number of insurance applicants, daily page views, and unique visitors.
  • Anti-fraud service is mainly used to monitor the benefits of bonus points and other cash benefits. It can also monitor and determine whether an action is performed by the users in real-time.

With the rapid development of business, real-time computing is increasingly demanding. It requires a platform with low latency, low resource consumption, high efficiency, high accuracy, and other capabilities. In addition to meeting the most basic business needs, we are also making full use of the features of Flink to enrich the input and output interfaces and ensure data quality. Based on the SQL version, the ML and Scala versions of Flink will also help real-time computing show its capabilities in anti-fraud and complex business scenarios.

Shenzhen Stock Exchange

The new generation of monitoring systems of the Shenzhen Stock Exchange focuses on core business, such as the supervision of abnormal trading behaviors and the clue screening of illegal behaviors. It fully supports the integrated supervision of transaction monitoring, investigation and analysis, and business research. The system follows the design principle of "safety, efficiency, sustainable evolution, open, autonomous, and controllable." It was created as a secure, efficient, flexible, easy-to-use, and highly inclusive distributed architecture technology system.

The real-time monitoring platform is the core subsystem of the monitoring system. In terms of architecture design, core technologies, computing capabilities, high availability, and disaster recovery design, Flink is believed to represent the future trend of real-time computing technology. Flink is the best choice for the real-time monitoring platform. Compared with Storm, Flink provides a powerful state management mechanism, more user-friendly programming interfaces, and Exactly-Once semantics. Compared with Spark Streaming, Flink provides a more powerful window computing capability and can better meet the performance requirement for low latency.

Flink helps applications manage their state, automatically saves checkpoints, and provides multiple backend implementations. When an application needs to maintain large states, it can use the state backend of RocksDB to greatly reduce memory overhead and facilitate garbage collection. In case of failure, the application state can be restored to the latest checkpoint.

The powerful abilities of Flink SQL greatly reduce the threshold for developing stream computing services. Flink SQL also can meet over 80% of the development needs for the real-time statistics and real-time alerting business of the monitoring system. The UDF, UDAF, and UDTF functions are extended to implement specific business functions, further simplifying business R&D.

Flink window computing supports business time, full computation, and incremental computation. Optimized internal algorithms make Flink window computing excellent in performance metrics, making it easy to compute interval indicators.

Flink is based on Chandy-Lamport's distributed snapshot algorithms and delivers automatic fault handling. When a system failure occurs, jobs can be recovered from the most recent state snapshot and can continue to run, which ensures the Exactly-Once semantics of internal data processing. This provides a solid foundation for the monitoring system to build a distributed real-time computing platform with high availability.

In 2019, the Shenzhen Stock Exchange signed a cooperation agreement with the Alibaba Realtime Compute team. The next-generation monitoring system's real-time computing platform has been running securely and reliably for nearly 300 days. By the end of April 2020, the number of raw business messages was more than 5,000 per second on average, with a peak of over 1.2 million per second. It takes 100 milliseconds on average to perform statistics on key services and system alerting, which provides strong support for the real-time monitoring of the core business.

Learn more about Alibaba Cloud Realtime Compute for Apache Flink

double 11 FinTech Stream Computing financial industry Apache flink Realtime Compute Data Collection Message-Oriented Middleware
0 0 0
Share on

Read previous post:

Architecture Evolution and Application Scenarios of Real-time Warehouses in the Cainiao Supply Chain

Read next post:

Apache Flink Best Practices: Constructing Real-Time Data Warehouses in the Financial Industry

Apache Flink Community

150 posts | 43 followers

You may also like

Comments

Apache Flink Community

150 posts | 43 followers

Related Products

More Posts by Apache Flink Community

See All