By Yijing
Image loading is the most common and basic feature of apps, but more importantly, it is also one of the determining factors of user experience. This function although seemingly simple, actually introduces many technical challenges. This article introduces the attempts made by the Xianyu technical team at Flutter image optimization, and shares the technical details of Xianyu's typical image processing solutions. Hopefully, this article can provide some inspiration for dealing with this function.
When we first used Flutter, we put our core concerns on images and the optimization of its functions. The image display imposes a huge impact on the user experience in Xianyu. Now, try to recall, have you ever encountered the following problems when loading images:
In response to these problems, Xianyu has been optimizing the image framework since the initial version of the Flutter service. Factors from the optimization of native code in the beginning to the advanced technology of external textures later on, and from memory usage to package size, will all be described in this article. I hope the ideas and methods about the optimizations can bring you some inspiration.
From a technical point of view, the goal of image loading, in simple terms, is nothing more than the maximization of loading efficiency. That is, to load as many images as possible, with the minimum possible resource cost at the maximum possible speed.
The initial version of Xianyu image processing is a purely native solution. If you do not want to modify much of the underlying logic, the native solution is definitely the simplest and most economical one. The functional modules of the native solution are as follows:
However, if you modify nothing before using it, you may find that its performance is not as expected. Then, what optimizations can be done at this early stage?
Yes, as you might have guessed, the first approach is to set up a cache. The best solution for image loading is to use a cache. First, the native image component supports custom image caching. The specific implementation class is ImageCache. You can set ImageCache in two dimensions:
By setting the ImageCache size, you can make full use of the caching mechanism to accelerate image loading. In addition, Xianyu has made two major optimizations:
After Xianyu was launched, we received feedback from online public opinion and found that it was not optimal to set a same cache size for all mobile phone models. In particular, if a large cache was set for low-end models, the user experience would deteriorate and the stability would be affected. Based on the practical situation, we have implemented a Flutter plug-in that can obtain basic machine information from the mobile phone system. The obtained information enables us to set different cache policies for different configurations of mobile phone models. In practice, we decreased the size of the image cache for low-end models and increased the size for high-end models as appropriate. In this way, we can obtain optimal cache performance on mobile phone models with different configurations.
If you are familiar with app development, you know that a mature image loading framework uses multi-level caching. In addition to the common memory cache, a file cache is available. In terms of loading efficiency, multi-level caching improves the loading speed by expanding cache space. In terms of stability, it will not consume too many valuable memory resources and cause out-of-memory (OOM) issues. However, the image loading framework of Flutter does not have a separate disk cache. Therefore, we have expanded the disk cache capability based on the native solution.
In this effort, we did not create a disk cache on our own for the specific architecture implementation. Instead, we chose to reuse the existing capability. First, we exposed the disk caching function of the native image loading framework through an interface. Then, we grafted the native disk caching capability onto the Flutter layer by means of bridging. When an image is loaded to Flutter and no hits can be found in the memory, another search attempt is made in the disk cache. If this attempt also has no hits, a network request is raised.
By increasing the disk cache capacity, the efficiency of Flutter image loading can be further improved.
CDN optimization is another important image optimization measure. The efficiency of CDN optimization depends on the minimized size of the transmitted image. This involves these common strategies:
In simple terms, the actual size of the image to load may be greater than the size of the actual display window. In this case, you do not need to load the complete size of the image, but to load an image at the size that fits the window. That is, you can crop unnecessary parts of the image to minimize the transmitted image. From the perspective of the end side, this improves the loading speed and reduces memory usage.
To this end, we can increase the compression ratio of images based on the actual situation. Under the premise of not compromising the display effect, we can further reduce the image size through compression.
I recommend that you use the webp format for images because it requires relatively fewer image resources. Flutter natively supports webp, including animated stickers. Here, note that a webp animated sticker has a much smaller size than a gif image and provides better transparency support. It is an ideal alternative to the gif format.
For the preceding reasons, the Xianyu image framework implements a set of CDN size matching algorithms on the Flutter side. With these algorithms, the requested image will be automatically matched to the most appropriate size and compressed according to the actual display window. If the image format allows, you should always try to convert the image format into webp before delivering it, which maximizes the efficiency of transmitting CDN images.
In addition to the preceding optimization strategies, we can further optimize the performance of images with Flutter in some other ways.
To display images as fast as possible, you can use the official preloading mechanism provided by Flutter, precacheImage. The precacheImage function can preload images to the memory so that they can be displayed instantly once requested.
This approach is actually a common optimization solution with Flutter. The WidgetUpdate overriding scheme compares the old and the new widget versions to determine whether to re-render the element. This avoids the repeated rendering of the same image.
ListView is the most common scrolling container in Flutter. The performance of ListView directly affects the final user experience.
The implementation idea of Flutter ListView is different from the implementation idea of the native solution. Its most significant highlight is that it introduces the viewPort concept. In this concept, any part that goes beyond viewPort will be forcibly reclaimed.
Based on this principle, we have put forth two suggestions:
1) Cell splitting
In this practice, avoid large cells as much as possible to minimize the performance loss during frequent cell creation. This is because, the cell size affects not only the image loading process, but also affects other components such as text processing and video processing which are prone to performance problems caused by overly complex cell lines.
2) Rational use of the buffer
With ListView, you can set the size of the preloaded content by setting cacheExtent. Preloading can accelerate view rendering. However, the cacheExtent value must be appropriate. A larger value does not suit all cases, because the workload of the overall memory of the page increases as the preloading cache size expands.
I should point out here that the native solution is perfect and sufficient for a pure Flutter app. However, for hybrid apps, it has two defects:
The native image solution is an independent image loading solution. As far as a hybrid app is concerned, the native solution is independent of the native image framework, and therefore its capabilities cannot be reused. For example, features such as CDN cropping and compression require repeated construction. In particular, some of the unique image decoding capabilities of the native solution are difficult to use for Flutter. As a result, the same app may provide differentiated support for different image formats.
From the perspective of the entire app, with the native image solution, we maintain two large cache pools, that is, the native image cache and the Flutter-side image cache. However, the two caches cannot interact with each other, resulting in huge waste. In particular, it greatly affects the peak memory performance.
After multiple optimizations, the performance of the mobile phone system-based solution has been greatly improved. However, the memory watermark of the entire app was still high, which is especially true for iOS. In this context, we have to further optimize the image framework. Based on the analysis on the shortcomings of the native solution, we had a bold idea: Could we fully reuse the native image loading capability?
When considering how to integrate the image capabilities of Flutter and the native solution, we thought of external textures. External textures are not developed by Flutter. It is a commonly used method to optimize audio and video performance.
In the current phase, we have developed texture plug-ins for Flutter and the native solution based on the shared-context solution. By adopting this solution, Flutter can obtain and display images loaded by the native image library through sharing textures. To implement the channel for texture sharing, we made in-depth customizations on the engine layer. The detailed process is as follows:
This solution not only integrates the native and Flutter image architectures, but also optimizes image loading performance throughout the process.
The application of external textures is a big leap forward in the image solution of Xianyu. With this technology, we can reuse the local capabilities of the image processing solution and also implement external textures for the video capability. This avoids heavy repetitive construction and improves the performance of the entire app.
In fact, this optimization strategy was created in desperation. After analyzing online data, we found something interesting about the Flutter page stack, that is, the page at the bottom of a stack is not released if the stack is a multi-page stack. This reclamation is not performed even when the free memory capacity becomes very low. As a result, as the number of pages increases, memory usage increases linearly. The largest proportion of memory consumption is due to image resources.
So, we are considering whether images on the page at the bottom of the page stack can be directly reclaimed?
Driven by this thought, we launched another round of optimization on the image architecture. All images in the entire image framework monitor the changes of the page stack. When an image finds itself not at the top of the stack, it automatically reclaims the corresponding image textures to release resources. This solution prevents the memory usage of images from increasing linearly with the number of pages. The following figure shows how it works:
Note that the page location judgment at this stage requires the additional interface provided by the page stack (hybrid stack specifically). The coupling degree between systems is high.
After we integrated the native and Flutter image frameworks, we received an unexpected gain, that is, the native solution and Flutter can share local image resources. This means that you no longer need to keep two copies of same image resources on Flutter and the mobile phone system. This greatly improves the reuse rate of local resources and reduces the overall package size. Based on this solution, we have realized a set of resource management functions, with which scripts can automatically synchronize local image resources on different ends. In this way, the utilization of local resources is improved and the package size is reduced.
Native images do not have the PlaceHolder feature. Therefore, to use the native solution, you need to use FadeInImage. Since we had made a lot of customizations for Xianyu scenarios, we implemented a PlaceHolder mechanism.
In terms of core functions, we introduced the following loading states:
For different states, you can control the display logic of PlaceHolder in a fine-grained manner.
As Xianyu services continue to advance, we must consider the cost of upgrading the engine. Our core demand is to realize the same function without changing the engine though it sounds impractical.
The external texture solution requires bridging for communication with native capabilities, including the delivery of image requests and synchronization of image loading states. The amount of data sent through the bridge is considerable, especially upon quick ListView sliding. In the current solution, the bridge is called separately when an image is loaded. This becomes a bottleneck when a large number of images are processed.
When we implement image reclamation, the current image loading solution requires the stack to provide the interface for querying whether the current location is the bottom of the stack. This leads to solution coupling, and it is difficult to abstract an independent image loading solution.
By 2020, we have gradually deepened our understanding of the basic capabilities of Flutter, and implemented an image framework with a better overall solution.
During the exploration, we came to another question: Can we use external textures without modifying the engine? The answer is yes.
In fact, Flutter provides an official external texture solution.
In addition, textures used by the native solution and textures displayed on the Flutter side point to same objects at the underlying layer and do not generate additional data copies. This ensures highly efficient texture sharing. Then, you may be wondering, why did the Xianyu team implement an independent external texture solution based on shared-context? Prior to version 1.12, the official iOS external texture solution had performance bugs. In earlier versions, during each rendering process regardless of whether textures were updated or not, CVPixelBuffer was frequently obtained, resulting in unnecessary performance loss due to lock-related loss in the process. This problem had been fixed in version 1.12 (with the official commit address on GitHub) so that the official solution is now sufficient for the needs. In this context, we used the official solution again to implement the external texture function.
As discussed earlier, previous versions of page stack-based image resource reclamation require interfaces that are strongly dependent on stack capabilities. This introduces unnecessary dependencies and, more importantly, prevents the overall solution from evolving to an independent universal solution. To solve this problem, we conducted in-depth research on the underlying layer of Flutter. We found that the layer of Flutter can constantly sense the changes of the page stack.
Each page uses the router object obtained from context as the identifier to reorganize all image objects on the page. Images that obtain the same router object are identified as the images on the same page. In this way, you can manage all images on a page basis. Overall, we simulated the virtual page stack structure by using the LRU algorithm, so that image resources for the page at the bottom of the stack can be reclaimed.
In this practice, we aggregated image requests in one frame as a unit, and then transferred the unit to the native image loading framework through a channel request, avoiding frequent bridge calls. This is especially effective in scenarios such as fast scrolling.
After using external textures for image loading, we found that reusing textures can further improve the performance. Let's use a simple scenario as an example. As we know, in an e-commerce shopping mall, labels and base pictures are often used for product display. In most cases, this type of images are repeatedly displayed on different commodities. In this case, we can reuse the rendered textures for different display components to further optimize GPU memory usage and avoid repeated GPU creation. To finely manage textures, we introduced a reference count algorithm to manage the reuse of textures. With these solutions, we realized the efficient reuse of textures across pages.
Moreover, we migrated the mapping between textures and requests to the Flutter side so that the reuse of textures can be done on the shortest path, further reducing the communication workload of the bridge.
Currently, the latest version is under grayscale testing and is available for only a small group of users. We will elaborate the specific optimization results later. Subordinate data is mainly for scheme 2.
Compared with the initial online version, the version that integrates native code has a 25% reduction in the abortion rate of iOS and significantly improves the user experience with the same display performance.
The memory optimization of multi-page stacks plays an important role in memory optimization in multi-page scenarios. To verify this, we performed an extreme test as follows, where the test environment was a non-Xianyu app:
As you can see, the optimization of the multi-page stack enables enhanced control of the memory usage of multiple Flutter pages.
By using external textures, local resources are better reused, and the package size is reduced by 1 MB. In the early stage of Xianyu's access to Flutter, we used to transform existing pages. In that case, resources were highly repeated. However, with the increasing number of new Xianyu Flutter services, we are seeing less repetition of resources of Flutter and the native solution. The influence of external textures on the package size has gradually weakened.
Optimization is an endless journey. Our optimization efforts for Xianyu image processing will continue. Regarding our latest solution, I have only given a preliminary introduction here. We will provide more technical details, including test data, later on. After the solution is fully optimized, we will gradually open its source code.
Flutter Analysis and Practice: Evolution and Innovation of Xianyu Technologies
56 posts | 4 followers
FollowXianYu Tech - September 11, 2020
Alibaba Clouder - September 21, 2020
XianYu Tech - January 13, 2021
XianYu Tech - August 10, 2021
XianYu Tech - August 10, 2021
XianYu Tech - March 11, 2020
56 posts | 4 followers
Follow
Offline Visual Intelligence Software Packages
Offline SDKs for visual production, such as image segmentation, video segmentation, and character recognition, based on deep learning technologies developed by Alibaba Cloud.
Learn More
CT Image Analytics Solution
This technology can assist realizing quantitative analysis, speeding up CT image analytics, avoiding errors caused by fatigue and adjusting treatment plans in time.
Learn More
CDN(Alibaba Cloud CDN)
A scalable and high-performance content delivery service for accelerated distribution of content to users across the globe
Learn More
ApsaraVideo Media Processing
Transcode multimedia data into media files in various resolutions, bitrates, and formats that are suitable for playback on PCs, TVs, and mobile devices.
Learn MoreMore Posts by XianYu Tech
