Flutter Analysis and Practice: Multimedia Capability Expansion Best Practices

With the booming development of mobile smart devices, a mobile multi-terminal development framework has become a general trend. Download the Flutter Analysis and Practice: Evolution and Innovation of Xianyu Technologies eBook for step-by-step analyses based on real-life problems, as well as clear explanations of the important concepts of Flutter.

2.3.1 Background

Albums are necessary for the development of image and video features because most developers need to obtain images or videos from albums. The most direct method is to call the system album API. However, the album API does not provide advanced features such as custom UI and selection of multiple images.

2.3.2 Design

Xianyu's album component API is easy to use, rich in features, and highly customizable. You can integrate the component as is or customize the UI and integrate the component.

Flutter designs the UI presentation layer while data are provided by the native platforms. This mode naturally isolates the UI from data. We usually use the Model View Controller (MVC) architecture when developing a native component. The ideas behind Flutter components are similar. Figure 2-13 shows the overall architecture.

Figure 2-13

As shown in Figure 2-13, the Flutter side shows a typical MVC architecture, in which Model is the entity class representing images and videos, Widget is View, and Controller calls the APIs of each platform. When Model changes, View is reconstructed to reflect the changes of Model. View events trigger Controller to obtain data from the native side and then update Model. The native side communicates with the Flutter side through MethodChannel. There is no strong dependency between the two layers. They only need to communicate with each other by following the protocols.

On the native side, UIAdapter is responsible for model matching and screen type recognition. Permission is responsible for processing media read and write permission requests. Cache is mainly responsible for caching GPU textures to increase the response speed of previewing large images. Decoder is responsible for parsing bitmaps. OpenGL is responsible for converting bitmaps to textures.

Note that the album component sees most images as GPU textures, which dramatically reduces the Java heap memory usage when compared to the old album implementation. If you use the native album on a device with low specifications, the app may be killed due to high memory usage when it is idle in the backend. What you see if the app restarts when you bring it out of the background. Using the Flutter album component will give you a better experience on a device with low specifications.

2.3.3 Difficulties

1) Image Loading Pagination

The album list needs to load a lot of images. Flutter's GridView has several constructors and a common mistake is to use the first one which requires a large number of widgets at the start. The second constructor should be used. Then GridView will call back IndexedWidgetBuilder to obtain the widgets during sliding, which is Load On Demand.

GridView.builder({
    ...
    List<Widget> children = const <Widget>[],
    ...
  })
GridView.builder({
    ...
    @required IndexedWidgetBuilder itemBuilder,
    int itemCount,
    ...
  })

After an image is out of sight, the resources must be recycled, which means deleting the texture. The memory usage of GridView remains constant after it increases to a certain level even if it continues to load more images. If a texture is quickly slid back and forth, data is repeatedly created and deleted, which causes memory jitters and bad user experience.

Therefore, we maintain an image state machine, with the following states: None, Loading, Loaded, Wait_Dispose, and Disposed. When image loading starts, the state changes from None to Loading, and a blank image or a placeholder is displayed. When data is called back, the state is set to Loaded and a widget tree is recreated to display image thumbnails. When an image is slid away, its state changes to Wait_Dispose. The image is not destroyed immediately. If the image is slid back, its state changes from Wait_Dispose back to Loaded and will no longer be destroyed. If the image is not slid back, it enters the Disposed state from Wait_Dispose. Once the image enters the Disposed state, displaying it again will require a new loading process.

2) Displaying Images

When you tap an image in GridView, the image is displayed. The resolution of images taken by a camera is very high. When the original image is fully loaded, memory usage would be very high. Therefore, the image is scaled to 1080p at the largest during bitmap decoding. The native Android bitmap decoding process is the same. The width and height of the bitmap are decoded first. Then the scaling multiple is calculated based on the target size. Finally, the desired bitmap is decoded.

Most images in the photo album on Android have rotation angles. If an image is displayed as is, it may be in the angle. Therefore, the bitmap needs to be rotated. It takes about 200 ms to rotate a 1080p image by using Matrix on the test machine. However, it only takes about 10 ms to rotate the image by using the OpenGL texture coordinates. Using OpenGL for rotation is clearly the better choice.

When an image is previewed as a large image, the album uses a horizontally sliding PageView. Generally, Flutter PageView does not actively load adjacent pages. In this case, a trick you can use is to set the ViewportFraction parameter to 0.9999 for PageController.

PageController(viewportFraction=0.9999)

You can also preload images on the native side. For example, when the fifth image is being loaded, the textures of the fourth and sixth images are loaded in advance. When you slide to the fourth and sixth images, the cached textures are directly used.

3) Memory

GPU textures for album images greatly reduce the usage of Java heap memory and improve app performance. However, GPU has limited memory. Therefore, the images must be deleted in time after use, to avoid memory leaks. On Android, delete the textures in GPU threads. Otherwise, the deletion is invalid.

On the GridView list page, the memory usage increases by 13 MB at the most. The difference is that the original native album uses the Java heap memory and the Flutter album uses the native memory or graphic memory.

2.3.4 Summary

The album component API is easy to use and highly customizable. The Flutter side has a clear hierarchy. If you need to customize UI, rewrite the widget. Also, this album component does not depend on the system album. It is complete and consistent with the UI and interaction of existing apps. It also is a good foundation for more album operations.

Since we are using GPU textures, supporting 4K HD images should not be a problem. This would not require too much memory on the client. However, it takes more time to convert 4K bitmap to textures. Therefore, you should implement load time animations as part of UI interactions.

Community

Flutter Analysis and Practice: Multimedia Capability Expansion Best Practices

2.3.1 Background

2.3.2 Design

2.3.3 Difficulties

2.3.4 Summary

Read previous post:

Read next post:

XianYu Tech

You may also like

Comments

XianYu Tech

Related Products

mPaaS

Alibaba Cloud Flow

Mobile Testing

Super App Solution for Telcos