aliyun-timestream is a plug-in developed by the Alibaba Cloud Elasticsearch team based on the features of time series products that are provided by the Elastic community. This plug-in is used to enhance the storage and usage performance of time series data. aliyun-timestream uses Prometheus Querying Language (PromQL) statements instead of domain-specific language (DSL) statements to query stored metric data. This helps simplify query operations and improve query efficiency. aliyun-timestream also reduces storage costs. This topic describes the background information about the aliyun-timestream plug-in, the use scenarios and advantages of the plug-in, and the results of the performance test performed on the plug-in.
Background information
If you want to store and use time series data in Elasticsearch, you must configure complex settings for time series scenarios in Elasticsearch, which requires high technical skills. In addition, you may encounter issues such as low write performance, high storage costs, and complex query operations of massive time series data in these scenarios. To resolve these issues, the Alibaba Cloud Elasticsearch team developed the aliyun-timestream plug-in. This plug-in provides the following advantages:
Supports the time series data model, provides APIs that can be used to create, modify, query, and delete time series indexes, and automatically applies the best practices of using Elasticsearch in a time series scenario when you create a time series index. This helps simplify the operations that are required to manage time series data in Elasticsearch and reduces the technical complexity for performing the operations.
Improves the write performance and query performance of time series data and reduces the space that is required to store time series data.
Allows you to execute PromQL statements to query data stored in Elasticsearch and supports seamless integration with Prometheus and Grafana. This can help further reduce storage and usage costs of time series data and improve efficiency and usability.
Use scenarios
The aliyun-timestream plug-in is suitable for scenarios in which you need to store and use time series data. For example, you can use aliyun-timestream if you want to store and query the data of system metrics and Internet of Things (IoT) devices.
Limits
The following table describes the requirements that your Elasticsearch cluster must meet if you want to use the aliyun-timestream plug-in.
Cluster version and kernel version | Region |
The version of the Elasticsearch cluster must be V7.16 or later, and the kernel version of the Elasticsearch cluster must be V1.7.0 or later. | China (Shenzhen), China (Chengdu), China (Guangzhou), China (Ulanqab), China North 2 Finance, and China (Hong Kong) |
The version of the Elasticsearch cluster must be V7.10 or later, and the kernel version of the Elasticsearch cluster must be V1.8.0 or later. | China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), and China (Zhangjiakou) |
The supported regions may change. You can view the supported regions in the Elasticsearch console.
Advantages
Improved efficiency in data management: The aliyun-timestream plug-in integrates with the templates for best practices of using Elasticsearch in time series scenarios based on the time series data model and the supported APIs that can be used to create, modify, query, and delete time series indexes. This significantly simplifies the operations that are required to manage time series data in Elasticsearch and reduces the technical complexity for performing the operations.
Improved query performance: The aliyun-timestream plug-in allows you to execute PromQL statements to query data stored in Elasticsearch and supports seamless integration with Prometheus and Grafana. The aliyun-timestream plug-in supports downsampling queries and time-based partitioning for data streams.
Optimized storage costs: The aliyun-timestream plug-in optimizes data compression and metadata storage. This way, the space that is required to store time series indexes is reduced by more than 80% compared with the space that is required to store common indexes of open source Elasticsearch.
Improved read and write performance: The aliyun-timestream plug-in helps improve the write TPS of time series indexes by approximately 40% compared with common indexes of open source Elasticsearch. In addition, the aliyun-timestream plug-in improves query and analysis performance of time series data by 5 times compared with open source Elasticsearch.
The following table describes the results of comparison between the time series scenarios in which the aliyun-timestream plug-in is used and in which the aliyun-timestream plug-in is not used. The comparison is performed from the following dimensions: data model, storage performance, and query performance.
Dimension | aliyun-timestream used | aliyun-timestream not used |
Data model | The time series data model is supported. | You must configure a large number of settings or perform a large number of operations that are related to metrics. For example, you must configure the related settings to generate a field that is used to specify the ID of a time series, use the ID and other time-related settings to sort indexes, or use the ID to perform shard routing. |
Storage | APIs that can be used to create, modify, query, and delete time series indexes are provided. When you create a time series index, the system automatically applies the best practices of using Elasticsearch in a time series scenario. This helps improve the data compression rate and significantly reduce storage costs. | When you use a relational data model to store time series data, you may encounter the issue that the volume of data that is stored is excessively large. Open source Elasticsearch proves that the space required by a relational data model to store data is 25 times larger than the space required by the time series data model to store data. |
Query statement | The aliyun-timestream plug-in allows you to execute PromQL statements to query time series data and can be seamlessly integrated with Prometheus and Grafana. | You must execute complex DSL statements to query time series data. |