Simple Log Service:Compatibility between Simple Log Service and Elasticsearch

Compatibility

Elasticsearch is an open source, distributed search and analytics engine that is built based on Apache Lucene. Elasticsearch is commonly used for data processing and analysis. If you switch the log engine from Elasticsearch to Simple Log Service, you may need to change your usage habits and upstream and downstream programs. To resolve these issues, Log Service provides the Elasticsearch-compatible API to ensure smooth migration of Elasticsearch query and analysis solutions and to reduce the difficulty in using Simple Log Service.

The Elasticsearch-compatible API of Simple Log Service translates the Elasticsearch DSL query syntax into the index-based query syntax and SQL analysis syntax of Simple Log Service. Then, Simple Log Service returns query and analysis results in a format that meets the Elasticsearch requirements. This way, the Elasticsearch query protocols can be implemented.

Note

• The Elasticsearch-compatible API of Simple Log Service is implemented based on the API specification of Elasticsearch 7.10.

• You can call an Elasticsearch-compatible API operation only when you use a Standard Logstore and you create at least one field index.

Supported Elasticsearch DSL

The following tables describe the Elasticsearch DSL that is compatible with the Elasticsearch-compatible API of Simple Log Service.

Differences between the Elasticsearch-compatible API and standard Elasticsearch API

• The Elasticsearch-compatible API of Simple Log Service does not support data updates.

• The Elasticsearch-compatible API of Simple Log Service does not support query statements that contain scripts.

• A Simple Log Service Logstore allows you to query petabytes of data. When you use the Elasticsearch-compatible API of Simple Log Service, Elasticsearch does not need to perform index rollovers.

• Simple Log Service does not automatically create indexes. You must create indexes by calling API operations or by using the Simple Log Service console. For more information, see Create indexes.

• When you use the Elasticsearch DSL to query fields, take note of the mappings between the fields of Elasticsearch and Simple Log Service. The following table describes the field mappings.

  Elasticsearch	Simple Log Service	Description
  @timestamp	__time__	If you specify the @timestamp field in a query statement, the __time__ field of Simple Log Service is mapped.
  _id	None	If the _id field exists in Simple Log Service, the _id field in Elasticsearch is mapped to the _id field in Simple Log Service. If the _id field does not exist in Simple Log Service, the system generates the uuid or _pack_meta field and maps the values of the _id field to the uuid or _pack_meta field.

  Grant permissions to a RAM user

  You can use the following methods to grant a RAM user the permissions to query or write data by using the Elasticsearch-compatible API.

  • Simple mode: You can use system policies to grant the required permissions to the RAM user. You do not need to configure parameters.

  • Custom mode: You can create custom policies and attach the policies to the RAM user. This mode allows you to perform fine-grained access control. However, the configurations in this mode are complex.

  Simple mode

  You can use system policies to grant a RAM user the permissions to query or write data by using the Elasticsearch-compatible API. The following table describes the system policies. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.

  System policy name	Query permissions	Write permissions
  AliyunLogFullAccess	Yes	Yes
  AliyunLogReadOnlyAccess	Yes	No

  Custom mode

  You can use custom policies to grant a RAM user the permissions to query or write data by using the Elasticsearch-compatible API. The following examples show how to configure custom policies. For more information, see Create a custom policy and Grant permissions to RAM users.

  • The following script shows a sample policy that grants a RAM user the permissions to query and write data by using the Elasticsearch-compatible API:

    {
        "Version": "1",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "log:GetIndex",
                    "log:GetLogStoreContextLogs",
                    "log:ListLogStores",
                    "log:GetLogStoreHistogram",
                    "log:GetLogstoreLogs",
                    "log:GetLogStoreContextLogs",
                    "log:GetCursorOrData",
                    "log:GetLogstore",
                    "log:PostLogStoreLogs"
                ],
                "Resource": "acs:log:*:*:project/*"
            }
        ]
    }

  • The following script shows a sample policy that grants a RAM user the permissions to query data by using the Elasticsearch-compatible API:

    {
        "Version": "1",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "log:GetIndex",
                    "log:GetLogStoreContextLogs",
                    "log:ListLogStores",
                    "log:GetLogStoreHistogram",
                    "log:GetLogstoreLogs",
                    "log:GetLogStoreContextLogs",
                    "log:GetCursorOrData",
                    "log:GetLogstore"
                ],
                "Resource": "acs:log:*:*:project/*"
            }
        ]
    }

  • The following script shows a sample policy that grants a RAM user the permissions to write data by using the Elasticsearch-compatible API:

    {
        "Version": "1",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "log:PostLogStoreLogs"
                ],
                "Resource": "acs:log:*:*:project/*"
            }
        ]
    }