Simple Log Service:Connect to Kibana

Deploy using Docker Compose

1. On your server, run the following commands to create a new directory named sls-kibana and a subdirectory named data in the sls-kibana directory. Then, change the permissions on the data directory to grant the Elasticsearch container read, write, and execute permissions.

   mkdir sls-kibana
   
   cd sls-kibana
   
   mkdir data
   chmod 777 data 

2. Create a file named .env in the sls-kibana directory. The following sample code shows the content of the file. Modify the parameters based on your actual requirements.

   ES_PASSWORD=aStrongPassword  # Modify the value based on your actual requirements.
   
   SLS_ENDPOINT=cn-huhehaote.log.aliyuncs.com
   SLS_PROJECT=etl-dev-7494ab****
   SLS_ACCESS_KEY_ID=xxx
   SLS_ACCESS_KEY_SECRET=xxx
   # ECS_ROLE_NAME="" # If you use an ECS instance RAM role to access Simple Log Service, specify the name of the ECS instance RAM role.
   #SLS_PROJECT_ALIAS=etl-dev # Optional. If the name of the SLS_PROJECT is too long, you can set an alias.
   #SLS_LOGSTORE_FILTERS="access*" # Optional. Filter the Logstores for which you want to automatically create index patterns. To specify multiple index patterns, separate them with commas (,), for example, "access*,error*". Note that the value must be enclosed in double quotation marks ("").
   #KIBANA_SPACE=default # Optional. Specify the space in which you want to create the index pattern. If the space does not exist, it is automatically created.
   
   # To add more projects, continue to add them. Note that if you add more than six projects, you must also add references in the docker-compose.yml file.
   #SLS_ENDPOINT2=cn-huhehaote.log.aliyuncs.com
   #SLS_PROJECT2=etl-dev2
   #SLS_ACCESS_KEY_ID2=xxx
   #SLS_ACCESS_KEY_SECRET2=xxx
   #SLS_PROJECT_ALIAS2=etl-dev2 # Optional. If the name of the SLS_PROJECT is too long, you can set an alias.
   #SLS_LOGSTORE_FILTERS2="test*log" # Optional. Filter the Logstores for which you want to automatically create index patterns. To specify multiple patterns, separate them with commas (,), for example, "access*,error*". Note that the value must be enclosed in double quotation marks ("").
   #KIBANA_SPACE2=default # Optional. Specify the space in which you want to create the index pattern. If the space does not exist, it is automatically created.

   Parameter	Description
   ES_PASSWORD	The password of Elasticsearch. This is also the password for Kibana.
   ECS_ROLE_NAME	The instance RAM role. For information about the permissions required for the RAM role, see RAM authorization.
   SLS_ENDPOINT	The endpoint of the project. For more information, see Manage projects.
   SLS_PROJECT	The name of the Simple Log Service project. For more information, see Manage projects.
   SLS_ACCESS_KEY_ID	The AccessKey ID of the RAM user that you created in the "Prerequisites" section. The RAM user must have the permissions to query data in the Logstore. For more information, see RAM authorization.
   SLS_ACCESS_KEY_SECRET	The AccessKey secret of the RAM user that you created in the "Prerequisites" section. The RAM user must have the permissions to query data in the Logstore. For more information, see RAM authorization.
   SLS_PROJECT_ALIAS	Optional. If the name of the SLS_PROJECT is too long, you can set an alias.
   SLS_LOGSTORE_FILTERS	Optional. Filter the Logstores for which you want to automatically create index patterns. To specify multiple index patterns, separate them with commas (,), for example, "access*,error*". Note that the value must be enclosed in double quotation marks ("").
   KIBANA_SPACE	Optional. Specify the space in which you want to create the index pattern. If the space does not exist, it is automatically created.

3. Create a file named docker-compose.yaml in the sls-kibana directory. The following sample code shows the content of the file.

   services:
     es:
       image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/elasticsearch:7.17.26
       #image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/elasticsearch:7.17.26-arm64
       environment:
         - "discovery.type=single-node"
         - "ES_JAVA_OPTS=-Xms2G -Xmx2G"
         - ELASTIC_USERNAME=elastic
         - ELASTIC_PASSWORD=${ES_PASSWORD}
         - xpack.security.enabled=true
       volumes:
         # TODO: The ./data directory must be created in advance. Make sure that you have run the mkdir data && chmod 777 data command.
         - ./data:/usr/share/elasticsearch/data
     kproxy:
       image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5
       #image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5-arm64
       depends_on:
         - es
       environment:
         - ES_ENDPOINT=es:9200
   
         - ECS_ROLE_NAME=${ECS_ROLE_NAME}
         # The first SLS project
         - SLS_ENDPOINT=${SLS_ENDPOINT}
         - SLS_PROJECT=${SLS_PROJECT}
         - SLS_LOGSTORE_FILTERS=${SLS_LOGSTORE_FILTERS}
         - KIBANA_SPACE=${KIBANA_SPACE}
         - SLS_PROJECT_ALIAS=${SLS_PROJECT_ALIAS}
         - SLS_ACCESS_KEY_ID=${SLS_ACCESS_KEY_ID}
         - SLS_ACCESS_KEY_SECRET=${SLS_ACCESS_KEY_SECRET}
   
         # The second SLS project
         - SLS_ENDPOINT2=${SLS_ENDPOINT2}
         - SLS_PROJECT2=${SLS_PROJECT2}
         - SLS_LOGSTORE_FILTERS2=${SLS_LOGSTORE_FILTERS2}
         - KIBANA_SPACE2=${KIBANA_SPACE2}
         - SLS_PROJECT_ALIAS2=${SLS_PROJECT_ALIAS2}
         - SLS_ACCESS_KEY_ID2=${SLS_ACCESS_KEY_ID2}
         - SLS_ACCESS_KEY_SECRET2=${SLS_ACCESS_KEY_SECRET2}
   
         - SLS_ENDPOINT3=${SLS_ENDPOINT3}
         - SLS_PROJECT3=${SLS_PROJECT3}
         - SLS_LOGSTORE_FILTERS3=${SLS_LOGSTORE_FILTERS3}
         - KIBANA_SPACE3=${KIBANA_SPACE3}
         - SLS_PROJECT_ALIAS3=${SLS_PROJECT_ALIAS3}
         - SLS_ACCESS_KEY_ID3=${SLS_ACCESS_KEY_ID3}
         - SLS_ACCESS_KEY_SECRET3=${SLS_ACCESS_KEY_SECRET3}
   
         - SLS_ENDPOINT4=${SLS_ENDPOINT4}
         - SLS_PROJECT4=${SLS_PROJECT4}
         - SLS_LOGSTORE_FILTERS4=${SLS_LOGSTORE_FILTERS4}
         - KIBANA_SPACE4=${KIBANA_SPACE4}
         - SLS_PROJECT_ALIAS4=${SLS_PROJECT_ALIAS4}
         - SLS_ACCESS_KEY_ID4=${SLS_ACCESS_KEY_ID4}
         - SLS_ACCESS_KEY_SECRET4=${SLS_ACCESS_KEY_SECRET4}
   
         - SLS_ENDPOINT5=${SLS_ENDPOINT5}
         - SLS_PROJECT5=${SLS_PROJECT5}
         - SLS_LOGSTORE_FILTERS5=${SLS_LOGSTORE_FILTERS5}
         - KIBANA_SPACE5=${KIBANA_SPACE5}
         - SLS_PROJECT_ALIAS5=${SLS_PROJECT_ALIAS5}
         - SLS_ACCESS_KEY_ID5=${SLS_ACCESS_KEY_ID5}
         - SLS_ACCESS_KEY_SECRET5=${SLS_ACCESS_KEY_SECRET5}
   
         - SLS_ENDPOINT6=${SLS_ENDPOINT6}
         - SLS_PROJECT6=${SLS_PROJECT6}
         - SLS_LOGSTORE_FILTERS6=${SLS_LOGSTORE_FILTERS6}
         - KIBANA_SPACE6=${KIBANA_SPACE6}
         - SLS_PROJECT_ALIAS6=${SLS_PROJECT_ALIAS6}
         - SLS_ACCESS_KEY_ID6=${SLS_ACCESS_KEY_ID6}
         - SLS_ACCESS_KEY_SECRET6=${SLS_ACCESS_KEY_SECRET6}
         # To add more projects, continue to add them. You can add a maximum of 255 projects.
     kibana:
       image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kibana:7.17.26
       #image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kibana:7.17.26-arm64
       depends_on:
         - kproxy
       environment:
         - ELASTICSEARCH_HOSTS=http://kproxy:9201
         - ELASTICSEARCH_USERNAME=elastic
         - ELASTICSEARCH_PASSWORD=${ES_PASSWORD}
         - XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED=true
       ports:
         - "5601:5601"
   
     # This service component is optional. It is used to automatically create Kibana index patterns.
     index-patterner:
       image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5
       #image: sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5-arm64
       command: /usr/bin/python3 -u /workspace/create_index_pattern.py
       depends_on:
         - kibana
       environment:
         - KPROXY_ENDPOINT=http://kproxy:9201
         - KIBANA_ENDPOINT=http://kibana:5601
         - KIBANA_USER=elastic
         - KIBANA_PASSWORD=${ES_PASSWORD}
   
         - ECS_ROLE_NAME=${ECS_ROLE_NAME}
   
         - SLS_PROJECT_ALIAS=${SLS_PROJECT_ALIAS}
         - SLS_ACCESS_KEY_ID=${SLS_ACCESS_KEY_ID}
         - SLS_ACCESS_KEY_SECRET=${SLS_ACCESS_KEY_SECRET}
   
         - SLS_PROJECT_ALIAS2=${SLS_PROJECT_ALIAS2}
         - SLS_ACCESS_KEY_ID2=${SLS_ACCESS_KEY_ID2}
         - SLS_ACCESS_KEY_SECRET2=${SLS_ACCESS_KEY_SECRET2}
   
         - SLS_PROJECT_ALIAS3=${SLS_PROJECT_ALIAS3}
         - SLS_ACCESS_KEY_ID3=${SLS_ACCESS_KEY_ID3}
         - SLS_ACCESS_KEY_SECRET3=${SLS_ACCESS_KEY_SECRET3}
   
         - SLS_PROJECT_ALIAS4=${SLS_PROJECT_ALIAS4}
         - SLS_ACCESS_KEY_ID4=${SLS_ACCESS_KEY_ID4}
         - SLS_ACCESS_KEY_SECRET4=${SLS_ACCESS_KEY_SECRET4}
   
         - SLS_PROJECT_ALIAS5=${SLS_PROJECT_ALIAS5}
         - SLS_ACCESS_KEY_ID5=${SLS_ACCESS_KEY_ID5}
         - SLS_ACCESS_KEY_SECRET5=${SLS_ACCESS_KEY_SECRET5}
   
         - SLS_PROJECT_ALIAS6=${SLS_PROJECT_ALIAS6}
         - SLS_ACCESS_KEY_ID6=${SLS_ACCESS_KEY_ID6}
         - SLS_ACCESS_KEY_SECRET6=${SLS_ACCESS_KEY_SECRET6}
   
         # To add more projects, continue to add them. You can add a maximum of 255 projects.

4. Run the following command to start the service.

   docker compose up -d

5. Run the following command to query the status of the service.

   docker compose ps

6. After the deployment is complete, enter http://${IP address of the server where Kibana is deployed}:5601 in a browser to open the Kibana logon page. Then, enter the username and password that you specified for Elasticsearch in Step 2.

   Important

   You must add a rule to the security group of the server to allow traffic on port 5601. For more information, see Add a security group rule.

   http://${IP address of the server where Kibana is deployed}:5601

   

Deploy using Helm

Deploy using Docker

sudo docker pull sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5

sudo docker run  -d --name proxy \
            -e ES_ENDPOINT=${IP address of the machine where Elasticsearch is deployed}:9200 \
            -e SLS_ENDPOINT=https://prjA.cn-guangzhou.log.aliyuncs.com/es/ \
            -e SLS_PROJECT=prjA \
            -e SLS_ACCESS_KEY_ID=${aliyunAccessId} \
            -e SLS_ACCESS_KEY_SECRET=${aliyunAccessKey} \
            -p 9201:9201 \
            -ti sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5

sudo docker pull sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5

sudo docker run  -d --name proxy \
            -e ES_ENDPOINT=${IP address of the machine where Elasticsearch is deployed}:9200 \
            -e SLS_ENDPOINT=https://prjA.cn-guangzhou.log.aliyuncs.com/es/ \
            -e SLS_ENDPOINT2=https://prjB.cn-guangzhou.log.aliyuncs.com/es/ \ 
            -e SLS_PROJECT=prjA \
            -e SLS_PROJECT2=prjB \
            -e SLS_ACCESS_KEY_ID=${aliyunAccessId} \ 
            -e SLS_ACCESS_KEY_SECRET=${aliyunAccessKey} \ 
            -e SLS_ACCESS_KEY_ID2=${aliyunAccessId} \ # If the value is the same as that of SLS_ACCESS_KEY_ID, you do not need to configure this parameter.
            -e SLS_ACCESS_KEY_SECRET2=${aliyunAccessKey} \ # If the value is the same as that of SLS_ACCESS_KEY_SECRET, you do not need to configure this parameter.
            -p 9201:9201 \
            -ti sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kproxy:2.1.5

curl http://${IP address of the machine where the proxy is deployed}:9201

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

sudo docker pull  sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kibana:7.17.26

sudo docker run -d --name kibana \
            -e ELASTICSEARCH_HOSTS=http://${IP address of the machine where the proxy is deployed}:9201 \
            -e ELASTICSEARCH_USERNAME=elastic \
            -e ELASTICSEARCH_PASSWORD=passwd \
            -e XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED=true \
            -p 5601:5601 \
            sls-registry.cn-hangzhou.cr.aliyuncs.com/kproxy/kibana:7.17.26

http://${IP address of the server where Kibana is deployed}:5601

Query and analyze data

1. In the navigation pane on the left, choose Analytics>Discover.

   Important

   If you use the Elasticsearch-compatible API to analyze Simple Log Service data in Kibana, you can only use the Discover and Dashboard modules.

   

2. In the upper-left corner of the page, select the destination index. In the upper-right corner of the page, select a time range to query log data.

   

Manually configure an index pattern (optional)

1. In the navigation pane on the left, choose Management >Stack Management.

2. In the navigation pane on the left, choose Kibana > Index Patterns.

3. The first time you use Kibana, click create an index pattern against hidden or system indices in the displayed dialog box.

   Note

   No data is displayed in the index pattern list by default. You must manually create index patterns in Kibana and map them to Logstores in Simple Log Service.

4. In the Create index pattern window, configure the parameters.

   

   Parameter name	Description
   Name	The name of the index. The naming convention is ${Simple Log Service project name}.${Logstore name}. Important The index name supports only exact match. Wildcard characters are not supported. You must enter a complete index name. For example, if the project name is etl-guangzhou and the Logstore name is es_test22, the index name is etl-guangzhou.es_test22.
   Timestamp field	The timestamp field. The value is fixed to @timestamp.

5. Click Create index pattern.