Simple Log Service:Connect Simple Log Service to Grafana

Version compatibility between Grafana and aliyun-log-grafana-datasource-plugin

The following table describes the version compatibility between Grafana and aliyun-log-grafana-datasource-plugin.

Usage notes

If you want to configure Security Token Service (STS)-based redirection and ensure security, take note of the following points:

● For the Resource Access Management (RAM) user whose AccessKey pair is used when you add a data source in Grafana, you must attach the AliyunRAMReadOnlyAccess policy to the RAM user.

● For the RAM role whose Alibaba Cloud Resource Name (ARN) is specified for the data source, you must attach only the AliyunLogReadOnlyAccess policy to the RAM role. For more information, see Embed console pages and share log data.

After you configure STS-based redirection in password-free mode, check whether public access and sharing are allowed for Grafana dashboards that are generated based on the data source. Public access may lead to increase in traffic costs and public exposure of log content.

Step 1: Install the Simple Log Service plug-in

The following procedure describes how to install the Simple Log Service plug-in for Grafana:

1. Run the following commands to decompress the project package to the plug-in directory of Grafana.

   • If Grafana is installed by using a YUM repository or an RPM package, run the following command:

     unzip aliyun-log-grafana-datasource-plugin-master.zip -d /var/lib/grafana/plugins

   • If Grafana is installed by using a .tar.gz file, run the following command:

     {PATH_TO} specifies the installation directory of Grafana.

     unzip aliyun-log-grafana-datasource-plugin-master.zip -d {PATH_TO}/grafana-8.0.6/data/plugins

2. Modify the configuration file of Grafana.

   1. Open the configuration file.

      • If Grafana is installed by using a YUM repository or an RPM package, open the /etc/grafana/grafana.ini file.

      • If Grafana is installed by using a .tar.gz file, open the {PATH_TO}/grafana-8.0.6/conf/defaults.ini file.

   2. Find [plugins] in the configuration file to configure the allow_loading_unsigned_plugins parameter.

      allow_loading_unsigned_plugins = aliyun-log-service-datasource

3. Restart the Grafana service.

   1. Run the kill command to terminate the Grafana process.

   2. Run the following commands to start the Grafana service:

      • If Grafana is installed by using a YUM repository or an RPM package, run the following command:

        systemctl restart grafana-server

      • If Grafana is installed by using a .tar.gz file, run the following command:

        ./bin/grafana-server web

Step 2: Add a data source for Grafana

The following procedure describes how to add the Simple Log Service plug-in as a data source for Grafana:

1. Log on to Grafana.

2. In the left-side navigation pane, choose > Data Sources.

3. On the Data Sources tab, click Add data source.

4. On the Add data source page, click Select in the LogService card.

5. Configure the data source.

   The following table describes the parameters.

   Parameter	Description
   Name	The name of the data source.
   Default	The Default switch. In this example, turn on the switch.
   Endpoint	The endpoint of the Simple Log Service project. Example: http://cn-qingdao.log.aliyuncs.com. Enter an endpoint based on your business requirements. For more information, see Endpoints.
   Project	The name of the project.
   Logstore	The name of the Logstore.
   AccessKeyId	The AccessKey ID provided by Alibaba Cloud. The AccessKey ID is used to identify the user. To ensure the security of your account, we recommend that you use the AccessKey pair of a RAM user. For more information about how to obtain an AccessKey pair, see AccessKey pair.
   AccessKeySecret	The AccessKey secret provided by Alibaba Cloud. The AccessKey secret is used to authenticate the key of the user. To ensure the security of your account, we recommend that you use the AccessKey pair of a RAM user. For more information about how to obtain an AccessKey pair, see AccessKey pair.

6. Click Save & Test.

Step 3: Add a dashboard

Perform the following steps to add a dashboard for Grafana:

1. In the left-side navigation pane, choose > Dashboards.

2. In the New Panel panel, click Choose Visualization.

3. Configure template variables.

   After you configure template variables in Grafana, you can select different variable values to view different results in the same panel.

   1. Configure a template variable for a time interval.

      1. In the upper-right corner of the New dashboard page, select a time interval and click the icon.

      2. Click Variables.

      3. Click Add variable.

      4. Configure the parameters for the template variable. Then, click Add.

         The following table describes the parameters.

         Parameter	Description
         Name	Enter the name of the template variable. Example: myinterval. If you use a template variable in a query criteria, you must add two dollar signs ($$) preceding the name of the template variable. Example: $$myinterval.
         Type	Select Interval.
         Label	Enter time interval.
         Values	Enter 1m,10m,30m,1h,6h,12h,1d,7d,14d,30d.
         Auto Option	Turn on Auto Option. Retain the default values of the other parameters.

   2. Configure a template variable for a domain.

      1. On the Variables page, click New.

      2. Configure the parameters for the template variable. Then, click Add. The following table describes the parameters.

         Parameter	Description
         Name	Enter the name of the template variable. Example: hostname. If you use a template variable in a query criteria, you must add a dollar sign ($) preceding the name of the template variable. Example: $hostname.
         Type	Select Custom.
         Label	Enter the name of the domain.
         Custom Options	Enter *,example.com,example.org,example.net. You can view information about access to all domains. You can also view information about access to the example.com, example.org, or example.net domain.
         Selection Options	Retain the default value of the parameter.

   3. In the left-side navigation pane, click Save.

4. Add panels based on your business requirements.

   • Add a graph panel to display page views (PVs) and unique visitors (UVs).

     1. In the upper-right corner of the page, click the icon.

     2. In the New Panel panel, click Add Query.

     3. Click the icon and select Graph from the Visualization drop-down list.

     4. Click the icon. Then, enter UV&PV in the Title field.

     5. Click the icon, select Logservice from the Query drop-down list, and then configure the parameters. The following table describes the parameters.

        Parameter	Description
        Query	Enter the query statement that you want to execute. Example: $hostname| select approx_distinct(remote_addr) as uv ,count(1) as pv , __time__ - __time__ % $$myinterval as time group by time order by time limit 1000 In the query and analysis results, $hostname is replaced with the name of the domain that you specify and $$myinterval is replaced with the time interval that you specify. Important Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.
        X-Column	Enter time. The value is a UNIX timestamp that is accurate to the second.
        Y-Column	Enter uv,pv.

     6. If the difference between the values of UVs and the values of PVs is significant, use a dual Y-axis panel to display UV and PV statistics.

     7. Click the icon to save your settings.

   • Add a graph panel to display inbound traffic and outbound traffic.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs) " section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select sum(body_byte_sent) as net_out, sum(request_length) as net_in,__time__ - __time__ % $$myinterval as time group by __time__ - __time__ % $$myinterval limit 10000 In the query and analysis results, $hostname is replaced with the name of the domain that you specify and $$myinterval is replaced with the time interval that you specify. Important Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.
     X-Column	Enter time. The value is a UNIX timestamp that is accurate to the second.
     Y-Column	Enter net_in,net_out.

   • Add a pie chart panel to display the distribution of requests by HTTP request method.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv ,method group by method In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter pie.
     Y-Column	Enter method,pv.

   • Add a pie chart panel to display the distribution of requests by HTTP status code.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv ,status group by status In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter pie.
     Y-Column	Enter status,pv.

   • Add a pie chart panel to display the request sources of frequently accessed pages.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv , referer group by referer order by pv desc In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter pie.
     Y-Column	Enter referer,pv.

   • Add a table panel to display the pages that are loaded at the highest latencies.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select URL as top_latency_URL ,request_time order by request_time desc limit 10 In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	You do not need to configure this parameter.
     Y-Column	Enter top_latency_url,request_time.

   • Add a table panel to display frequently accessed pages.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv, split_part(URL,'?',1) as path group by split_part(URL,'?',1) order by pv desc limit 20 In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	You do not need to configure this parameter.
     Y-Column	Enter path,pv.

   • Add a table panel to display the pages that have the most access requests with HTTP status code 200 not returned.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname not status:200| select count(1) as pv , url group by url order by pv desc In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	You do not need to configure this parameter.
     Y-Column	Enter url,pv.

   • Add a singlestat panel to display the average latency.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select avg(request_time) as response_time, avg(upstream_response_time) as upstream_response_time ,__time__ - __time__ % $$myinterval as time group by __time__ - __time__ % $$myinterval limit 10000 In the query and analysis results, $hostname is replaced with the name of the domain that you specify and $$myinterval is replaced with the time interval that you specify. Important Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.
     X-Column	Enter time.
     Y-Column	Enter upstream_response_time,response_time.

   • Add a logs panel to display the details about logs.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

     Note

     Each page can display up to 100 logs. Therefore, the maximum value of the Logs Per Page parameter is 100.

   • Add a pie chart panel to display client statistics.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv, case when regexp_like(http_user_agent , 'okhttp') then 'okhttp' when regexp_like(http_user_agent , 'iPhone') then 'iPhone' when regexp_like(http_user_agent , 'Android') then 'Android' else 'unKnown' end as http_user_agent group by http_user_agent order by pv desc limit 10 In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter pie.
     Y-Column	Enter http_user_agent,pv.

   • Add a graph panel to display the number of requests for which each HTTP status code is returned over a 1-minute time range.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select to_unixtime(time) as time,status,count from (select time_series(__time__, '1m', '%Y-%m-%d %H:%i', '0') as time,status,count(*) as count from log group by status,time order by time limit 10000) In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter time.
     Y-Column	Enter col1#:#col2. col1 is the aggregated column, and col2 is one of the other columns.

   • Add a worldmap panel to display the distribution of source IP addresses.

     For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

     Parameter	Description
     Query	Enter the query statement that you want to execute. Example: $hostname | select count(1) as pv ,geohash(ip_to_geo(arbitrary(remote_addr))) as geo,ip_to_country(remote_addr) as country from log group by country having geo <>'' limit 1000 In the query and analysis results, $hostname is replaced with the name of the domain that you specify.
     X-Column	Enter map.
     Y-Column	Enter country,geo,pv.

     Parameter	Description
     Location Data	Select geohash.
     Location Name Field	Enter country.
     geo_point/geohash Field	Enter geo.
     Metric Field	Enter pv.

5. View the results.

   In the upper section of the Dashboard page, you can select a time interval. You can also filter the results by using the time interval and hostname query criteria.

FAQ

• Where are Grafana logs stored?

  Grafana logs are stored in the following directories:

  • macOS: /usr/local/var/log/grafana

  • Linux: /var/log/grafana

• What do I do if aliyun-log-plugin_linux_amd64: permission denied appears in logs?

  Grant the EXECUTE permission on the dist/aliyun-log-plugin_linux_amd64 directory of the plug-in directory.