How to obtain the IP2Location library from OSS and enrich IP address data - Simple Log Service

You can use the data transformation feature of Simple Log Service to obtain the IP2Location library from Object Storage Service (OSS) and enrich IP address data in logs. After the enrichment, the data includes the country, province, and city to which an IP address belongs.

Prerequisites

An AccessKey pair that has the read-only permissions on OSS is created. The AccessKey pair is used to obtain IP address files from OSS buckets. For more information, see Create an AccessKey pair. For more information about authorization policies, see RAM policy.
An AccessKey pair that has the write-only permissions on OSS is created. The AccessKey pair is used to upload IP address files to OSS buckets. For more information, see Upload objects. IP address files are downloaded from the official website of IP2Location and uploaded to OSS buckets.

Background information

IP2Location provides a global IP address library that you can use to identify the geographic locations of IP addresses all around the world. You can download IP address files from the official website of IP2Location and upload the files to OSS. Then, you can use the data transformation feature to obtain the IP address files from OSS and enrich IP address data in Simple Log Service. After the enrichment, the data includes the country, province, and city to which an IP address belongs.

Procedure

Perform the following operations on the data transformation page. For more information, see Create a data transformation job.

Use the res_oss_file function to obtain the IP2Location library from an OSS bucket. For more information, see res_oss_file.
Use the geo_parse function to parse an IP address in the IP2Location library. For more information, see geo_parse.
Use the e_set function to add new fields that are obtained after parsing to a log to enrich the IP address data. For more information, see e_set.

Raw log
```
ip: 192.0.2.1
```

Transformation rule

e_set(
    "geo",
    geo_parse(
        v("ip"),
        ip_db=res_oss_file(
            endpoint="http://oss-cn-hangzhou.aliyuncs.com",
            ak_id=res_local("AK_ID"),
            ak_key=res_local("AK_KEY"),
            bucket="test",
            file="your ip2location bin file",
            format="binary",
            change_detect_interval=20,
        ),
        provider="ip2location",
    ),
)
e_json("geo")

The following table describes the fields that are related to the res_oss_file function. For more information, see res_oss_file.

Field	Description
endpoint	The endpoint that is used to access the OSS bucket. For more information, see Regions and endpoints.
ak_id	The AccessKey ID that has the read-only permissions on OSS. For security purposes, we recommend that you set the value to `res_local("AK_ID")`. In this case, the system obtains the ID from the `Advanced Parameter Settings` field. For more information about how to configure the `Advanced Parameter Settings` field, see Create a data transformation job.
ak_key	The AccessKey secret that has the read-only permissions on OSS. For security purposes, we recommend that you set the value to res_local("AK_KEY"). In this case, the system obtains the ID from the Advanced Parameter Settings field.
bucket	The OSS bucket that is used to store your IP address file.
file	The name of the uploaded IP address file.
format	The format of the IP address file in the IP2Location library that is obtained from OSS. Use format='binary'.

Result

ip: 192.0.2.1
city: Dearborn
province: Michigan
country: United States