Obtain the IPIP library from OSS and enrich IP address data - Simple Log Service

You can use the data transformation feature of Simple Log Service to obtain the IPIP library from Object Storage Service (OSS) and use the library to identify the city, state, and country to which an IP address belongs.

Prerequisites

AccessKey pairs are created to access an OSS bucket. For more information, see Create an AccessKey pair.
We recommend that you create an AccessKey pair that has read-only permissions on the OSS bucket and an AccessKey pair that has write-only permissions on OSS. This way, you can use the first AccessKey pair to read data from the OSS bucket and use the second AccessKey pair to write data to the OSS bucket. For more information, see RAM policies.
The IPIP library is downloaded from the IPIP.NET website and uploaded to OSS. For more information, see Upload objects.
We recommend that you use an AccessKey pair that has write-only permissions on OSS to upload the library.

Background information

IPIP.NET provides a global IP address library that allows you to identify geographic locations of IP addresses around the world. You can download the IP address library from the IPIP.NET website, and upload the library to OSS. If you want to identify the city, state, and country to which an IP address belongs during data transformation, you can use the res_oss_file function to obtain the IPIP library. Then you can use the Structured data functions function to parse IP addresses and use the e_set function to add new fields to log entries.

Examples

Raw log entry:
```
ip: 192.0.2.1
```

Transformation rule:

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id=res_local("AK_ID"),
                                                 ak_key=res_local("AK_KEY"),
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                 format='binary',change_detect_interval=200)))

The following table describes the fields in the res_oss_file function.

Field	Description
endpoint	The endpoint of OSS. For more information, see Regions and endpoints.
ak_id	The AccessKey ID that has read permissions on OSS. For security concerns, we recommend that you set the value to res_local("AK_ID") in the Advanced Parameter Settings field. For information about how to set the Advanced Parameter Settings field, see Create a data transformation job.
ak_key	The AccessKey secret that has read permissions on OSS. For security concerns, we recommend that you set the value to res_local("AK_KEY") in the Advanced Parameter Settings field.
bucket	The OSS bucket used to store the IP address library.
file	The name of the IP address library that you have uploaded.
format	The format of the data obtained from the IP address library. Set the value to binary.

Result:

ip: 192.0.2.1
city: Hangzhou
province: Zhejiang
country: China