All Products
Search

This Product

    Simple Log Service:Use the e_table_map function to enrich HTTP response status codes

    Document Center

    Simple Log Service:Use the e_table_map function to enrich HTTP response status codes

    Last Updated:Sep 01, 2024

    NGINX logs record important information that can be used for website O&M. Simple Log Service provides the e_table_map function that you can use to enrich HTTP response status codes for NGINX log analysis. This topic describes how to use the e_table_map function to enrich HTTP response status codes.

    Prerequisites

    NGINX logs are collected. For more information, see Data collection overview.

    Example scenario

    An enterprise developed Application A and defined HTTP response status codes to maintain the application. The data is updated at irregular intervals. Only the http_code field in raw NGINX logs indicates the status of HTTP requests. This field cannot be used to identify issues in an efficient manner.

    In this case, a mapping table of HTTP response status codes is required. You can use the e_table_map function to enrich logs. Then, you can identify the status of HTTP requests in an efficient manner.

    • Sample raw log

      body_bytes_sent:1750
host:www.example.com
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

      This raw log is stored in a Logstore named nginx-demo. The value of the http_code field is an HTTP response status code.

    • Mapping table of HTTP response status codes

      The following table is a mapping table of typical HTTP response status codes.

      code

      alias

      category

      desc

      100

      1xx

      Informational

      Continue

      200

      2xx

      Success

      OK

      300

      3xx

      Redirection

      Multiple Choices

      400

      4xx

      Client Error

      Bad Request

    • Enriched log

      body_bytes_sent:1750
host:www.example.com
http_code:200
http_code_alias:2xx
http_code_category:Success
http_code_desc:OK
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

    Available solutions

    Transformation process

    image

    1. Convert HTTP response status codes into table objects.

    2. Use the e_table_map function to transform and enrich data.

    Recommended solutions

    The following table describes the solutions that you can use to enrich data.

    Solution

    Supported data volume

    Incremental update

    Batch update

    Scenario

    (Recommended) Enrich data based on a Logstore

    Large

    Supported

    Supported

    The mapping table contains a large volume of data and is frequently updated.

    Enrich data based on a MySQL table

    Medium

    Not supported

    Supported

    The mapping table is frequently updated.

    Enrich data based on an Object Storage Service (OSS) object

    Medium

    Not supported

    Supported

    The mapping table is infrequently updated.

    Enrich data based on embedded code

    Small

    Not supported

    Not supported

    The mapping table contains typical HTTP response status codes.

    Solution 1: (Recommended) Enrich data based on a Logstore

    1. Use an SDK to write HTTP response status codes to a Logstore named http_code.

      The following example shows a log that contains an HTTP response status code in the http_code Logstore:

      __source__:203.0.XXX.XX
__tag__:__receive_time__:1595424194
__topic__:
code:200
alias:2xx
description:OK
category:Success

      For more information, see Overview of Simple Log Service SDKs.

    2. Obtain the name and endpoint of the http_code Logstore and the required AccessKey pair. The obtained information is used to write a data transformation statement.

      For more information about Simple Log Service endpoints and AccessKey pairs, see Endpoints and AccessKey pair.

    3. Go to the data transformation page of the nginx-demo Logstore that stores raw logs.

      For more information, see Create a data transformation job.

    4. In the editor box, enter a data transformation statement.

      Read data from the http_code Logstore and use the e_table_map function to return the values of the matched fields. 

      e_table_map( res_log_logstore_pull("cn-hangzhou-intranet.log.aliyuncs.com",
        res_local("AK_ID"),res_local("AK_KEY"),"live-demo","http_code",
        ["code","alias","description","category"]),
              [("http_code","code")],
              [("alias","http_code_alias"), ("description","http_code_desc"), 
              ("category","http_code_category")])
      Important

      To ensure data security, we recommend that you specify an AccessKey pair in the Advanced Parameter Settings field. For more information about how to configure the Advanced Parameter Settings field, see Create a data transformation job.

      • The res_log_logstore_pull function pulls data from another Logstore when you transform data in a Logstore. For more information, see res_log_logstore_pull.

      • The e_table_map function maps the value of an input field to a row in a specified table and returns a new field. For more information, see e_table_map.

    5. Click Preview Data.

      After the raw NGINX log is enriched, new fields that are related to the HTTP response status code are included in the log. 

      body_bytes_sent:1750
host:www.example.com
http_code:200
http_code_alias:2xx
http_code_category:Success
http_code_desc:OK
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

    6. Create a data transformation job.

      For more information, see Create a data transformation job.

    Solution 2: Enrich data based on a MySQL table

    1. Save HTTP response status codes to an ApsaraDB RDS for MySQL database.

      The following figure shows the mapping table of HTTP response status codes that are stored in the ApsaraDB RDS for MySQL database.mysql

    2. Obtain the host address, username, password, database name, and table name of the ApsaraDB RDS for MySQL database. The obtained information is used to write a data transformation statement.

    3. Go to the data transformation page of the nginx-demo Logstore that stores raw logs.

      For more information, see Create a data transformation job.

    4. In the editor box, enter a data transformation statement.

      Read data from the MySQL database and use the e_table_map function to return the values of the matched fields. 

      e_table_map(res_rds_mysql(address="MySQL host address", 
                  username="Username", password="Password",
                  database="Database name",table="Table name", refresh_interval=300),
              [("http_code","code")],
              [("alias","http_code_alias"), ("description","http_code_desc"), 
              ("category","http_code_category")])
      Important

      To ensure data security, we recommend that you specify an AccessKey pair in the Advanced Parameter Settings field. For more information about how to configure the Advanced Parameter Settings field, see Create a data transformation job.

      • The res_rds_mysql function pulls data from a specified table in an ApsaraDB RDS for MySQL database. For more information, see res_rds_mysql.

      • The e_table_map function maps the value of an input field to a row in a specified table and returns a new field. For more information, see e_table_map.

    5. Click Preview Data.

      After the raw NGINX log is enriched, new fields that are related to the HTTP response status code are included in the log. 

      body_bytes_sent:1750
host:www.example.com
http_code:200
http_code_alias:2xx
http_code_category:Success
http_code_desc:OK
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

    6. Create a data transformation job.

      For more information, see Create a data transformation job.

    Solution 3: Enrich data based on an OSS object

    1. Save HTTP response status codes to an object named http_code.csv and upload the object to an OSS bucket.

      For more information, see Upload objects.

    2. Obtain the name and endpoint of the OSS bucket to which the http_code.csv object is uploaded and the required AccessKey pair. The obtained information is used to write a data transformation statement.

      For more information about OSS endpoints, see Regions and endpoints.

    3. Go to the data transformation page of the nginx-demo Logstore that stores raw logs.

      For more information, see Create a data transformation job.

    4. In the editor box, enter a data transformation statement.

      Read data from the OSS bucket and use the e_table_map function to return the values of the matched fields. 

      e_table_map(
      tab_parse_csv(
           res_oss_file(endpoint="oss-cn-shanghai-internal.aliyuncs.com",
              ak_id=res_local("AK_ID"), ak_key=res_local("AK_KEY"), 
              bucket="ali-sls-etl-test", 
              file="http_code.csv", format='text')),
              [("http_code","code")],
              [("alias","http_code_alias"),
               ("description","http_code_desc"),
               ("category","http_code_category")])
      Important

      To ensure data security, we recommend that you specify an AccessKey pair in the Advanced Parameter Settings field. For more information about how to configure the Advanced Parameter Settings field, see Create a data transformation job.

      • The res_oss_file function pulls data from an object in a specified OSS bucket. The data can be updated at regular intervals. For more information, see res_oss_file.

      • The tab_parse_csv function creates a table from a comma-separated values (CSV) file. For more information, see tab_parse_csv.

      • The e_table_map function maps the value of an input field to a row in a specified table and returns a new field. For more information, see e_table_map.

    5. Click Preview Data.

      After the raw NGINX log is enriched, new fields that are related to the HTTP response status code are included in the log. 

      body_bytes_sent:1750
host:www.example.com
http_code:200
http_code_alias:2xx
http_code_category:Success
http_code_desc:OK
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

    6. Create a data transformation job.

      For more information, see Create a data transformation job.

    Solution 4: Enrich data based on embedded code

    1. Prepare a mapping table of HTTP response status codes in the CSV format.

    2. Go to the data transformation page of the nginx-demo Logstore that stores raw logs.

      For more information, see Create a data transformation job.

    3. In the editor box, enter a data transformation statement.

      Use the tab_parse_csv function to create a table from the CSV file and use the e_table_map function to return the values of the matched fields. 

      e_table_map(tab_parse_csv("code,alias,category,description\n100,1xx,Informational,Continue\n101,1xx,Informational,Switching Protocols\n102,1xx,Informational,Processing (WebDAV)\n200,2xx,Success,OK\n201,2xx,Success,Created\n202,2xx,Success,Accepted\n203,2xx,Success,Non-Authoritative Information\n204,2xx,Success,No Content\n205,2xx,Success,Reset Content\n206,2xx,Success,Partial Content\n207,2xx,Success,Multi-Status (WebDAV)\n208,2xx,Success,Already Reported (WebDAV)\n226,2xx,Success,IM Used\n300,3xx,Redirection,Multiple Choices\n301,3xx,Redirection,Moved Permanently\n302,3xx,Redirection,Found\n303,3xx,Redirection,See Other\n304,3xx,Redirection,Not Modified\n305,3xx,Redirection,Use Proxy\n306,3xx,Redirection,(Unused)\n307,3xx,Redirection,Temporary Redirect\n308,3xx,Redirection,Permanent Redirect (experimental)\n400,4xx,Client Error,Bad Request\n401,4xx,Client Error,Unauthorized\n402,4xx,Client Error,Payment Required\n403,4xx,Client Error,Forbidden\n404,4xx,Client Error,Not Found\n405,4xx,Client Error,Method Not Allowed\n406,4xx,Client Error,Not Acceptable\n407,4xx,Client Error,Proxy Authentication Required\n408,4xx,Client Error,Request Timeout\n409,4xx,Client Error,Conflict\n410,4xx,Client Error,Gone\n411,4xx,Client Error,Length Required\n412,4xx,Client Error,Precondition Failed\n413,4xx,Client Error,Request Entity Too Large\n414,4xx,Client Error,Request-URI Too Long\n415,4xx,Client Error,Unsupported Media Type\n416,4xx,Client Error,Requested Range Not Satisfiable\n417,4xx,Client Error,Expectation Failed\n418,4xx,Client Error,I'm a teapot (RFC 2324)\n420,4xx,Client Error,Enhance Your Calm (Twitter)\n422,4xx,Client Error,Unprocessable Entity (WebDAV)\n423,4xx,Client Error,Locked (WebDAV)\n424,4xx,Client Error,Failed Dependency (WebDAV)\n425,4xx,Client Error,Reserved for WebDAV\n426,4xx,Client Error,Upgrade Required\n428,4xx,Client Error,Precondition Required\n429,4xx,Client Error,Too Many Requests\n431,4xx,Client Error,Request Header Fields Too Large\n444,4xx,Client Error,No Response (Nginx)\n449,4xx,Client Error,Retry With (Microsoft)\n450,4xx,Client Error,Blocked by Windows Parental Controls (Microsoft)\n451,4xx,Client Error,Unavailable For Legal Reasons\n499,4xx,Client Error,Client Closed Request (Nginx)\n500,5xx,Server Error,Internal Server Error\n501,5xx,Server Error,Not Implemented\n502,5xx,Server Error,Bad Gateway\n503,5xx,Server Error,Service Unavailable\n504,5xx,Server Error,Gateway Timeout\n505,5xx,Server Error,HTTP Version Not Supported\n506,5xx,Server Error,Variant Also Negotiates (Experimental)\n507,5xx,Server Error,Insufficient Storage (WebDAV)\n508,5xx,Server Error,Loop Detected (WebDAV)\n509,5xx,Server Error,Bandwidth Limit Exceeded (Apache)\n510,5xx,Server Error,Not Extended\n511,5xx,Server Error,Network Authentication Required\n598,5xx,Server Error,Network read timeout error\n599,5xx,Server Error,Network connect timeout error\n"),
              [("http_code","code")],
              [("alias","http_code_alias"), ("description","http_code_desc"), 
              ("category","http_code_category")])
      Important

      To ensure data security, we recommend that you specify an AccessKey pair in the Advanced Parameter Settings field. For more information about how to configure the Advanced Parameter Settings field, see Create a data transformation job.

      • The tab_parse_csv function creates a table from a CSV file. For more information, see tab_parse_csv.

      • The e_table_map function maps the value of an input field to a row in a specified table and returns a new field. For more information, see e_table_map.

    4. Click Preview Data.

      After the raw NGINX log is enriched, new fields that are related to the HTTP response status code are included in the log. 

      body_bytes_sent:1750
host:www.example.com
http_code:200
http_code_alias:2xx
http_code_category:Success
http_code_desc:OK
http_referer:www.example.aliyundoc.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XXX.XX
remote_addr:203.0.XXX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

    5. Create a data transformation job.

      For more information, see Create a data transformation job.