Add logs of cloud services that belong to the same or a different Alibaba Cloud account, or a third-party cloud account - Security Center

After you enable the cloud threat detection and response (CTDR) feature, you can add logs of cloud services to the feature to monitor and analyze alerts and logs across resources in a centralized manner. The cloud services can belong to the same Alibaba Cloud account as Security Center, a different Alibaba Cloud account from Security Center, or a third-party cloud account. After you add logs, the CTDR feature monitors and analyzes the added logs, identifies attacks, builds complete attack chains, and generates security events. This improves the analysis and handling efficiency of alerts.

Prerequisites

The CTDR feature is enabled. For more information, see Purchase and enable threat analysis and response.
Simple Log Service is activated for the cloud service other than Security Center whose logs you want to add to the CTDR feature. For more information, visit the Documentation center.
Note
If you add logs of Security Center, you do not need to separately purchase the log analysis feature.

Add logs of Alibaba Cloud services

If you want to add the logs of cloud services that belong to the current Alibaba Cloud account, you need to only find the required cloud services and log types and add logs on the Service Integration page.
If you want to configure a log collection policy to add logs of cloud services that belong to different Alibaba Cloud accounts, you must configure multi-account management settings and log on to the Security Center console by using the global administrator account that you specify. Then, go to the Service Integration page and select Global Account View. Then, perform the following operations to add logs. For more information, see Use the multi-account management feature.

In the left-side navigation pane, choose CTDR > Service Integration.
On the Service Integration page, find the required cloud service and click Access Settings in the Actions column.
In the panel that appears, find the required log type and click the number in the Associated Accounts column.
You can also select multiple log types and click the button in the lower part to select multiple accounts from which you add these types of logs at a time.
In the Access Settings panel, find the required log type and click Select in the Import Account column.
Note
If the current logon account passed only individual real-name verification, only the current logon account is displayed in the Select Account panel. You can select the accounts that are managed by the CTDR feature only when the current logon account is the global administrator account and Global Account View is selected.
- If a cloud service such as Security Center supports only a dedicated Logstore, you need to only select the current logon account. You do not need to select a Logstore. After you select the current logon account, the logs of the cloud service are automatically stored in the dedicated Logstore.
- If a cloud service also supports custom Logstores, you must select the current logon account and the required Logstore from the drop-down list in the LogStore (Format: regionId.project.logStore) column. Alternatively, you can copy and paste the name of the custom Logstore that you want to use. The name of a Logstore is in the regionId.project.logStore format.
Turn on or turn off the switch in the Automatically Associate New Accounts column based on your business requirements.
If you turn on the switch for a log type and a new Alibaba Cloud account is added to the CTDR feature, the feature automatically collects the logs of the log type of the cloud services within the new account.
Note
Only the global administrator account can turn on the switch after Global Account View is selected.

Add logs of third-party cloud services

If your business is deployed on Alibaba Cloud and a third-party cloud service and you want to manage alerts across cloud environments, you can add your third-party cloud account to the CTDR feature to implement centralized alert monitoring and operations management. Supported providers of third-party cloud services are Huawei Cloud and Tencent Cloud.

1. Configure a third-party cloud account

Configure a Huawei Cloud sub-account

Create two custom policies named siemBasePolicy and siemNormalPolicy. For more information, see Create a custom policy.

Note

When you create a custom policy on Huawei Cloud, you cannot select global-level and project-level cloud services at the same time. In this case, you must create two policies to comply with the principle of least privilege.

siemBasePolicy: the permissions on global-level cloud services. The following code shows the content of the policy:

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:roles:listRoles",
                "iam:roles:getRole",
                "iam:groups:listGroupsForUser",
                "iam:groups:listGroups",
                "iam:users:getUser",
                "iam:groups:getGroup"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "rms:resources:list",
                "rms:resources:summarize"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "obs:object:GetObject",
                "obs:bucket:GetBucketLocation",
                "obs:bucket:HeadBucket",
                "obs:object:GetObjectVersionAcl",
                "obs:bucket:ListAllMyBuckets",
                "obs:bucket:ListBucket",
                "obs:object:GetObjectVersion",
                "obs:object:GetObjectAcl"
            ]
        }
    ]
}

siemNormalPolicy: the permissions on project-level cloud services. The following code shows the content of the policy:

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cfw:ipGroup:list",
                "cfw:acl:list",
                "cfw:ipMember:put",
                "cfw:ipMember:create",
                "cfw:ipGroup:create",
                "cfw:instance:get",
                "cfw:ipGroup:put",
                "cfw:ipMember:list",
                "cfw:ipGroup:get",
                "cfw:ipMember:delete"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "waf:whiteBlackIpRule:list",
                "waf:whiteBlackIpRule:put",
                "waf:ipgroup:get",
                "waf:whiteBlackIpRule:get",
                "waf:ipgroup:list",
                "waf:whiteBlackIpRule:create",
                "waf:whiteBlackIpRule:delete"
            ]
        }
    ]
}

Create user groups named siemUser and readonlyuser, and grant the required permissions to the user groups. The following table describes the required permissions. For more information, see Create a user group and assign permissions.

User Group	Required permission
siemUser	Custom policies: siemBasePolicy and siemNormalPolicy.
readonlyuser	LTS ReadOnlyAccess: the read-only permission on Log Tank Service (LTS). OBS OperateAccess: the permissions to perform basic operations on Object Storage Service (OBS) of Huawei Cloud. The following operations are included: view the bucket list, obtain bucket metadata, list objects in a bucket, query bucket locations, upload objects, obtain objects, delete objects, and obtain the ACL configurations of an object. OBS ReadOnlyAccess: the read-only permissions on OBS. The following operations are supported: view the bucket list, obtain bucket metadata, list objects in a bucket, and query the location of a bucket. CFW ReadOnlyAccess: the read-only permissions on Cloud Firewall. WAF ReadOnlyAccess: the read-only permissions on Web Application Firewall (WAF).

Create an Identity and Access Management (IAM) user and associate the IAM user with the siemUser user group. For more information, see Create an IAM user.
Create an AccessKey pair for the IAM user. For more information, see Manage AccessKey pairs for an IAM user.

Configure a Tencent Cloud sub-account

Create a custom policy named siemPolicy based on policy syntax.

The following code shows the content of the policy.

{
    "statement": [
        {
            "action": [
                "cfw:DescribeAclApiDispatch",
                "cfw:DescribeBorderACLList",
                "cfw:CreateAcRules"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "waf:DescribeDomains",
                "waf:DescribeIpAccessControl",
                "waf:DeleteIpAccessControl",
                "waf:UpsertIpAccessControl",
                "waf:PostAttackDownloadTask"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:DescribeDatahubGroupOffsets",
                "ckafka:DescribeGroup",
                "ckafka:DescribeGroupInfo",
                "ckafka:DescribeGroupOffsets",
                "ckafka:CreateDatahubGroup",
                "ckafka:ModifyDatahubGroupOffsets",
                "ckafka:ListConsumerGroup"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "cam:GetUser",
                "cam:CheckSubAccountName",
                "cam:CheckUserPolicyAttachment",
                "cam:GetAccountSummary",
                "cam:GetPolicy",
                "cam:GetPolicyVersion",
                "cam:ListAllGroupsPolicies",
                "cam:ListAttachedGroupPolicies",
                "cam:ListAttachedRolePolicies",
                "cam:ListAttachedUserAllPolicies",
                "cam:ListAttachedUserPolicies",
                "cam:ListGroupsPolicies",
                "cam:ListPolicies",
                "cam:ListUsers"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        }
    ],
    "version": "2.0"
}

Create a sub-account. For more information, see Create a sub-account.
Attach the siemPolicy policy to the created sub-account. For more information, see Authorization management.
Create an AccessKey pair for the sub-account. For more information, see AccessKey pair.

2. Transfer required logs to a specific cloud service

Before you can use CTDR, you must transfer the logs of cloud services to storage or messaging cloud services such as OBS and TDMQ for CKafka (CKafka). This way, CTDR can directly read and analyze the logs from the cloud services. You must transfer logs to a cloud service based on the log type. The following table describes how to transfer logs.

Cloud service provider	Cloud service log	Transfer destination service	Transfer configuration	Description of log collection delay
Huawei Cloud	Alert logs of Cloud Firewall Alert logs of WAF	obs	Transfer logs stored in LTS to OBS. For more information, see Transfer logs to OBS. The following list describes the key parameters: Custom Log Transfer Path: Set this parameter to Enabled and the last time variable in the custom path to %M, which indicates minutes. Example: `/LogTanks/cn-north-4/CFW/lts-topic-cfw-0001//%Y/%m/%d/%H/%M`. Log Transfer Interval: Set this parameter to 2 minutes. Important You must configure the parameters based on the preceding information. CTDR collects data from OBS based on the file directory. If the collection frequency that you configure does not fit the directory structure, the system may repeatedly pull the same data. CTDR cannot collect data stored in encrypted buckets. Do not transfer logs to encrypted buckets.	Data collected from OBS is offline data, which leads to delays in data collection. In the current mechanism of the system, data collection is delayed by three specified collection intervals compared with the current system time. For example, if you specify 2 minutes as the collection interval and a collection task is started at 17:58 on September 10, 2024, the system retrieves data from the directory /2024/09/10/17/52. The data in the directory is from 6 minutes ago, which corresponds to three collection intervals. This mechanism ensures data integrity. You must wait for three collection intervals. This helps prevent incomplete data or data loss due to ongoing data write operations, particularly in scenarios in which a large amount of data is processed.
Tencent Cloud	Alert logs of Cloud Firewall (Only logs of intrusion prevention are supported.)	ckafka	Transfer logs to specific CKafka topics. For more information, see Log shipping.	Data is collected in real time. No collection delay occurs.
Tencent Cloud	Alert logs of WAF	None	CTDR calls WAF API operations to collect logs every 10 minutes. You do not need to manually transfer logs.	The delay in data collection is more than 10 minutes.

3. Add the third-party cloud account to CTDR

You must add the third-party cloud account to the CTDR feature by entering the AccessKey pair of the sub-account. This way, the feature can obtain the alert logs of third-party cloud assets.

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
Grant permissions to the sub-account.
Security Center obtains the read permissions on third-party cloud assets and synchronizes the information about third-party cloud assets by using the AccessKey pair of the sub-account.
1. In the left-side navigation pane, choose CTDR > Service Integration.
2. In the Multi-cloud Service Access section, move the pointer over the icon of the required third-party cloud service provider and click Grant Permission.
3. In the Edit Multi-cloud Configuration panel, select Manual Configuration, select Threat Analysis in the Permission Description section, and then click Next.
4. In the Submit AccessKey Pair step, enter the AccessKey pair of the sub-account and click Next.
5. In the Policy Configuration step, configure the AK Service Status Check parameter and click OK.
Add the sub-account to the CTDR feature.
1. In the left-side navigation pane, choose CTDR > Service Integration.
2. In the Multi-cloud Service Access section, move the pointer over the icon of the required third-party cloud service provider and click Add Account.
3. In the Add Account panel, click Add.
4. In the Account Association Settings panel, enter the name and ID of the master account for the sub-account, select the AccessKey ID of the sub-account, and then click Associate Account and Associate Data Source.
5. In the Data Source Settings panel, specify the cloud services whose logs you want to add.
  - Huawei Cloud: A data source can store data only from one OBS bucket. If you want to import data from multiple buckets, create the required number of data sources. Otherwise, you need to only create one data source.
    1. In the Data Source Settings - Huawei Cloud panel, configure the Access Method, Data Source Name, Region, and Bucket Name parameters. Then, click Save Data Source.
    2. Click Add Log Type, select the log type that you want to add in the Log Type column, enter the path to the required OBS bucket in the OBS File Path field, and then click Save Log Type.
      Set the last time variable in the custom path of the OBS File Path parameter to %M, which indicates minutes. Example: /LogTanks/cn-north-4/CFW/lts-topic-cfw-0001//%Y/%m/%d/%H/%M.
      Save the configurations of the log type. If the required logs of Cloud Firewall and WAF are transferred to OBS buckets, you must re-click Add Log Type to add another log type.
  - Tencent Cloud: The collection methods for alert logs from Cloud Firewall and WAF vary. If you want to add both types of logs, you must separately create a data source for each log type. In this topic, the alert logs of Cloud Firewall are used. If you want to add the alert logs of WAF, complete configurations as prompted.
    1. In the Data Source Settings - Tencent Cloud panel, configure the Access Method, Data Source Name, Internet URL, Username, and Password parameters. Then, click Save Data Source.
    2. Click Add Log Type, configure the Log Topic and Consumer Group Name parameters, select a log type in the Log Type column, and then click Save Log Type.

4. Add the logs of cloud services within the third-party cloud account

In the left-side navigation pane, choose CTDR > Service Integration.
On the Service Integration page, find the third-party cloud service whose logs you want to add and click Access Settings in the Actions column.
In the panel that appears, find the required log type and click the value in the Associated Accounts column.
In the panel that appears, select the required account and click OK.
Turn on or turn off the switch in the Automatically Associate New Accounts column based on your business requirements.
If you turn on the switch for a log type and a new third-party cloud account is added to the CTDR feature, the feature automatically collects the logs of the log type of the cloud services within the new account.
Note
Only the global administrator account can turn on the switch after Global Account View is selected.

References

After you add logs of cloud services to CTDR, you can configure detection rules to aggregate multiple related alerts into security events that contain complete attack chains. This reduces the number of alerts and improves the analysis and handling efficiency of alerts. For more information, see Use detection rules.
You can use the charts on the dashboard provided by the CTDR feature to centrally monitor and manage the security status of your enterprise across cloud platforms, accounts, and cloud services. You can also review the performance of security operations. For more information, see Dashboard.
You can use the log management feature of CTDR to quickly query logs and view information about logs. This helps simplify log management in a multi-resource environment. For more information, see Log management.
You can call API operations to submit multiple cloud service adding tasks or log adding tasks at the same time, or view cloud accounts that are added to the CTDR feature. For more information, see Log Management.
Does the threat analysis and response feature support devices in a data center?
What do I do if the amount of log data that is added and stored exceeds the purchased log capacity?