Simple Log Service allows you to collect data from multiple sources, such as servers, applications, open source software, IoT devices, mobile devices, and Alibaba Cloud services. You can also collect data that is transferred over standard protocols. This topic describes the data sources that are supported by Simple Log Service.
Collection methods
Collection method | Description | References |
Collection by using Logtail | Logtail is a log collection agent that is provided by Simple Log Service. Logtail has been used to collect data from major services of Alibaba Group for several years. Logtail collects real-time data from millions of servers and processes dozens of petabytes of data every day. Logtail has demonstrated its stability and reliability during Double 11 events over the years. | |
Collection from Alibaba Cloud services | Simple Log Service can collect logs from various Alibaba Cloud services, such as elastic computing, storage,A security, and database services. The logs record statistics such as user operations, running status, and business dynamics of Alibaba Cloud services. | |
Data import | Simple Log Service can import data of Object Storage Service (OSS) objects, Amazon Simple Storage Service (S3) objects, MySQL databases, Elasticsearch clusters, and Kafka clusters. | |
Other collection methods | You can use the Kafka protocol, Syslog protocol, or Loggie to upload logs to Simple Log Service. You can also use the web tracking feature or Simple Log Service SDK to upload logs to Simple Log Service. | |
Collection of common logs | Simple Log Service can collect Log4j logs, Python logs, Node.js logs, WordPress logs, and Unity3D logs. |
Data sources
Category | Source | Collection method | References |
Application | Program output | None | |
Access log | |||
Trace data | None | ||
Metric data | None | ||
Programming language | Java | None | |
Log4J Appender | None | ||
LogBack Appender | None | ||
C | None | ||
Python | None | ||
Python Logging | None | None | |
PHP | None | ||
.Net | None | ||
C++ | None | ||
Go | None | ||
NodeJS | None | ||
JS | None | ||
OS | Linux | None | |
Windows | None | ||
Mac/Unix | None | ||
Docker file | None | ||
Docker output | None | ||
Database | MySQL Binlog | None | |
MySQL query result | None | ||
SQL Server query result | None | ||
PostgreSQL query result | None | ||
Mobile device | iOS and Android | None | |
Web page | None | ||
Intelligent IoT | None | ||
Standard protocol | HTTP | ||
Syslog | None | ||
Kafka | None | ||
OpenTelemetry | None | ||
Imported data | Object Storage Service (OSS) data | None | |
MySQL data | None | ||
Elasticsearch data | None | ||
CloudMonitor data | None | ||
Third-party software | Logstash | None | |
Flume | None | ||
Beats | None | ||
Fluentd | None | ||
Telegraf | None | ||
Alibaba Cloud service | Services such as Elastic Compute Service (ECS) and OSS | None |
Network types and endpoints
Alibaba Cloud internal network: An Alibaba Cloud internal network provides gigabits of shared bandwidth. Transmission of log data over an internal network is stable and fast. Alibaba Cloud internal networks include virtual private clouds (VPCs) and the classic network.
Internet: Transmission of log data over the Internet is limited by the network bandwidth. Network issues such as jitters, latency, and packet loss may also occur. This affects the speed and stability of data transmission.
Transfer acceleration: This network type can be used to accelerate log collection by using the edge nodes of Alibaba Cloud CDN (CDN). Compared with log collection over the Internet, log collection based on transfer acceleration is more stable, and the transmission latency is lower.
For more information about the scenarios in which you can select a specific network type, see Select a network type. For more information about endpoints, see Endpoints. For more information about how to enable the transfer acceleration feature, see Use the transfer acceleration feature.
FAQ
Which network type do I need to select if I use Express Connect together with Simple Log Service?
We recommend that you select the internal network type.
Can I obtain public IP addresses when I collect data from the Internet?
Yes, you can turn on Log Public IP for your Logstore to obtain public IP addresses. For more information, see Create a Logstore.
Which network type do I need to select if I want to collect ECS logs from Region A to a Simple Log Service project in Region B?
We recommend that you select the Internet type and specify the region name of Region B in the command used to install Logtail on your Elastic Compute Service (ECS) instance in Region A. This allows Logtail to access the project in Region B over the Internet. For more information about how to select a network type in other scenarios, see Select a network type.
How do I check whether an endpoint is accessible?
You can run the following command. If output is returned, the endpoint is accessible.
curl $myproject.cn-hangzhou.log.aliyuncs.com$myproject: the name of the project.cn-hangzhou.log.aliyuncs.com: the endpoint.