Simple Log Service:Collect IoT or embedded development logs

Requirements in embedded software development

IoT or embedded software developers must have profound knowledge and rich experience to manage, monitor, and troubleshoot a large number of IoT devices. Embedded software development has the following requirements:

• Data collection: Collect data from millions or tens of millions of devices distributed around the world in real time.

• Debugging: Use a set of solutions to collect data online and debug software in real time.

• Online troubleshooting: Locate an online IoT device and identify the error cause if an error occurs on the device.

• Monitoring: Monitor the number, statuses, and locations of online IoT devices, and configure alert rules for the devices.

• Real-time data analysis: Visualize the data generated by IoT devices to create user profiles by connecting the devices with real-time computing platforms and big data warehouses.

Major challenges in the IoT field

Traditional software development solutions cannot meet the preceding requirements in the IoT industry due to the following challenges:

• A large number of IoT devices are deployed. In traditional O&M, managing 10,000 servers is a big challenge. However, in the IoT industry, managing 100,000 online IoT devices is only a basic requirement.

• IoT devices are widely distributed in different places within a country or around the world.

• IoT devices are in unknown statuses in most cases and are difficult to access and debug.

• IoT devices are equipped with low-end hardware to reduce costs. For example, an IoT device may have a total memory size of 32 MB. As a result, traditional log collection solutions for PCs do not apply to IoT devices.

C Producer

A log data collection solution customized by Simple Log Service

Logtail is the agent of Simple Log Service and is deployed on millions of x86 servers. In addition, Simple Log Service provides a variety of collection solutions:

• Mobile SDK: You can use the SDKs to collect data from Android or iOS platforms with tens of millions of daily active users (DAUs).

• Web tracking (JavaScript): Web tracking is a lightweight solution that is similar to Baidu Tongji and Google Analytics. You can use web tracking to collect data without a signature.

In these solutions, C Producer Library is a solution that is customized based on Logtail to collect log data from IoT devices. The solution is compatible with the CPU, memory, disk, network, and application mode of IoT devices. The following figure shows the features of these solutions.

Features of C Producer Library

C Producer Library serves as a lightweight Logtail to offer high stability, high performance, and low resource consumption. Compared with Logtail, C Producer Library does not support real-time configuration management. However, C Producer Library inherits the following features of Logtail:

• Multiple tenants: C Producer Library can process multiple types of logs such as Metric, DebugLog, and ErrorLog based on their priorities. You can configure multiple clients for C Producer Library. You can also configure a collection priority, destination project, and destination Logstore for each client.

• Contextual query: Logs generated by the same client are in the same context. You can view the relevant logs before and after a specified log.

• Concurrent sending and resumable upload: You can set the maximum cache size. If the size of cached logs reaches the upper limit, no more logs can be written to the cache.

In addition, C Producer Library provides the following features that are specific to IoT devices:

• Local debugging: You can export logs to local devices. You can configure the log rotation, log quantity, and rotation size.

• Fine-grained resource control: You can set cache sizes and aggregate modes for different types of data or logs.

• Log cache compression: If log data fails to be sent to Logstores, the log data can be compressed to reduce the memory usage of IoT devices.

Advantages of C Producer Library

As a custom solution for IoT devices, C Producer Library has the following advantages:

• Highly concurrent write traffic: You can configure the thread pool of clients to write hundreds of thousands of logs per second to Simple Log Service. For more information, see the "Performance test" section in this topic.

• Low resource consumption: Only 70% of the CPU resources are required to write 200,000 logs per second. For low-performance hardware such as Raspberry Pi, the CPU resources are not affected even if 100 logs are generated per second.

• Direct log shipping: After logs are generated, they are directly sent to Simple Log Service.

• Logic isolation between computing and I/O: Logs are generated in an asynchronous manner without blocking worker threads.

• Multiple priorities: You can configure different priorities for your clients to ensure that logs with higher priorities are sent first.

• Local debugging: You can configure local debugging to test your applications if the network connection is unavailable.

C Producer Library simplifies application development. You do not need to consider the implementation of log collection or the impact on your business. This makes data collection much easier.

The following table compares C Producer Library and other embedded collection solutions.

C Producer Library + Simple Log Service solution

C Producer Library can be integrated with Simple Log Service to provide a full range of log collection solutions for IoT devices.

• Large scale

  • Writes hundreds of millions of logs from clients to Simple Log Service in real time.

  • Writes petabytes of data per day.

• High speed

  • Fast collection: Logs can be consumed immediately after being written to Simple Log Service.

  • Fast query: Billions of data records can be processed and queried within 1 second by executing a complex query statement in which you specify five conditions.

  • Fast analysis: Hundreds of millions of data records can be aggregated and analyzed within 1 second by executing a complex analysis statement that includes five aggregate functions and the GROUP BY clause.

• High compatibility

  • Seamlessly integrated with various Alibaba Cloud services.

  • Compatible with various open source storage, computing, and visualization systems.

Download and use C Producer Library

Download URL: GitHub

You can create multiple producers for each application and create multiple clients for each producer. You can configure the destination IP address, log level, local debugging, cache size, custom identifier, and topic for each client.

For more information about how to install C Producer Library, see README.

Performance test

The following figure shows the configuration details.

C Producer Library configuration

• ARM (Raspberry Pi)

  • Cache size: 10 MB

  • Aggregation time: 3 seconds. If the aggregation time, size of the aggregated packet, or number of aggregated logs reaches the specified thresholds, the data is encapsulated and sent to Simple Log Service.

  • Size of the aggregated packet: 1 MB

  • Number of aggregated logs: 1,000

  • Number of sending threads: 1

  • Number of custom tags: 5

• X86

  • Cache size: 10 MB

  • Aggregation time: 3 seconds. If the aggregation time, size of the aggregated packet, or number of aggregated logs reaches the specified thresholds, the data is encapsulated and sent to Simple Log Service.

  • Size of the aggregated packet: 3 MB

  • Number of aggregated logs: 4096

  • Number of sending threads: 4

  • Number of custom tags: 5

__source__: 192.0.2.1
__tag__:1: 2
__tag__:5: 6
__tag__:a: b
__tag__:c: d
__tag__:tag_key: tag_value
__topic__: topic_test
_file_: /disk1/workspace/tools/aliyun-log-c-sdk/sample/log_producer_sample.c
_function_: log_producer_post_logs
_level_: LOG_PRODUCER_LEVEL_WARN
_line_: 248
_thread_: 40978304
LogHub: Real-time log collection and consumption
Search/Analytics: Query and real-time analysis
Interconnection: Grafana and JDBC/SQL92
Visualized: dashboard and report functions