Design Principles for Observability

Observability design refers to the design for better monitoring, analysis, and management of system operation status. As technologies such as cloud-native and microservices become more and more popular, the observability of systems becomes more and more complex. Cloud observability is mainly designed from five aspects: monitoring metrics, distributed tracing, logging, monitoring dashboards, and event alarms, in order to realize the construction of a comprehensive observability system on the cloud.

Monitoring Metrics

The system needs to collect and display metrics about its operational status, such as CPU usage, memory usage, network traffic, etc. Monitoring metrics can help organizations understand the health and performance of the system, so that problems can be quickly discovered when the system encounters anomalies. Monitoring metrics can be implemented through monitoring tools, and they allow alerts to be sent when anomalies occur. There are many monitoring tools available, such as Prometheus, Grafana, Zabbix, and Alibaba Cloud CloudMonitor. These tools can collect metrics at regular intervals, provide visualized metric reports, and send alerts to help organizations discover problems in a timely manner.

Distributed Tracing

When a system encounters issues, it is necessary to be able to trace the behavior and interaction of each component in the system. By implementing distributed tracing in the system, problems can be quickly located and effectively troubleshooted. Distributed tracing can be achieved by adding tracing identifiers in the system. When a request enters the system, the identifier is added to the request and passed throughout the system. Each component can add the identifier to its logs, so that troubleshooting can be performed when problems occur. Distributed tracing can be implemented using open-source tools such as Jaeger, Zipkin, SkyWalking, and CAT. Alibaba Cloud provides ARMS to achieve distributed tracing.

Logging

The system needs to record critical events and failures to help diagnose problems and resolve failures. For a system, logging is critical. It can record everything happening in the system, including successful operations, error operations, warning messages, and more. Therefore, logging is one of the basic requirements in observability design. By recording events and error messages to log files or databases, troubleshooting and problem diagnosis can be easily done. However, logging alone is not enough. Effective management and analysis of logs are also required. If there are too many logs, it becomes burdensome because they occupy storage space and it takes a long time to find useful information. Therefore, logs need to be filtered and archived to manage them better.

Monitoring Dashboards

To better understand the operation status of the system, monitoring metrics and tracing information need to be visualized. Visualization can be achieved through charts, dashboards, and other means. Visualization can help us better understand the operation status and performance of the system. Through visualization, we can quickly identify problems in the system and take appropriate measures to solve them. Visualization can be implemented using various tools such as Grafana, Kibana, etc.

Event Alarms

The system needs to monitor security events and behaviors, such as unauthorized access or malicious attacks. Security monitoring can be achieved through log recording and real-time alarms. Security log recording can help organizations understand security events and behaviors in the system. By analyzing security logs, security vulnerabilities and attack behaviors can be identified, and corresponding measures can be taken to protect system security. Real-time alarms can notify relevant personnel of potential security threats, allowing quick actions to be taken.

In summary, the requirements for observability design are to improve the reliability, stability, and performance of the system. By implementing the above functions, the operational status of the system can be effectively monitored and managed. Observability has become an essential design requirement, and every software system needs to consider observability design.