You can use the log management feature of Simple Message Queue (formerly MNS) to push operation logs of messages to Simple Log Service (SLS). Then, you can troubleshoot online issues by using SLS to query and analyze operation logs.
Scenarios
The log management feature can be applied in the following scenarios:
Track the messages sent to a message queue if consumer clients cannot receive the messages.
Determine the consumer clients that consume a message and the number of times that the message is consumed.
Determine when a message can be consumed again if it fails to be consumed due to consumer client breakdown.
Identify the cause if endpoints cannot receive messages that are published to a topic.
You can use the log management feature of SMQ to achieve these purposes by using the following method:
Push logs to Simple Log Service and view the complete message traces in the Simple Log Service console.
Specify the parameters to view operation logs of messages by using the log query tool provided by Alibaba Cloud.
Billing
You are not charged for using the log management feature of SMQ.
SMQ pushes logs to Simple Log Service. You are charged for using Simple Log Service based on the used storage space, traffic, and number of requests. For more information, see Billing overview.
Log format of queue messages
Operation logs of queue messages are generated when operations are performed on queue messages. For example, such logs are generated when you send, consume, and delete messages. Each operation log contains multiple fields, each of which has a specific meaning. The fields contained in an operation log vary based on different operations. The following tables describe the fields in operation logs of queue messages and the presence of each field in the logs for specific operation types.
Log fields
Each operation log contains multiple fields. The following table describes the fields.
Field
Description
Time
The time when the operation is performed.
MessageId
The ID of the message involved in the operation.
QueueName
The name of the queue involved in the operation.
AccountId
The Alibaba Cloud account or Resource Access Management (RAM) user to which the queue belongs.
RemoteAddress
The IP address of the client from which the operation is initiated.
NextVisibleTime
The time when the message is visible after the operation is complete.
ReceiptHandleInRequest
The value of the ReceiptHandle parameter that is passed into the operation when the client performs the operation.
ReceiptHandleInResponse
The value of the ReceiptHandle parameter that is returned from the client after the operation is complete.
ProcessTime
The duration in which the operation is performed.
RequestId
The ID of the operation.
Action
The specific action of the operation, such as delete or send.
Fields of different operations
Logs that are generated by different operations contain different fields. The following table describes the log fields of different operations.
Operation
Time
QueueName
AccountId
MessageId
RemoteAddress
NextVisibleTime
ReceiptHandleInResponse
ReceiptHandleInRequest
SendMessage or BatchSendMessage
Yes
Yes
Yes
Yes
Yes
Yes
No
No
PeekMessage or BatchPeekMessage
Yes
Yes
Yes
Yes
Yes
No
No
No
ReceiveMessage or BatchReceiveMessage
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
ChangeMessageVisibility
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
DeleteMessage or BatchDeleteMessage
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Log format of topic messages
Operation logs of topic messages are generated when operations are performed on topic messages. For example, such logs are generated when you publish and push messages. The following tables describe the fields in operation logs of topic messages and the presence of each field in the logs for specific operation types.
Log fields
Each operation log contains multiple fields. The following table describes the fields.
Field
Description
Time
The time when the operation is performed.
MessageId
The ID of the message involved in the operation.
TopicName
The name of the topic involved in the operation.
SubscriptionName
The name of the subscription to the message.
AccountId
The Alibaba Cloud account or RAM user to which the topic belongs.
RemoteAddress
The IP address of the client from which the operation is initiated.
NotifyStatus
The status code or error message returned when SMQ pushes a message to a client.
ProcessTime
The duration in which the operation is performed.
MessageTag
The tags of the message.
RequestId
The ID of the operation.
Action
The specific action of the operation, such as delete or send.
Fields of different operations
Logs that are generated by different operations contain different fields. The following table describes the log fields of different operations.
Operation
Time
MessageId
TopicName
SubscriptionName
AccountId
RemoteAddress
NotifyStatus
SubscriptionName
PublishMessage
Yes
Yes
Yes
No
Yes
Yes
No
No
Notify
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
NotifyStatus
The NotifyStatus field is specific to the operation logs that are generated when you push messages. It can help you determine why SMQ fails to push messages to endpoints. The following table describes the status codes and the corresponding solutions.
Status code
Description
Recommended solution
2xx
The message is pushed.
None.
Other HTTP status code
After the message is pushed to the endpoint, an HTTP status code other than 2xx is returned.
Check the processing logic of the endpoint.
InvalidHost
The endpoint specified in the subscription is invalid.
Use the curl or telnet command to check whether the endpoint specified in the subscription is valid.
ConnectTimeout
The connection between SMQ and the endpoint specified in the subscription timed out.
Use the curl or telnet command to check whether the endpoint specified in the subscription is accessible.
ConnectFailure
SMQ failed to connect to the endpoint specified in the subscription.
Use the curl or telnet command to check whether the endpoint specified in the subscription is accessible.
UnknownError
An unknown error occurred.
Contact SMQ technical support.
Prerequisites
Before you use the log management feature, make sure that the following requirements are met:
A SMQ queue and an SMQ topic are created. For more information, see the "Create a queue" section of the Manage queues in the console topic and the "Create a topic" section of the Manage topics in the console topic.
A Simple Log Service project and a Logstore are created. For more information, see the "Create a project" section of the Manage a project topic and the "Create a Logstore" section of the Manage a Logstore topic.
You can push operation logs of queue messages and topic messages to a project that resides in the same region as the queues and topics.
SMQ is authorized to assume the AliyunMNSLoggingRole role to push logs to Simple Log Service.
You can go to the Cloud Resource Access Authorization page and complete the authorization as prompted.
WarningTo ensure that SMQ operation logs can be pushed to Simple Log Service, do not revoke permissions from the RAM role or delete the RAM role.
Enable logging
Enable logging for a queue
Enable logging for a topic
Push logs to Simple Log Service
View logs
Log on to the Simple Log Service console.
In the Projects section, click the project that you want to manage.
In the left-side navigation pane, click Log Storage. In the Logstores list, click the Logstore that you want to manage.
Enter a search statement and specify Last 15 Minutes as the time range to query. Then, click Search & Analyze. For more information, see Query and analyze logs.
Examples of log analysis
View the trace of a queue message
This example describes the trace of a queue message that is queried by specifying the queue name and message ID in the $QueueName and $MessageId format, such as log and EED287A265726135146E6A9CADC8XXXX.
The following query results record the process from the time when the message is sent to the time when the message is received.
View the trace of a topic message
This example describes the trace of a topic message that is queried by specifying the topic name and message ID in the $TopicName and $MessageId format, such as logtest and 8798453B65727FC6433E6AB4F746XXXX.
The following query results record the process from the time when the message is sent to the time when the message is received.
View statistics about sent queue messages
This example describes the statistics about sent queue messages that are queried by specifying the queue name and the name of the send operation in the $QueueName and (SendMessage or BatchSendMessage) format, such as log and (SendMessage or BatchSendMessage).
The following figure shows the query results. Three queue messages are sent by the message producer to a queue named log in the specified time range.
View statistics about sent topic messages
This example describes the statistics about sent topic messages that are queried by specifying the topic name and the name of the publish operation in the $TopicName and PublishMessage format, such as logtest and PublishMessage.
The following figure shows the query results. Three topic messages are sent by the message producer to a topic named logtest in the specified time range.
View statistics about consumed queue messages
This example describes the statistics about consumed queue messages that are queried by specifying the queue name and the name of the consume operation in the $QueueName and (ReceiveMessage or BatchReceiveMessage) format, such as log and (ReceiveMessage or BatchReceiveMessage).
The following figure shows the query results. Twelve messages in a queue named log are consumed in the specified time range.
View statistics about deleted queue messages
This example describes the statistics about deleted queue messages that are queried by specifying the queue name and the name of the delete operation in the $QueueName and (DeleteMessage or BatchDeleteMessage) format, such as log and (DeleteMessage or BatchDeleteMessage).
The following figure shows the query results. Sixty-one queue messages are deleted from a queue named log in the specified time range.
View statistics about messages that are processed by a client
This example describes the statistics about messages processed by a client that are queried by specifying the client IP address in the $ClientIP format, such as 10.10.10.0.
If you want to query a specific type of operation log on a client, you can use multiple keywords, such as $ClientIP and (SendMessage or BatchSendMessage).
The following figure shows the query results. The client processed 66 messages in the specified time range.