IIS Web Service is also one mainstream HTTP service software. Many Microsoft applications run on IIS and some users run their .Net Framework applications on the platform provided by IIS. Many users also configure a WAF before the IIS server to secure websites and prevent attacks. This time, one of our users encountered an interesting problem.
Many web applications (identified by SNI) are running on the IIS Web Service. A WAF is configured before the IIS server to implement back-to-origin. According to user feedback, an IIS web site does not implement back-to-origin by using the WAF.
The main symptom is that the browser keeps loading when accessing this website and then returns error codes like 502/504 and 499.
Because the IIS Web service is basically like a black box and almost no error logs and access logs are generated when the problem occurs, we can only turn to the "bold assumption and careful verification" approach to locate the problem step by step. Considering the whole system, problems can only exist in the following aspects.
We test the connectivity of all the related IP addresses and corresponding ports. telnet and tcpping show that the three-way handshake is normal.
We have basically ruled out server problems through direct access to the IIS server. The following is the access record of the host test.
curl -v https://hello.test.domain:8443/ -o /dev/null -s --trace-time --resolve hello.test.domain:8443:ip.add.rss. 227
Compared with other websites with the same configuration, this WAF does not have special configuration options applied. Even if the configuration is completely consistent with that of normal websites, the problem still exists. After we check the WAF log, the problem seemingly goes back to the IIS server, because the log clearly records the 120s latency.
Because a conflict exists between the WAF and the IIS server, we analyze the interaction between these two services by capturing some packets. Unfortunately, we cannot decrypt the results of the first packet capture and therefore cannot get a clear picture of the interaction between the WAF and the IIS server. So, we continued with the following attempts:
Detailed instructions on the preceding operations can be found on the Internet and therefore are not explained here. Finally, we capture the normal and abnormal HTTPS data flows. Decrypted data packets are as follows:
Normal: The client directly connects to IIS.
Abnormal: The client connects to IIS through WAF.
Obviously, the problem is that the server sends a HELLO REQUEST to the WAF in an SSL tunnel but the WAF does not respond, causing IIS to continuously wait in the RE-NEGOTIATE phase. Normally, the client can process this request and send CLIENT HELLO.
After reviewing the RFC specifications for TLS, we notice that the RFC specifications do not clearly define the behavior of a client receiving a HELLO REQUEST.
https://tools.ietf.org/html/rfc5246
In the IIS implementations on Windows Server 2008 R2, having IIS continuously wait for a response is not necessarily a proper behavior.
Now we have found the cause of this problem and know that the HELLO REQUEST is not required. We can see the IIS help documents and locate the corresponding Ignore configuration to prevent IIS from sending a HELLO REQUEST.
Windows Networking Troubleshooting 4: Slow Response of .NET Framework Applications
Windows Networking Troubleshooting 6: The Mystery of Disappearing IP Addresses
Tim Chen - May 22, 2019
Tim Chen - May 22, 2019
Alibaba Clouder - December 18, 2017
Alibaba Cloud Community - December 8, 2021
Alibaba Cloud Community - January 10, 2024
Alibaba Cloud Community - January 15, 2024
A cloud firewall service utilizing big data capabilities to protect against web-based attacks
Learn MoreWeb App Service allows you to deploy, scale, adjust, and monitor applications in an easy, efficient, secure, and flexible manner.
Learn MoreAlibaba Cloud offers an accelerated global networking solution that makes distance learning just the same as in-class teaching.
Learn MoreConnect your business globally with our stable network anytime anywhere.
Learn MoreMore Posts by Tim Chen