What do I do if the error message "Because of a protocol error, this session will be disconnected. Please try connecting to the remote computer again" appears when I connect to a Windows instance by using Remote Desktop? - Elastic Compute Service

This topic describes how to resolve the following issue: The error message "Because of a protocol error, this session will be disconnected. Please try connecting to the remote computer again" appears when you connect to a Windows Elastic Compute Service (ECS) instance by using Remote Desktop.

Problem description

When you connect to a Windows instance, you are prompted with a protocol error and receives an error message.

Causes

This issue may occur because of various reasons, including the following ones:

The Remote Desktop session settings on the instance are incorrect. In this case, you are prompted with a protocol error when you connect to the instance. To resolve the issue, perform the operations that are described in the Solution 1: Modify the Remote Desktop session settings on the instance section.
The port that is used for Remote Desktop connections is occupied by another application. By default, port 3389 is used for Remote Desktop connections in Windows. If the port is occupied by another application, the Remote Desktop service cannot run as expected. To resolve the issue, perform the operations that are described in the Solution 2: Change the port number used for Remote Desktop connections section or the Solution 3: Terminate the process that is occupying the port used for Remote Desktop connections section.

Solutions

Use one of the following solutions to resolve the issue. In the following examples, an instance that runs Windows Server 2016 is used.

Solution 1: Modify the Remote Desktop session settings on the instance

Connect to the Windows instance by using Virtual Network Computing (VNC).
For more information, see Connect to a Windows instance by using a password.
Right-click the icon and select Run. In the Run dialog box, enter gpedit.msc to start Local Group Policy Editor.
In the Local Group Policy Editor window, choose Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Security.
In the Security directory, double-click Require use of specific security layer for remote (RDP) connections.
In the dialog box that appears, select Enabled, set Security Layer to RDP, and then click OK.
Run the following command to update Group Policy settings and apply the preceding configurations.
1. Right-click the icon and select Run. In the Run dialog box, enter cmd to start Command Prompt.
2. Run the gpupdate command to update Group Policy settings.
Connect to the instance again and make sure that the issue is resolved.

Solution 2: Change the port number used for Remote Desktop connections

Connect to the Windows instance by using VNC.
For more information, see Connect to a Windows instance by using a password.
Right-click the icon and select Run. In the Run dialog box, enter regedit to start Registry Editor.

In the Registry Editor window, navigate to the following path:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp

In the RDP-Tcp directory, double-click the PortNumber registry key.
In the dialog box that appears, select Decimal in the Base section and modify the value in the Value data field. This field specifies the number of port that is used by Remote Desktop.
Note
Port 3389 shown in the preceding figure is the default port and is provided only for reference. Change the value to the number of port that you want to use for Remote Desktop.
Use one of the following methods to restart the instance for the configurations to take effect.
- Method 1: Restart the Windows instance in the ECS console. For more information, see Restart instances.
- Method 2: On the instance, choose > > Restart.

Solution 3: Terminate the process that is occupying the port used for Remote Desktop connections

Connect to the Windows instance by using VNC.
For more information, see Connect to a Windows instance by using a password.
Right-click the icon and select Run. In the Run dialog box, enter cmd to start Command Prompt.
Run the following command to view the ID of the process that is occupying the port used for Remote Desktop connections:
```
netstat -ano |findstr 3389
```
Note
In this example, the default port 3389 is used for Remote Desktop connections. Specify the actual number of the port that you use for Remote Desktop connections.
Run the following command with the obtained process ID (PID) to view the process that is occupying the port used for Remote Desktop connections:
In normal cases, the port used for Remote Desktop connections is occupied by the svchost.exe process. If the port is occupied by a process that is not svchost.exe, the port cannot be used for Remote Desktop connections as expected.
```
tasklist |findstr [$PID]
```
Note
Set to [$PID] to the PID that you obtained in preceding step.
Run the following command to terminate the process and release the port.
Important
This step is a high-risk operation. Before you perform this step, make sure that terminating the process does not affect your business on the instance.
```
taskkill /f /pid [$PID]
```
Connect to the instance again and make sure that the issue is resolved.