If for some reason you have real trouble connecting to your server via RDP, you could be being attacked by a bot, constantly trying to break into your server and preventing you from connecting.

Take a look at your security log in Event Viewer to see what’s going on. If you see lots of audits saying Logon Failure then that’s probably what’s happening. Change your port from 3389 first, then if it still happens use Hamachi to create a VPN.  It’s the easiest way to hide your remote server from hackers.