This tripped me up hardcore at work recently, because I was trying to SSH into a VM using a non-standard port, but the port-forwarding went from the non-standard port into the usual 22 on the VM. So when I got "Connection Refused" on port 22, I thought that meant the connection was getting through the NAT layer but then failing at the VM for some reason.
NOPE I'd just specified the option wrong!