I’m the only network engineer at this company and so I have no one else to bounce ideas off of, so I’m coming to you, r/networking. The company I work for is in Germany, we have 2 Palo Alto Firewalls and we use GlobalProtect as our VPN with a gateway on both PAs (one is physical on site and the other is a VM in a cloud). The company hires a few people living in other countries and they just work remotely. I’ve never heard any issues with this until this week. Someone working from Uzbekistan cannot reach any of our internal sites.
After a lot of investigating, we found that the connection through the VPN is so slow that the DNS requests come to late or not at all and after a failed request from our DNS his PC sends a request to his local modem, which obviously only resolves external sites. I checked on the PA for his public IP that his connecting with for GlobalProtect and checked the security logs from that IP and more that half of the connections are being dropped because of no answer. So I’m now thinking that his internet or the connection from his country to us is just not good enough, but he can quickly resolve and load any other German/European websites.
Is there anything I can even do here? It’s odd to me that he can easily reach other pages in other countries but only our GlobalProtect connection is bad. It’s also good to note that the 2 PAs don’t use the same internet, because one is in the cloud it just uses whatever that cloud provider has. Otherwise I would have contacted our internet provider to see if they had anything going on that could be causing this.
Any ideas are greatly appreciated. I’m hitting a wall here and you are my last hope for fixing this.
Some simple troubleshooting that might help. Try using a MTR utility to ping the path and collect loss information across all hops.
Also, is it SSL or IPsec VPN? SSL VPN shares the same port with HTTPS and their ISP might be rate limiting or inspecting the traffic in some way that is breaking the connection. It would be worth changing to IPsec to see if that helps too.
Otherwise u/Reece_56 is right, performance issues escalate to his ISP. They may need to escalate further upstream so best to open a ticket asap.
Your DNS-names look different from outside your network as compared when you are inside the network. Your employee might not see certain domain-names before he connects. Or might might stop seeing them as soon as he connects. Or the ip-addresses might be different.
Recursive routes. It might be that as soon as the vpn-tunnel connects, new routes to your internal network get installed on your employee’s PC. These include a route to the vpn-tunnel endpoint. Now the PC tries to route tunnel-packets via the tunnel itself. That won’t end well.
Not sure it’s either one of these 2 issues. But it’s worth checking out. If you ever find out what went wrong, let us know. TIA.
Did you manage to find what was causing the issue? I have a similar problem. I am working from abroad for a couple of days (I am now in Germany) and I am connecting to our PaloAlto VPN, but this causes the speed of the internet to be decreased drastically and I cannot do my work productively. If I disconnect from the VPN, there is no such issue, but I need to be connected to the VPN to access the internal URLs. If I am working outside of the office back home, I can use the VPN without having to deal with such a low speed. I tried to change the gateway, but this did not change anything.
Get the user to do a tracert from the command line of his laptop to your vpn public address. Check for any very slow hops. Compare against tracert to those german websites that are ok.
Then get the user to tether thru his mobile phone internet and tracert again.
That won’t fix anything but might show up any very slow hops.
Are you sure that the ISP is not traffic shaping for certain types of traffic and lowering the QOS and increasing priority for other traffic on the ISP end (http, https and icmp traffic for higher priority)? I would have the end user in that country ping your server (public ip address via wan and not via the vpn connection) and see if the latency and performance issue is seen there.
You assume he has the same setup as every other employee. Did you check? End-users can often change the settings themselves. Without telling you. Or he forgot. When I connected to my company’s network over a vpn, I changed the setting myself, to do split-routing. I wasn’t supposed to do that, but I did. I could have easily messed up, and done something that would have broken my vpn.
You assume that: 1) nothing on the PC of the employee is different, and 2) nothing at the company’s end is different, and that 3) therefor the problem must be the network in the middle. Ask people here how often they were told “the network is broken”, while in fact the problem was with one of the two endpoints.
Is he the only one in Uzbekistan? Without being able trace, or establish where the high latency is coming from. Only other way is basically do a packet capture at the source and destination. And that way you can prove that your UDP 500 and IP sec traffic is getting dropped.
In that case you gotta gather some data from, your user in Uzbekistan and some of the others, and make a presentation for your boss, explaining the issue.
Do have some “plans” and some “costs” associated with those plans.
Bosses hate the simple reply “no”, since IT and Technology is like magic, and so everything is possible. Pull them into reality, but never JUST say “no it’s not possible”.
Ay, it’s the trouble of being a one-man department. You are also the department-manager, and have the responsible for justifying all spending, and in a lot of cases also negotiating with the company, what the budget is.
Keeping all these threads aligned and nice is mostly a matter of experience.
But hey, it’s hard to be jobless when working in IT!
