The Brief:
Trying to put in place a solution to allow more of our staff to work at home. Rather than simple mobile vpn, ‘The Bosses’ want permanent connections up that allow file share access/printing/backups. The workstations (that will be remote) are a mixture of Win10, MacOS, and Linux (mostly Ubuntu).
For the one off’s and actual mobile workers, I’ve got IkeV2 Chap mobile vpn setup and working very well for Win10 and Linux users (mac users, however, no, but that’s for another day) and the transfer speeds are in line with the remote workers access speeds (reasonable, with no complaints). The problem lies now with the site to site IPSEC configuration – those speeds are *miserable*.
A Bit of Detail:
On the office side, we’ve a repurposed Dell Poweredge r220 (xeon E3-1220 v3 3.1 GHz - 4C4T, 8GB RAM) running pfSense 2.4.5-r-p1 on 2, 200MB dedicated fiber lines.
On the clients side, I’ve purchased a couple SG-1100’s to test things out before I roll them to end users (with all sorts of different speeds and internet providers). I’m testing the client on a modest 10MB ADSL line. The ISP provider’s router is a ‘business’ type which at least allows me to either port forward or put the SG-1100 in the DMZ, allowing direct access to the internet (this is what I have currently configured). I have been using the second incoming fiber line to host the Site to Site VPN (as if I use the primary, the mobile VPN stops working, something else to ask about later?)
Here’s the current config I’m trying (and failing) with:
Office router details:
| Interfaces | WAN1/WAN2/LAN | |
|---|---|---|
| Gateways | WAN1 | WAN2 |
| Gateway Groups | WAN_LoadBalancer | |
| WAN_LinkFailover | ||
| Static Routes | None | |
Phase 1:
| Key Exchange | IKEv2 |
|---|---|
| IP | IPv4 |
| Interface | Secondary WAN interface |
| Remote Gateway | IP Address for my remote office (static) |
| Auth Method | Mutual PSK |
| My Identifier - IP Address | External IP address for 2nd WAN |
| Peer Identifier | Peer IP address |
| Preshared Key | Matches on both sides |
| Phase 1 Proposal | AES128-GCM / 128 / SHA256 / DH14 |
| NAT Traversal | Auto |
| Mobike | Enable |
| Enable Dead Peer Detection | Active |
| Enable Max MSS - Active | 1500 |
| Auto Exclude LAN Address | Active |
| Asynchronous Cryptography | Active |
Office Router Phase 2:
| Mode | Tunnel IP¨v4 |
|---|---|
| Local Network | LAN Subnet |
| NAT/BINAT Translation | None |
| Remote Network : Network | 10.0.20.0 / 24 |
| Phase 2 Proposal Protocol | ESP |
| Encryption Algorithms | AES128-GCM / 128 / No Hash / PFS key grp 14 |
Remote office setup on the SG1100 is a copy of the above Phase1/Phase2 with the exception of Remote Gateway (being set to the external IP for my secondary WAN link) and the Remote Network being set to Network:10.0.1.0/24.
I’ve tried (originally) the encryption settings in the pfSense Guide (https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-psk.html)
phase 1: AES / 256 / SHA256 / DH 2
phase 2 AES256-GCM / 128 / No Hash / PFS Off
The current settings I’ve tried after going over the article at https://medium.com/@dEad0r/measuring-performance-of-site-to-site-vpn-tunnels-between-pfsense-vms-b484ba425aff
I’ve tried MSS settings 1400 and deactivated as well as turning off Asynchronous Cryptography.
The Symptoms:
Accessing Web pages on remote (office site) servers (such as pfSense admin interface for the office installation of pfSense and other web administration portals) takes upwards of 30 seconds to load, with some just timing out.
Access File shares (smb) on remote files servers succeeds in listing the files/folders, but any attempt to open or copy results in a time out.
Iperf results:
Baseline from Remote office to ping.online.net (directly connected to internet):
------------------------------------------------------------
Client connecting to ping.online.net, TCP port 5001
TCP window size: 93.5 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.94 port 43356 connected with 62.210.18.40 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.9 sec 1280 KBytes 118 KBytes/sec
Baseline when remote office is connected to main office via Mobile VPN (IKEv2 Chap) and testing performance to file server:
iperf3 -c <file-srv-ip> -f K
Connecting to host <file-srv-ip>, port 5201
[ 4] local 10.0.3.1 port 59152 connected to <file-srv-ip> port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 255 KBytes/sec
[ 4] 1.01-2.00 sec 0.00 Bytes 0.00 KBytes/sec
[ 4] 2.00-3.00 sec 128 KBytes 128 KBytes/sec
[ 4] 3.00-4.01 sec 128 KBytes 128 KBytes/sec
[ 4] 4.01-5.00 sec 0.00 Bytes 0.00 KBytes/sec
[ 4] 5.00-6.00 sec 128 KBytes 128 KBytes/sec
[ 4] 6.00-7.00 sec 0.00 Bytes 0.00 KBytes/sec
[ 4] 7.00-8.02 sec 128 KBytes 126 KBytes/sec
[ 4] 8.02-9.01 sec 0.00 Bytes 0.00 KBytes/sec
[ 4] 9.01-10.00 sec 128 KBytes 129 KBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 896 KBytes 89.6 KBytes/sec sender
[ 4] 0.00-10.00 sec 693 KBytes 69.3 KBytes/sec receiver
iperf Done.
Results from remote office when connected to SG-1100 with Site-to-Site connection to Main Office Active and testing performance to the same file server:
iperf3 -c <file-srv-ip> -f K
Connecting to host <file-srv-ip>, port 5201
[ 5] local 10.0.20.54 port 32934 connected to <file-srv-ip> port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 164 KBytes 164 KBytes/sec 0 17.0 KBytes
[ 5] 1.00-2.00 sec 124 KBytes 124 KBytes/sec 0 22.6 KBytes
[ 5] 2.00-3.00 sec 62.2 KBytes 62.2 KBytes/sec 1 24.0 KBytes
[ 5] 3.00-4.00 sec 124 KBytes 124 KBytes/sec 4 22.6 KBytes
[ 5] 4.00-5.00 sec 62.2 KBytes 62.2 KBytes/sec 9 11.3 KBytes
[ 5] 5.00-6.00 sec 62.2 KBytes 62.2 KBytes/sec 0 15.6 KBytes
[ 5] 6.00-7.00 sec 124 KBytes 125 KBytes/sec 1 14.1 KBytes
[ 5] 7.00-8.00 sec 62.2 KBytes 62.2 KBytes/sec 0 17.0 KBytes
[ 5] 8.00-9.00 sec 62.2 KBytes 62.2 KBytes/sec 3 15.6 KBytes
[ 5] 9.00-10.00 sec 62.2 KBytes 62.2 KBytes/sec 3 15.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 911 KBytes 91.1 KBytes/sec 21 sender
[ 5] 0.00-10.25 sec 829 KBytes 80.8 KBytes/sec receiver
iperf Done.
I’ve posted the results from when I am connected via the Mobile VPN (where the file transfer speed and access to web portals is acceptable for my overall connection speed) to compare the iperf result to those where the speed is hugely diminished (when connected to the Site-to-Site) The iperf results are very similar.
Now, knowing, and reading that SMB suffers from high latency connections I’ve tried a scp of a small file (190kb) while connected via Mobile VPN and the site-to-site with those results below:
MobileVPN:
scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf
<USERNAME>@<file-srv>'s password:
<REMOTE-FILE-NAME> 100% 190KB 277.5KB/s 00:00
Site-to-Site
scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf
<USERNAME>@<file-srv>'s password:
<REMOTE-FILE-NAME> 100% 190KB 5.0KB/s 00:38
Obviously a huge, and unacceptable difference, and this is using scp. I have not yet tried wget, but I expect similar, disappointing results.
In Conclusion :
I’m definitely using this experience to build my experience around this topic and have done countless searches (probably using the wrong terms) and have seen other people with similar issues and have seen the conclusion that IPSEC site to site just does not work with SMB, with no real constructive takeaways (and I’ve shown above, it’s affecting more than just SMB, so I’m thinking I’m missing something) so I’m turning to you, blessed redditors to help me grok where I am going wrong (and hopefully before I’ve torn out *all* my hair and resign myself to self doubt!).
Thanks for any feedback/advice you may have, even if it is pointing to an article I may have missed or some settings I may or may have not yet tried!!
EDIT: Cross-posting over to the Netgate forums as well
