In Response to "I expose all my services to open web"

That post is here

Summary of that post is that OP is using mTLS on the open internet to host his services, rather than a VPN.

My creds: I am a security engineer with specialization in offensive embedded systems security research.

mTLS, or “client certificate authentication”, on a web server is equally as secure as running a VPN. In fact, OpenVPN can be configured to use mTLS just like a web server can. There was a lot of misinformation in that thread and I’d like to address it here:

1: If you use TailScale, it is only an outbound connection from your home so no ports are exposed.

This is a half-truth. With TailScale, TailScale itself exposes ports. You authenticate and connect to those ports, which then connect you back to the reverse connection from your home. Ports are exposed at TailScale. If your security requirements and threat model allow for using TailScale then it’s totally fine to use it, but the idea that TailScale doesn’t expose ports is a half-truth.

2: If you use a reverse proxy the way OP does, attackers will be able to scan your web server, identify web server vulnerabilities, and pop into your network!

No. mTLS requires the attacker to have a valid private key to authenticate to the reverse proxy. If a valid private key and certificate are not there, then the attacker cannot begin scanning the web app. The mTLS handshake happens before the attacker can probe the web service. If you don’t believe me, use WireShark and see how a TLS connection works. Even over regular TLS, you will see that the TLS connection happens first, before any HTTP traffic is transmitted. Better yet, host your own mTLS instance, scan 443 without a private key and see what data you get back.

3: If you expose a port, even if it requires a private key to connect to it, you are less secure than if you use WireGuard, which requires an authenticated packet before it responds.

No. WireGuard allows you to avoid confirming or denying that a port is open, since it’s over UDP and most systems don’t respond if you try to interact to a nonexistent service over UDP. This, on its own, does not make WireGuard more secure than say TCP OpenVPN or mTLS. It does, however, prevent people looking at your IP address from knowing if you are running some sort of authentication-required service. If this increases your risk, then you can choose to use WireGuard, instead, but this is not the case for a vast majority of people.

For more information on mTLS, see Hello mTLS by the awesome people at Smallstep. They also have a cool tutorial on using Yubikeys with mTLS here to connect back to the homelab, similar to how OP is running his homelab.

The great part about using Yubikeys for mTLS is it allows you to have a hardware-backed, two-factor authentication method at layer 6, rather than traditional MFA which is at layer 7. This allows MFA with a lower attack surface, since the attacker can’t look for any web vulnerabilities to bypass MFA.

I was feeling like I did some unholy thing creating that post. My mistake was I did not mention mTLS in my post. Thank you for all the clarifications.

Edit: Another mistake I did was mentioning that I expose my all my services to open web when I only expose the reverse proxy.

Thanks to u/scrug for pointing that out.

I love this subreddit

I’m glad you’re making these points, yeah there’s fundamentally no real difference in cryptographic security between logging on with a TLS cert to a VPN and logging on with a TLS cert to an application - although there’s one caveat: if you’re hosting multiple services, you are somewhat enlarging your attack surface by exposing >1 application, while a VPN is only one.

But as also mentioned in the other topic, carefully designed firewall rules keep virtually all random attackers from even reaching the application and attempt a login in the first place. That also in principle allows you to finetune access per-app, while a VPN entry would be one-fits-all.

The main drawback that I see of mTLS is support of apps (not web apps that you view with a browser, but e.g. actual mobile apps running on phones that don’t have it implemented).

I guess I’ll throw in my 2 cents as an embedded engineer who’s been responsible for implementing mTLS and peripherally involved in the security architecture. On a technical level, I agree with everything in your post. If anything, mTLS will give you even more flexibility to authenticate clients.

That said… in my homelab, I just use WireGuard. :smiley:

The main reasons are:

  • My main skillset is not sysadmin or devops. I don’t have enough confidence to setup and maintain a mTLS reverse proxy exposing my private data, especially when I’m also trying to learn and experiment. WireGuard is a single port / service, and I can easily hit the big red button if I want to cut off access. (I know, I know, a reverse proxy would just be one port as well…)

  • Extra complexity in wrapping other protocols. I’m not sure how I’d do something like exposing Plex via mTLS. I have some ideas, but I’d need to do a lot more research to know that they’re actually the correct answer.

  • Access from mobile. For apps that I access from my phone, provisioning a self-signed client cert seems really annoying. VPN split proxy is much easier.

But yeah, if set up properly, mTLS is perfectly secure. It secures a huge chunk of the Internet, after all. If anything, companies are moving away from VPN solutions in favor of zero trust.

I think this sub is wild, there is more thought put into security here than 90% of businesses. I think most people are fine exposing a reverse proxy and building up to 2FA, no attacker really cares about a jellyfin server. Seeing all these post about wireguard, VPN, key based authentication just scares away people that may take an interest in self hosting.

I’d argue that mTLS supports a zero trust foundation better than having a VPN into a system and full on reign after you get in.

And while I’ve seen it, ensured it was implemented for services at my place of employment, and even read on it, my pea brain didn’t think of using it in my homelab.

Thanks, I have something to obsess over this weekend

This is one of my favorite subs. Thanks to both you and u/a_sugarcane for a great discussion on this topic!

What’s this, a security engineer who threat models and takes a real risk based approach to determining a control’s suitability?
They really exist?

Not someone who looks at it and says “wireguard doesn’t even answer unauth’d packets, therefore it’s more secure as it mitigates the discoverability risk, you must implement wireguard over all other solutions”

Can you come work here?

I’d love to hear your take on Cloudflare Tunnels. I have a few services exposed via cloudflare tunnel but they’re behind their authentication service + geo ip locked to the country I’m in.

People’s opinion seems to differ wildly about that.

What do you think about exposing services like that:

  1. Cloudflare tunnel →
  2. Traefik →
  3. Authentik →
  4. Docker container of the service

Thank you for taking the time to write this up :heart:

So can you give me some advice if I just want to host a game server without making my network Public? I want to expose the panel for managing the server and the game servers access port itself (pterodactyl).

What really confused me is that the game adds itself to a server browser and I didn’t understand how you can hide the IP if the software itself is connecting to the server browser. I guess you’d need a VPN?

A lot of these things end up adding latency and Im just not sure what best practice would be. I typically hosted a lot of stuff on VPs cloud instances but moved to self hosting because I wanted to learn more.

I was setting up a reverse proxy with nginx recently but tailscale also seemed like a good option.

You guys are awesome. It’s impressive that you know these details to such a degree.

All of what you said went completely over my head, and I have no idea what any of it means. I am slightly concerned about the (probable lack of) security on my home lab, but there’s nothing I can do about it, as I don’t have the capacity to learn all what you just said. But I wanted to say that I appreciate your knowledge!

How does Tailscale work behind NAT? I haven’t port forwarded anything to my server and can still connect from the outside.

I’m not giving up Wireguard anytime soon… TailScale came in a close second, but I just didn’t want to be bothered with installing agents on everything in order to connect to them.

Interesting post. Personally, I think this topic is also overloaded with emotions.

I have some services running for 6+ years, just plain Docker, Traefik as a reverse proxy and opened ports, redirecrion from http to https, crowdsec as middleware.

Many people including myself also get this feeling in their guts if people say, option A is insecure, option B is the only one that works and think they are doing a big mistake. But from my experience, and I think you can also second this, the main security risk beside my server is still a non tech savy user who clicks on every shit he sees.

The only attacks I have seen were some random bot logins, for any real person, I am just too uninteresting to be targeted as a little fish. There are conpanies hosting stuff even more insecure and still survive. We should not cook this topic warmer than needed.

Hell, people buy devices that send their fingerprint over the internet to open their door and get heated when you dont use a VPN to access your network. Half of devices in the network are security risks anyway, every shitty smart TV, my photovoltaic power converter for 3k € that shows itself in the network with the hostname “espressif”. These monkeys didnt even care to change the one liner in some code they copied from the internet to change the device name from the 2 dollar microcontroller that pronably runs the whole firmware for this thing. We are not the government/worthy companies. We are not interesting enough for targeted attacking.

What does your client certificate setup look like? I have gotten the server side to run as i want, but i cannot find a mobile(android) browser that supports the save storage and access of the client certificates. (I.e. standard browser just prompts for a list of certificates to send to the server) ideally i would want somerhing that can select the right certificate for the website from a save (like biometrically locked) location.

Thanks for your post!