I definitely noticed the performance boost on my Pixel 8, for some reason it seems to really not like wireguard-go, it struggled to pull even 100mbps, maybe something unoptimized on Google's custom hardware. With the new GotaTun version I can pull 500mbps+, though unfortunately it also seems to have introduced a bug that randomly prevents the phone from entering a deep sleep state, so occasionally my battery will randomly start draining at 10x normal speed if I have it enabled until I reboot.
I'm surprised by this comment. I have wireguard on 24/7 on my shitty Samsung A5 and it lasts forever. By comparison the Pixel 8 is a beast. Sounds like an Android bug more than wireguard.
Pixel 6 here. Vanilla wireguard app. It sucks the life out of my phone and nearly halves the already half-life battery (thanks Google for your crappy OEM producers!)
Thank samsung for their shitty modems in the pixels.
However, there’s going to be a large discrepancy for all devices on battery usage based on whether VPN is on wifi or cellular, and additionally when on cellular how close to the tower they are. I live near cell edge and VPN’s roast my batts on cellular no matter the make, in city it’s almost not noticeable to have VPN on. Better to use wifi when far from towers, cellular more efficient if it’s strong signal.
AFAIK the C implementation is a kernel module that's not shipped in stock Android releases. The WireGuard Android app uses that module when available, but otherwise uses wireguard-go.
Good knowledge here, was unaware of this feature of the app. Would there be any case of the app defaulting to the wireguard kernel module if it's not included by any OEM Android release? I would assume that means most users are actually running wireguard-go.
Android kernels 4.19 and higher should all have support included for WireGuard unless the OEM specifically disables it: [0]. The Pixel 8 ships with the android 14 6.1 kernel so it most definitely should have WireGuard kernel support. You can check this in the WireGuard app BTW, if you go to settings it will show the backend that's in use.
Kernel support should have no bearing as the apps are purely userspace apps. You can use the kernel mode if you root the phone, but that's not a typical scenario.
Well, the issue isn't kernel vs user space, but you are correct that you still need a custom ROM and/or root unfortunately. I had assumed Android had also allowed netlink sockets for WireGuard but alas they did not. So the app can't communicate with the kernel module, bummer.
It's very likely that VPNs like this are not CPU-bound, even on somewhat whimpy CPUs. I'd wager even some microcontrollers could sling 500megabits/sec around without trouble.
That CPU is pretty much a toy compared to (say) a brand-new M5 or EPYC chip, but it similarly eclipses almost any MCU you can buy.
Even with fast AES acceleration on the CPU/MCU — which I think some Cortex MCUs have — you’re really going to struggles to get much over 100Mbits of encrypted traffic handling, and that’s before the I/O handling interrupts take over the whole chip to shuttle packets on and off the wire.
Modern crypto is cheap for what you get, but it’s still a lot of extra math in the mix when you’re trying to pump bytes in and out of a constrained device.
You're looking at the wrong thing, WireGuard doesn't use AES, it uses ChaCha20. AES is really, really painful to implement securely in software only, and the result performs poorly. But ChaCha only uses addition rotation and XOR with 32 bit numbers and that makes it pretty performant even on fairly computationally limited devices.
For reference, I have an implementation of ChaCha20 running on the RP2350 at 100MBit/s on a single core at 150Mhz (910/64 = ~14.22 cycles per bytes). That's a lot for a cheap microcontroller costing around 1.5 bucks total. And that's not even taking into account using the other core the RP2350 has, or overclocking (runs fine at 300Mhz also at double the speed).
You’re totally right; I got myself spun around thinking AES instead of of ChaCha because the product I work on (ZeroTier) started with the initially and moved to AES later. I honestly just plain forgot that WireGuard hadn’t followed the same path.
An embarrassing slip, TBH. I’m gonna blame pre-holiday brain fog.
For most any 5G network you should be safe to 1420 - 80 = 1340 bytes if using IPv6 transport or 1420 - 60 = 1360 bytes if using IPv4 transport.
For testing I recommend starting from 1280 as a "does this even work" baseline and then tweaking from there. I.e. 1280 either as the "outside" MTU if you only care about IPv4 or as the "inside" MTU if you want IPv6 to work through the tunnel. This leverages that IPv6 demands a 1280 byte MTU to work.
Hah! I just ran into this recently and can confirm. The coax to my DOCSIS ISP was damaged during a storm, which was causing upstream channels to barely work at all. (Amusingly, downstream had no trouble.) While waiting for the cable person to come around later in the week, I hooked my home gateway device up to an old phone instead of the modem. I figured there would be consequences, but surprisingly, everything went pretty smoothly... But my Wireguard-encapsulated connections all hung during the TLS handshake! What gives?
The answer is MTU. The MTU on my network devices were all set to 1500, and my Wireguard devices 1420, as is customary. However, I found that 1340 ( - 80) was the maximum I could use safely.
Wait, though... Why in the heck did that only impact Wireguard? My guess is that TCP connections were discovering the correct MSS value automatically. Realistically that does make sense, but something bothers me:
1. How come my Wireguard packets seemed to get lost entirely? Shouldn't they get fragmented on one end and re-assembled on the other? UDP packets are IP packets, surely they should fragment just fine?
2. Even if they don't, if the Linux TCP stack is determining the appropriate MSS for a given connection then why doesn't that seem to work here? Shouldn't the underlying TCP connection be able to discover the safe MSS relatively easily?
I spelunked through Linux code for a while looking for answers but came up empty. Wonder if anyone here knows.
My best guess is that:
1. A stateless firewall/NAT somewhere didn't like the fragmented UDP packets because it couldn't determine the source/dest ports and just dropped them entirely
2. Maybe MSS discovery relies on ICMP packets that were not able to make it through? (edit: Yeah, on second thought, this makes sense: if the Wireguard UDP packets are not making it to their destination, then the underlying encapsulated packets won't make it out either, which means there won't be any ICMP response when the TCP stack sends a packet with Don't Fragment set.)
But I couldn't find anything to strongly support that.
Basically the only parts of the Internet which actually work reliably, around the globe, are the bits needed so that web pages basically kinda work. If you break literally everything else your service is crap, and some customers might notice, but many won't and also some won't have a choice so, sucks to be them. But if you break the Web, now everybody notices that you broke stuff and they're angry.
This is why DoH (DNS over HTTPS) is a thing. It obviously makes no actual sense to use the web protocol to move DNS packet, but, this works and most things don't work for everybody so eh, this is what we have. Smashing the Path MTU discovery doesn't break the web.
Breaking literally everything so long as the web pages work even means you can't upgrade parts of the web unless you get creative. TLS 1.3 the modern security protocol that is used for most of your web pages today, would not work for most people if it admitted that it's TLS 1.3, if you send packets with TLS version 1.3 on them people's "intelligent" "best in classs security" protective garbage (in the industry we call these "middle boxes") thinks it is being attacked by some unknown and unimaginable dastardly foe and kills the data. So TLS 1.3 really, I am not making this up, always pretends it is a TLS 1.2 re-connection, and despite the fact that no such connection ever existed these same "best in class security" technologies just have no idea what's happening and wave it through. It's very very stupid that they do that, but it was needed to make the web work, which matters, whereas actual security eh, suckers already bought the device, who cares.
This situation is deeply sad but, one piece of good news is that while "This Iranian woman can't even talk confidentially to her own mother without using code words because the people in charge there intercept her communications" won't attract as much sympathy as you'd like from some bearded white guy who has never left Ohio, the fact that those people broke his network protocol to do that interception infuriates him, and he's well up for ensuring they can't do that to the next version.
Your ultimate conclusion is correct, to my understanding. I know wireguard sought to be ultra minimal but I do wish they had included DPLPMTUD as something which is required to be supported (but not mandated to be used e.g. if the user wants to hard set it as they would currently) because it's one of those cases where "do it yourself separately the UNIX way™" or "have the tunneled things do it if they need it" instead are both significantly more complex and fragile.
On that note, from the TCP layer it should just look like an ICMP blackhole, which makes me wonder if enabling `net.ipv4.tcp_mtu_probing` will magically make TCP connections under Wireguard work even with the MTU set wrong. I'd try it, but unfortunately with a similar configuration I am unable to get the fragmentation behavior I was getting before; which makes me wonder if it was my UniFi Security Gateway that actually didn't like the fragmented packets.
Oh, this is the reason the Mullvad app on my Pixel 6a was suddenly able to connect in less than a second where before it would take 5-10 seconds, nice!
Nice, I love WireGuard. I ended up building WrapGuard [1] to run applications without root access to the host and choose Go to write it in. I don't really know Rust, but does it make more sense for firmware/networking type software? Is there even a difference?
Yes, lots of firmware runs on hardware where a GC doesn't make sense. Because of limited memory and performance constraints. Sometimes having predictable timings (i.e. not a GC with pauses) is nice. I believe compiler and library support is also just better for many embedded platforms in rust.
> networking type software
Rust is a much more aggressively optimizing compiler, and thus will typically be faster, in the places where that matters. GC pauses might also be a point against golang in some places here. Rust's idioms provide slightly less opportunity for bugs in places where reliability matters (e.g. having a type system that requires you check for errors instead of just patterns that encourage it).
So there's a difference, but generally go is a good enough language for networking software and it would be rare that I wouldn't suggest that "use what you know" is more important than the differences between the languages for non-firmware network software.
I've implemented a few protocols in rust (and plenty in go and other languages).
One thing others haven't mentioned that I like rust for in this space:
The typestate pattern makes it really nice to work with protocols that have state. You encode your state machine logic into types, and your transitions into methods with move semantics, and you have a nice way to make sure your higher level code is using your protocol library correctly.
Another nice thing is that you can keep the number of copies and allocations way down if you're careful about how you use your buffers.
Very cool. I may use this, but also curious what the best choice would be if you don't need encryption. I'm specifically wanting to enable some local container networking using apple's new container tool [1]. I know I could just use Docker...
Correct me if I’m wrong, but if you use LD_PRELOAD, presumably it will not work for applications that circumvent libc, such as Go binaries (at least those with CGo disabled)?
Tor does this the right way on Linux. You make a separate user namespace with access only to the WireGuard network adapter and run the program inside of that. You want the kernel involved if you want any sort of guarantee:
How does this work in something like Kubernetes where you have a sidebar container configuring the network for the main container without affecting others on the same host?
I think all containers share the same netns in a pod. You restrict the pod to only the Wireguard peer IP, and have a (NET_ADMIN) sidecar container create an interface (tun/kernel wg) and update the routing tables for the netns. Then I believe the traffic from the other containers in the pod is tunneled.
Can you use user namespaces to create a network namespace with the VPN active and stick applications in that namespace?
From a quick search, https://blog.thea.codes/nordvpn-wireguard-namespaces/ sees to have at least the bones of a decent solution, though I've not had a chance to dig very far. A lot of results use root to set up the namespace, but I was pretty sure that shouldn't be needed with a new kernel and user namespaces enabled
I have no idea. I’ve never messed with it, but maybe something like eBPF to intercept network syscalls? Not sure if that’s a thing—especially without root access? Mostly I was just thinking the project page could use a disclaimer since, in Go, it is common to bypass libc. :shrug:
This seems like a very cool, useful project though!
One usecase I've always wanted is being able to combine multiple tunnels into one shared connection, for instance airVPN allows 5 simultaneous users per sub, it would be awesome if I could run 5x connections and combine their traffic, but I dunno how I would do this with wg / nmcli
VPNs are level 3 while interface bonding is level 2. You’d have to create a vxlan over wireguard. It sounds like a nightmare but it would be interesting to implement.
I believe you are making use of gVisor’s userspace TCP implementation. I’m not sure if there is something similar in Rust that would be so easy to set up like this.
GotaTun is specific to Mullvad and the features they usually add make sense for a public VPN provider. Unlikely projects such as Tailscale adopt it.
Besides, engineers at Tailscale, I don't think, strike me as startled by any hurdle too tall to debug, improve Go-based libraries. In fact, they pushed wireguard-go past 10gbps on Linux-based platforms back in April 2023! https://tailscale.com/blog/more-throughput
That's more of a job for an encapsulating protocol. (shadowsocks or similar) Wireguard isn't designed to be obfuscating alone. It's just a simple l3 udp tunnel with a minimal attack surface.
That's the traditional answer parroted in the Wireguard documentation but a few hours' serious thought and design is enough to reveal the fatal flaw: any encapsulating protocol will have to reinvent and duplicatively implement all of the routing logic. Perr-based routing is at least 50% of wireguard's value proposition. Having to reimplement it at the higher level defeats the purpose. No, obfuscation _has_ to be part of the same protocol as routing.
(Btw, same sort of thing occurs with zfs combining raid and filesystem to close the parity raid write hole. Often strictly layered systems with separation of concerns are less than the sum of their parts.)
In this case with the, I believe it’s called quantum tunneling by mullvad, it’s actually a good thing. Because the encapsulation protocol is just UDP/IP, a well established existing protocol that can masquerade as any kind of internet traffic easily.
amnezia-wg is quite cool and they have built the kmod too, I did some test so far they can works even in my location which block wireguard server quickly.
The mullvad apps do offer obfuscation options (shadowsocks, etc) but i agree it would be nice if something was baked into wireguard itself. I recently went through setting up shadowsocks over wg for my homelab and it was a good bit of effort
WireGuard is a protocol that, like all protocols, makes necessary trade-offs. This page summarizes known limitations due to these trade-offs.
Deep Packet Inspection
WireGuard does not focus on obfuscation. Obfuscation, rather, should happen at a layer above WireGuard, with WireGuard focused on providing solid crypto with a simple implementation. It is quite possible to plug in various forms of obfuscation, however.
If anyone working on the implementation is here, was it not possible to upstream your changes to BoringTun? The blog mentions some changes but doesn't go into detail on that aspect.
I'm guessing because BoringTun has been in a state of "currently undergoing a restructuring" for something like 3 years by now, I'm guessing Mullvad wasn't too keen to maybe/maybe not be able to contribute, and much more prefer being in 100% control of their own implementation.
As someone who wants to see Wireguard succeed and in even wider use, this move makes sense from that perspective too. The more implementations we have available, the more we can trust that the protocol is secure and stable enough. Personally I also have about 100x more trust in Mullvad than Cloudflare both in terms of security but more importantly privacy, but that's just the cherry on top.
BoringTun is unmaintained. There are various forks being developed.
I work at Obscura VPN and faced with boringtun bugs a few years ago we evaluated a few of the forks and switched our client to be based on top of NepTUN (https://github.com/NordSecurity/NepTUN).
I am curious why Mullvad started their own fork rather than building on top of one of the existing ones. It would be nice if there could be reconsolidation somewhere.
Its funny, this is another of the billions of reasons why Mullvad should be the VPN of choice. But so many fucking people can't ever get over that their favorite social media influencer/Youtuber is offering a code for 200% off of NordShark VPN, now with extra AI.
Mullvad is great for privacy. But it's blocked by pretty much every VPN block list. NordVPN at the very least bypasses all the ones I regularly encounter.
I do use Mullvad for most web browsing though. But Imgur for example is blocked on it, and it's blocked in the UK, so I need NordVPN if I want to see any images there.
Most people's VPN usage is literally just geolocation restrictions and Nord is really good at that.
System wide proxy configuration doesn’t actually always work system wide.
A VPN tends to have more success in encapsulating all application traffic (or all desired application traffic, if you’re so inclined to configure your system)
I love and use mullvad myself but I don't think they are very competitive for the average person. They mostly just care about getting around geo blocks on websites and streaming services, which mullvad puts 0 effort into facilitating.
Currently using airVPN, but ye gods, their eddie client is atrocious on linux. I wind up using wg / nmcli, but then have to block traffic going outside of the vpn with iptable rules because it leaks for some reason.
I miss mullvad dearly, and I might try proton after my 3y sub is up.
Not only Eddie, their account control panels and site in general look like something from the 90s, and it seriously hampers their business. I can't recommend them to anyone that isn't highly technical. And even then, as a technical user, why do I manually have to select one of 10-20 servers within a city or region, why am I being asked to manually load balance? Why is there no Wireguard over port 53 or 443?
It makes more sense when you know they're privacy activists first, businessmen second. But Mullvad shows you can be pro privacy and still offer great UX and a sleek site and client.
Btw, if you're managing things in CLI, you could take a look at their Hummingbird Suite. AFAIK it has a killswitch.
What sucks with Proton is that you can't share the VPN account with friends, because it is tied to your Proton account. They should create a vpn.proton.me subdomain that you can create a special managed account on that can only touch the VPN settings.
They're planning to introduce OpenVPN Data Channel Offload (DCO) support to more servers once Linux 6.18 starts becoming more mainstream.
With DCO, OpenVPN can perform almost as well as Wireguard, sometimes even better. Although with more performance overhead so not the best choice for laptops and phones.
Tangentially related but I kind of wish Wireguard looked more toward the future and had included AES as alternative to ChaCha20. At the time of development, many ARM devices didn't yet have AES acceleration which is why ChaCha20 was needed for wide hardware support, but they do since ARMv8 which became widespread in 2015. Intel and AMD have had AES acceleration for a long time. And then ChaCha20 would have been the fallback on MIPS and RISC-V.
I would just pirate at that point. You're paying for the streaming service anyways. Use mullvad to download the torrent :). I'm pretty sure they ignore dmca requests. Not that they even know their customer's names if you pay with Mullvad amazon card.
Been buying mullvad for the last 4-5 years but oftentimes I can’t even browse the fucking New York Times website due to low bandwidth, let alone stream anything. At this point, I just keep adding time to my account just in case, without using it.
Mullvad seems to care and be competent about privacy, but most average VPN users aren’t seeking the most extreme privacy. They just want something cheap that lets them do geolocation things or access the most websites.
The average VPN user is knowledge-less. At best their internet usage data is being sold to third party analytics companies. At worst third parties are routing their own bots through their local connection.
I think the general consensus is that it improves security of the protocol, but obviously that won't matter much if the implementation gets something wrong or has worse security by itself.
Issues in the protocol itself would need all implementations to change, but issues in the implementation would obviously be isolated to one implementation. For something like Wireguard, I'd wager a guess that issues in the implementations are more common than issues in the protocol, at least at this stage.
If the implementation gets it wrong that can also be a sign of ambiguity in the protocol / standard and as such result in clarifications and an overall more well specified protocol
The increased attack surface mostly only affects that one particular implementation though. So, yes, twice as many implementations that may contain exploitable bugs, but each new implementation could only be used to exploit a fraction of the total user base
Competitions helps in multiple ways. It improve tooling, test suites, CVE response time, documentation and evolution of the protocol. There are some counter examples where compatibility suck, like DLNA but the problem often come from the spec.
That's really good because it means it will be able to have more exposure, more exposure means more improvement, more improvement eventually dig out bad bugs and reduces the attack surface in the long run
dont fix if it ain't broken. look at sudo-rs and other rust ports.
ofc, thats a cynical view.
i personally think its a bad idea to duplicate efforts. better combine them. otherwise u risk making mistakes that were already solved. missing lessons already learnt.
sudo-rs itself is not a bad idea, Canonical’s premature shipping of it in Ubuntu was the bad idea. sudo-rs was transparent with how far it had gotten in compatibility and feature parity
if anyone else is more familiar with go (I only really do rust) is there no solution to preventing stack smashing on goroutines? https://github.com/mullvad/mullvadvpn-app/pull/7728
I understand that go routines have a smaller stack size (the whole green thread problem) but there's no way to fix this?
I would not have guessed that iOS allows enough access to APIs to implement anything vpp-based. Very cool to see. I also enjoyed working with vpp (for the brief 6 months that I had with it).
I was thinking that's hard, but I noticed that vpp get ported to FreeBSD using epoll shim library, and I learnt apple Darwin use some some userland of FreeBSD to do POSIX compatibility, then after some tests and hacking, most related to minor POSIX API adaptation such as mmap and one major coroutine need add some assembly code, and it work! But I think most disappointed to me is that apple do lack some vectorized network IO unless do some kernel extension or other sort non standard ways.
I would love to see more root cause analysis data on the crashes they were seeing with wireguard-go. I wonder if it was bugs in the library itself, or the FFI.
I definitely noticed the performance boost on my Pixel 8, for some reason it seems to really not like wireguard-go, it struggled to pull even 100mbps, maybe something unoptimized on Google's custom hardware. With the new GotaTun version I can pull 500mbps+, though unfortunately it also seems to have introduced a bug that randomly prevents the phone from entering a deep sleep state, so occasionally my battery will randomly start draining at 10x normal speed if I have it enabled until I reboot.
I'm surprised by this comment. I have wireguard on 24/7 on my shitty Samsung A5 and it lasts forever. By comparison the Pixel 8 is a beast. Sounds like an Android bug more than wireguard.
Pixel 6 here. Vanilla wireguard app. It sucks the life out of my phone and nearly halves the already half-life battery (thanks Google for your crappy OEM producers!)
Thank samsung for their shitty modems in the pixels.
However, there’s going to be a large discrepancy for all devices on battery usage based on whether VPN is on wifi or cellular, and additionally when on cellular how close to the tower they are. I live near cell edge and VPN’s roast my batts on cellular no matter the make, in city it’s almost not noticeable to have VPN on. Better to use wifi when far from towers, cellular more efficient if it’s strong signal.
They must have fixed it in recent versions. My pixel 9 pro battery seems the same with proton VPN (wirrguard) on or off.
What app are you using?
It's just called WireGuard, by the "WireGuard Development Team" off google play.
Pretty sure thats the c implementation not the go one
AFAIK the C implementation is a kernel module that's not shipped in stock Android releases. The WireGuard Android app uses that module when available, but otherwise uses wireguard-go.
Good knowledge here, was unaware of this feature of the app. Would there be any case of the app defaulting to the wireguard kernel module if it's not included by any OEM Android release? I would assume that means most users are actually running wireguard-go.
I hope so.
Android kernels 4.19 and higher should all have support included for WireGuard unless the OEM specifically disables it: [0]. The Pixel 8 ships with the android 14 6.1 kernel so it most definitely should have WireGuard kernel support. You can check this in the WireGuard app BTW, if you go to settings it will show the backend that's in use.
[0] https://android-review.googlesource.com/c/kernel/common/+/14...
Kernel support should have no bearing as the apps are purely userspace apps. You can use the kernel mode if you root the phone, but that's not a typical scenario.
Well, the issue isn't kernel vs user space, but you are correct that you still need a custom ROM and/or root unfortunately. I had assumed Android had also allowed netlink sockets for WireGuard but alas they did not. So the app can't communicate with the kernel module, bummer.
Same behavior on raspberry pi 5. Might be just lack of arm optimizations.
It's very likely that VPNs like this are not CPU-bound, even on somewhat whimpy CPUs. I'd wager even some microcontrollers could sling 500megabits/sec around without trouble.
You’re in for a surprise then once you actually go look at the performance.
A Raspberry Pi 4 can manage something like 70Mbps of raw AES en/decryption flow: https://github.com/lelegard/aesbench/blob/main/RESULTS.txt
That CPU is pretty much a toy compared to (say) a brand-new M5 or EPYC chip, but it similarly eclipses almost any MCU you can buy.
Even with fast AES acceleration on the CPU/MCU — which I think some Cortex MCUs have — you’re really going to struggles to get much over 100Mbits of encrypted traffic handling, and that’s before the I/O handling interrupts take over the whole chip to shuttle packets on and off the wire.
Modern crypto is cheap for what you get, but it’s still a lot of extra math in the mix when you’re trying to pump bytes in and out of a constrained device.
You're looking at the wrong thing, WireGuard doesn't use AES, it uses ChaCha20. AES is really, really painful to implement securely in software only, and the result performs poorly. But ChaCha only uses addition rotation and XOR with 32 bit numbers and that makes it pretty performant even on fairly computationally limited devices.
For reference, I have an implementation of ChaCha20 running on the RP2350 at 100MBit/s on a single core at 150Mhz (910/64 = ~14.22 cycles per bytes). That's a lot for a cheap microcontroller costing around 1.5 bucks total. And that's not even taking into account using the other core the RP2350 has, or overclocking (runs fine at 300Mhz also at double the speed).
You’re totally right; I got myself spun around thinking AES instead of of ChaCha because the product I work on (ZeroTier) started with the initially and moved to AES later. I honestly just plain forgot that WireGuard hadn’t followed the same path.
An embarrassing slip, TBH. I’m gonna blame pre-holiday brain fog.
Yeah no, this is very much not true, even more so for a Go-based implementation and energy consumption optimized ARM devices.
MTU strikes again. 1320.
Why 1320 and not larger?
For most any 5G network you should be safe to 1420 - 80 = 1340 bytes if using IPv6 transport or 1420 - 60 = 1360 bytes if using IPv4 transport.
For testing I recommend starting from 1280 as a "does this even work" baseline and then tweaking from there. I.e. 1280 either as the "outside" MTU if you only care about IPv4 or as the "inside" MTU if you want IPv6 to work through the tunnel. This leverages that IPv6 demands a 1280 byte MTU to work.
Hah! I just ran into this recently and can confirm. The coax to my DOCSIS ISP was damaged during a storm, which was causing upstream channels to barely work at all. (Amusingly, downstream had no trouble.) While waiting for the cable person to come around later in the week, I hooked my home gateway device up to an old phone instead of the modem. I figured there would be consequences, but surprisingly, everything went pretty smoothly... But my Wireguard-encapsulated connections all hung during the TLS handshake! What gives?
The answer is MTU. The MTU on my network devices were all set to 1500, and my Wireguard devices 1420, as is customary. However, I found that 1340 ( - 80) was the maximum I could use safely.
Wait, though... Why in the heck did that only impact Wireguard? My guess is that TCP connections were discovering the correct MSS value automatically. Realistically that does make sense, but something bothers me:
1. How come my Wireguard packets seemed to get lost entirely? Shouldn't they get fragmented on one end and re-assembled on the other? UDP packets are IP packets, surely they should fragment just fine?
2. Even if they don't, if the Linux TCP stack is determining the appropriate MSS for a given connection then why doesn't that seem to work here? Shouldn't the underlying TCP connection be able to discover the safe MSS relatively easily?
I spelunked through Linux code for a while looking for answers but came up empty. Wonder if anyone here knows.
My best guess is that:
1. A stateless firewall/NAT somewhere didn't like the fragmented UDP packets because it couldn't determine the source/dest ports and just dropped them entirely
2. Maybe MSS discovery relies on ICMP packets that were not able to make it through? (edit: Yeah, on second thought, this makes sense: if the Wireguard UDP packets are not making it to their destination, then the underlying encapsulated packets won't make it out either, which means there won't be any ICMP response when the TCP stack sends a packet with Don't Fragment set.)
But I couldn't find anything to strongly support that.
Basically the only parts of the Internet which actually work reliably, around the globe, are the bits needed so that web pages basically kinda work. If you break literally everything else your service is crap, and some customers might notice, but many won't and also some won't have a choice so, sucks to be them. But if you break the Web, now everybody notices that you broke stuff and they're angry.
This is why DoH (DNS over HTTPS) is a thing. It obviously makes no actual sense to use the web protocol to move DNS packet, but, this works and most things don't work for everybody so eh, this is what we have. Smashing the Path MTU discovery doesn't break the web.
Breaking literally everything so long as the web pages work even means you can't upgrade parts of the web unless you get creative. TLS 1.3 the modern security protocol that is used for most of your web pages today, would not work for most people if it admitted that it's TLS 1.3, if you send packets with TLS version 1.3 on them people's "intelligent" "best in classs security" protective garbage (in the industry we call these "middle boxes") thinks it is being attacked by some unknown and unimaginable dastardly foe and kills the data. So TLS 1.3 really, I am not making this up, always pretends it is a TLS 1.2 re-connection, and despite the fact that no such connection ever existed these same "best in class security" technologies just have no idea what's happening and wave it through. It's very very stupid that they do that, but it was needed to make the web work, which matters, whereas actual security eh, suckers already bought the device, who cares.
This situation is deeply sad but, one piece of good news is that while "This Iranian woman can't even talk confidentially to her own mother without using code words because the people in charge there intercept her communications" won't attract as much sympathy as you'd like from some bearded white guy who has never left Ohio, the fact that those people broke his network protocol to do that interception infuriates him, and he's well up for ensuring they can't do that to the next version.
Your ultimate conclusion is correct, to my understanding. I know wireguard sought to be ultra minimal but I do wish they had included DPLPMTUD as something which is required to be supported (but not mandated to be used e.g. if the user wants to hard set it as they would currently) because it's one of those cases where "do it yourself separately the UNIX way™" or "have the tunneled things do it if they need it" instead are both significantly more complex and fragile.
On that note, from the TCP layer it should just look like an ICMP blackhole, which makes me wonder if enabling `net.ipv4.tcp_mtu_probing` will magically make TCP connections under Wireguard work even with the MTU set wrong. I'd try it, but unfortunately with a similar configuration I am unable to get the fragmentation behavior I was getting before; which makes me wonder if it was my UniFi Security Gateway that actually didn't like the fragmented packets.
Oh, this is the reason the Mullvad app on my Pixel 6a was suddenly able to connect in less than a second where before it would take 5-10 seconds, nice!
I think the problem is on the Android side. I noticed the same behavior with ZeroTier and even with MizuDroid, all totally unrelated.
Do you have wireguard keepalives on?
Nice, I love WireGuard. I ended up building WrapGuard [1] to run applications without root access to the host and choose Go to write it in. I don't really know Rust, but does it make more sense for firmware/networking type software? Is there even a difference?
1. https://github.com/puzed/wrapguard
> firmware
Yes, lots of firmware runs on hardware where a GC doesn't make sense. Because of limited memory and performance constraints. Sometimes having predictable timings (i.e. not a GC with pauses) is nice. I believe compiler and library support is also just better for many embedded platforms in rust.
> networking type software
Rust is a much more aggressively optimizing compiler, and thus will typically be faster, in the places where that matters. GC pauses might also be a point against golang in some places here. Rust's idioms provide slightly less opportunity for bugs in places where reliability matters (e.g. having a type system that requires you check for errors instead of just patterns that encourage it).
So there's a difference, but generally go is a good enough language for networking software and it would be rare that I wouldn't suggest that "use what you know" is more important than the differences between the languages for non-firmware network software.
I've implemented a few protocols in rust (and plenty in go and other languages).
One thing others haven't mentioned that I like rust for in this space:
The typestate pattern makes it really nice to work with protocols that have state. You encode your state machine logic into types, and your transitions into methods with move semantics, and you have a nice way to make sure your higher level code is using your protocol library correctly.
Another nice thing is that you can keep the number of copies and allocations way down if you're careful about how you use your buffers.
Pick the devil you know, as they say.
from TFA, the main advantage would be for embedded (as a library) use case, FFI with Go is harder.
Very cool. I may use this, but also curious what the best choice would be if you don't need encryption. I'm specifically wanting to enable some local container networking using apple's new container tool [1]. I know I could just use Docker...
[1] https://github.com/apple/container/issues/670
Very cool project. Is it always an LD_PRELOAD or can it function as a standalone SOCKS proxy similar to wireproxy?
Thanks chjj. Yeah it's always LD_PRELOAD. There is wireproxy [1] though that might do what you want?
1. https://github.com/whyvl/wireproxy
Correct me if I’m wrong, but if you use LD_PRELOAD, presumably it will not work for applications that circumvent libc, such as Go binaries (at least those with CGo disabled)?
Yeah you are right. Can you think of any way we could capture that traffic too?
Tor does this the right way on Linux. You make a separate user namespace with access only to the WireGuard network adapter and run the program inside of that. You want the kernel involved if you want any sort of guarantee:
https://blog.torproject.org/introducing-oniux-tor-isolation-...
How does this work in something like Kubernetes where you have a sidebar container configuring the network for the main container without affecting others on the same host?
I think all containers share the same netns in a pod. You restrict the pod to only the Wireguard peer IP, and have a (NET_ADMIN) sidecar container create an interface (tun/kernel wg) and update the routing tables for the netns. Then I believe the traffic from the other containers in the pod is tunneled.
Can you use user namespaces to create a network namespace with the VPN active and stick applications in that namespace?
From a quick search, https://blog.thea.codes/nordvpn-wireguard-namespaces/ sees to have at least the bones of a decent solution, though I've not had a chance to dig very far. A lot of results use root to set up the namespace, but I was pretty sure that shouldn't be needed with a new kernel and user namespaces enabled
It would be a non-trivial amount of work but syscall user dispatch lets you intercept syscalls on modern linux if you really want to.
https://docs.kernel.org/admin-guide/syscall-user-dispatch.ht...
I have no idea. I’ve never messed with it, but maybe something like eBPF to intercept network syscalls? Not sure if that’s a thing—especially without root access? Mostly I was just thinking the project page could use a disclaimer since, in Go, it is common to bypass libc. :shrug:
This seems like a very cool, useful project though!
ptrace: https://github.com/hmgle/graftcp
One usecase I've always wanted is being able to combine multiple tunnels into one shared connection, for instance airVPN allows 5 simultaneous users per sub, it would be awesome if I could run 5x connections and combine their traffic, but I dunno how I would do this with wg / nmcli
VPNs are level 3 while interface bonding is level 2. You’d have to create a vxlan over wireguard. It sounds like a nightmare but it would be interesting to implement.
I’m just gonna leave this here: https://docs.zerotier.com/protocol/#bridging
Disclosure: I work at ZeroTier :)
I believe you are making use of gVisor’s userspace TCP implementation. I’m not sure if there is something similar in Rust that would be so easy to set up like this.
There isn't something as mature as gVisor afaik. https://github.com/smoltcp-rs/smoltcp implements many of the same abstractions as gVisor.
Fingers crossed that GotaTun will also make its way into the Tailscale Android app (since that's what I use to connect to Mullvad).
GotaTun is specific to Mullvad and the features they usually add make sense for a public VPN provider. Unlikely projects such as Tailscale adopt it.
Besides, engineers at Tailscale, I don't think, strike me as startled by any hurdle too tall to debug, improve Go-based libraries. In fact, they pushed wireguard-go past 10gbps on Linux-based platforms back in April 2023! https://tailscale.com/blog/more-throughput
I wish they would improve wireguard-the-protocol as well: wireguard doesn't stand a chance against gov/isp blocks.
That's more of a job for an encapsulating protocol. (shadowsocks or similar) Wireguard isn't designed to be obfuscating alone. It's just a simple l3 udp tunnel with a minimal attack surface.
That's the traditional answer parroted in the Wireguard documentation but a few hours' serious thought and design is enough to reveal the fatal flaw: any encapsulating protocol will have to reinvent and duplicatively implement all of the routing logic. Perr-based routing is at least 50% of wireguard's value proposition. Having to reimplement it at the higher level defeats the purpose. No, obfuscation _has_ to be part of the same protocol as routing.
(Btw, same sort of thing occurs with zfs combining raid and filesystem to close the parity raid write hole. Often strictly layered systems with separation of concerns are less than the sum of their parts.)
In this case with the, I believe it’s called quantum tunneling by mullvad, it’s actually a good thing. Because the encapsulation protocol is just UDP/IP, a well established existing protocol that can masquerade as any kind of internet traffic easily.
Makes it difficult to block by censors. Great video I saw here: https://youtu.be/pZiG8r-diTM?si=wy35elqMt1T6euq0
This also means wg is just doing one thing instead of a dozen it doesn’t “need” to.
> It's just a simple l3 udp tunnel
Wait, isn’t UDP L4? Am I missing something?
Wireguard is a L3 VPN that uses UDP (L4) for tunneling. Thats probably what was meant.
Yes, but it tunnels arbitrary IP packets encapsulated in UDP.
There are forks of wg because of this. Like amnezia-wg
This is a neat project!
https://docs.amnezia.org/documentation/amnezia-wg/
amnezia-wg is quite cool and they have built the kmod too, I did some test so far they can works even in my location which block wireguard server quickly.
The mullvad apps do offer obfuscation options (shadowsocks, etc) but i agree it would be nice if something was baked into wireguard itself. I recently went through setting up shadowsocks over wg for my homelab and it was a good bit of effort
Anywhere I can read more about this?
Mullvad does offers several obfuscation methods well geared towards the scenarios you mention.
Known Limitations
WireGuard is a protocol that, like all protocols, makes necessary trade-offs. This page summarizes known limitations due to these trade-offs.
Deep Packet Inspection
WireGuard does not focus on obfuscation. Obfuscation, rather, should happen at a layer above WireGuard, with WireGuard focused on providing solid crypto with a simple implementation. It is quite possible to plug in various forms of obfuscation, however.
tl;dr Read the docs.
Mullvad does exactly this.
WireGuard limitations hurt the attempt however.
For example, multi-hop betrays the actual exit node to your ISP (or MITM) due to the port used.
To clarify, this is refering to Mullvad multi-hop feature. Doing your own multihop setup doesn't have this issue, right?
Correct. Note that the MTU will be further reduced and that WireGuard DIY multi-hop may be inferred.
If anyone working on the implementation is here, was it not possible to upstream your changes to BoringTun? The blog mentions some changes but doesn't go into detail on that aspect.
I'm guessing because BoringTun has been in a state of "currently undergoing a restructuring" for something like 3 years by now, I'm guessing Mullvad wasn't too keen to maybe/maybe not be able to contribute, and much more prefer being in 100% control of their own implementation.
As someone who wants to see Wireguard succeed and in even wider use, this move makes sense from that perspective too. The more implementations we have available, the more we can trust that the protocol is secure and stable enough. Personally I also have about 100x more trust in Mullvad than Cloudflare both in terms of security but more importantly privacy, but that's just the cherry on top.
BoringTun is unmaintained. There are various forks being developed.
I work at Obscura VPN and faced with boringtun bugs a few years ago we evaluated a few of the forks and switched our client to be based on top of NepTUN (https://github.com/NordSecurity/NepTUN).
I am curious why Mullvad started their own fork rather than building on top of one of the existing ones. It would be nice if there could be reconsolidation somewhere.
Its funny, this is another of the billions of reasons why Mullvad should be the VPN of choice. But so many fucking people can't ever get over that their favorite social media influencer/Youtuber is offering a code for 200% off of NordShark VPN, now with extra AI.
Mullvad is great for privacy. But it's blocked by pretty much every VPN block list. NordVPN at the very least bypasses all the ones I regularly encounter.
I do use Mullvad for most web browsing though. But Imgur for example is blocked on it, and it's blocked in the UK, so I need NordVPN if I want to see any images there.
Most people's VPN usage is literally just geolocation restrictions and Nord is really good at that.
I regularly go to imgur via mullvad, exit Netherlands.
Aren't proxies good enough for that purpose?
The user experience differs for proxies.
System wide proxy configuration doesn’t actually always work system wide.
A VPN tends to have more success in encapsulating all application traffic (or all desired application traffic, if you’re so inclined to configure your system)
I love and use mullvad myself but I don't think they are very competitive for the average person. They mostly just care about getting around geo blocks on websites and streaming services, which mullvad puts 0 effort into facilitating.
It became less of a choice for many after they sadly had to disable port forwarding.
Yeah, their reasoning is solid (easy to abuse) but it is still a very useful feature.
AFAIK, at the moment your choices are AirVPN and ProtonVPN. AirVPN has static port forwarding and Proton has UPNP port forwarding.
private internet access has port forwarding too
PIA is not to be trusted after their buyout, IMHO
Currently using airVPN, but ye gods, their eddie client is atrocious on linux. I wind up using wg / nmcli, but then have to block traffic going outside of the vpn with iptable rules because it leaks for some reason.
I miss mullvad dearly, and I might try proton after my 3y sub is up.
Not only Eddie, their account control panels and site in general look like something from the 90s, and it seriously hampers their business. I can't recommend them to anyone that isn't highly technical. And even then, as a technical user, why do I manually have to select one of 10-20 servers within a city or region, why am I being asked to manually load balance? Why is there no Wireguard over port 53 or 443?
It makes more sense when you know they're privacy activists first, businessmen second. But Mullvad shows you can be pro privacy and still offer great UX and a sleek site and client.
Btw, if you're managing things in CLI, you could take a look at their Hummingbird Suite. AFAIK it has a killswitch.
What sucks with Proton is that you can't share the VPN account with friends, because it is tied to your Proton account. They should create a vpn.proton.me subdomain that you can create a special managed account on that can only touch the VPN settings.
>Btw, if you're managing things in CLI, you could take a look at their Hummingbird Suite. AFAIK it has a killswitch.
Hummingbird doesn't support wireguard iirc, which is a deal breaker
They're planning to introduce OpenVPN Data Channel Offload (DCO) support to more servers once Linux 6.18 starts becoming more mainstream.
With DCO, OpenVPN can perform almost as well as Wireguard, sometimes even better. Although with more performance overhead so not the best choice for laptops and phones.
Tangentially related but I kind of wish Wireguard looked more toward the future and had included AES as alternative to ChaCha20. At the time of development, many ARM devices didn't yet have AES acceleration which is why ChaCha20 was needed for wide hardware support, but they do since ARMv8 which became widespread in 2015. Intel and AMD have had AES acceleration for a long time. And then ChaCha20 would have been the fallback on MIPS and RISC-V.
I wish I could use Mullvad. But their IPs are banned from many streaming services and they don't change them often enough so I am stuck with Nord.
I would just pirate at that point. You're paying for the streaming service anyways. Use mullvad to download the torrent :). I'm pretty sure they ignore dmca requests. Not that they even know their customer's names if you pay with Mullvad amazon card.
Been buying mullvad for the last 4-5 years but oftentimes I can’t even browse the fucking New York Times website due to low bandwidth, let alone stream anything. At this point, I just keep adding time to my account just in case, without using it.
Not to mention holding companies which snap up 15 competing VPNs and whitelabel most of them.
Mullvad seems to care and be competent about privacy, but most average VPN users aren’t seeking the most extreme privacy. They just want something cheap that lets them do geolocation things or access the most websites.
The average VPN user is knowledge-less. At best their internet usage data is being sold to third party analytics companies. At worst third parties are routing their own bots through their local connection.
You do know that NordSec maintains its own rust fork of BoringTun: https://github.com/NordSecurity/NepTUN ? :)
There is also https://github.com/firezone/boringtun which is a fork by https://www.firezone.dev/
It looks like GH Issues are disabled. https://github.com/mullvad/gotatun
It's unclear where to report problems, suggestions, etc.
One meta thing I've always wondered ... Are multiple implementations of the same protocol good or bad for security?
Probably naively, I'm thinking:
What do the security folks out there think of the topic?I think the general consensus is that it improves security of the protocol, but obviously that won't matter much if the implementation gets something wrong or has worse security by itself.
Issues in the protocol itself would need all implementations to change, but issues in the implementation would obviously be isolated to one implementation. For something like Wireguard, I'd wager a guess that issues in the implementations are more common than issues in the protocol, at least at this stage.
If the implementation gets it wrong that can also be a sign of ambiguity in the protocol / standard and as such result in clarifications and an overall more well specified protocol
I wouldn't say that multiple implementations are duplicating the attack surface since most users will not end up running them in parallel.
I meant at a global level (think as if you're attacking all wireguard users, not a single one)
The increased attack surface mostly only affects that one particular implementation though. So, yes, twice as many implementations that may contain exploitable bugs, but each new implementation could only be used to exploit a fraction of the total user base
> could only be used to exploit a fraction
If anything this is a even a good thing, since it means that each individual vulnerability an attacker finds is less valuable to them.
Diversity is a fantastic thing for security. It limits the impact when a bug drops and gives the possibility to migrate or run a mix of systems.
Competitions helps in multiple ways. It improve tooling, test suites, CVE response time, documentation and evolution of the protocol. There are some counter examples where compatibility suck, like DLNA but the problem often come from the spec.
That's really good because it means it will be able to have more exposure, more exposure means more improvement, more improvement eventually dig out bad bugs and reduces the attack surface in the long run
Is having Mac OS and Linux a decrease or increase in security over just having windows only?
dont fix if it ain't broken. look at sudo-rs and other rust ports.
ofc, thats a cynical view.
i personally think its a bad idea to duplicate efforts. better combine them. otherwise u risk making mistakes that were already solved. missing lessons already learnt.
sudo-rs itself is not a bad idea, Canonical’s premature shipping of it in Ubuntu was the bad idea. sudo-rs was transparent with how far it had gotten in compatibility and feature parity
Hope tailscale adopts this
the linked issues are quite interesting, why does go have to page in so much memory for the GoString? Is this for some sort of optimization? https://github.com/mullvad/mullvadvpn-app/pull/6727
if anyone else is more familiar with go (I only really do rust) is there no solution to preventing stack smashing on goroutines? https://github.com/mullvad/mullvadvpn-app/pull/7728 I understand that go routines have a smaller stack size (the whole green thread problem) but there's no way to fix this?
It was solved in another PR by using an alternate stack by setting the flag SA_ONSTACK.
I thought Wireguard runs inside the kernel on Android since it ships as part of Linux now.
I think it has to be enabled as a module, and the android kernel has it disabled.
Apparently the module is there: https://www.xda-developers.com/google-adds-wireguard-vpn-and...
But you need to be rooted to use it: https://lists.zx2c4.com/pipermail/wireguard/2022-September/0...
As far as I know, you need root permissions to add an interface. Even though the module is enabled, it's not used.
The official wireguard app also mentions wireguard-go
For the similar reason I do not using any go based proxy code in my MintFlow app, and use rust to implement some proxy protocols.
But my app’s wireguard is natively implemented by fdio vpp plugin, so it’s based on C.
I would not have guessed that iOS allows enough access to APIs to implement anything vpp-based. Very cool to see. I also enjoyed working with vpp (for the brief 6 months that I had with it).
I was thinking that's hard, but I noticed that vpp get ported to FreeBSD using epoll shim library, and I learnt apple Darwin use some some userland of FreeBSD to do POSIX compatibility, then after some tests and hacking, most related to minor POSIX API adaptation such as mmap and one major coroutine need add some assembly code, and it work! But I think most disappointed to me is that apple do lack some vectorized network IO unless do some kernel extension or other sort non standard ways.
Is there any way to switch to this implementation for generic WireGuard users?
I tried downloading their Android app, but it's not generally usable for people who host our own WireGuard, which is fair enough.
The github repo is linked in the post which has build instructions: https://github.com/mullvad/gotatun
I would love to see more root cause analysis data on the crashes they were seeing with wireguard-go. I wonder if it was bugs in the library itself, or the FFI.
Yeah I'm surprised by that. I thought Wireguard was so simple and wireguard-go was so popular that it wouldn't crash. It's just UDP packets.
Can you use DAITA with just gotatun (on linux) or do you require the Mullvad daemon?
Now that's how you name things!
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
They didn't. They forked an old unmaintained thing already written in rust to add new features...
https://dosaygo-studio.github.io/hn-front-page-2035/news-hon...
[flagged]
[flagged]