UXG Port Forwarding vs DNAT

07 Mar 2025 in Tech / Uxg / Networking / Dnat 2025-03-12

What started out as an excercise in getting my HAProxy logs to be sent to Fluent-bit, turned into a rabbit hole (as usual) in which I discovered that Port Forwarding to hosts that were routed via BGP (Anycast HAProxy environment and Proxmox SDN environment), were using the source address of the upstream gateway. So basically my HAProxy logs were a bit useless as they didn’t actually show the source of the connectoion.

Natually I spent time digging into X-Forwarded-For with HAProxy and I refined my HAProxy logging config quite a lot - no bad thing.

In summary, the log component of my haproxy.cfg looks like this:

  defaults
    option forwardfor
    option redispatch
    option httplog
    log etselogs.mgmt.etse.me:514 local0 info

I also fixed an issue where the logs were showing the wrong source IP with a funny ::ffff: prefix like this:

2025 Mar  4 00:00:15: hap01.mgmt.etse.me (local0/info) [haproxy] ::ffff:192.168.2.1:33638 [04/Mar/2025:00:00:15.763] https-in~ https-in/<NOSRV> -1/-1/-1/-1/0 400 0 - - PR-- 1/1/0/0/0 0/0 "<BADREQ>"
2025 Mar  4 00:00:16: hap01.mgmt.etse.me (local0/info) [haproxy] ::ffff:192.168.2.1:41036 [04/Mar/2025:00:00:16.054] https-in~ gitea_pool/abc-1 0/0/0/43/43 200 132 - - ---- 1/1/0/0/0 0/0 "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/1.1"

Solution HERE

Now my front end config stanzas look like this:

frontend https-in
    http-request set-header X-Forwarded-Proto https if { ssl_fc } 
    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

    bind :443 ssl crt /etc/haproxy/certs/
    bind :::443 v6only ssl crt /etc/haproxy/certs/
    
    mode http
    option http-server-close

However I digress, the source IP problem persisted. The logs now looked like the below; no ::ffff: but still the wrong source IP:

Mar  5 23:59:45 hap01.mgmt.etse.me haproxy[1157507]: 192.168.2.1 - - [05/Mar/2025:15:59:45 +0000] "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/1.1" 200 132 "" "" 34770 400 "https-in~" "gitea_pool" "gitea-1" 0 0 0 42 42 ---- 2 2 0 0 0 0 0 "" "" "192.168.2.1" 
Mar  5 23:59:45 hap01.mgmt.etse.me haproxy[1157507]: 192.168.2.1 - - [05/Mar/2025:15:59:45 +0000] "PROPFIND /remote.php/dav/files/jacko/ HTTP/1.1" 207 874 "" "" 53343 420 "https-in~" "drop_pool" "drop-1" 0 0 0 57 57 ---- 2 2 0 0 0 0 0 "" "" "192.168.2.1"

Note the 4th field above is stating the client IP as 192.168.2.1 which is the IP of the UXG on that VLAN.

At this point I decided it was time to break out tcpdump / wireshark to see if I could identify what was happening at the network level. Using tcpdump -i eth0 -s 65535 tcp port 80 -w http.pcap CREDIT this is what the captured packets looked like:

PCAP

PCAP Detail

So whilst I don’t consider myself an expert of pulling apart pcap’s, from this I was able to redirect the troubleshooting efforts. It occurred to me that, unlike a normal “Port Forwarding” pattern from a Firewall / home router, in this case I was port forwarding to an IP address that was only reachable via BGP 192.168.4.4. Normally you are Port Forwarding to a segment which is directly connected. It got me wondering.

I stumbled across this helpful video: Ubiquiti UniFi Gateway - DNAT and Port Forwarding (NAT/Destination NAT) which points out that Port Forwarding is exactly the same as Destination NAT. So acting on a hunch, I decided to test out using Destination NAT to forward external traffic on tcp port 8080 to tcp port 80 on a test server on my SDN network range. Here is what the rule looked like:

DNAT Config Example

To verify the config, I initiated a connection to my public IP on port 8080 from an external host using curl http://123.123.123.5:8080 and ran tcpdump -i ens18 -s 65535 tcp port 80 on the host. This is what tcpdump output:

tcpdump tcpdump tcpdump

BINGO! NOW I can see the actual source IP in the Packet Capture - so Port Forwarding and DNAT are NOT exactly the same?!

So naturally I went right ahead and switched my port 80/443 rules to use DNAT rather than Port Forwarding and tested from outside my network with everything looking just chipper. However, shortly there after I opened Home Assistant to do something and low and behold, internal hosts were no longer able to talk the Anycast Reverse Proxy! What now.

I soon discovered that all of the DNS records that I have pointing to my Public IP which normal just work internally due to Hairpin NAT, were no longer working. So it would seem that the one disadvantage of DNAT as that Hairpin NAT doesn’t work? I’m not sure. One day I may dig into it (or be forced to if something breaks) but for now, the fix is simply to create Internal DNS records for those names on the UXG, pointing to the Internal IP of the Anycast Reverse Proxy 192.168.4.4.

Now everything is working just nicely and my HAProxy logs show the correct source IP:

Mar  6 21:16:10 hap01.mgmt.etse.me haproxy[1222631]: 10.192.54.72:50501 [06/Mar/2025:21:16:10.686] https-in~ drop_pool/drop-1 0/0/1/45/46 207 887 - - ---- 3/3/0/0/0 0/0 "PROPFIND /remote.php/dav/files/jacko/ HTTP/1.1"
Mar  6 21:16:11 hap01.mgmt.etse.me haproxy[1222631]: 2406:3400:658:100:109d:43ff:fec0:4232:40004 [06/Mar/2025:21:16:11.444] https-in~ etse_pool/etse-1 0/0/0/0/1 200 30536 - - --NI 3/3/0/0/0 0/0 "GET / HTTP/1.1"
Mar  6 21:16:11 hap01.mgmt.etse.me haproxy[1222631]: 192.168.254.97:37214 [06/Mar/2025:21:16:11.469] https-in~ semaphore_pool/sem-1 0/0/0/0/1 200 1020 - - ---- 3/3/1/1/0 0/0 "GET / HTTP/1.1"

I was buzzing all day from this fix. One day maybe I will be able to understand it fully, probably not though 🤠

Templates:

Error