Migrating from USG to UCG Ultra.

02 Jan 2025 in Tech / Usg / Ucg / Unifi / Bgp 2025-01-07

A friend of mine and I were recently looking into setting up the Unifi SDWAN between our Homelabs, his in Sydney, mine in Perth. We discovered that the Unifi Security Gateway line of Unifi Routers was not supported and a Unifi Cloud Gateway or Dream Machine was required. Being a little trigger happy, he immediately ordered 2 UCG Ultra’s! I had a mild panic as my Homelab is very tightly integrated into the Unifi Security Gateway and the config.gateway.json way of managing the non-supported config (BGP routing, Custom DNSMasq entries being the main ones). After a little digging around, I found that the UCG Ultra, if using the Unifi Early access firmware branch, could do everything that I needed and would probably make support easier and my network faster.

Migration Approach

Develop frr.conf
Upgrade UCG to Early Access Release Channel
Backup and Restore Unifi Config
Create Custom DNS Records
Enable ssh
Enable BGP
Update Port Forwarding and Firewall Rules
Add more BGP config for Proxmox SDN

Develop frr.conf

My thinking here was that the best way to create a valid Free Range Routing config frr.conf that wasn’t going to screw up the UCG was to develop it using the Virtual Router Shell vtysh on a dev machine. So on a spare Debian 12 machine setup:

sudo apt install frr
sudo sed -i -e 's/bgpd\=no/bgpd\=yes/g' /etc/frr/daemons # change bgpd=no to bgpd=yes in /etc/frr/daemons
sudo systemctl enable frr
sudo systemctl start frr

Now we have the BGP binaries setup and a working vtysh to use, we can now start the Virtual Router Shell and create our BGP configuration. To start the Virtual Router Shell:

debtestvm# sudo vtysh

Hello, this is FRRouting (version 8.4.4..
Copyright 1996-2005 Kunihiro Ishiguro, et al.

The Virtual Router Shell uses the Cisco sytle configration syntax. So you can do things like show run to see the running configuration and conf t to “Configure Terminal” and make changes to the configuration.

debtestvm# show run
Building configuration...

Current configuration:
!
frr version 8.4.4
frr defaults traditional
hostname debtestvm
log syslog informational
service integrated-vtysh-config
!
end

Ok, lets get started. Recall the design shared in the post Anycast Reverse Proxy with ExaBGP USG and HAProxy.

Anycast Reverse Proxy diagram

Basic BGP Config

We can see from the diagram that this router is part of BGP Autonomous System 65002. It is peering with 2 BGP neighbors (neighbors), 192.168.2.2 and 192.168.3.3. In this dev environment, this router has no connectivity to those peers but that doesn’t matter, what we are trying achieve is a valid configration file that we can port to our Prod router. Not shown in the diagram is that this router will have the IP 192.168.254.1 so that will be its BGP router-id. Lets start. First we set our local router-id, hostname and domainname; then we enter router config mode for AS 65002 and add basic configuation for our BGP neighbours. Being iBGP, we share the same AS as our neighbors. Don’t forget to commit the changes! wr me or do wr me if you’re config mode:

debtestvm# conf t
debtestvm(config)# router-id 192.168.254.1
debtestvm(config)# hostname Cloud-Gateway-Ultra
debtestvm(config)# domainname mgmt.etse.me
debtestvm(config)# router bgp 65002
debtestvm(config-router)# neighbor 192.168.2.2 remote-as 65002
debtestvm(config-router)# neighbor 192.168.3.3 remote-as 65002
debtestvm(config-router)# do wr me
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
[OK]

Prefix Lists

Ok so we have enough configuration now to setup a peering relationship with our neighbours but at this point we are not accepting any IPv4 or IPv6 prefixes from them. Before we do that, we need to put some prefix lists in place that we can use to filter incoming routes to just those that we want and block anything that might cause us routing problems. For our Reverse Proxy use case, this is quite simple as we only want accept one IPv4 address and one IPv6 address (192.168.4.4/32 & 2406:3400:658:105::4:4/128).
To create them, firstly drop out of the router config using exit. We tend to name the prefix lists by the service and the direction. Eg haproxy-in denotes that the routes are for haproxy and the “IN” indicates that we are receiving the routes from a peer, rather than sending them to a peer.

debtestvm(config-router)# exit
debtestvm(config)# ip prefix-list HAProxy-IN seq 5 permit 192.168.4.4/32 
debtestvm(config)# ipv6 prefix-list HAProxy6-IN seq 5 permit 2406:3400:658:105::4:4/128

To apply those prefix lists to our neighbours:

debtestvm(config)# router bgp 65002
debtestvm(config-router)# address-family ipv4 unicast
debtestvm(config-router-af)# neighbor 192.168.2.2 prefix-list HAProxy-IN in
debtestvm(config-router-af)# neighbor 192.168.2.2 soft-reconfiguration inbound
debtestvm(config-router-af)# neighbor 192.168.3.3 prefix-list HAProxy-IN in
debtestvm(config-router-af)# neighbor 192.168.2.2 soft-reconfiguration inbound
debtestvm(config-router-af)# exit
debtestvm(config-router)# address-family ipv6 unicast
debtestvm(config-router-af)# neighbor 192.168.2.2 prefix-list HAProxy6-IN in
debtestvm(config-router-af)# neighbor 192.168.2.2 activate
debtestvm(config-router-af)# neighbor 192.168.3.3 prefix-list HAProxy6-IN in
debtestvm(config-router-af)# neighbor 192.168.3.3 activate
debtestvm(config-router-af)# exit

Notes:

Route Maps

Route Maps can be used for all sorts of things with BGP. In our case, we are simply going to use route maps to tell BGP to prefer the routes receievd from the BGP Peer on our “primary” HAProxy node. We set the local-preference attribute - the higher number is the preferred Peer. First we create the route map, then we apply the route map to the neighbour. Firstly make sure you are in config mode, not config-router mode. The prompt should be debtestvm(config)#.

debtestvm(config-router)# exit
debtestvm(config)# route-map hap01-in permit 5
debtestvm(config-route-map)# set local-preference 200
debtestvm(config-route-map)# exit
debtestvm(config)# route-map hap02-in permit 5
debtestvm(config-route-map)# set local-preference 100
debtestvm(config-route-map)# exit

Now to apply the route maps to the Peers, specifiying that the direction of the routes is in:

debtestvm(config)# router bgp 65002
debtestvm(config-router)# address-family ipv4 unicast
debtestvm(config-router-af)# neighbor 192.168.2.2 route-map hap01-in in
debtestvm(config-router-af)# neighbor 192.168.3.3 route-map hap02-in in
debtestvm(config-router-af)# exit
debtestvm(config-router)# exit
debtestvm(config)# do wr me
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
[OK]

Review Config

Now lets have a look at what our final configuration for our HAProxy BGP relationship looks like:

debtestvm(config)# do show run
Building configuration...

Current configuration:
!
frr version 8.4.4
frr defaults traditional
router-id 192.168.254.1
hostname Cloud-Gateway-Ultra
domainname mgmt.etse.me
log syslog informational
service integrated-vtysh-config
!
ip router-id 192.168.254.1
!
router bgp 65002
 neighbor 192.168.2.2 remote-as 65002
 neighbor 192.168.3.3 remote-as 65002
 !
 address-family ipv4 unicast
  neighbor 192.168.2.2 soft-reconfiguration inbound
  neighbor 192.168.2.2 prefix-list HAProxy-IN in
  neighbor 192.168.2.2 route-map hap01-in in
  neighbor 192.168.3.3 route-map hap02-in in
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 192.168.2.2 activate
  neighbor 192.168.2.2 prefix-list HAProxy6-IN in
  neighbor 192.168.3.3 prefix-list HAProxy6-IN in
 exit-address-family
exit
!
ip prefix-list HAProxy-IN seq 5 permit 192.168.4.4/32
!
ipv6 prefix-list HAProxy6-IN seq 5 permit 2406:3400:658:105::4:4/128
!
route-map hap01-in permit 5
 set local-preference 200
exit
!
route-map hap02-in permit 5
 set local-preference 100
exit
!
end

Backup Dev Config File

Take a copy of the /etc/frr/frr.conf that we have been creating on our dev machine. Eg:

scp /etc/frr/frr.conf user@hostname:~/

Prepare

Configure a Guest Network port on your existing LAN, ensuring that Guest isolation is enabled to block it from seeing the rest of your network.
Plug in the Cloud Gateway Ultra WAN Port to the Guest LAN port
Setup the Unifi Cloud Gateway Ultra as a new temporary Unifi Site

Enable Early Access Release Channel

Access the Management Console at https://192.168.1.1 and login with default creds ubnt/ubnt
Enable Early Access Release Channel: Management UI -> Settings -> Control Plane. Under Unifi OS click on “UCG Ultra” and in the popup menu, change Relase Channel to “Early Access”.
Update the UCG Ultra to the latest Early Access release which (at time of writing) is UCG Ultra: 4.1.13 / Network Application: 9.0.108

Backup and Restore Unifi Config

Take a final backup copy of your Unifi Application via the GUI and download it to your machine. I excluded the history, YMMV.
Power on the Cloud Gateway Ultra and connect laptop to one of the LAN ports.
Access the Management Console at https://192.168.1.1 and login with default creds ubnt/ubnt
Restore the Unifi Network Application Backup from Step 1

Cutover to UCG Ultra

Shutdown the Unifi Network Application LXC / VM
Poweroff the Unifi Security Gateway
Reroute WAN and LAN links from USG to UCG.
Power on UCG

At this point, I allowed some time for my network to come back up and did some checks that my devices were working.
A ended up restarting my Proxmox VM’s and LXC’s so that that DHCP would re-register their names in DNSMasq

Create Custom DNS Records

Now that the UCG is in Prod, and assuming that your Internet connection came back up, you can manage your Unifi system as https://unifi.ui.com. Login using your UI creds.
Go-to Settings -> Routing -> DNS
Create appropriate entries, as per your config.gateway.json dns forwarding options. Only limitation at tome of writing is CNAME records and PTR records.

Enable SSH Access

Goto Settings -> System -> Advanced
Tick “Device SSH Authentication”
Provide a user name and password to use for ssh. Key based ssh didn’t work for me, not sure why.

Enable BGP

ssh to the UCG Ultra ssh root@192.168.254.1 (note for me, the username supplied in the step above didn’t apply, i needed to use root)
sed -i -e 's/bgpd\=no/bgpd\=yes/g' /etc/frr/daemons to change bgpd=no to bgpd=yes in /etc/frr/daemons
Copy our Dev Config file to the UCG: scp ~/frr.conf root@Cloud-Gateway-Ultra:/etc/frr/frr.conf
systemctl enable frr enable the Free Range Routing service so that it will start automatically on UCG boot.
systemctl start frr start the Free Range Routing service
vtysh use the Virtual Router Shell to confirm that we are receieving routes:

frr# show ip route
Codes: K - kernel route, C - connected, S - static, O - OSPF,
       B - BGP, T - Table, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 10.10.0.2/32 is directly connected, tun1, 4d02h54m
C>* 10.64.0.0/24 is directly connected, br64, 5d00h00m
C>* 10.192.0.0/18 is directly connected, br192, 5d00h00m
C>* 10.254.0.0/26 is directly connected, br254, 5d00h00m
K>* 192.168.1.0/24 [0/0] via 10.10.0.2, tun1, 4d02h54m
C>* 192.168.2.0/24 is directly connected, br2, 5d00h00m
C>* 192.168.3.0/24 is directly connected, br3, 5d00h00m
B>* 192.168.4.4/32 [200/100] via 192.168.2.2, br2, weight 1, 5d00h00m
C>* 192.168.254.0/24 is directly connected, br0, 5d00h00m
C>* 220.233.1.69/32 is directly connected, ppp0, 4d02h42m

Note the route prefixed with B which denotes a route learned from BGP: B>* 192.168.4.4/32 [200/100] via 192.168.2.2, br2, weight 1, 5d00h00m

We can look at the IPv6 routes also, once again prefixed with a B - note that there appears to be an issue with weighting on the IPv6 routes, something I need to fix:

frr# show ipv6 route
Codes: K - kernel route, C - connected, S - static, B - BGP,
       T - Table, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 2406:3400:600:69:9ab:1bb3:91ae:5856/128 is directly connected, ppp0, 4d02h43m
C>* 2406:3400:658:100::/64 is directly connected, br0, 5d00h02m
C>* 2406:3400:658:101::/64 is directly connected, br192, 5d00h02m
C>* 2406:3400:658:102::/64 is directly connected, br2, 5d00h02m
C>* 2406:3400:658:103::/64 is directly connected, br254, 5d00h02m
C>* 2406:3400:658:104::/64 is directly connected, br3, 5d00h02m
B>* 2406:3400:658:105::4:4/128 [200/100] via 2406:3400:658:102:be24:11ff:feda:1670, br2, weight 1, 2d11h43m
  *                                      via 2406:3400:658:104:be24:11ff:fef6:b89, br3, weight 1, 2d11h43m
C * fe80::/64 is directly connected, tun1, 4d02h55m
C * fe80::/64 is directly connected, eth4, 4d02h57m
C * fe80::/64 is directly connected, br64, 5d00h02m
C * fe80::/64 is directly connected, br3, 5d00h02m
C * fe80::/64 is directly connected, br254, 5d00h02m
C * fe80::/64 is directly connected, br2, 5d00h02m
C * fe80::/64 is directly connected, br192, 5d00h02m
C * fe80::/64 is directly connected, br0, 5d00h02m
C * fe80::/64 is directly connected, switch0.64, 5d00h02m
C * fe80::/64 is directly connected, switch0.3, 5d00h02m
C * fe80::/64 is directly connected, switch0.254, 5d00h02m
C * fe80::/64 is directly connected, switch0.2, 5d00h02m
C * fe80::/64 is directly connected, switch0.192, 5d00h02m
C * fe80::/64 is directly connected, switch0.1, 5d00h02m
C * fe80::/64 is directly connected, eth3, 5d00h02m
C * fe80::/64 is directly connected, eth2, 5d00h02m
C * fe80::/64 is directly connected, eth1, 5d00h02m
C * fe80::/64 is directly connected, eth0, 5d00h02m
C>* fe80::/64 is directly connected, switch0, 5d00h02m
C>* fe80::3e08:f6ff:fee5:0/128 is directly connected, ppp0, 4d02h43m
C>* fe80::9c3a:5edf:4d32:4bf9/128 is directly connected, ppp0, 4d02h43m

Add Port Forward and Firewall Rule for HAProxy

This was originally in my config.gateway.json so now I need to use the GUI to add a Port Forwarding fule for TCP/80 and TCP/443 to 192.168.4.4
As above, for the IPv6 access, use the GUI to add a firewall rule to allow Source Any, Dest 2406:3400:658:105::4:4/128 on TCP/80 and TCP/443.

MORE BGP Configuration (Proxmox SDN)

A little addendum to the BGP work. I also need to move the BGP config from the Unifi Security Gateway for the BGP Peering with Proxmox SDN so that the L2VPN / VXLAN zones are reachable to the rest of the network.

Proxmox SDN BGP

For this work, I am going to start by configuring the prefix lists so that they are ready when I setup the BGP neighbours. In this instance, the danger of a route leaking from the SDN network into my Prod network is very real and indeed did cause a “Denial of BBL” incident for a 90 minute period when I set this up (essentially the Proxmox SDN network was advertising a defaul route which the UCG was not setup to filter - oops!).

Proxmox SDN currently has two Zones configured which both live within the 10.100.0.0/16 IP range. So my prefix lists need simply to filter everything except that prefix. I assume this is best practise but I am also going to setup a prefix list to explicitly deny a default route 0.0.0.0/0 coming from the Proxmox SDN BGP Peers. Note the ge 16 below. This says that it will accept a prefix that is greater than 16 bits long. So any prexies with mask of /16 - /32 are acceptable.

Connect to your UCG via ssh ssh root@192.168.254.1 then enter the Virtual Router Shell vtysh:

debtestvm# conf t
debtestvm(config)# ip prefix-list PVE-IN seq 10 permit 10.100.0.0/16 ge 16
debtestvm(config)# ip prefix-list PVE-IN seq 20 deny 0.0.0.0/0
debtestvm(config)# do wr me

Now we can setup the BGP neighbours:

debtestvm(config)# router bgp 65002
debtestvm(config-router)# neighbor 192.168.254.21 remote-as 65005
debtestvm(config-router)# neighbor 192.168.254.167 remote-as 65005

…and configure the route propogation:

redistribute connected / redistribute static tells BGP to advertise any directly connected, or any static routed networks, to its Peers.
default-originate tells BGP to advertise itself as a default route to its BGP Peers. We want this so that our Proxmox SDN VM’s have a route to the internet and the rest of the network.
soft-reconfiguration inbound explained better than I understand here -> BGP Soft Reconfiguration

debtestvm(config-router)# address-family ipv4 unicaset
debtestvm(config-router-af)# network 10.100.0.0/16
debtestvm(config-router-af)# redistribute connected
debtestvm(config-router-af)# redistribute static
debtestvm(config-router-af)# neighbor 192.168.254.21 prefix-list PVE-IN in
debtestvm(config-router-af)# neighbor 192.168.254.167 prefix-list PVE-IN in
debtestvm(config-router-af)# neighbor 192.168.254.21 default-originate
debtestvm(config-router-af)# neighbor 192.168.254.167 default-originate
debtestvm(config-router-af)# neighbor 192.168.254.21 soft-reconfiguration inbound
debtestvm(config-router-af)# neighbor 192.168.254.167 soft-reconfiguration inbound
debtestvm(config-router-af)# do wr me