Skip to content

1. Prerequisites

Before touching the GUI, have these ready. Getting the physical and addressing plan right first is what makes the rest a checklist rather than a debugging session.

1.1 Hardware & cabling (per node ×2)

Confirm each node is built and cabled per OPNSENSE.md §2:

  • [ ] 8× SFP+, 2× 1G RJ45 (i350), separate BMC/IPMI port.
  • [ ] Each redundant pair split across two different NIC cards (§1) — one card failure degrades, never halves both legs.
  • [ ] 1× USB NIC for OOB (see below), rear USB3 port, no hub.
Role Port Peer MTU
Core A (eBGP) SFP+ (combo) CR-COLO-01 (/31) 1500
Core B (eBGP) SFP+ (X710 #1) CR-COLO-02 (/31) 1500
Server LACP a SFP+ (X710 #2) VM-switch-A (MLAG) 9000
Server LACP b SFP+ (X710 #3) VM-switch-B (MLAG) 9000
WAN → bdr-1 SFP+ (combo) border-1 (/31) 1500
WAN → bdr-2 SFP+ (X710 #1) border-2 (/31) 1500
pfSync a/b RJ45 ×2 (i350) direct Cat6 to peer node — no switch 1500
OOB USB NIC OOB CRS326 switch 1500
IPMI/BMC BMC port OOB CRS326 switch 1500

The USB OOB NIC is a deliberate departure from OPNSENSE.md §7

§2/§7 originally spent both RJ45 on pfSync and declared "no dedicated OOB NIC". We add a USB NIC so pfSync keeps its redundant bond and you get a dedicated OOB port. Its one caveat: USB NICs enumerate as ue0/ue1, and a replug/bus reset can re-enumerate and silently break the assignment. One NIC per node, rear USB3, no hub. The BMC remains the true break-glass if the USB NIC ever drops — OOB via USB is a convenience path, not the last line. Prefer an ASIX AX88179 (axge) or Realtek RTL8153 (ure) — both solid on FreeBSD 14 / OPNsense 25.x.

1.2 Diverse power & path

  • [ ] FW-01 and FW-02 on different PDUs/feeds.
  • [ ] Each node's two core links reach the two CCRs over diverse cabling (§8).

1.3 Images & access

  • [ ] OPNsense installer (current stable, 25.x) on USB for each node.
  • [ ] BMC/KVM reachable on the OOB switch for both nodes (console during install).
  • [ ] The Site values table filled in — at minimum all the DECIDED rows, and ideally the CHOOSE rows settled.

1.4 Upstream/peer coordination (do this early — it has lead time)

These are other people's config and gate your cutover:

  • [ ] Colo CCRs (CR-COLO-01/02) configured to peer eBGP with both nodes' core /31 addresses (AS <OPN_AS>), BFD enabled. See net-design/example-configs/target-colo-rr.rsc.
  • [ ] WAN borders (bdr-1/2) configured to peer with both nodes' WAN /31 addresses; they send default only, accept only <PUBLIC_BLOCK>.
  • [ ] RPKI ROAs + IRR objects published for <PUBLIC_BLOCK> (origin = the announcing AS) before cutover, or upstreams filter you (§5c).
  • [ ] VM CRS326 MLAG pair has a LACP bond configured facing each node (lacp-rate 1sec, layer-3-and-4 hash, MTU 9000) — must match §3 settings.
  • [ ] OOB gateway gw-oob-1 source-NATs VPN clients into <OOB_NET> — this is what makes OOB "no routing" work (DESIGN §7.3). Confirm it exists.

Why coordinate BGP peers now

OPNsense uses CARP-gated FRR (§3): the borders/CCRs have neighbour config for both nodes waiting, but only the master's sessions ever come up. If the far ends aren't pre-configured for both nodes, failover won't converge. Get them staged before you reach page 5.