Skip to content

WAN & Internet Edge

The corporate internet edge terminates on the OPNsense pair at the colo, with a Starlink link reserved as a payments-only backup. A separate retail ISP business also lands at the colo but is L3-isolated from the corporate network.

Corporate WAN edge

  • ISP: a 10G link with routed public subnets. The BGP option is used (not static-to-VIP), run from both OPNsense nodes via FRR — announce the routed public subnets, accept a default (or default + partials).
  • OPNsense pair: CARP for stateful services (NAT, captive-portal VIP); eBGP AS 65510 → both colo CCRs. OPNsense originates 0.0.0.0/0, the public subnets and the server-room zone subnets; the CCRs export the 10/8 aggregate and venue prefixes.
  • Public routed subnets terminate on OPNsense; public-facing server-room hosts sit in a DMZ zone behind it.
  • RB4011 at the colo, eBGP AS 65502 to both colo CCRs.
  • In normal state it announces nothing (or payment-provider prefixes at low pref). On WAN failure, payment / critical prefixes become the best path via Starlink.
  • Deliberately no default route via Starlink — bulk traffic can never fail over to it; only whitelisted payment prefixes do.

The default route is emergent

Nothing statically points at a backup. When the WAN dies, OPNsense withdraws 0.0.0.0/0, the RRs stop originating it, and it disappears fleet-wide. Bulk traffic blackholes; payment traffic survives because the Starlink route to the payment prefixes is still present. Test this quarterly.

The Starlink box currently peers to OPNsense only — the target moves it to peer directly with the CCRs so the backup survives an OPNsense failure.

Retail ISP / WAN business (separate)

This is a different business that shares the metal but not the routing. It is documented here only where it touches the corporate metro.

Fact Value
ASN 204258
IPv4 185.109.40.0/22
IPv6 2a06:4e80::/29
Internal (never announced) 100.64.0.0/16
Upstream AS 51945 (ConnetU, "via NoOne") on both transits — path diversity, not provider diversity (accepted risk)
  • Edge: bdr-1 (new CCR2004) + bdr-2 (re-used CR-LEVEL-04 CCR2004), iBGP AS 204258, 2×25G bonded (LACP), next-hop-self.
  • BNG: bng-1 (CCR1016), private AS 65520 eBGP to both borders; ~20 subscribers at 100–200 Mb, PPPoE per POP (max-mtu/max-mru 1500, RFC 4638).
  • Transit A: 10G at the colo → bdr-1. Transit B: 1G, enters the Mathew St hub and is delivered to the colo as an L2 circuit over the metro VPLS → bdr-2.
  • The metro is only an L2 circuit supplier to this business — there is no BNG in the metro routing/addressing/Terraform. POP subscriber ports ride per-POP service VLANs bridged into vpls-pop<n>, dual-handed at the colo with same-site-id VPLS multihoming.

Before cutover

  • RPKI ROAs for 185.109.40.0/22 and 2a06:4e80::/29.
  • Drop the 185.109.40.0/24 more-specific (it currently pulls all inbound through the 1G Transit B) and switch prepend policy ×1 → ×2.
  • Confirm the IPv6 transit /127 addressing with the upstream (the legacy CERs use per-circuit /64s today).
  • WAN-EXISTING.md §4.2/§4.4 contain live PPPoE and WireGuard credentials — excluded from these docs; rotate on migration.