Skip to content

colo-rr — Colo core / route reflector

The two colo core routers. Worked example: CR-COLO-01 (CR-COLO-02 mirrors it).
Model 2× CCR2004-1G-12S+2XS, RouterOS 7.19+
NetBox role / platform colo-rr / routeros, status active
Ansible playbook colo-rr.yml (roles: backup, baseline, bgp_instance, transport, rr, static_routes, venue_handoff, vpls hub)
OOB 172.16.201.22/24 on ether1

Function

  • OSPF / LDP / MPLS transport.
  • Two independent BGP route reflectors (cluster-id = own loopback), address families ip + l2vpn.
  • eBGP to the edge firewall (AS 65510) and the Starlink gateway (AS 65502).
  • VPLS hub (guest + POP).
  • Venue hand-off for colo-cabled venues.

Interfaces (CR-COLO-01)

Port Role Address / peer
loopback router-id, iBGP source 10.255.255.1/32
sfp28-1 inter-core bond A (25G) 10.254.96.0/31 → CR-COLO-02, OSPF cost 4, +LDP+BFD+MPLS, L2MTU 9000
sfp28-2 inter-core bond B (25G, ECMP) 10.254.96.2/31
sfp-sfpplus1/2 edge (eBGP) → OPNsense fw-colo-1/2 /31
sfp-sfpplus3 eBGP-peer 10.254.96.12/31 → CR-STARLINK-1 (from-starlink / to-starlink)
sfp-sfpplus5 venue-handoff 10.254.97.0/31 → rtr-lvl-1
sfp-sfpplus6 core-link → CR-COLOROOF-01
sfp-sfpplus7 guest-trunk → guest FW (VLAN per venue, guest-ho-<tri> = 2000+venue_id)
sfp-sfpplus8–11 venue-handoff colo-fed venues
ether1 OOB 172.16.201.22/24

No direct server-switch link

The CCRs have no link to the VM/server switches. Server traffic goes venue → CCR → OPNsense (eBGP) → VM switch → server.

BGP

  • as=65500 router-id=10.255.255.1, cluster-id = own loopback (two independent RRs). The as/router-id now live on the BGP instance (split off the template/connection in RouterOS 7.20+; per-connection local.role=ibgp-rr enables reflection) — validated on 7.23.
  • Edge FW peer (config context): remote_as=65510, 10.254.96.4 ↔ .5.
  • Default aggregate 10.128.0.0/16; originates 0.0.0.0/0 conditional on the edge session being up.
  • iBGP is BFD-free (RR→client and RR↔RR): these sessions are loopback-to-loopback (multihop), so they carry no BFD (use_bfd: False). Enabling it makes RouterOS reject the session ("BFD forbidden for destination address") and the peer hangs unestablished; liveness comes from single-hop link BFD + OSPF loopback withdrawal
  • BGP hold timers. See iBGP & route reflectors.
  • iBGP uses next-hop-self: the RR's own transport connection to the peer RR sets nexthop-choice=force-self, and PE-originated routes arrive already carrying the originating PE's loopback (the PE applied force-self). Either way eBGP-learned venue/edge prefixes have an OSPF-resolvable loopback next-hop and install active on the RR rather than inactive.
  • VPLS hub site-id 1000 on both colos → multihoming / DF election (the RD differs per box, so the same site-id is advertised twice).