Skip to content

Move Plan — Old Network → New Core

Version: 1.0 — 2026-07-06 Sequences the physical and logical migration from the as-built network (DISCOVERY.md) to the target design (DESIGN.md). Strategy per agreed approach: interconnect old and new racks → move WAN → re-home links backups-first → decommission. IP network first (Stage 1); guest VPLS + PPPoE follow later (Stage 2, CONFIG-GUIDE §9) and appear here only as constraints.

Golden rule for every link move: the new core port adopts the old core-side /30 address. The far end (venue/metro router) needs zero config in the move window — light comes back, its existing static default still points at the same gateway IP, traffic flows. Routing modernisation (eBGP conversion) happens afterwards, remotely, per device.

1. Device fates

Device As-built Fate
CR-LEVEL-01 (CCR1016, ROS6) WAN NAT + public IPs + main hub Decommission — WAN → OPNsense; links → CR-COLO-01/02
CR-LEVEL-02 (CCR2004, ROS 6.48) Second hub (Seel/Charlotte/Boston/L03/L04 + 3 venues) Re-use as CR-HOLMES-01 — once drained (uplinks D2#1–3, venues D3), wipe → netinstall ROS 7.19 → target config (loopback .22) → replaces the Boston CCR1009 in the D2#4 window. No new CCR2004 purchase for Holmes
CR-LEVEL-03 (RB5009, 7.16) Links Charlotte-01, LEVEL-02, Old Bank roof, Irish House Re-role → CR-COLOROOF-01 (colo roof) — needs loopback assigned (has none!)
CR-LEVEL-04 (CCR2004, 7.16) Only one uplink (LEVEL-02); WireGuard hub (12 peers), public /30, gre-bolton Re-role as bdr-2 (WAN business border router 2 — WAN-DESIGN §4; not yet purchased otherwise). Blockers first: WireGuard peers → OPNsense (WIREGUARD.md §4), gre-bolton confirmed dead. Then wipe → ROS 7.19 → bdr-2 (AS 204258, loopback 100.64.0.2); its 2×25G XS ports become the bdr-1↔bdr-2 bond. .233 loopback returns to the pool
CR-BOSTON-01 (CCR1009, ROS 6.48) Holmes hub, dual fibre to LEVEL-01+02 Replace in place with the re-used LEVEL-02 CCR2004 (→ CR-HOLMES-01); CCR1009 retired to spares
CR-SEELSTREET-01 (CCR2004, 7.19.6) Seel hub, 8 venues Keep — re-home uplink; add 60 GHz to Holmes later
CR-CHARLOTTESTREET-002 (CCR2004, 7.19.4) New Charlotte hub Keep — becomes the only Charlotte router
CR-CHARLOTTESTREET-01 (RB5009UPr+S+) Already the Charlotte roof RB5009 — PoE, powers the dishes; carries venue CAT6 backups (ROK/NFL/RBL), colo-roof dish, Irish House 2nd dish, LHGG segment. No link to -002 today — roof island uplinks only via 60 GHz Keep — it is the target Charlotte roof PE (later rename CR-CHARLOTTEROOF-01, renumber into .50–.79). Build the 01↔02 link by repurposing one venue's CAT6 run — closes the site ring (colo→fibre→-002→roof→60 GHz→colo roof), so a fibre cut is survived at transport level by the whole site. The donor venue becomes single-homed to -002 (acceptable: the ring now covers it for fibre cuts; pick a non-POS-critical venue, and NOT Celtic Corner, which already loses its Celtic-roof path). Link config: OSPF p2p cost 100, L2MTU ≥1600, LDP. Celtic-roof links (ether4/7) die at Celtic removal; LHGG segment to untangle; confirm the Temple Court 60 GHz (design expects it on this roof — not seen in discovery)
CR-CELTICROOF-02 (RB5009) Celtic roof; venue CEL (dual-homed already) Remove (CEL already has a path via Charlotte-002 sfp-sfpplus4)
CR-LORDSTREETROOF-01 Offline, loopback unreachable Remove — confirm RTR-LOR-001 (10.151.14.4/30 via LEVEL-01 sfpplus1.4) has/gets another path first
CR-MATHEWSTREET-01 (CCR2004, 7.19.3) Mathew hub via VLAN 601 L2 Keep — 601-swap (§4)
CR-OLDBANKROOF-01 (RB5009, 7.16) 60 GHz to LEVEL-01 + LEVEL-03; venues OLB, CAS Keep — re-terminate colo-side 60 GHz on CR-COLOROOF-01; fibre to Fenwick later
SERVER-ROUTER (RB4011, 10.255.255.249) Old server room gateway Decommission — replaced by new server-room fabric + OPNsense zones (services renumber to 10.128/16 in Phase 7)
RTR-BNG-001 (RB4011, ROS6) A venue, oddly attached via LEVEL-01 sfpplus1.4 Re-home as normal venue on plain WAN/venue port; not the future BNG
Venue routers (RTR-*) Mostly RB4011, ROS 7.1x–7.19 Keep — far end untouched during moves; eBGP conversion after; ROS6 stragglers (YNK 6.47, RBL 6.39, FUS/EIN/OLB/CAS/WDS/SOH…) upgraded to ROS7 at conversion

Known gaps (fill before their move windows): Temple Court and Fenwick St routers never appeared in discovery (they sit behind Mathew St, which we only have from export) — collect them once Mathew St's mgmt list admits us. Passwords unknown: CAS, LOR, MCC, BPL. 60 GHz antenna inventory (LEVEL_, ANT-) unverified.

2. Stage A — Build the new colo (no impact on live network)

  1. Rack + cable per DESIGN §5.1: CR-COLO-01/02 (CCR2004), 2×25G bonded inter-core, OPNsense pair (CARP), OOB CRS326-24G + gateway + console server (serial to everything), server-room CRS fabrics.
  2. Deploy config via Terraform (TERRAFORM-PLAN §8 step order): baseline + transport + RR modules. Loopbacks from registry (new: CR-COLO-01 .245? — assign from free space; do not reuse .255/.250 until their owners are dead).
  3. New core runs target routing from day one: OSPF area 0 between the pair, iBGP RR templates ready, input filters on, bgp-networks empty.
  4. Burn-in: 48h idle with monitoring/syslog attached; btest across the bond.

3. Stage B — Interconnect old ↔ new racks

  1. Run 2× fibre LEVEL rack ↔ colo: one to CR-COLO-01, one to CR-COLO-02.
  2. Each is a hybrid port: routed /30 (new allocations from 10.254.9.x free space) + tagged VLANs for L2 stretch (WAN hand-off VLAN, and anything else that must ride between racks during transition).
  3. Enable OSPF (cost 10) on both /30s: old LEVEL-01/02 ↔ new CR-COLO-01/02 become one OSPF domain; also add both new CCRs to the old iBGP mesh (two more peers on LEVEL-01/02 only — venues/metros don't change). New core now learns every prefix; old network learns nothing new (new core originates nothing yet).
  4. Verify: from CR-COLO-01, traceroute to a venue loopback in every hub (via old paths).

Rollback: disable the two OSPF interfaces. Nothing depends on them yet.

4. Stage C — Move the WAN (the 601-swap)

Current: upstream fibre lands at Mathew St; WAN + VLAN 601 (metro L2 to LEVEL-01 sfp2.601 ↔ CR-MATHEWSTREET-01 sfp-sfpplus1, 10.254.254.0/30) are delivered over it to LEVEL-01, which does NAT and holds the public IPs (185.109.x on the CCR1016).

Target: your fibre runs the metro natively (jumbo-capable); WAN rides it as a tagged VLAN to the colo OPNsense.

  1. Prep (remote): inventory everything on LEVEL-01 that touches WAN: NAT rules, public IP bindings, dst-nat forwards, the CR01-LIV0x upstream sessions on sfpplus1.x, Starlink policy, hotspot walled-garden IPs (185.109.40.8/9 — the portal lives in public space today). Recreate all of it, disabled, on OPNsense. We hold LEVEL-01's raw state in discovery/raw/ — do a fresh /export too.
  2. Parallel WAN path: upstream hand-off VLAN extended: Mathew St → (601-path) → LEVEL rack → (interconnect trunk VLAN) → OPNsense WAN port. OPNsense comes up with a secondary public IP; test outbound + BGP/static with upstream while LEVEL-01 still carries production.
  3. Cutover window (morning, ~1h): upstream moves the routed subnets/default to OPNsense (BGP announce or gateway swap); OPNsense NAT enabled; LEVEL-01 default route replaced by static toward CR-COLO (which now originates default from OPNsense); venue traffic hairpins old→new→OPNsense. Confirm: venue POS test transaction, hotspot login, public dst-nat services.
  4. Rollback: upstream re-points to LEVEL-01 VIP; re-enable its NAT. Keep the rollback path warm for a week.
  5. Physical 601-swap (separate window, after §5 Mathew re-home): when the Mathew St fibre is re-patched as your native link into CR-COLO-01, the WAN VLAN moves onto it and the upstream's L2 601 service is cancelled. MTU on the native fibre → 9092; verify with DF pings before any Stage-2 VPLS work.
  6. Starlink: RB4011 moves to the colo (eBGP to both CR-COLOs per design §2.4/§4.4) in this stage too — it's currently useless if LEVEL-01 dies.

Principle: move the backup path to the new core first; prove it; then move the primary while the proven backup carries traffic. New-core port adopts the old core-side /30 address in every case. One link per window; verify (far-end ping, OSPF/BGP state, POS traceroute) before the next.

D1 — 60 GHz backups → CR-COLOROOF-01 (ex-LEVEL-03)

# Link (as-built) Move New termination
1 Old Bank roof ether3 ↔ LEVEL-01 sfp11 (10.254.5.28/30) Colo-side antenna feed re-patches from LEVEL-01 to CR-COLOROOF-01 CR-COLOROOF-01 (keeps 10.254.5.29)
2 Old Bank roof ether1 ↔ LEVEL-03 ether6 (10.254.8.24/30) Already on the RB5009 being re-roled — renumber identity/loopback only stays
3 Charlotte roof chain (Charlotte-01 ether4/ether7 ↔ Celtic roof) Do not move — dies with Celtic roof removal (§6) after Charlotte venues re-home —
4 Irish House 60 GHz (10.254.8.28/30 on LEVEL-03) Stays on CR-COLOROOF-01 stays

Also in D1: give CR-COLOROOF-01 and CR-LEVEL-04 loopbacks (neither has one), upgrade CR-COLOROOF-01 to 7.19.x, wire it to both CR-COLO CCRs per design.

D2 — Metro hub fibres (one hub per window; hub's 60 GHz backup must be green first)

# Hub As-built port New port /30 carried over Notes
1 Seel St LEVEL-02 sfp-sfpplus8 CR-COLO-02 10.254.70.0/30 First — cleanest hub, ROS7 already. Backup: a Boston↔Seel 60 GHz dish pair already exists but is offline (creds/WIRELESS-60G) — revive it before this move and Seel is no longer single-homed. Investigate why it's dormant first.
2 Charlotte LEVEL-02 sfp-sfpplus9 CR-COLO-01 10.254.70.4/30 Backup via Celtic-roof chain still alive at this point
3 Holmes/Boston link A LEVEL-02 sfp-sfpplus2 CR-COLO-02 10.254.8.12/30 Move A while B carries
4 Holmes/Boston link B + router swap LEVEL-01 sfp5 CR-COLO-01 10.254.8.4/30 Same window: CCR1009 → re-used LEVEL-02 CCR2004 (CR-HOLMES-01, ROS 7.19, target config, pre-staged); venues MCC/BPL(/BLR) re-patch like-for-like, /30s preserved. Dependency: LEVEL-02 must be fully drained first (D2#1–3 + its D3 venues KEL/WDS/SOH) — so this is the LAST D2 window
5 Mathew St (fibre swap) LEVEL-01 sfp2.601 CR-COLO-01 (native, jumbo) 10.254.254.0/30 The 601-swap physical step (§4.5). Backup for its venues = Temple Court 60 GHz — verify Temple Court state first (discovery gap)
6 LEVEL-04 LEVEL-02 sfp-sfpplus7 (link decommissions) 10.254.9.4/30 Resolved: LEVEL-04 → bdr-2 (WAN business). No re-home — after its WireGuard peers move to OPNsense, the box is pulled, wiped and rebuilt as bdr-2; this /30 dies with it

D3 — Colo-fed venue fibres (LEVEL-01/02 SFP ports → CR-COLO-01/02)

Batch 2–3 per morning window; far end untouched; new port adopts the old address. Port assignments per DESIGN §5.1 budget (new venues default to CR-COLO-02).

Venue As-built /30 Venue router (state)
Fusion LEVEL-01 sfp1 10.254.4.4/30 RTR-FUS CCR1009 ROS6 — needs ROS7/refresh at eBGP conversion
Yankees (YNK) LEVEL-01 sfp3 10.254.4.0/30 RTR-YNK 6.47
Einstein LEVEL-01 sfp4 10.254.4.12/30 RTR-EIN CCR1009 ROS6
LEVEL venue LEVEL-01 sfp9 10.254.0.20/30 RTR-LVL 7.11
Kells LEVEL-02 sfp-sfpplus3 10.254.5.64/30 RTR-KEL 7.15
Wood Street (WDS) LEVEL-02 sfp-sfpplus4 10.254.7.0/30 RTR-WDS ROS6
SOHO LEVEL-02 sfp-sfpplus5 10.254.4.8/30 RTR-SOH 6.44
Lord St venue LEVEL-01 sfpplus1.4 (L2 via upstream!) 10.151.14.4/30 RTR-LOR — rides an upstream L2 leg; needs its own path decided before Lord St roof removal
RTR-BNG-001 (venue) LEVEL-01 sfpplus1.4 10.254.6.8/30 re-home to plain venue port / WAN per its needs
Cheers (RTR-CHR) — currently a flat 10.222.2.0/24 on the Vive trunk LEVEL-01 sfpplus1.6 n/a (flat L2) untangle: give it a proper /30 + port at re-home
SERVER-ROUTER LEVEL-01 sfp8 10.199.199.0/30 stays until Phase 7 server migration, then decom

D4 — Routing modernisation (remote, after each site's cable move)

Per venue, in any order, no site visit: ROS7 if needed → eBGP CE conversion (CONFIG-GUIDE §2.3b: blackhole anchor, sessions to metro PE/colo, default-only in-filter, BFD) → remove its static routes from old and new cores → Terraform import + venue module (TERRAFORM-PLAN). When the last venue /16 static is gone from the new core, enable RR default-originate and drop the temporary mesh sessions to LEVEL-01/02.

6. Stage E — Decommission (only when its column is empty)

Order: Celtic roof (after Charlotte venues confirmed on -002 paths; CEL loses one of two uplinks — verify its Charlotte-002 path first) → Lord St roof (already offline; confirm RTR-LOR re-homed) → Boston CCR1009 (leaves in D2#4) → LEVEL-02 (drained, re-deployed as CR-HOLMES-01) → LEVEL-04 (WireGuard migrated, re-deployed as bdr-2) → LEVEL-01 (last: it holds the public IPs until §4 is proven stable) → SERVER-ROUTER (after Phase 7 renumbering) → cancel upstream 601 L2 service. (CR-CHARLOTTESTREET-01 is NOT decommissioned — it stays as the Charlotte roof PE.)

For each: final /export archived, licence/hardware recorded, loopback returned to registry, monitoring/Oxidized/NetBox updated, old iBGP mesh entries removed fleet-wide (Terraform makes this a one-file change once the registry drives peers).

7. Window discipline & verification

  • Windows: venue-impacting moves Mon–Thu mornings; WAN cutover Sunday early AM; never Fri/Sat (trade nights).
  • Before any move: fresh /export of both ends; backup path proven by disabling primary for 60s (where one exists); rollback = re-patch, old port config left disabled-not-deleted for 1 week.
  • After every move: far-end loopback ping; OSPF neighbour count restored; routes_summary count matches pre-move; POS venue: test transaction; hotspot venue: portal login.
  • Stage gates: C complete before D2 (WAN must not depend on LEVEL-01 while its links drain); D2 hub complete before that hub's D3 venues; D4 fleet-complete + 601 native MTU verified = Stage 1 done → Stage 2 (VPLS/PPPoE) may begin.

8. Open decisions / actions for Lewis

  1. ~~CR-LEVEL-04's role~~ Resolved: LEVEL-04 → bdr-2; LEVEL-02 → CR-HOLMES-01. Remaining action: schedule the WireGuard migration (blocks bdr-2) and drain LEVEL-02 before the Holmes swap.
  2. Seel St ordering — the Seel↔Holmes backup dish already exists but is offline; revive it (find why it's dormant) before moving Seel, rather than "build new" or "accept risk". Cheapest of the three options.
  3. Confirm Temple Court / Fenwick reachability + mgmt access (blocks D2#5 verification).
  4. Passwords for CAS/LOR/MCC/BPL; mgmt-list fix for the nine firewalled venue routers (can be done at their D4 conversion).
  5. New loopback assignments: CR-COLO-01/02, CR-COLOROOF-01, CR-HOLMES-01 — allocate in registry.
  6. Rotate the legacy credential everywhere it survives (it's burned).