Skip to content

Physical Layer

Fibre, patching and transceivers between the metro sites and the colo, plus the NIC firmware practices for the edge/hypervisor cards.

Fibre & patching

The colo migration pulls existing venue and metro fibres into a new colo room via two feeds:

  • Old server room (venue fibres): 7 venue fibres × 2 cores = 14 cores now, ~24 target → pull a 24-core (48 for growth).
  • Meet-me room (metro + WAN transit): Mathew St, Charlotte, Seel, Holmes A+B, Old Bank, the WAN primary and the backup internet = 16 cores → ~20 → pull a 24-core.

Order: 2× 24-core single-mode (48 cores), or collapse to ~8 with in-room aggregation switches. The WAN → OPNsense hand-off is an in-room SFP+ DAC (0 inter-room cores). All plant is single-mode; BiDi optics use 1 core, duplex use 2.

Field-walk TODOs

  • Optic type (BiDi vs duplex) per run — the Type column in FIBRE-PATCH.md is the walk list.
  • Whether roof-to-rack runs are fibre or Cat6.
  • Reconcile SOHO dual-homing (a possibly-stale comment).
  • Whether the WAN hand-off is re-terminated or gets a new cross-connect.
  • Diverse-path cores for Boston/Seel/Charlotte/WAN.

Transceivers & inter-core

  • Inter-core: 2× 25G on the CCR SFP28 (2XS) ports, bonded to 50G, via SFP28 DAC.
  • Fibre infra: L2MTU ≥ 1600 (fibre supports 9000 — used on transport links).
  • CR-COLOROOF-01 can run a 2.5G port to a CCR via an RJ45 SFP, or a 2×SFP+ roof aggregator.

Intel X710 NICs

The edge firewalls and hypervisors use Intel 700-series (X710/XL710) cards. INTEL-X710.md is the firmware/crossflash runbook (run from a Linux live image over iDRAC).

  • Firmware string: ethtool -i reports NVM 0x<ETrackID> <FW/API> (e.g. 9.20 0x8000d969 22.0.9). Both ports of a card share one flash → identical firmware; a mismatch across a card's ports signals a counterfeit or bad flash.
  • PCI IDs: 8086:1572 = X710 10GbE SFP+; subsystem 1137 = Cisco, 1028 = Dell, 8086 = retail. Driver i40e (Linux) / ixl (FreeBSD).
  • Cisco/Dell crossflash: the Intel NVM tool refuses OEM cards — edit each step's .cfg REPLACES: to the card's current ETrackID, and ladder by ETrackID chain, not version number (8.15 is older than 8.50), cold power-cycling between every rung.
  • SFP+/DAC unlock: xl710-unlocker clears the module-auth bit in NVM (persists). Flash first, unlock last, and never run nvmupdate64e -rd on an unlocked card (it re-locks).
  • Pre-5.05 firmware has a security vuln (5.05 is the minimum fix); NVM ≥ 6.01 + driver ≥ 22.6 recommended. Disable the firmware LLDP agent (ethtool --set-priv-flags <iface> disable-fw-lldp on) or it breaks LLDP discovery / PXE. The MAC/VLAN-filter erratum on NVM 8.40+ affects the 10GBASE-T variants, not the SFP+ DA2 (1572) used here.