Information¶
Pub Invest Group have a Mikrotik network for a core/metro/office design.
There is an OPNSense edge firewall pair in HA, with 8x 10GB SFP+ ports. WAN is 10GB link with some routed subnets, we can do BGP to the ISP or static routes to a HA VIP.
I am re-organising the network, with a new server room. Extending fibres, new fibres, new routers etc.
I would like a full specification and design for what the end goal is.
Network Definition¶
We run a network on 10.0.0.0/8.
Each hub for the metro has at least one fibre connection, and a 60Ghz conenction. There can be multiple routers on a hub site.
From the metro, each venue has a /30 link from at least one metro router, and sometimes a secondary metro router.
A venue router has a subnet of 10.x.0.0/16 assigned, with a VLAN for different things, such as POS, Office/Corporate, Network managemetn (unifi), Staff untrust wifi and Guest wifi.
The goal for Guest Wifi is captive portal. Either on the mikrotik itself on the venue, or a VPLS/MPLS over BGP to the OPNSENSE pair with Captive on there.
Within a venue we don't allow inter-VLAN traffic, and then firewall what the site can get to on the server room networks. (eg POS to POS servers/AD servers, Unifi to controller)
THe server room has multiple VLANs for OOB, Backup, ISCSI/Data/Ceph, VM Traffic (with a sensitivity scope on multple VLANs)
Ideally as much zero trust on the design.
Sites and links¶
The below sites are what we have
Hubs - Colocation Facility (18-20 Fleet Street) - Mathew St Metro Rack - Charlotte Street Metro Rack - Seel Street Metro Rack - Temple Court Metro Rack - Old Bank Metro Roof - Fenwick Street Metro Rack - Holmes Building Metro Rack
Colocation Facility¶
This site contains:
- ccr2004-1g-12s+2xs - x2 routers
- CRS326-24S+2Q+RM - x2 for ceph, x2 for vm traffic
- on roof RB5009 connected to ccr2004
- opnsense hosts with 8x10GB SFP+ per node
WAN is fed from here for all network. We also have a Starlink for backup, that is connected to an RB4011, which is then connected with BGP to OPNSENSE and we can then announce routes that will go via starlink, eg payment services when wan issues. using routes to not use default route to stop bulk traffic.
This site serves the following venues: - LEVEL Night Club (10GB Fibre) - SOHO (10GB Fibre) - Fusion (10GB Fibre) - Kells of Eden (10GB Fibre) - Electrik (10GB Fibre) - Diego Demise (10GB Fibre) - Playbox (10GB Fibre) - Cheers (10GB Fibre) - Einstein Bier Haus (10GB Fibre) - Irish House (60ghz roof link)
It has metro links to - Holmes Building - Mathew street - Seel Street - Charlotte Street
on roof we have - Irish house 60ghz - Backup 60ghz to Charlotte St RB5009
Mathew St Metro Rack¶
This site has a 10GB L2 link from Colo facility, with up to 9000 MTU if needed. (upstream run a fibre link and WAN links are here too)
- ccr2004-1g-12s+2xs x1 router
This site then serves the venues: - Sgt Peppers - Rubber Soul - McCooleys - Revolver - Legends - Temple Tavern
The above sites also have 60Ghz backup to the Temple Court metro as backup
This site also takes in the metro fibre from Fenwick Street
Charlotte Street metro¶
This site has a 10GB Fibre back to Colo facility. It also has a CAT6 to an RB5009 on the roof, with a 60Ghz to the Temple Court metro RB5009.
- ccr2004-1g-12s+2xs x1 router
Venues served from here: - Rocking Horse - Nelly Foleys - Celtic Corner - Smokies - Ruby Blues
Each site has a CAT6 to roof RB5009 too as backup.
Seel Street Metro¶
This site has a 10GB Fibre back to colo facility.
- ccr2004-1g-12s+2xs x1 router
Venues served from here: - McNastys - Lago - Crooked Crown - Moloko - Brooklyn Mixer - Dirty o Sheas
This site can, when re-worked have a 60GHZ backup link via a 60Ghz to Holmes Building metro.
Temple Court Metro¶
This site has a 10GB Fibre to Mathew St Metro rack and 60Ghz to Charlotte St (via the RB5009 roof)
It has 60Ghz links for primary sites - 27 Stanley St - 1 Stanley St - 42 Stanley St - 48 Stanley St
and backup 60ghz for - Sgt Peppers - Rubber Soul - McCooleys MS - Revolver - Legends - Temple Tavern
Old Bank Metro¶
This is an RB5009 on the roof, with 2x 60Ghz links to Colo facility. It will soon have a 10GB Fibre from here to Fenwick street core too.
- RB5009 POE 10 port
Venues served - Old Bank bar - Castle St apartments
Fenwick street metro¶
This site has a 10GB Fibre to Mathew St Metro rack and 60Ghz to LEVEL (via the Old Bank roof RB5009 metro)
- ccr2004-1g-12s+2xs x1 router
Venues served - Old bank 2nd link - McSwiggins
Holmes building metro¶
This site has 10GB Link to Colo facility x2 with diverse paths.
- ccr2004-1g-12s+2xs x1 router
Venues served:
- McCooleys Concert Square
- Black Rabbit Shot Co
- Boston Pool Loft
Network info¶
We run currently OSPF + BGP on the metro core, with a 10.255.255.x loopback for each device. Each site has a 10.254.y.0/24 for /30s to assign. All IPv4
We also want to bring in a second network we have where we sell intenret and give them a public IP via PPPoE over the network. Do we do that with a VPLS and break out to the equipment on a VLAN per site where needed? or another way?
The core will do all routing, no firewalling. Just input firewall to metro router.
Example configs¶
In example-configs there are configs we have now on some routers.