When configuring NSX, BGP and ECMP there are a few configuration requirements you need to keep in mind:
BGP neighbors
ESG Firewall must be disabled
BGP Timers
BGP Graceful Restart
Static Routes on the ESGs
Static Routes on the DLR
Anti-Affinity Rules
iBGP vs eBGP
BGP neighbors
Equal Cost Multi-Pathing allows a router to choose from multiple paths to forward packets to. So a DLR with four ESGs configured with ECMP will see four paths it can spread traffic across.
The DLR will see routes for all possible paths for each ESG.
B 0.0.0.0/0 [20/0] via 10.10.10.10 B 0.0.0.0/0 [20/0] via 10.10.10.10 B 0.0.0.0/0 [20/0] via 10.10.10.10 B 0.0.0.0/0 [20/0] via 10.10.10.10 B 0.0.0.0/0 [20/0] via 10.10.10.11 B 0.0.0.0/0 [20/0] via 10.10.10.11 B 0.0.0.0/0 [20/0] via 10.10.10.11 B 0.0.0.0/0 [20/0] via 10.10.10.11
You will need to create Neighbors for each adjacent IP Pair – in the example above each ESG sees each router twice, which requires a neighbor configuration for each of the router IPs the ESG can see.
If the ESG has only one neighbor configured for a router it can access with two IPs, the connection might not become established as the router could use either IP to establish the neighbor adjacency.
ESG Firewall must be disabled
When ECMP is enabled on an Edge you must disable the firewall on that Edge. Previous version (<6.1.2 I believe) disabled the ESG firewall when ECMP was enabled.
If you have four ESGs and have the firewalls enabled even with a default allow, only one of the ESGs will pass traffic.
BGP Graceful Restart
If you’re using BGP with ECMP you need to disable Graceful Restart or the BGP timers you just set won’t do any good – you’ll have a ~2 minute / 125 second failover.
BGP Timers
Best practice is to reduce the BGP keep alive and hold down timers from their default. You can find VMware resources suggesting 1 second/3 seconds (especially between DLR/ESG) but you’ll want to decide the trade-offs between possible route flapping and time-to-fail for your environment. As you’ll want to set the same value for each side of a BGP neighbor pair, you should check with your hardware vendor for their recommendation also.
Static Routes on the ESGs
If the Control VM fails over or is redeployed, when the ESGs see it drop they will remove all routes to the networks behind the DLR. Creating a static route on the ESGs will prevent lost of connectivity in the event the Control VM goes offline.
Setting a weight/distance of 61 (or whatever your neighbor-configured weight is plus “1”) will ensure the static route is only used if the redistributed routes are unavailable.
Static Routes on the DLR
Don’t create static routes to the ESGs on the DLR or in the event of a ESG+Control VM failure (like both are on the same host) the DLR on the hosts can keep routing traffic to the dead ESG.
Anti-Affinity Rules
Rules should be created to ensure ESGs and Control VMs don’t run on the same hosts. This will prevent losing multiple ESGs at once and will prevent losing a ESG and Control VM at the same time, which will result in hosts still sending traffic to the down ESG.
Note that you’ll need to either delete the NSX-created control-VM-HA rule and add those VMs to the new rule, or add your ESGs to the NSX-created rule. You’ll also need to check it periodically as redeploying Edges will rename them and break your rules.
iBGP vs eBGP
BGP between different AS areas is called eBGP, BGP between routers with the same AS area is iBGP. The options are pretty limited for NSX so the differences are minimal, but now you know what your network guy is talking about.
