Free NSX books from VMware

VMware NSX Micro-segmentation: Day 1 Guide

VMware NSX Micro-segmentation: Day 2 Guide

VMware Operationalizing NSX

Automating NSX for vSphere with PowerNSX

Posted in Network, Scripting, Security, Virtualization, VMware | Tagged , | 1 Comment

VMware badges – 2017 edition (vROps / vSAN)

[Edit: Added time/#Qs to vSAN after someone tried it and responded to me]

VMware certification has announced a series of “badges” that existing VCPs can add to demonstrate knowledge in either vROps or vSAN.

vSAN was announced last week

Right now the portal claims the exams are only available during VMworld US (Aug 27-29). My guess would be the price will go up after VMworld with a new date range.

Key points:
VCP required
$250 for vSAN or $125 at VMworld
$125 for vROps until 8/29 then ??

  • (note: I was told vROps has a 40% discount for the next few months, but that is not reflected at checkout, so YMMV)

at a Pearson center for vSAN
online for vROps

vSAN time: 110min plus 30min time extension for ESL.
vSAN quesitons: 60

vROps: The number of questions and time allowed are not currently posted. if anyone pays the $125 and finds out please let me know!

Note that vSAN claims a “high score” is required to pass but generally

Prep Guides (with sample questions and outline)
VMware vRealize Operations 2017 Specialist

VMware vSAN 2017 Specialist

If you are interested in taking an exam just as expensive and time consuming as a VCP that only counts as a “badge” let me know in the comments.

Posted in Certification, Virtualization, VMware | Tagged | 3 Comments

Get your VMware certification URL and PDFs

After all that hard work, it’s time to show off.  You can obtain PDFs of your VMware certifications or a URL listing all of them in the VMware Certification Manager.

The URL will look like and can be send to prospective employers, or added to LinkedIn as a verification of your skill set.

URL steps:
Step 1:
Login to VMware Education.

Step 2:
Click on the Certification Manger link.

Step 3:
Click Manage your Transcripts.

Step 4:
If you don’t have a transcript listed, create a new one.

Step 4a:
Leave all the defaults and enter an expiration for the transcript.  Note that the URL won’t work after this date.


Step 5:
Click the Link icon and copy the URL listed.

Step 6:


PDF Steps:

Step 1:
Login to VMware Education.

Step 2:
Click on the Certification Manger link.

Step 3:
Click Track your certification status.

Step 4:
Each certification (note that VCIX is a “badge” and doesn’t have a PDF) will have a PDF link next to it.

Step 5:
Print and hang on the fridge for all to see.

Posted in Certification, VMware | Tagged | Leave a comment

NSX, BGP, ECMP quick hits

When configuring NSX, BGP and ECMP there are a few configuration requirements you need to keep in mind:

BGP neighbors
ESG Firewall must be disabled
BGP Timers
BGP Graceful Restart
Static Routes on the ESGs
Static Routes on the DLR

Anti-Affinity Rules
iBGP vs eBGP

BGP neighbors
Equal Cost Multi-Pathing allows a router to choose from multiple paths to forward packets to.  So a DLR with four ESGs configured with ECMP will see four paths it can spread traffic across.

The DLR will see routes for all possible paths for each ESG.

B [20/0] via
B [20/0] via
B [20/0] via
B [20/0] via
B [20/0] via
B [20/0] via
B [20/0] via
B [20/0] via

You will need to create Neighbors for each adjacent IP Pair – in the example above each ESG sees each router twice, which requires a neighbor configuration for each of the router IPs the ESG can see.

If the ESG has only one neighbor configured for a router it can access with two IPs, the connection might not become established as the router could use either IP to establish the neighbor adjacency.

ESG Firewall must be disabled

When ECMP is enabled on an Edge you must disable the firewall on that Edge.  Previous version (<6.1.2 I believe) disabled the ESG firewall when ECMP was enabled.

If you have four ESGs and have the firewalls enabled even with a default allow, only one of the ESGs will pass traffic.

BGP Graceful Restart
If you’re using BGP with ECMP you need to disable Graceful Restart or the BGP timers you just set won’t do any good – you’ll have a ~2 minute / 125 second failover.

BGP Timers
Best practice is to reduce the BGP keep alive and hold down timers from their default.  You can find VMware resources suggesting 1 second/3 seconds (especially between DLR/ESG) but you’ll want to decide the trade-offs between possible route flapping and time-to-fail for your environment.  As you’ll want to set the same value for each side of a BGP neighbor pair, you should check with your hardware vendor for their recommendation also.

Static Routes on the ESGs
If the Control VM fails over or is redeployed, when the ESGs see it drop they will remove all routes to the networks behind the DLR.  Creating a static route on the ESGs will prevent lost of connectivity in the event the Control VM goes offline.

Setting a weight/distance of 61 (or whatever your neighbor-configured weight is plus “1”) will ensure the static route is only used if the redistributed routes are  unavailable.

Static Routes on the DLR
Don’t create static routes to the ESGs on the DLR or in the event of a ESG+Control VM failure (like both are on the same host) the DLR on the hosts can keep routing traffic to the dead ESG.

Anti-Affinity Rules
Rules should be created to ensure ESGs and Control VMs don’t run on the same hosts.   This will prevent losing multiple ESGs at once and will prevent losing a ESG and Control VM at the same time, which will result in hosts still sending traffic to the down ESG.

Note that you’ll need to either delete the NSX-created control-VM-HA rule and add those VMs to the new rule, or add your ESGs to the NSX-created rule.  You’ll also need to check it periodically as redeploying Edges will rename them and break your rules.

iBGP vs eBGP
BGP between different AS areas is called eBGP, BGP between routers with the same AS area is iBGP.  The options are pretty limited for NSX so the differences are minimal, but now you know what your network guy is talking about.


Posted in Network, NSX, Virtualization, VMware | Tagged , , | Leave a comment

Checking NSX DFW rules and rule sets

The new VMware Docs page has a cheatsheet of CLI commands but here’s what you need to list the rules enforced on a VMs vnic.

SSH to NSX Manager

{Note that you can enable SSH if needed from the “Summary” page of the appliance config page – but not from the Web Client.}

show cluster all (to get the cluster IDs)

manager> show cluster all
No. Cluster Name Cluster Id Datacenter Name Firewall Status
1 NSXCluster domain-c7 Datacenter Enabled

show cluster <cluster-id> (to get the host IDs)

manager> show cluster domain-c7
Datacenter: Datacenter
Cluster: NSXCluster
No. Host Name Host Id Installation Status
1 esxitwo.corp.local host-21 Enabled
2 esxione.corp.local host-10 Enabled

show host <host-id> to get the VM IDs

manager> show host host-21
Datacenter: Datacenter
Cluster: NSXCluster
Host: esxitwo.corp.local
No. VM Name VM Id Power Status
1 VMware vRealize Network Insight Platform vm-144 off
2 TinyOne vm-52 off
3 VMware vRealize Network Insight Proxy vm-145 off
4 Tiny10 vm-203 off
5 NSX_Controller_5d671e3b-91ca-4351-9c1b-e13277d873f7 vm-124 on

show vm <vm-id> to get the filters list

manager> show vm vm-60
Datacenter: Datacenter
Cluster: NSXCluster
Host: esxione.corp.local
Host-ID: host-10
VM: Tiny10
Virtual Nics List:
Vnic Name Tiny10 - Network adapter 1
Vnic Id 5039be1d-ac25-cdda-5d37-2f5435146776.000
Filters nic-650978-eth0-vmware-sfw.2

what you want is the Filters nic-650978-eth0-vmware-sfw.2 as that with the host ID will get you the rules.

show dfw host <host-ID filter <filterID> rules

manager> show dfw host host-10 filter nic-650978-eth0-vmware-sfw.2 rules
ruleset domain-c7 {
 # Filter rules
 rule 1006 at 1 inout protocol tcp from addrset ip-securitygroup-13 to addrset ip-securitygroup-10 port 443 accept;
 rule 1006 at 2 inout protocol tcp from addrset ip-securitygroup-13 to addrset ip-securitygroup-10 port 80 accept;
 rule 1005 at 3 inout protocol tcp from any to addrset ip-securitygroup-10 port 22 accept;
 rule 1003 at 4 inout protocol ipv6-icmp icmptype 136 from any to any accept;
 rule 1003 at 5 inout protocol ipv6-icmp icmptype 135 from any to any accept;
 rule 1002 at 6 inout protocol udp from any to any port 68 accept;
 rule 1002 at 7 inout protocol udp from any to any port 67 accept;
 rule 1001 at 8 inout protocol any from any to any accept;
ruleset domain-c7_L2 {
 # Filter rules
 rule 1004 at 1 inout ethertype any from any to any accept;

show dfw host <host-ID filter <filterID> addrsets

manager> show dfw host host-10 filter nic-650978-eth0-vmware-sfw.2 addrsets
addrset ip-securitygroup-10 {
addrset ip-securitygroup-13 {
ip fe80::250:56ff:feb9:db4d,

Note that powered off VMs won’t be included in the addrsets, neither will the VM’s own IP address even if its in the group.

Posted in CLI, Firewall, Network, NSX, Virtualization, VMware | Tagged , , | Leave a comment

Hardware VTEPs supported by NSX

I had a discussion with a client about this a little while ago and it took me some time to find the link, but here’s VMware’s current list of supported Hardware VXLAN Gateways.

I’ve also been told there are some Cisco devices supported that aren’t on the list:

With OS version 7.0(3)I4(2):

The 9300-EX should join that list soon.  I have no idea if or when supported Cisco devices will be made public.

There is an HoL (HOL-1703-SDC-1) that has a module on configuring NSX to work with an Arista (note that its a simulation, not a “real” lab) if you want some almost-hands-on experience.

If you have some physical devices you want to participate on a VXLAN network (maybe L2 connectivity is required or you just can’t change IPs), hardware VTEPs are one possible solution.


Posted in Network, NSX, Virtualization, VMware | Tagged , , | 4 Comments

Powershell report on NSX IP Pools

While troubleshooting and NSX install I began wondering about the NSX IP pools and what was being used in them.  I still need to look up the allocated IPs and report on the objects using them (name, type) but for now the (very basic) report spits back IP pool name, the configured ranges, the total IPs in the ranges, the number of IPs used and which specific IPs are used.

Pool name: VTEP Pool
Configured range(s): -
Total IPs: 101
Number of IPs used: 4
IPs used:,,,

Note when you go to configure pools, if you want to allocate disjointed ranges you need to separate them with commas – this is no longer shows in the UI with 6.3. Also, to add single IPs just make a range out of them like:,,

The script leverages a shorter version of the NSXAPI function I stole from Chris Wahl a while back.

I’ve only tested with Powershell 5.0 – the original NSXAPI function created a new type which errors in PS5:

add-type : Cannot add type. The type name ‘TrustAllCertsPolicy’ already exists.

so I chopped that part of the code out along with most comments.

I’ll update this post when I tweak the script to report fully on the IPs allocated.

function NSXAPI {
Process {
$Username = "admin"
$Password = "vmware"
$NSXManager = "nsxmanager.corp.local"

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

$r = Invoke-WebRequest -Uri $Request -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$rxml = $r.Content
} # End of process
} # End of function

foreach ($pool in $pools) {
$name = $
$universal = $pool.isUniversal
$total = $pool.totaladdresscount
$used = $pool.usedaddresscount
if (($pool.ipranges.iprangedto).count) {
$range = ""
foreach ($r in $pool.ipranges.iprangedto) {
if ($range -eq "") {
$range = $range + "$($r.startaddress) - $($r.endaddress)"
} else {
$range = $range + ", $($r.startaddress) - $($r.endaddress)"
} Else {
$range = "$($pool.ipranges.iprangedto.startaddress) - $($pool.ipranges.iprangedto.endaddress)"
$string= "https://nsxmanager.corp.local/api/2.0/services/ipam/pools/$($pool.objectid)/ipaddresses"
$allocated = (NSXAPI $string).allocatedIpAddresses.allocatedIpAddress
$ips = ""
foreach ($a in $allocated) {
if ($ips) {
$ips= $ips + ", $($a.ipaddress)"
} else {
$ips= "$($a.ipaddress)"

"Pool name: $name"
"Configured range(s): $range"
"Total IPs: $total"
"Number of IPs used: $used"
"IPs used: $ips"
" "
" "

Posted in Network, NSX, PowerShell, Scripting, Virtualization, VMware | Tagged , | Leave a comment

NSX distributed logical router appliance Part 2

In a previous post I talked about UDLR/DLR differences, why you might need to deploy an appliance with your (U)DLR and how high availability works for the optional appliance.

This post will cover more on IP addressing for the appliance and configuration changes after a brief rant on naming.

What’s in a name

Would a DLR that wasn’t referred to as an Edge smell as sweet? – W. Shakespeare

So in the 6.3 documentation, the DLR falls under Routing / Add Logical (Distributed) Router

And there it mentions An NSX Edge appliance provides dynamic routing ability if needed.

In the GUI you find the (U)DLR under NSX Edges even though it’s not on “the edge” and deploying a VM based off an Edge appliance isn’t required for a distributed logical router.

In the KB article referenced in the docs the appliance is called a DLR Control VM which the Design Guide also call it.  Most VMware presentations refer to it as a “Control” appliance.

At least it’s explicitly mentioned in the docs now, it was kind of glossed over in the install section previously (6.0/6.1).

Speaking of names, it’s still referenced repeatedly as a vShield Edge appliance.

vApp details

First time logging in – and second.

Could we please pick one name and stick with it?


As mentioned in my last post each DLR VM created will show all configured IPs for the DLR.

However, the only one that normally sees traffic is the HA heartbeat APIPA address, and in the event of a failover, the HA Interface IP will be ARPed.

If you configure OSPF or BGP, you will be asked for a Protocol Address and a Forwarding Address.

BGP config also sets up the Forwarding and Protocol Addresses.

The Forwarding address is the “next hop” for all networks managed by the DLR.  The networks and “next hop” that will be pushed out to the OSPF neighbors.

OSPF Forwarding Address is the Transit IP for the DLR.

The Protocol Address will be assigned to the DLR VM and that is how the DLR VM will exchange routing information with the OSPF neighbors.

DLR VM with OSPF Protocol Address replacing Transit Uplink

Looking at the interfaces on the DLR VM, all internal links are added to a VDR interface and used to generate the dynamic routing updates.

The DLR HA interface gets added it’s own interface (assuming its not on the same network as another uplink)

While uplink DLR interfaces (whether or not a Protocol address is created) each get their own interface.

Uplink vs Internal

Dynamic routing protocol (BGP and OSPF) neighbors are supported only on uplink interfaces.  And of course dynamic routing protocols are the only reason to depoy a DLR VM.

Firewall rules are applicable only on uplink interfaces of the DLR VM and are limited to control and management traffic that is destined to the DLR VM.


The VM appears to be configured via Tools.  NSX Manager sends an update to the host holding the VM, which pushes the update via VMware Tools.

While the Design Guide states on pg 53:

The controller leverages the control plane with the ESXi hosts to push the new DLR configuration, including LIFs and their associated IP and vMAC addresses.

I don’t believe that statement is correct.

Because without a Controller:

Adding a new OSPF area:

The new changes show up on the DLR VM:

I believe updates are pushed via the “message bus” to the DLR VIB and DLR, which would explain why changes can be sent with no Controllers and no network connectivity to the DLR VM.

Speaking of controllers, in the NSX Troubleshooting guide it says Controller Cluster allocates one of the Controller nodes to be the master for this DLR instance. and references the Controller cluster receiving route tables and updates from the DLR VM and sending that information on to the hosts.

This does appear to be the case, as deleting my controllers stopped route updates from being delivered to the hosts.


Posted in Network, NSX, Virtualization, VMware | Tagged | Leave a comment

NSX distributed logical router appliance Part 1

When you create a Universal or “regular” Distributed Logical Router in NSX you have the option of deploying an appliance along with it.

This post will cover why to deploy an appliance along with a DLR, configuring the appliance for high availability and how the DLR high availability feature works.

Deploy if you need dynamic routing on your DLR
Enable HA and add an IP address on a dedicated logical switch
Disable vSphere HA for the DLR appliance

Pick Universal if you'll eventually have cross-vcenter and will only connect to logical switches (no dPGs)

Continue reading

Posted in Network, NSX, Virtualization, VMware | Tagged | Leave a comment

Moving an NSX DLR control VM (datastore or host)

If you have deployed an appliance along with your NSX distributed logical router you might need to move it around in the vSphere environment.

However, using the normal vSphere migration method causes more than a few issues.

Oh, that doesn’t look good!

Not to worry, because NSX makes it pretty simple to relocate them.

First go to Networking & Security / NSX Edges and double click on the DLR in question.

Under configuration, select the appliance instance to move (the “Name” field will match the vSphere name) and click the pencil/edit button.

Then choose the new values.

The table will be updated with the changes …

and if the datastore was changed you can go to vSphere and see the new appliance deployed and the old one removed.

when the deployed VM is only ~1GB total it is pretty quick to replace!

Note that the same method works to move all NSX components other than NSX Manager. Note also that the same recreate/delete is used for all moves even when just changing folders!

Speaking of folders, if an NSX object is moved into a folder,  it can’t later be moved back to “no folder” using the web client.

Posted in Network, NSX, Virtualization, VMware | Tagged | Leave a comment