So you may have seen the recent ACI migration post. There was a couple of very interesting points in there - namely that ACI uses a modified Nexus 1000v called the AVS and Nutanix is an upcoming partner! Many people use the Nexus 1000v on Nutanix today already, but there are not many guides on what the configuration should look like. This post will go over the Nexus 1000v configuration needed to get the Nutanix cluster working optimally.
So now that we know the full config, what does this look like logically and how are these VLANs used? Let's take a look into that now:
What is the Nexus 1000v?
The Nexus 1000v is a virtual switch built off of the VMware DVS architecture (and more recently the Hyper-V extensible switch and the Openstack OVS). Building off of the Nexus 7k architecture, there are 2 Virtual Supervisors (VSMs) and each host is a Virtual Ethernet Module (VEM).
The VEMs request their configuration from the VSMs over the network and then switch based on that config. Check out this youtube video for a guide on the installation. For the purpose of this post here's what you need to know:
VSM (Virtual Supervisor Module): VM that runs either on a physical appliance (1010) or inside your virtual enviornment
VEM (Virtual Ethernet Module): VIB that is installed on each host, either manually or by VUM when you choose the "add hosts" wizard
VUM: VMware update manager. Automatically installs VIBs when requested
Port-profile: A collection of switchport configuration. Equivalent to a VMware Port-group. By using a port-profile you only have to configure each type of traffic once rather then once per port. A port-profile has all of the normal switching configuration (access/trunk, ACLs, QoS, VLAN lists, etc.). The VMware admin simply assigns this to the necessary VMs.
What Config do I Need?
With the N1k we're going to duplicate the configuration that the Nutanix nodes put on the standard switches by default. Here's how we'll do this:
#type Ethernet specifies that this is for physical ports
port-profile type
Ethernet system-uplink
#these are commands that enable
the port-profile
#in vCenter and will always be present
vmware port-group
no shutdown
state enabled
#this config should match the
upstream switch.
switchport mode trunk
switchport trunk allowed vlan [allowed vlan list]
switchport trunk native vlan [upstream native vlan]
#system vlans are VLANs that
should be up no matter
#what the state of the n1k is
#in our system this should be
the management VLAN (CVM/ESXi)
#since that will bring up our storage as well
system vlan [system vlan list]
#this configuration is the
same as saying
#“route based on originating port ID” on the standard switch
channel-group auto mode on mac-pinning
The other config
we need is for vmk0 and the the controller VM port-group:
#type vEthernet specifies that this is a
#port-group vms/vmks will
connect to
#this one will be specifically for the vmks,
#so we will need to create
a second one for the CVMs
port-profile type
vethernet vmk
#these are commands that enable
the port-profile
#in vCenter and will always be present
vmware port-group
no shutdown
state enabled
#these commands specify the
connectivity,
#so they should be set to the management VLAN
switchport mode access
switchport access vlan [management VLAN]
#since we need this port up at
all times we will set the system vlan
system vlan [management VLAN]
#cisco recommends l3 mode for
connectivity now
#so we need the following command to tell the N1k
#to use this
interface
capability l3control
#we will manually specify the
pinning here
#this is the same as setting
active/standby adapters in VMware
#the number here is the vmnic
number
#so this will change depending on how the 10gb links are numbered
#or if you are using the 1gb
links
#see below for a note on
pinning
pinning-id [primary-SGID] backup
[backup-SGID]
#now we will create a port-profile for the CVMs.
#this will be identical to the vmk port-profile
#minus the capability
command
Port-profile type
vethernet VM-Networking
vmware port-group
no shutdown
state enabled
switchport mode access
switchport access vlan [management VLAN]
system vlan [management VLAN]
pinning-id [primary-SGID] backup [backup-SGID]
So here’s the
full config minus the comments:
port-profile type
Ethernet system-uplink
vmware port-group
no shutdown
state enabled
switchport mode trunk
switchport trunk allowed vlan [allowed vlan
list]
switchport trunk native vlan [upstream native
vlan]
system vlan [system vlan list]
channel-group auto mode on mac-pinning
port-profile type
vethernet vmk
vmware port-group
no shutdown
state enabled
switchport mode access
switchport access vlan [management VLAN]
system vlan [management VLAN]
capability l3control
pinning-id [primary-SGID] backup [backup-SGID]
port-profile type
vethernet VM-Networking
vmware port-group
no shutdown
state enabled
switchport mode access
switchport access vlan [management VLAN]
system vlan [management VLAN]
pinning-id [primary-SGID] backup [backup-SGID]
And that's it! Using the "Add Hosts" wizard in vCenter will add the ports to the correct groups and call VUM to install the correct VIB.
Note on Pinning
With mac-pinning, each ethernet port is assigned a SGID (sub-group id) based on it's vmnic number. Thus vmnic0 is SGID 0, vmnic1 is SGID 1 and so on. When a VM is powered on it is assigned a SGID number in a round-robin fashion so that traffic is balanced by the number of VMs.
In the above config we manually specified the SGID to pin to for the CVM and the management VMK. This is because we want to reduce the number of hops that the storage traffic has to take, thus improving latency.
Consider the following:
The ideal path for CVM/Host traffic is Host - Switch - Host. In the worst case we'll do Host - Switch - Switch - Host. To prevent this we will use pinning:
port-profile type vethernet VM-Networking
pinning-id 0 backup 1
This tells any ports in the vmk port-profile to use vmnic0 first and if that's not available use vmnic1. This allows us to control the traffic and keep the ideal traffic flow.
VLANs and their interactions
So now that we know the full config, what does this look like logically and how are these VLANs used? Let's take a look into that now:
VLANs/Port-profiles used:
CVM Eth0 (External network): This uses the "VM-Network" port-profile defined above. This should be in the same subnet as the host vmk0 and is used by the CVMs to communicate across the cluster and for access to prism/ncli
Host vmk0 (External network): Uses the "vmk" port-profile above. This is used for all managment of the ESXi host and for NDFS access if the local CVM is down.
VSM Management: Used by the VSM to talk to vCenter and the VEMs (in L3 mode). This can be a different VLAN then the CVM/Host management. This is the second adapter on the VSM
VSM Control: Used by the VSMs to talk to each other to exchange HA state. This is the first adapter on the VSM and has no dependencies on any other VLAN.
All of these VLANs should be a system VLAN in their port-profile and in system-uplink. Do NOT touch vSwitch1 (the NutanixvSwitch). VMK1 and eth1 of the CVM should be left there.