Cisco UCS VN-Link Hardware Implementation Walk Through

Here is another video blog on Cisco UCS. In this video I walk through the implementation of VMware VN-Link in hardware on the Cisco UCS VIC (Palo) adapter.

http://www.screencast.com/t/YWU3ZGVmYzAt

Command line output of the ESX configuration

esxupdate query
—-Bulletin ID—- —–Installed—– ————Summary————-
ESX400-Update01     2010-03-25T22:03:33 VMware ESX 4.0 Complete Update 1
VEM400-200912272-BG 2010-03-25T22:35:03 Cisco Nexus 1000V  4.0(4)SV1(2)

vem version
Package vssnet-esx4.1.0-00000-release
Version 4.0.4.1.2.0.80-1.9.179
Build 179
Date Wed Dec 9 08:13:45 PST 2009

vem status

VEM modules are loaded

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          4           32                1500    vmnic0,vmnic1
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
UCS-dvSwitch0  256         230         256               vmnic3,vmnic2

VEM Agent (vemdpa) is running

esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          4           32                1500    vmnic0,vmnic1

PortGroup Name      VLAN ID  Used Ports  Uplinks
Service Console     0        1           vmnic0,vmnic1

DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
UCS-dvSwitch0  256         230         256               vmnic3,vmnic2

DVPort ID           In Use      Client
1628                1           vmnic2
1629                1           vmnic3
1630                0
1631                0

esxcfg-nics -l
Name    PCI      Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  08:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:a9 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic1  09:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:ba 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic2  0a:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:aa 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic3  0b:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:bb 1500   Cisco Systems Inc 10G Ethernet NIC

lspci | egrep -i “cisco.*pass”
0c:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0d:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0e:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0f:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
10:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
11:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
12:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
13:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
14:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
15:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
16:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
17:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
18:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
19:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1a:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1b:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1c:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1d:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1e:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1f:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
20:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
21:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
22:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
23:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)

Advertisement

Cisco UCS Manager Equipment Tab Video Walkthrough

My first attempt at a video blog demonstrating the UCSM Equipment tab. I hope to do more of these and get better as I go.

Click the following link to view the Screencast video
UCSM Equipment Tab Walkthrough

Cisco UCS vNIC Failover

When you create a vNIC in UCSM with Cisco UCS you have the option to pin that vNIC to Fabric A or Fabric B and an option to Enable Failover. Most of the servers we deploy on UCS blades are ESX 4 and with ESX we always create two vNICs, one on Fabric A and one on Fabric B and then let ESX handle the NIC teaming and failover. With the new Palo adapter we create 4 vNICs (eth0-eth3), assign two for SC/vMotion in a Standard vSwitch and two for VM networking in a Distributed vSwitch or Nexus 1000v.

I was curious about how the UCS level vNIC failover worked so I built a Windows 2008 R2 blade. In the Service Profile I presented only on vNIC to it and checked the Enable Failover option. I was wondering if Windows would see two NICs or one because someone else had told me that Windows will see two NICs but will auto failover between them if the uplink on Fabric A goes down.

After I loaded the Cisco enic Palo drivers in Windows it only sees one NIC,  here is a screen shot of Device Manager and the Network Connections window.

To test the failover I first SSH into our Cisco 3750E switch to see witch 10GB uplink the MAC address was coming across and it was using Te1/0/1.

Cisco3750#show mac address-table | include 0025.b50c.95ab
1    0025.b50c.95ab    DYNAMIC     Te1/0/1

I then started an extended ping to the server and shutdown switch port Te1/0/1, failover happened almost immediately with only one ping missed. I then check where the MAC address was and it was on the other uplink for Fabric B.

Cisco3750#show mac address-table | include 0025.b50c.95ab
1    0025.b50c.95ab    DYNAMIC     Te1/0/2

Wow!! that was cool, NIC failover without having to mess with NIC teaming drivers in Windows. This is a much cleaner solution than using the old Intel ProSet, HP or Broadcom tools to create the third virtual NIC. It has to be more efficient using hardware based failover as well.

I then wanted to see what would happened if I brought switch port Te1/0/1 back online. To my surprise it switched back over to Fabric A almost immediately after enabling the switch port but this time there were 0 pings missed.

I checked the Windows event log and didn’t see any event where Windows detected a loss of network connectivity. This is really cool that Windows never knew anything about the uplink failure.

Very cool stuff.

Presenting 4 vNICs to VMware ESXi 4 with the Cisco UCS VIC (Palo) Adapter

We got our own Cisco UCS lab at the office last week complete with two 6210S, two server blades and the new Cisco VIC (Palo) adapter.

Here is a diagram I copied from the Chassis 1 –> Hybrid Display tab in UCS Manager.

What is not in the diagram are two Cisco MDS 9124s and a Cisco 3750E with two Ten Gigabit uplinks.

After we got our first ESXi 4.1 blade up booting from a CX4-120 LUN I was itching to present more than two 10GB adapters to the ESXi host.

When I initially looked at how to add more than two vNICs to a Service Profile I couldn’t figure out how to do it. I was thinking there was some new configuration screen somewhere where you had to go to and enable the additional vNICs. I was also unable to find any good documentation on how to do it so a posted a question to the Unified Computing Cisco Support Community – https://supportforums.cisco.com/community/netpro/data-center/unified-computing?view=discussions&start=0

If you haven’t checked out this community and you are interested in Cisco UCS you should definitely browse through it. There are some good tips in there.

This is a very active community and the two times I posted a question it was answered within 12 hours.

I posted a question on how to configure the new VIC (Palo) adapter and to my surprise it is a lot easier to configure than what I initially thought.

All that I had to was add two more vNICs to my Service Profile template. I don’t know why I didn’t just try that first.

I went into my updating Service Profile Template and added eth2 to fabric A and eth3 to fabric B, here is a screen shot

Now be careful because when you add vNICs or modify the storage of a Service Profile or an updating Service Profile Template it will power cycle the blade that is bound to the template. I don’t know if there is a way to change this behavior but I think this is dangerous.

After my ESX 4i server rebooted I first checked the vmnic list using the esxcfg-nics –l command. Here is the output

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:08:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:af 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic1  0000:09:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:bf 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic2  0000:0a:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:ae 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic3  0000:0b:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:be 1500   Cisco Systems Inc VIC Ethernet NIC

AWESOME!!!

Next I logged into my vSphere Client and checked the Network Adapters and added a new vSwitch for VM traffic.

Now I can keep my Management Network (Service Console) on a standard vSwitch and put my VM networks in a Distributed vSwitch or in Nexus 1000v without worrying about loosing access because a configuration error.