We got our own Cisco UCS lab at the office last week complete with two 6210S, two server blades and the new Cisco VIC (Palo) adapter.
Here is a diagram I copied from the Chassis 1 –> Hybrid Display tab in UCS Manager.
What is not in the diagram are two Cisco MDS 9124s and a Cisco 3750E with two Ten Gigabit uplinks.
After we got our first ESXi 4.1 blade up booting from a CX4-120 LUN I was itching to present more than two 10GB adapters to the ESXi host.
When I initially looked at how to add more than two vNICs to a Service Profile I couldn’t figure out how to do it. I was thinking there was some new configuration screen somewhere where you had to go to and enable the additional vNICs. I was also unable to find any good documentation on how to do it so a posted a question to the Unified Computing Cisco Support Community – https://supportforums.cisco.com/community/netpro/data-center/unified-computing?view=discussions&start=0
If you haven’t checked out this community and you are interested in Cisco UCS you should definitely browse through it. There are some good tips in there.
This is a very active community and the two times I posted a question it was answered within 12 hours.
I posted a question on how to configure the new VIC (Palo) adapter and to my surprise it is a lot easier to configure than what I initially thought.
All that I had to was add two more vNICs to my Service Profile template. I don’t know why I didn’t just try that first.
I went into my updating Service Profile Template and added eth2 to fabric A and eth3 to fabric B, here is a screen shot
Now be careful because when you add vNICs or modify the storage of a Service Profile or an updating Service Profile Template it will power cycle the blade that is bound to the template. I don’t know if there is a way to change this behavior but I think this is dangerous.
After my ESX 4i server rebooted I first checked the vmnic list using the esxcfg-nics –l command. Here is the output
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:08:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:af 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic1 0000:09:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:bf 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic2 0000:0a:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:ae 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic3 0000:0b:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:be 1500 Cisco Systems Inc VIC Ethernet NIC
AWESOME!!!
Next I logged into my vSphere Client and checked the Network Adapters and added a new vSwitch for VM traffic.
Now I can keep my Management Network (Service Console) on a standard vSwitch and put my VM networks in a Distributed vSwitch or in Nexus 1000v without worrying about loosing access because a configuration error.
Cool! You got your kit! Looking forward to your posts and around the UCS community as an early adopter!
I’m sure there’s some trouble we can collaborate on 🙂
Pingback: Virtualization Short Take #37 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers
Pingback: Virtualization Short Take #37 | Free Techie Blog
Hey
Why are you separating management and VM traffic with virtual switches, esp with UCS? I’m trying to work out what’s wrong with having a single vDS and two vNICs, one to each fabric, and everything trunked down into it.
Cheers
For ESX hosts on UCS I have actually been presenting 6 vNICs. There are a few reasons for this; 1). QoS, I have QoS policies tied to the different vNICs, 2 for vMotion, this keeps the vMotion traffic from hogging all of the bandwidth on the FEX links from SAN and VM taffic. 2). To keep SC/MGMT and vMotion on standard vSwitches while the 2 for VM traffic on Nexus 1000v or a VMware vDS.
3).Logical separation.
How many UCS vNICs can one present to esxi 4.1?
Take a look at this site for limitations, roughly 54 for Gen 1 and 128 for Gen 2
http://www.mseanmcgee.com/2011/07/ucs-2-0-cisco-stacks-the-deck-in-las-vegas/