We got our own Cisco UCS lab at the office last week complete with two 6210S, two server blades and the new Cisco VIC (Palo) adapter.
Here is a diagram I copied from the Chassis 1 –> Hybrid Display tab in UCS Manager.
What is not in the diagram are two Cisco MDS 9124s and a Cisco 3750E with two Ten Gigabit uplinks.
After we got our first ESXi 4.1 blade up booting from a CX4-120 LUN I was itching to present more than two 10GB adapters to the ESXi host.
When I initially looked at how to add more than two vNICs to a Service Profile I couldn’t figure out how to do it. I was thinking there was some new configuration screen somewhere where you had to go to and enable the additional vNICs. I was also unable to find any good documentation on how to do it so a posted a question to the Unified Computing Cisco Support Community – https://supportforums.cisco.com/community/netpro/data-center/unified-computing?view=discussions&start=0
If you haven’t checked out this community and you are interested in Cisco UCS you should definitely browse through it. There are some good tips in there.
This is a very active community and the two times I posted a question it was answered within 12 hours.
I posted a question on how to configure the new VIC (Palo) adapter and to my surprise it is a lot easier to configure than what I initially thought.
All that I had to was add two more vNICs to my Service Profile template. I don’t know why I didn’t just try that first.
I went into my updating Service Profile Template and added eth2 to fabric A and eth3 to fabric B, here is a screen shot
Now be careful because when you add vNICs or modify the storage of a Service Profile or an updating Service Profile Template it will power cycle the blade that is bound to the template. I don’t know if there is a way to change this behavior but I think this is dangerous.
After my ESX 4i server rebooted I first checked the vmnic list using the esxcfg-nics –l command. Here is the output
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:08:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:af 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic1 0000:09:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:bf 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic2 0000:0a:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:ae 1500 Cisco Systems Inc VIC Ethernet NIC
vmnic3 0000:0b:00.00 enic Up 10000Mbps Full 00:25:b5:0c:95:be 1500 Cisco Systems Inc VIC Ethernet NIC
AWESOME!!!
Next I logged into my vSphere Client and checked the Network Adapters and added a new vSwitch for VM traffic.
Now I can keep my Management Network (Service Console) on a standard vSwitch and put my VM networks in a Distributed vSwitch or in Nexus 1000v without worrying about loosing access because a configuration error.
Jeremy Waldrop's Blog 
Cool! You got your kit! Looking forward to your posts and around the UCS community as an early adopter!
I’m sure there’s some trouble we can collaborate on 🙂
Pingback: Virtualization Short Take #37 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers
Pingback: Virtualization Short Take #37 | Free Techie Blog
Hey
Why are you separating management and VM traffic with virtual switches, esp with UCS? I’m trying to work out what’s wrong with having a single vDS and two vNICs, one to each fabric, and everything trunked down into it.
Cheers
For ESX hosts on UCS I have actually been presenting 6 vNICs. There are a few reasons for this; 1). QoS, I have QoS policies tied to the different vNICs, 2 for vMotion, this keeps the vMotion traffic from hogging all of the bandwidth on the FEX links from SAN and VM taffic. 2). To keep SC/MGMT and vMotion on standard vSwitches while the 2 for VM traffic on Nexus 1000v or a VMware vDS.
3).Logical separation.
How many UCS vNICs can one present to esxi 4.1?
Take a look at this site for limitations, roughly 54 for Gen 1 and 128 for Gen 2
http://www.mseanmcgee.com/2011/07/ucs-2-0-cisco-stacks-the-deck-in-las-vegas/