UCS VIC 1340/1380 PCI Placement

I recently did a UCS implementation that included the B460-M4 blades. If you aren’t familiar with these beasts you should look them up. They are two B260 full-width blades connected together with a scalability connector on the front to create one giant server. Each of the B260s had a VIC 1340 MLOM to give the server two VIC 1340s.

download

I did the initial design and configuration consistent with our standard UCS design and logical build out.

We have a standard vNIC/vHBA design for ESXi hosts where there are a minimum of 6 vNICs and 2 vHBAs. The vNICs/vHBAs are split between Fabric A and Fabric b and then map to vSphere vSwitches (Standard and Distributed).

Here is a screen shot of our standard vNIC deign for ESXi on blades with a single VIC

Screen Shot 2016-02-12 at 12.03.17 PM

With two VICs we have a different configuration so that we make use of both VICs. In this configuration we place all of Fabric A vNICs/vHBAs on vCon 1 and all of Fabric B vNICs/vHBAs on vCon 2. With this configuration the vNIC to vmnic numbering changes and so does the vSwitch to vmnic uplinks

Picture1

In this design I implemented this two adapter configuration and built out my templates, pools and policies as usual.

Everything was going well until we built our first ESXi 6 host and couldn’t get management connectivity working. Upon further investigation we realized the vNIC to vmnic mappings were not correct.

After some research I came across this Cisco Bug ID that described our problem to a T – https://quickview.cloudapps.cisco.com/quickview/bug/CSCut78943

I personally wouldn’t call this bug but more of an explanation of the configuration options on the new VICs.

The new VIC 1340/1380s have two PCI channels (1 and 2) and you can control which specific channel the vNIC/vHBA is created on. This new configuration option is called “Admin Host Port” and by default is set to AUTO. With the AUTO setting UCS will round robin each vNIC/vHBA across both Admin Host Ports per vCon.

This Round Robin configuration will place every other vNIC on Admin Host Port 1. This causes an issue because the installed Operating System detects all vNICs on Admin Host Port 1 first and Admin Host Port 2 second.

With the two VICs per blade the configuration that worked like we wanted was to place all of the vNICs on Fabric A on Admin Host Port 1 and the vHBA on Admin Host Port 2 on vCon1 and then the same for Fabric B.

We placed the vHBAs on Admin Host Port 2 so that we could make full use of both PCI lanes.

To configure this on the Service Profile template:

  • Go do the Network tab
  • Click the Modify vNIC/vHBA Placement link
  • Set the placement to Specify Manually
  • Place all Fabric A vNICs/vHBAs on vCon 1
  • Set the Admin Host Port for all vNICs to 1
  • Set the Admin Host Port for the vHBA to 2
  • Place all Fabric B vNICs/vHBAs on vCon 2
  • Set the Admin Host Port for all vNICs to 1
  • Set the Admin Host Port for the vHBA to 2

Here are two screen shots showing this configuration

Picture1

Picture1

Here is a screen shot of the applied configuration, notice all Fabric A vNICs are on the Desired Placement of 1 and Fabric B vNICs on 2. Also notice that the Admin Host Port is 1 for all vNICs.

Picture1

 

 

 

Advertisements

7 thoughts on “UCS VIC 1340/1380 PCI Placement

  1. I am having a hard time getting the Admin Host Port to change while using a LAN/SAN connectivity policy. Do you know if this works?
    I have tried to modify the Admin Host port on Service Profiles and Service Profile Templates. It seems to take the change but never actually changes.

  2. Was an OS already installed on the blade before you changed the admin host ports? If so you will need to reload the OS or force it to re-enumerate the PCI devices. If ESX you can do this by selecting reset system configuration in the DCUI

  3. Hi Jeremy,
    Thanks for the post. I have a question please.
    With a blade with VIC 1380, can you have individual 16 X 10Gbps channels to the FI through IOM 2208?
    Can we create 16 vNICs on VMWare ESXi host each mapping to the unique channels (above)

  4. Is this B200-M3/4 with both the MLOM 1340 and an optional VIC 1380 in the mezzanine slot? Your UCS Manager chassis discovery policy should be set to port channel so that the physical uplinks from the IOMs to the FIs are port channelled. The ports from the VIC to the IOM backplane ports are also port channelled. When you create vNICs in UCSM those are pinned to the port channel upstream to the IOM/FI. The PCI channels on the VIC 1340/1380 have nothing to do with the upstream backplane to IOM and IOM to FI. These are just channels on the VIC where vNICs can be placed. If you have 6 vNICs and 2 vHBA you can put half on PCI channel 1 and the other half on PCI channel 2. Also look at this PDF for how the port channeling from the VIC to the IOM works, start on page 56 – http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/b200m4-specsheet.pdf

  5. Hi Jeremy, I have a C460 M4 using Dual VIC’s under UCSM. My service profile is using 2 vCON’s. What I am seeing is my VIF’s are only using 1 patch/port on each VIC. So I have 2 cables pretty much doing nothing. Is there a way to spead the vNICs evenly between all paths when using Dual VICs?

    Fabric A:
    PCI 4 / 1 – All “A” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)
    PCI 9 / 1 – Nothing

    Fabric B:
    PCI 4 / 2 – Nothing
    PCI 9 / 2 – All “B” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)

    Thanks,
    Joe

  6. hey, check to see what vCons the two adapters are mapped to. You might need to modify the vNICs that you want on the second adapter to be vCon 3 or 4. SSH into UCSM.
    scope org /
    scope service-profile SERVICEPROFILE-NAME
    show vcon-mapping

  7. Thanks! I did try using all 4 vCons and that got me the bandwidth and faillover paths I wanted. I was hung up on thinking I could only have a vCon per physical adapter.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s