UCS VIC 1340/1380 PCI Placement

I recently did a UCS implementation that included the B460-M4 blades. If you aren’t familiar with these beasts you should look them up. They are two B260 full-width blades connected together with a scalability connector on the front to create one giant server. Each of the B260s had a VIC 1340 MLOM to give the server two VIC 1340s.

download

I did the initial design and configuration consistent with our standard UCS design and logical build out.

We have a standard vNIC/vHBA design for ESXi hosts where there are a minimum of 6 vNICs and 2 vHBAs. The vNICs/vHBAs are split between Fabric A and Fabric b and then map to vSphere vSwitches (Standard and Distributed).

Here is a screen shot of our standard vNIC deign for ESXi on blades with a single VIC

Screen Shot 2016-02-12 at 12.03.17 PM

With two VICs we have a different configuration so that we make use of both VICs. In this configuration we place all of Fabric A vNICs/vHBAs on vCon 1 and all of Fabric B vNICs/vHBAs on vCon 2. With this configuration the vNIC to vmnic numbering changes and so does the vSwitch to vmnic uplinks

Picture1

In this design I implemented this two adapter configuration and built out my templates, pools and policies as usual.

Everything was going well until we built our first ESXi 6 host and couldn’t get management connectivity working. Upon further investigation we realized the vNIC to vmnic mappings were not correct.

After some research I came across this Cisco Bug ID that described our problem to a T – https://quickview.cloudapps.cisco.com/quickview/bug/CSCut78943

I personally wouldn’t call this bug but more of an explanation of the configuration options on the new VICs.

The new VIC 1340/1380s have two PCI channels (1 and 2) and you can control which specific channel the vNIC/vHBA is created on. This new configuration option is called “Admin Host Port” and by default is set to AUTO. With the AUTO setting UCS will round robin each vNIC/vHBA across both Admin Host Ports per vCon.

This Round Robin configuration will place every other vNIC on Admin Host Port 1. This causes an issue because the installed Operating System detects all vNICs on Admin Host Port 1 first and Admin Host Port 2 second.

With the two VICs per blade the configuration that worked like we wanted was to place all of the vNICs on Fabric A on Admin Host Port 1 and the vHBA on Admin Host Port 2 on vCon1 and then the same for Fabric B.

We placed the vHBAs on Admin Host Port 2 so that we could make full use of both PCI lanes.

To configure this on the Service Profile template:

  • Go do the Network tab
  • Click the Modify vNIC/vHBA Placement link
  • Set the placement to Specify Manually
  • Place all Fabric A vNICs/vHBAs on vCon 1
  • Set the Admin Host Port for all vNICs to 1
  • Set the Admin Host Port for the vHBA to 2
  • Place all Fabric B vNICs/vHBAs on vCon 2
  • Set the Admin Host Port for all vNICs to 1
  • Set the Admin Host Port for the vHBA to 2

Here are two screen shots showing this configuration

Picture1

Picture1

Here is a screen shot of the applied configuration, notice all Fabric A vNICs are on the Desired Placement of 1 and Fabric B vNICs on 2. Also notice that the Admin Host Port is 1 for all vNICs.

Picture1

 

 

 

Advertisements

20 thoughts on “UCS VIC 1340/1380 PCI Placement

  1. I am having a hard time getting the Admin Host Port to change while using a LAN/SAN connectivity policy. Do you know if this works?
    I have tried to modify the Admin Host port on Service Profiles and Service Profile Templates. It seems to take the change but never actually changes.

  2. Was an OS already installed on the blade before you changed the admin host ports? If so you will need to reload the OS or force it to re-enumerate the PCI devices. If ESX you can do this by selecting reset system configuration in the DCUI

  3. Hi Jeremy,
    Thanks for the post. I have a question please.
    With a blade with VIC 1380, can you have individual 16 X 10Gbps channels to the FI through IOM 2208?
    Can we create 16 vNICs on VMWare ESXi host each mapping to the unique channels (above)

  4. Is this B200-M3/4 with both the MLOM 1340 and an optional VIC 1380 in the mezzanine slot? Your UCS Manager chassis discovery policy should be set to port channel so that the physical uplinks from the IOMs to the FIs are port channelled. The ports from the VIC to the IOM backplane ports are also port channelled. When you create vNICs in UCSM those are pinned to the port channel upstream to the IOM/FI. The PCI channels on the VIC 1340/1380 have nothing to do with the upstream backplane to IOM and IOM to FI. These are just channels on the VIC where vNICs can be placed. If you have 6 vNICs and 2 vHBA you can put half on PCI channel 1 and the other half on PCI channel 2. Also look at this PDF for how the port channeling from the VIC to the IOM works, start on page 56 – http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/b200m4-specsheet.pdf

  5. Hi Jeremy, I have a C460 M4 using Dual VIC’s under UCSM. My service profile is using 2 vCON’s. What I am seeing is my VIF’s are only using 1 patch/port on each VIC. So I have 2 cables pretty much doing nothing. Is there a way to spead the vNICs evenly between all paths when using Dual VICs?

    Fabric A:
    PCI 4 / 1 – All “A” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)
    PCI 9 / 1 – Nothing

    Fabric B:
    PCI 4 / 2 – Nothing
    PCI 9 / 2 – All “B” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)

    Thanks,
    Joe

  6. hey, check to see what vCons the two adapters are mapped to. You might need to modify the vNICs that you want on the second adapter to be vCon 3 or 4. SSH into UCSM.
    scope org /
    scope service-profile SERVICEPROFILE-NAME
    show vcon-mapping

  7. Thanks! I did try using all 4 vCons and that got me the bandwidth and faillover paths I wanted. I was hung up on thinking I could only have a vCon per physical adapter.

  8. Thanks Jeremy for your posts, I faced similar issue but till now I cannot fixed.
    •Hardware: UCSB-B420-M4, UCS-SP-FI48, Cisco UCS VIC 1380 mezzanine, Cisco UCS VIC 1340 modular LOM, Cisco UCS Port Expander Card (mezz) with UCSM 3.2.1d (infrastructure and servers) adtapters showing as adpter 1 and adapter 3

    •I have create below nics from 00 to 05 are mapped to vCon1 then FI-A and from 06 to 11 are mapped to vCon3 then FI-B

    vmnic00_esxi_A
    vmnic01_vmotion_A
    vmnic02_FT_A
    vmnic03_NSX_A
    vmnic04_Prod_A
    vmnic05_Backup_A
    vmnic06_esxi_B
    vmnic07_vmotion_B
    vmnic08_FT_B
    vmnic09_NSX_B
    vmnic10_Prod_B
    vmnic11_Backup_B

    I put all these vNICs in LAN Connectivity Policy in same order, then in SP template I did vNIC/vHBA placement manually and put the host port as below

    vCon1
    vmnic00_esxi_A 1
    vmnic01_vmotion_A 1
    vmnic02_FT_A 1
    vmnic03_NSX_A 1
    vmnic04_Prod_A 1
    vmnic05_Backup_A 1
    vHBA_A 2

    vCon3
    vmnic06_esxi_B 1
    vmnic07_vmotion_B 1
    vmnic08_FT_B 1
    vmnic09_NSX_B 1
    vmnic10_Prod_B 1
    vmnic11_Backup_B 1
    vHBA_B 2

    Still showing from ESXI (Vmware-ESXi-6.5.0-5969303-Custom-Cisco-6.5.1.1) not in order but at least now when i did host port it bring all vNICs in vCon3 first but not ordered then all vNics in vCon1 but not ordered also.

  9. Hello Muhammad, your issue is close to mine. I would first look at how the vNIC’s are being are being mapped, adjustment may need made to your placement policy. In UCS Manager navigate to Equipment –> Servers –> Your Server –> and then select the VIF Paths tab. In this screen you will see how many paths per VIC are present and where they are going. Each path would represent a vCON. From this you should be able to learn how to adjust you vNIC / vHBA placement to set the desired order.

  10. thanks Joe, i did this already and from VIF path all vNICs is in correct order also i can see it from CLI service profile connection and all in correct order but from ESXI 6.5 unfortunately its not in order, by the way i tried esxi 5.5 it was in correct order !!

  11. Muhammad, did you configure the Admin Host ports so that all of the vNICs for vCon1 are Admin Host port 1 and all of the vNICs for vCon3 are Admin Host port 2?
    Also make sure the VICs are not vCon1 and vCon2, the adapter number you see in UCSM
    doesn’t correlate to the vCon number. There is a cli cmd you can use to check this,
    ssh into UCSM
    scope service-profile server 1/1 (where 1/1 is the chassis/blade associated to the profile in question)
    show vcon-mapping
    show vcon-assign is another good cmd

  12. hello Jeremy, Yes tried
    *First all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA.

    *Second all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 1 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 2 for vNICS and Admin Host port 2 for vHBA.

    *Third all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon2 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA. (But this option put all vNICs under one adapter so i didn’t continue with it)

  13. Hey, after you changed vCon and Admin host port settings how did you test to see if it worked? You must either rebuild ESX or from the DCUI Reset System Configuration. Either of these options will re-enumerate the PCI devices. Just rebooting the host doesn’t update any change you made in UCSM.

  14. For below option only I reinstall ESXI (and this was the last thing I tired today) – Also I tried same with ESXI 5.5 it was working fine !!

    *First all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA.

  15. Ok, maybe I misunderstood. Are you saying that if you install ESX 5.5 or 6 the 6 the vmnic numbering is correct but not with 6.5? That means VMware changed the way PCI device are enumerated. This would require some trial and error to see how 6.5 does things so that the UCS side matches up.

  16. I wish I had access to a UCS environment that had blades with 2 VICs so that I could test this. I am sure this will come up at my customers once they start rolling out 6.5.

  17. If I figure some thing out I will try to share it. I did the implementation with 2 VICs many times and it was working fine with esxi 5.x and UCSM 2.X but now unfortunately some thing wrong.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s