I recently did a UCS implementation that included the B460-M4 blades. If you aren’t familiar with these beasts you should look them up. They are two B260 full-width blades connected together with a scalability connector on the front to create one giant server. Each of the B260s had a VIC 1340 MLOM to give the server two VIC 1340s.
I did the initial design and configuration consistent with our standard UCS design and logical build out.
We have a standard vNIC/vHBA design for ESXi hosts where there are a minimum of 6 vNICs and 2 vHBAs. The vNICs/vHBAs are split between Fabric A and Fabric b and then map to vSphere vSwitches (Standard and Distributed).
Here is a screen shot of our standard vNIC deign for ESXi on blades with a single VIC
With two VICs we have a different configuration so that we make use of both VICs. In this configuration we place all of Fabric A vNICs/vHBAs on vCon 1 and all of Fabric B vNICs/vHBAs on vCon 2. With this configuration the vNIC to vmnic numbering changes and so does the vSwitch to vmnic uplinks
In this design I implemented this two adapter configuration and built out my templates, pools and policies as usual.
Everything was going well until we built our first ESXi 6 host and couldn’t get management connectivity working. Upon further investigation we realized the vNIC to vmnic mappings were not correct.
After some research I came across this Cisco Bug ID that described our problem to a T – https://quickview.cloudapps.cisco.com/quickview/bug/CSCut78943
I personally wouldn’t call this bug but more of an explanation of the configuration options on the new VICs.
The new VIC 1340/1380s have two PCI channels (1 and 2) and you can control which specific channel the vNIC/vHBA is created on. This new configuration option is called “Admin Host Port” and by default is set to AUTO. With the AUTO setting UCS will round robin each vNIC/vHBA across both Admin Host Ports per vCon.
This Round Robin configuration will place every other vNIC on Admin Host Port 1. This causes an issue because the installed Operating System detects all vNICs on Admin Host Port 1 first and Admin Host Port 2 second.
With the two VICs per blade the configuration that worked like we wanted was to place all of the vNICs on Fabric A on Admin Host Port 1 and the vHBA on Admin Host Port 2 on vCon1 and then the same for Fabric B.
We placed the vHBAs on Admin Host Port 2 so that we could make full use of both PCI lanes.
To configure this on the Service Profile template:
- Go do the Network tab
- Click the Modify vNIC/vHBA Placement link
- Set the placement to Specify Manually
- Place all Fabric A vNICs/vHBAs on vCon 1
- Set the Admin Host Port for all vNICs to 1
- Set the Admin Host Port for the vHBA to 2
- Place all Fabric B vNICs/vHBAs on vCon 2
- Set the Admin Host Port for all vNICs to 1
- Set the Admin Host Port for the vHBA to 2
Here are two screen shots showing this configuration
Here is a screen shot of the applied configuration, notice all Fabric A vNICs are on the Desired Placement of 1 and Fabric B vNICs on 2. Also notice that the Admin Host Port is 1 for all vNICs.
Jeremy Waldrop's Blog 
I am having a hard time getting the Admin Host Port to change while using a LAN/SAN connectivity policy. Do you know if this works?
I have tried to modify the Admin Host port on Service Profiles and Service Profile Templates. It seems to take the change but never actually changes.
Was an OS already installed on the blade before you changed the admin host ports? If so you will need to reload the OS or force it to re-enumerate the PCI devices. If ESX you can do this by selecting reset system configuration in the DCUI
Hi Jeremy,
Thanks for the post. I have a question please.
With a blade with VIC 1380, can you have individual 16 X 10Gbps channels to the FI through IOM 2208?
Can we create 16 vNICs on VMWare ESXi host each mapping to the unique channels (above)
Is this B200-M3/4 with both the MLOM 1340 and an optional VIC 1380 in the mezzanine slot? Your UCS Manager chassis discovery policy should be set to port channel so that the physical uplinks from the IOMs to the FIs are port channelled. The ports from the VIC to the IOM backplane ports are also port channelled. When you create vNICs in UCSM those are pinned to the port channel upstream to the IOM/FI. The PCI channels on the VIC 1340/1380 have nothing to do with the upstream backplane to IOM and IOM to FI. These are just channels on the VIC where vNICs can be placed. If you have 6 vNICs and 2 vHBA you can put half on PCI channel 1 and the other half on PCI channel 2. Also look at this PDF for how the port channeling from the VIC to the IOM works, start on page 56 – http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/b200m4-specsheet.pdf
Hi Jeremy, I have a C460 M4 using Dual VIC’s under UCSM. My service profile is using 2 vCON’s. What I am seeing is my VIF’s are only using 1 patch/port on each VIC. So I have 2 cables pretty much doing nothing. Is there a way to spead the vNICs evenly between all paths when using Dual VICs?
Fabric A:
PCI 4 / 1 – All “A” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)
PCI 9 / 1 – Nothing
Fabric B:
PCI 4 / 2 – Nothing
PCI 9 / 2 – All “B” Fabric Interfaces (If I unplug this interface, the vNICs are down, they don’t move)
Thanks,
Joe
hey, check to see what vCons the two adapters are mapped to. You might need to modify the vNICs that you want on the second adapter to be vCon 3 or 4. SSH into UCSM.
scope org /
scope service-profile SERVICEPROFILE-NAME
show vcon-mapping
Thanks! I did try using all 4 vCons and that got me the bandwidth and faillover paths I wanted. I was hung up on thinking I could only have a vCon per physical adapter.
Thanks Jeremy for your posts, I faced similar issue but till now I cannot fixed.
•Hardware: UCSB-B420-M4, UCS-SP-FI48, Cisco UCS VIC 1380 mezzanine, Cisco UCS VIC 1340 modular LOM, Cisco UCS Port Expander Card (mezz) with UCSM 3.2.1d (infrastructure and servers) adtapters showing as adpter 1 and adapter 3
•I have create below nics from 00 to 05 are mapped to vCon1 then FI-A and from 06 to 11 are mapped to vCon3 then FI-B
vmnic00_esxi_A
vmnic01_vmotion_A
vmnic02_FT_A
vmnic03_NSX_A
vmnic04_Prod_A
vmnic05_Backup_A
vmnic06_esxi_B
vmnic07_vmotion_B
vmnic08_FT_B
vmnic09_NSX_B
vmnic10_Prod_B
vmnic11_Backup_B
I put all these vNICs in LAN Connectivity Policy in same order, then in SP template I did vNIC/vHBA placement manually and put the host port as below
vCon1
vmnic00_esxi_A 1
vmnic01_vmotion_A 1
vmnic02_FT_A 1
vmnic03_NSX_A 1
vmnic04_Prod_A 1
vmnic05_Backup_A 1
vHBA_A 2
vCon3
vmnic06_esxi_B 1
vmnic07_vmotion_B 1
vmnic08_FT_B 1
vmnic09_NSX_B 1
vmnic10_Prod_B 1
vmnic11_Backup_B 1
vHBA_B 2
Still showing from ESXI (Vmware-ESXi-6.5.0-5969303-Custom-Cisco-6.5.1.1) not in order but at least now when i did host port it bring all vNICs in vCon3 first but not ordered then all vNics in vCon1 but not ordered also.
Hello Muhammad, your issue is close to mine. I would first look at how the vNIC’s are being are being mapped, adjustment may need made to your placement policy. In UCS Manager navigate to Equipment –> Servers –> Your Server –> and then select the VIF Paths tab. In this screen you will see how many paths per VIC are present and where they are going. Each path would represent a vCON. From this you should be able to learn how to adjust you vNIC / vHBA placement to set the desired order.
thanks Joe, i did this already and from VIF path all vNICs is in correct order also i can see it from CLI service profile connection and all in correct order but from ESXI 6.5 unfortunately its not in order, by the way i tried esxi 5.5 it was in correct order !!
Muhammad, did you configure the Admin Host ports so that all of the vNICs for vCon1 are Admin Host port 1 and all of the vNICs for vCon3 are Admin Host port 2?
Also make sure the VICs are not vCon1 and vCon2, the adapter number you see in UCSM
doesn’t correlate to the vCon number. There is a cli cmd you can use to check this,
ssh into UCSM
scope service-profile server 1/1 (where 1/1 is the chassis/blade associated to the profile in question)
show vcon-mapping
show vcon-assign is another good cmd
hello Jeremy, Yes tried
*First all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA.
*Second all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 1 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 2 for vNICS and Admin Host port 2 for vHBA.
*Third all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon2 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA. (But this option put all vNICs under one adapter so i didn’t continue with it)
Hey, after you changed vCon and Admin host port settings how did you test to see if it worked? You must either rebuild ESX or from the DCUI Reset System Configuration. Either of these options will re-enumerate the PCI devices. Just rebooting the host doesn’t update any change you made in UCSM.
For below option only I reinstall ESXI (and this was the last thing I tired today) – Also I tried same with ESXI 5.5 it was working fine !!
*First all vNIC/vHBA which mapped to FI-A put all in vCon1 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA and all vNIC/vHBA which mapped to FI-B put all in vCon3 with Admin Host port 1 for vNICS and Admin Host port 2 for vHBA.
“or from the DCUI Reset System Configuration” how can I do it?
https://pubs.vmware.com/vsphere-50/index.jsp#com.vmware.vsphere.install.doc_50/GUID-AD450DA3-881B-4916-9231-9B6998AAC3AA.html
Ok, maybe I misunderstood. Are you saying that if you install ESX 5.5 or 6 the 6 the vmnic numbering is correct but not with 6.5? That means VMware changed the way PCI device are enumerated. This would require some trial and error to see how 6.5 does things so that the UCS side matches up.
hmmm , So it seems I need to do some trial and error 😦 okay I will try .. Any way Thanks for support.
I wish I had access to a UCS environment that had blades with 2 VICs so that I could test this. I am sure this will come up at my customers once they start rolling out 6.5.
If I figure some thing out I will try to share it. I did the implementation with 2 VICs many times and it was working fine with esxi 5.x and UCSM 2.X but now unfortunately some thing wrong.
hey jeremy, first time poster to your site but your site has saved me many times 🙂
i have similar issues as above.
ucsb 200 m4, vic 1340 cards with mezz expanders.
when adding 8 vnics (2 mgmt, 2 server, 2 iscsi, 2 vmotion) they spread out using admin host port 1 & 2 (which is mlom and expander i believe) and believe this shows as just vcon1? Install esxi and all good.
if i add more vnics it shunts the order within esxi and mapping etc.
But if i put all to ‘admin host port 1’ (mlom) i can add no problems without issues to esxi whatsoever.
even with esxi installed i can make the after changes to host admin port 2 to 1.
is there any risk to this configuration with just using the mlom it appears and not the expander?
whats the best way here?
thanks greatly
mark
Hey, so you have 2 VICs? Two VICs are a real PIA. Here is how I configure vNICs to vcon/admin ports.
Place all Fabric A vNICs/vHBAs on vCon1/admin host port 1
Place all Fabric B vNICs/vHBAs on vCon2/admin host port 2
If you have to add vNICs later you will have to add them to vCon 2/admin host port 2 or your ESXi VMNICs will be all screwed up.
Hope that helps
Hi Jeremy thanks for the reply!
No I only have one Vic card with a port expander so essentially only vcon1.
I have vnic1 & 2 as Mgmt, vnic3&4 as server uplinks, (these are on admin host port 1) and vnic5 & 6 as iscsi, vnic 7 & 8 as vmotion (these are on admin port 2) – this is how they were auto placed.
Now everything was fine until I wanted to add a few more vnics… It did the Nic reordering in esxi and lost access to my storage!!
From what I have read in this scenario the first few Nics always stay the same and when you add more it puts them in the middle and starts the reordering?!?
If I place everything on admin host port 1 I don’t seem to have any issues?!
So is there a risk of everything being on admin host port 1?
Thanks
So for a single VIC config I always manually place them like this
vmnic0-mgmt-a – vCon1/Admin Host Port 1
vmnic1-mgmt-b – vCon1/Admin Host Port 1
vmnic2-vmo-a – vCon1/Admin Host Port 1
vmnic3-vmo-b – vCon1/Admin Host Port 1
vmnic4-vm-a – vCon1/Admin Host Port 2
vmnic5-vm-b – vCon1/Admin Host Port 2
vmnic6-iscsi-a – vCon1/Admin Host Port 2
vmnic7-iscsi-b – vCon1/Admin Host Port 2
If you add more make sure you manually place them at the end of this list vCon1/Admin host port 2
Ok 1 last thing if I may….
So the host ports aren’t for load balancing in anyway as if on host port 1 that’s the MLOM and host port 2 is the expander?
So like you for example I have Mgmt on same host port?
Mark
The two admin host ports are two different PCI channels and are not tied to the port expander mezz card. The port expander mezz add more bandwidth via 4 additional backplane connections in a port channel. Take a look at this on page 64 – https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/b200m4-specsheet.pdf
Hmm….on the KVM of the esxi host in network adapters when split over both admin ports it shows 4 mlom and 4 chassis ports…but when change to all host admin port 1 they all change to mlom?