This post will walk through the configuration of FCoE multihop in UCS and cover how to migrate from traditional FC to FCoE.
The diagram below is an example of the topology we have in our lab. We opted for dedicated FCoE links instead of sharing the northbound LAN uplinks. There are a few reasons for this:
- When using vPC for northbound uplinks it isn’t supported to also run FCoE over these links.
- Having dedicated LAN and FCoE uplinks is easier to troubleshoot.
- Provides consistent performance with guaranteed bandwidth.
Our pre-FCoE configuration consisted of 4 8G FC connections configured in a SAN Port Channel in each Fabric.
Nexus Configuration Steps
- Cable up 2 additional twinax cables in each Fabric, doesn’t have to be twinax but that is the most cost effective way to go. You can also use 10G-SR SFPs and LC/LC fibre.
- Configure an FCoE VLAN on Nexus:
- This must be a new VLAN
- Must be the same FCoE VLAN used in UCS for the current VSANs
- Must only exist on the Nexus switch for which the VSAN is on. For example in the diagram above VLAN 100 only exists on Fabric A Nexus and VLAN 101 only exists on Fabric B Nexus.
- Prune the FCoE VLANs from all trunks except for the FCoE port channels, this includes the vPC peer link.
- Map the VLAN to the VSAN
- Create a Virtual FC interface (vfc)
- Add the vfc to the appropriate VSAN
- Bind the vfc to the FCoE port channel
Here is a screen shot of the relevant configuration (Note the port channel and vfc will not be and logged in until the UCS side is completed)
Show commands that are useful for verification and troubleshooting
show run ipqos – It is very important that the default FCoE QoS policies are applied to the system qos. If they are not the vfc interface will never initialize and login.
The output of show run ipqos should look like this:
show queuing interface e1/20
show vlan fcoe
show fcoe database
show flogi database
show interface vfc 20
show interface vfc 20 trunk vsan
show interface port-channel 20
show interface port-channel 20 trunk
show interface port-channel 20 fcoe
If properly planned and implemented the migration from FC to FCoE can be performed with little or no disruption in service.
- Admin down the FC SAN port channel in Fabric A on both sides
- Admin up the FCoE port channel on both sides
- Verify the UCS server WWPNs logged in over the vfc interface, this will look similar to this output.
Once Fabric A is migrated over check your host SAN access and multipathing software to verify all paths are logged in.
4 thoughts on “Cisco UCS FCoE Multihop Configuration and Migration”
Pingback: Cisco UCS Multihop FCoE QoS Gotcha | Jeremy Waldrop's Blog
Hi Jeremy, I enjoyed reading your article. Drop me an email as I’ve updated the above topology diagram a bit. – Craig
Ive followed the cisco guide on FCoE multihop. My scenario is as follows:
UCS FI –> N5k fabric –> EMC VNX 5300
Im having performance issues – to upload an ISO to a datastore which is around 2.5gb in size taking 90 minutes. The link are as above 20gig on each side and the blade has x2 vhbas and multipathing (vmware rr, emc ALUA 4) etc etc.
Any ideas why it would be so slow? ive disabled any qos policies on UCS only the default config for qos on UCS side. For the N5k the default is running – N5ks running NX-OS 6.x.
UCS running newest firmware as is vmware enic / fnic drivers.
Ive got a TAC case open, real head scratcher..
Pingback: Cisco UCS FCoE Multihop Configuration and Migration – skminhaj