Cisco UCS FCoE Multihop Configuration and Migration

This post will walk through the configuration of FCoE multihop in UCS and cover how to migrate from traditional FC to FCoE.

The diagram below is an example of the topology we have in our lab. We opted for dedicated FCoE links instead of sharing the northbound LAN uplinks. There are a few reasons for this:

  1. When using vPC for northbound uplinks it isn’t supported to also run FCoE over these links.
  2. Having dedicated LAN and FCoE uplinks is easier to troubleshoot.
  3. Provides consistent performance with guaranteed bandwidth.

image

Our pre-FCoE configuration consisted of 4 8G FC connections configured in a SAN Port Channel in each Fabric.

Nexus Configuration Steps

  1. Cable up 2 additional twinax cables in each Fabric, doesn’t have to be twinax but that is the most cost effective way to go. You can also use 10G-SR SFPs and LC/LC fibre.
  2. Configure an FCoE VLAN on Nexus:
    1. This must be a new VLAN
    2. Must be the same FCoE VLAN used in UCS for the current VSANs 
    3. Must only exist on the Nexus switch for which the VSAN is on. For example in the diagram above VLAN 100 only exists on Fabric A Nexus and VLAN 101 only exists on Fabric B Nexus.
    4. Prune the FCoE VLANs from all trunks except for the FCoE port channels, this includes the vPC peer link.
  3. Map the VLAN to the VSAN
  4. Create a Virtual FC interface (vfc)
  5. Add the vfc to the appropriate VSAN
  6. Bind the vfc to the FCoE port channel

Here is a screen shot of the relevant configuration (Note the port channel and vfc will not be and logged in until the UCS side is completed)

image

Show commands that are useful for verification and troubleshooting

show run ipqos – It is very important that the default FCoE QoS policies are applied to the system qos. If they are not the vfc interface will never initialize and login.

The output of show run ipqos should look like this:

image

show queuing interface e1/20

show vlan fcoe

show fcoe database

show flogi database

show interface vfc 20

show interface vfc 20 trunk vsan

show interface port-channel 20

show interface port-channel 20 trunk

show interface port-channel 20 fcoe

Migration Steps

If properly planned and implemented the migration from FC to FCoE can be performed with little or no disruption in service.

  1. Admin down the FC SAN port channel in Fabric A on both sides
  2. Admin up the FCoE port channel on both sides
  3. Verify the UCS server WWPNs logged in over the vfc interface, this will look similar to this output.

image

Once Fabric A is migrated over check your host SAN access and multipathing software to verify all paths are logged in.

About these ads

3 thoughts on “Cisco UCS FCoE Multihop Configuration and Migration

  1. Pingback: Cisco UCS Multihop FCoE QoS Gotcha | Jeremy Waldrop's Blog

  2. Hi Jeremy, I enjoyed reading your article. Drop me an email as I’ve updated the above topology diagram a bit. – Craig

  3. Hey Jeremy,

    Ive followed the cisco guide on FCoE multihop. My scenario is as follows:
    UCS FI –> N5k fabric –> EMC VNX 5300

    Im having performance issues – to upload an ISO to a datastore which is around 2.5gb in size taking 90 minutes. The link are as above 20gig on each side and the blade has x2 vhbas and multipathing (vmware rr, emc ALUA 4) etc etc.

    Any ideas why it would be so slow? ive disabled any qos policies on UCS only the default config for qos on UCS side. For the N5k the default is running – N5ks running NX-OS 6.x.
    UCS running newest firmware as is vmware enic / fnic drivers.

    Ive got a TAC case open, real head scratcher..
    -Nick

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s