I happened across an interesting scenario this morning while I was doing configuring jumbo MTU of 9216 on the Nexus 5500 switches in our lab.
I wanted to enable jumbo frames and with Nexus 5500s you have to do this with a QoS policy map. Here are the steps:
See my post on configuration and migration to multihop FCoE for details on my lab setup – http://jeremywaldrop.wordpress.com/2013/04/11/cisco-ucs-fcoe-multihop-configuration-and-migration/
When I first configured UCS multihop FCoE I experienced terrible SAN performance. It was so bad that it took 20 minutes to boot a single virtual machine.
This post will walk through the configuration of FCoE multihop in UCS and cover how to migrate from traditional FC to FCoE.
I was doing some testing of the new vSphere 5.1 features and came across something odd with the new Distributed Switch Health Check feature with Cisco UCS.
Cisco released UCS Firmware 2.1 last weekend with a host of new features. Along with this update the UCS Central management software was also released.
This post will summarize most of the new features, I left out the less exciting/ground breaking features that most don’t care about.
With ESXi 5 VMware made it very easy to create custom installation ISOs. I have been doing a lot of upgrades from ESXi 4.1 in our customer sites that have UCS, Nexus 1000v and PowerPath VE so I decided to create a custom ISO that includes the current UCS enic/fnic drivers and Nexus 1000v/PowerPath VE.
When I first started doing these upgrades I would remove the host from the Nexus 1000v VDS, uninstall Nexus 1000v VEM and uninstall PowerPath VE. After upgrading/rebuilding the host I would then use VUM to install Nexus 1000v VEM add it back to the VDS and then install PowerPath VE.
As a follow up from yesterday’s post (UCS PowerShell Health check HTML report) I figured I would post a video demonstrating a UCS configuration script that I just finished.
The video starts with me showing an un-configured UCS cluster, I go through all of the tabs, pools, policies and templates showing that there is no configuration.
I have been slack on my blog posts of late, mostly because of motivation but I have also been very busy with very little free time to spare.
I like being busy and I have been working on some cool projects, mostly with UCS, Nexus, vSphere and EMC storage.
A few weeks ago I finally had a few days in the lab so I decided to take a look a the Cisco UCS Powertool. I didn’t really have anything big planned I was more just curious about it.
One of the exciting new updates from Cisco recently is the 2nd generation UCS hardware.
UCS gen 2 hardware included updated:
For details on the new hardware check out Sean McGee’s post – http://www.mseanmcgee.com/2011/07/ucs-2-0-cisco-stacks-the-deck-in-las-vegas/
This post will focus on configuration of Unified Ports that are part of the 6248 hardware. As with most hardware/software capabilities of the Fabric Interconnects, Unified Ports came from the Cisco Nexus 5500 line that has been around for about a year now.
One of the questions that a lot of our UCS customers ask us is “What is the difference between Update Firmware and Active Firmware?”
I work for a Cisco/EMC/VMware VAR named Varrow and we do a fair amount of VMware SRM projects.
One of the challenges we face in doing SRM failover testing is being able to route between VMs that are brought up at the recovery site in a test bubble. Not all of our customers that use SRM need to be able to test this as a lot of them just need to verify that the VMs boot and can access storage.
For the few that need to be able to do more extensive testing and need to be able to route between VMs on different VLANs we have come up with a simple solution.
This solution will work in any VMware environment but when the customer has Cisco UCS at their recovery site there are additional benefits and functionalities that can be realized.
The solution utilizes a free VM router appliance from Vyatta and can be downloaded from the VMware Virtual Appliance Market – http://www.vmware.com/appliances/directory/va/383813
The advantage you get when you have Cisco UCS at the recovery site is that you can easily create a new vNIC and the way the layer 2 switching works within UCS allows you to be able to route between VMs across multiple ESX hosts.
For non-UCS environments it will not be possible to route between VM on different ESX hosts without some additional hardware; pNIC and L2 switch.
To test this out in our lab here are the steps I followed:
configure
set interfaces ethernet eth0 address 10.120.10.254/24
set interfaces ethernet eth0 description “VLAN-120-SRM-TEST”
set interfaces ethernet eth1 address 10.130.17.253/24
set interfaces ethernet eth1 description “VLAN-117-SRM-TEST
set interfaces ethernet eth2 address 10.13.7.245/24
set interfaces ethernet eth2 description “VLAN-107-SRM-TEST”
commit
save
After the interface configuration I issued these commands to verify configuration and routing
From my 2 test VMs I was then able to ping between them across ESXi hosts
After upgrading our UCS lab to 1.4 all of my Service Profiles and Service Profile Templates were in a warning state with blue boxes around them. There wasn’t an outage and all of our blades were still functioning.
The warnings were due to they way Cisco changed the Serial over LAN Policy. My current Service Profile Template had the Serial over LAN Policy set to <Not Set> but in 1.4 that isn’t a valid option. I changed the Serial over LAN Policy to “No Serial over LAN Policy” and all of the warnings disappeared.
I have to say now that I have had a chance to implement Cisco UCS firmware 1.4 and look at the new features I am blown away. Cisco should have made this version 2.0.
Here is a list of my favorite new features included in 1.4:
SAN Port Channeling:
This allows you to bundle all of the FC connections on a 6120 so that there is 1 logical uplink to the northbound FC switch. Port channels provide faster convergence when there is link failure because a server vHBA doesn’t have to get re-pinned to another uplink.
Maintenance Policies:
Remember when making a change on a Service Profile rebooted the server without asking? Or even worse making a change on an updating Service Profile template caused all of the Service Profiles bound to that template to reboot?
Well Maintenance Policies prevent these unplanned reboots by forcing the user to acknowledge the reboot or you can even schedule it to happen after hours.
Policy Usage Reporting:
Ever wondered what Service Profiles and Service Profile templates were using one of the many UCS policies? Well wonder no more there is now a Show Policy Usage report on every policy in UCSM. I wish there was a similar feature for templates, I am guessing that will be included in a future update.
Enhanced Active Directory Support:
You no longer have to extend the Active Directory schema and you can map UCS roles to Active Directory groups.
Multiple Authentication Sources:
Pre-1.4 you could only have one authentication source at a time. Now you can create authentication domains and have the option to login to all of them.
Local File System Download Option:
A remote SCP, SFTP server is no longer required for downloading new firmware or saving backups.
There are lots of other new features in 1.4 that I didn’t mention here, these were just some of my favorites.
Thanks to Mike Laverick the Ultimate Deployment Appliance now supports ESXi 4.1. – http://www.rtfm-ed.co.uk/vmware-content/ultimate-da/
I developed a script for automating the installation of ESXi 4.1 on Cisco UCS boot from SAN. The UDA template I setup uses a subtemplate for the host names, management IP and vMotion IP.
Here is my configuration:
And these vHBAs
Here is the subtemplate I am using
SUBTEMPLATE;IPADDR;HOSTNAME;VMOTIONIP
UCSESX1;10.150.15.11;ucsesx1;10.150.12.11
UCSESX2;10.150.15.12;ucsesx2;10.150.12.12
Here is the script I am using
accepteula
rootpw –iscrypted $1$NvqID7HA$mkw26HiBQgbso6jk1jX014
clearpart –alldrives –overwritevmfs
autopart –firstdisk=fnic –overwritevmfs
reboot
install url http://[UDA_IPADDR]/[OS]/[FLAVOR]
network –bootproto=static –ip=[IPADDR] –gateway=10.150.15.254 –nameserver=10.150.7.3 –netmask=255.255.255.0 –hostname=[HOSTNAME].domain.com –addvmportgroup=0
%firstboot –unsupported –interpreter=busybox
#Set DNS
vim-cmd hostsvc/net/dns_set –ip-addresses=10.150.7.3,10.150.7.2
#Add pNIC vmnic1 to vSwitch0
esxcfg-vswitch -L vmnic1 vSwitch0
#Add new vSwitch for vMotion
esxcfg-vswitch -a vSwitch1
#Add vMotion Portgroup to vSwitch1
esxcfg-vswitch -A vMotion vSwitch1
#Add pNIC vmnic2 to vSwitch1
esxcfg-vswitch -L vmnic2 vSwitch1
#Add pNIC vmnic3 to vSwitch1
esxcfg-vswitch -L vmnic3 vSwitch1
#Assign ip address to vMotion vmk1
esxcfg-vmknic -a -i [VMOTIONIP] -n 255.255.255.0 -p vMotion
#Assign VLAN to vMotion PortGroup
esxcfg-vswitch -v 12 -p vMotion vSwitch1
#Enable CDP listen and advertise
esxcfg-vswitch -B both vSwitch0
esxcfg-vswitch -B both vSwitch1
sleep 5
#Enable vMotion to vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1
#Set NIC order policy for vMotion port groups
vim-cmd hostsvc/net/vswitch_setpolicy –nicorderpolicy-active=vmnic3 –nicorderpolicy-standby=vmnic2 vSwitch1
#enable TechSupportModes
vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm
vim-cmd hostsvc/enable_local_tsm
vim-cmd hostsvc/start_local_tsm
vim-cmd hostsvc/net/refresh
# NTP time config
echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 0.vmware.pool.ntp.org >> /etc/ntp.conf
echo server 2.vmware.pool.net.org >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/sbin/chkconfig –-level 345 ntpd on
/etc/init.d/ntpd stop
/etc/init.d/ntpd start
#One final reboot
reboot
The four additional vNICs will be used as uplinks to the Cisco Nexus 1000v dvSwitch.
I recently have had a few people ask me what Cisco UCS adapter placement policies are used for and how/when to use them. This post will hopefully answer those questions and give a few examples.
First I will start with the Cisco definition of what vNIC/vHBA placement policies are. This definition was copied from the Cisco UCS Manager GUI Configuration guide – Cisco UCS GUI Configuration Guide
vNIC/vHBA placement policies are used to assign vNICs or vHBAs to the physical adapters on a server. Each vNIC/vHBA placement policy contains two virtual network interface connections (vCons) that are virtual representations of the physical adapters. When a vNIC/vHBA placement policy is assigned to a service profile, and the service profile is associated to a server, the vCons in the vNIC/vHBA placement policy are assigned to the physical adapters. For servers with only one adapter, both vCons are assigned to the adapter; for servers with two adapters, one vCon is assigned to each adapter.
You can assign vNICs or vHBAs to either of the two vCons, and they are then assigned to the physical adapters based on the vCon assignment during server association. Additionally, vCons use the following selection preference criteria to assign vHBAs and vNICs:
- All
The vCon is used for vNICs or vHBAs assigned to it, vNICs or vHBAs not assigned to either vCon, and dynamic vNICs or vHBAs.
- Assigned-Only
The vCon is reserved for only vNICs or vHBAs assigned to it.
- Exclude-Dynamic
The vCon is not used for dynamic vNICs or vHBAs.
- Exclude-Unassigned
The vCon is not used for vNICs or vHBAs not assigned to the vCon. The vCon is used for dynamic vNICs and vHBAs.
For servers with two adapters, if you do not include a vNIC/vHBA placement policy in a service profile, or you do not configure vCons for a service profile, Cisco UCS equally distributes the vNICs and vHBAs between the two adapters
Usage Scenarios
Half-width Blades (B200-Mx)
If you have half-width blades (B200-Mx) then you will only ever have a single mezzanine card or vCon1. In this case you would only use a vNIC/vHBA placement policy in these two scenarios:
Full-width Blades (B250-Mx, B440-M1)
It is important to note that only the Cisco UCS VIC M81KR (aka Palo) allows you to have more than 2 vNICs/vHBAs per adapter and is the only card that allows for VN-Link in hardware where you have up to 54 Dynamic vNICs that are dynamically assigned to VMs that are configured to be part of the UCSM Managed Distributed Virtual Switch. – VN-Link in Cisco UCS
