UCS gen 2 hardware included updated:

1. Fabric Interconnect – 6248s
2. Fabric Extender – 2208
3. Mezzanine Adapter – VIC 1280

For details on the new hardware check out Sean McGee’s post – http://www.mseanmcgee.com/2011/07/ucs-2-0-cisco-stacks-the-deck-in-las-vegas/

This post will focus on configuration of Unified Ports that are part of the 6248 hardware. As with most hardware/software capabilities of the Fabric Interconnects, Unified Ports came from the Cisco Nexus 5500 line that has been around for about a year now.

Unified Ports, or UP, gives the ability to configure any of the 32 fixed ports or 16 ports on the expansion module as either 1G, 10G Ethernet or 4G, 8G Fiber Channel. All that it is required is that you have the correct SFP needed for the specific port type.

When I first heard about Unified Ports I thought that the port type was automatically configured based on the SFP that was inserted. After some research I realized that this is not the case.

When I first rebuilt our UCS lab with 6248s they were still on beta code as the new hardware and firmware 2.0 had not been release yet. The configuration guides were not available so it took me a while to figure out how to configure a port to be an uplink FC port.

I finally found the new Unified Ports wizard that is located on the Fabric Interconnect General tab. Here is a screen shot with the “Configure Unified Ports” action highlighted.

When you click the Configure Unified Ports action this message pops up

I guess this is why the ports are not automatically configured based on the type of SFP inserted. That would be dangerous.

When you click Yes the Configure Unified Ports window appears.

At first this interface was very confusing and it is hard to read the key at the bottom due to the light green color of the port type. I didn’t even see the slider bar so I was trying to right-click on the ports to configure them.

When I did finally see the slider I understood how to configure the ports.

The window has 2 buttons at the bottom for configuring the 32 fixed ports and another for configuring the 16 expansion ports.

By default the slider is in the right most position. In this position all of the ports are in the Unconfigured Ethernet mode.

In my environment I wanted ports 9-16 on the expansion module to be FC and the rest to be Ethernet.

To accomplish this I went to the expansion module page and dragged the slider to the middle.

WARNING****If there is a change to the port mode of any of the ports on the fixed module (first 32 ports) and you click finish the Fabric Interconnect will reboot. If there is a change to just the expansion module only that module is reset.

It is best if you can decide up front which ports you want as Ethernet and which ports you want as FC so that you do not have a disruption.

I recommend start Ethernet from left to right and FC from right to left. This usually means that you only need to change the expansion module configuration.

The first section deals with the supported Nexus 5000/5500 FEX topologies. The next section will show the Nexus 7000 FEX topologies.

The following topologies are valid for these NX-OS versions.

Nexus 5000/5500: version 5.x and up

Nexus 7000: version 5.1 and up. Also the M-Series line card N7K-M132XP-12/12L is required. I have heard rumor that when NX-OS 5.2 comes out server PC/vPC will be supported.

The Nexus 2248 FEX is supported with all of these topologies.

Hopefully this post will help clarify the supported topologies.

Nexus 5000/5500 Topologies

Single attached host and single attached FEX with static FEX uplink pinning.

Single attached host and single attached FEX with port channel FEX uplink pinning.

Single attached FEX with static FEX uplink pinning, multi-attached, port channel host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Single attached FEX with port channel FEX uplink pinning, multi-attached host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Single attached FEX with static FEX uplink pinning, dual-homed host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Single attached FEX with port channel FEX uplink pinning, dual-homed host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Dual-attached FEX, single attached host.

Dual-attached FEX, dual-attached host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Dual-attached FEX, dual-attached host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Single attached FEX with static FEX uplink pinning, vPC attached host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Single attached FEX with port channel FEX uplink pinning, vPC attached host.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

All of the Nexus 5000/5500 supported topologies in one image

Nexus 7000 Topologies

You have a lot fewer options when it comes to connecting a FEX to a Nexus 7000. This is mainly due to the more redundant architecture of the Nexus 7000 as compared to the Nexus 5000/5500. The need for dual-attached FEX is not needed.

Single attached host with port channel FEX uplink pinning.

Dual-attached active/standby teaming host with port channel FEX uplink pinning.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Dual-attached active/active teaming host with port channel FEX uplink pinning.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

Dual-attached active/standby teaming host with port channel FEX uplink pinning to a single Nexus 7000.

**HOST SIDE NIC TEAMING SOFTWARE REQUIRED**

There are 2 classes of I/O Modules for the Nexus 7000; M-Series and F-Series.

Below is a list of the modules and their capabilities.

Here are the M-Series Modules:

1. N7K-M132XP-12/12L – http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/Data_Sheet_C78-437757.html
• 32 Ports
• 80 GB bandwidth
• 10 GB interfaces only
• SFP+ interfaces
• Layer 2 and Layer 3
• Required for Nexus 2248 FEX uplinks (This is the only I/O module that supports FEX)
2. N7K-M108X2-12L  – http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-574915.html
• 8 Ports
• 80 GB bandwidth
• 10 GB interfaces only
• X2 interfaces
• Layer 2 and Layer 3
3. N7K-M148GT-11/11L – http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/Data_Sheet_C78-437763.html
• 48 Ports
• 46 GB bandwidth
• 1 GB interfaces only
• Copper interfaces
• Layer 2 and Layer 3
4. N7K-M148GS-11/11L – http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/Data_Sheet_C78-437763.html
• 48 Ports
• 46 GB bandwidth
• 1 GB interfaces only
• SFP fiber interfaces
• Layer 2 and Layer 3

M-Series L vs XL – XL Requires the Scalable Feature License

• N7K-F132XP-15 – http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-605622.html
• 32 Ports
• 320 GB bandwidth
• 10 and 1 GB interfaces
• SFP+ interfaces
• Layer 2 only (an M-Series I/O module is required for Layer 3)
• Supports FabricPath (TRILL)
• Supports FCoE (FCoE N7K-FCOEF132XP License required)

NX-OS Software Licenses for Nexus 7000 – http://www.cisco.com/en/US/prod/collateral/iosswrel/ps9494/ps9372/data_sheet_c78-437306.html

Update vs Activate – On the firmware management tab in UCS Manager there are 2 operations; Activate Firmware and Update Firmware. It is easy to get these 2 operations confused. Hopefully the information below will help clear up any confusion about these operations.

1. Update Firmware  – This is a staging operation where the new firmware version is copied to the backup slot of the component. Think of this in terms of vCenter Update Manager where you have a Staging operation and a Remediation operation. The staging operation copies the new updates to an ESXi host in preparation for performing the remediation.

   The update operation is a non-disruptive operation that can be performed at anytime. When I do firmware updates I perform the staging of all the new firmware before hand so that when I am performing the activations I don’t have to wait for the firmware to be copied to the backup slot. If you have a lot of chassis/blades this can save an hour or 2 of time.

   Not all UCS components have backup slots, only the ones that do not reside on the Fabric Interconnects have backup slots. These components have backup slots:

1. CIMC (Cisco Integrated Management Controller)
2. Adapters
3. I/O Modules (FEX)
   
   
2. Activate Firmware – This operation is when the firmware in the backup slot is swapped with the startup slot. Depending on the component and configuration of your environment this can be a non-disruptive operation. Hopefully you have everything multi-pathed with failover and the user-acknowledge maintenance policy in place so that there is minimal disruption during activation.

   For more information refer to the Cisco UCS Install Guides – http://www.cisco.com/en/US/products/ps10281/prod_installation_guides_list.html 
   

One of the challenges we face in doing SRM failover testing is being able to route between VMs that are brought up at the recovery site in a test bubble.  Not all of our customers that use SRM need to be able to test this as a lot of them just need to verify that the VMs boot and can access storage.

For the few that need to be able to do more extensive testing and need to be able to route between VMs on different VLANs we have come up with a simple solution.

This solution will work in any VMware environment but when the customer has Cisco UCS at their recovery site there are additional benefits and functionalities that can be realized.

The solution utilizes a free VM router appliance from Vyatta and can be downloaded from the VMware Virtual Appliance Market – http://www.vmware.com/appliances/directory/va/383813

The advantage you get when you have Cisco UCS at the recovery site is that you can easily create a new vNIC and the way the layer 2 switching works within UCS allows you to be able to route between VMs across multiple ESX hosts.

For non-UCS environments it will not be possible to route between VM on different ESX hosts without some additional hardware; pNIC and L2 switch.

To test this out in our lab here are the steps I followed:

1. Created 3 new test VLANs that only exist in UCS. It is important that these VLANs do not exist on your northbound layer switch.
   
2. Created a new vNIC template in UCS Manager named vmnic8-srm-b and added it to my ESXi Service Profile Template. This vNIC is configured to use Fabric B as primary but with failover enabled so that if B is down it will failover to A. I normally configure 2 vNICs per VMware vSwitch and let VMware handle the failover but with this solution I needed a vSwitch with only 1 uplink so that routing between VMs across multiple ESX host could be achieved.
   
3. After a reboot of my UCS hosted ESXi host the new vmnic8 was present
   
4. Created a new vSwitch and uplinked vmnic8 to it.
5. Created 3 new VM port groups on the new vSwitch; one for each test VLAN.
   
6. Imported the Vyatta OVF into vCenter and placed the 3 default vNICs into each of the new port groups.
   
7. Powered on the Vyatta VM and logged into the console as root with the default password of vyatta.
8. Configured the 3 Ethernet interfaces using these commands

configure
set interfaces ethernet eth0 address 10.120.10.254/24
set interfaces ethernet eth0 description “VLAN-120-SRM-TEST”
set interfaces ethernet eth1 address 10.130.17.253/24
set interfaces ethernet eth1 description “VLAN-117-SRM-TEST
set interfaces ethernet eth2 address 10.13.7.245/24
set interfaces ethernet eth2 description “VLAN-107-SRM-TEST”
commit
save

After the interface configuration I issued these commands to verify configuration and routing

From my 2 test VMs I was then able to ping between them across ESXi hosts

The warnings were due to they way Cisco changed the Serial over LAN Policy. My current Service Profile Template had the Serial over LAN Policy set to <Not Set> but in 1.4 that isn’t a valid option. I changed the Serial over LAN Policy to “No Serial over LAN Policy” and all of the warnings disappeared.

Here is a list of my favorite new features included in 1.4:

SAN Port Channeling:

This allows you to bundle all of the FC connections on a 6120 so that there is 1 logical uplink to the northbound FC switch. Port channels provide faster convergence when there is link failure because a server vHBA doesn’t have to get re-pinned to another uplink.

Maintenance Policies:

Remember when making a change on a Service Profile rebooted the server without asking? Or even worse making a change on an updating Service Profile template caused all of the Service Profiles bound to that template to reboot?
Well Maintenance Policies prevent these unplanned reboots by forcing the user to acknowledge the reboot or you can even schedule it to happen after hours.

Policy Usage Reporting:

Ever wondered what Service Profiles and Service Profile templates were using one of the many UCS policies? Well wonder no more there is now a Show Policy Usage report on every policy in UCSM. I wish there was a similar feature for templates, I am guessing that will be included in a future update.

Enhanced Active Directory Support:

You no longer have to extend the Active Directory schema and you can map UCS roles to Active Directory groups.

Multiple Authentication Sources:

Pre-1.4 you could only have one authentication source at a time. Now you can create authentication domains and have the option to login to all of them.

Local File System Download Option:

A remote SCP, SFTP server is no longer required for downloading new firmware or saving backups.

There are lots of other new features in 1.4 that I didn’t mention here, these were just some of my favorites.

During the course of the install the customer stated that in a few weeks they will be purchasing another chassis of blades and will be using these for additional ESXi 4.1 hosts. To demonstrate the power of the UCS stateless model I went ahead and provisioned all 8 of the new ESXi hosts even though the hardware did not exist.

Sounds like a magic trick right? Well sort of, but using the power of UCS Service Profiles and boot from SAN here is how we accomplished this:

1. Built out the UCS Service Profiles for the hosts from the Service Profile template.
2. Put 4 of the new ESXi hosts into maintenance mode and powered them down.
3. Disassociated the service profiles from those 4 blades.
4. Using these 4 free blades we associated them with 4 of the Service Profiles for the hardware that did not yet exist.
5. Created the boot LUNs
6. Installed ESXi 4.1.
7. Added the hosts to vCenter.
8. Applied our vSphere Host Profile to configure the new hosts.
9. Powered down the 4 new servers.
10. Pre-built the next 4 the same way.
11. Powered those down and associated the original service profiles with the 4 blades.

Now when the customers gets the new chassis and blades all they have to do is associate the service profiles and they are done.

Most of our clients that are on vCenter 2.5 are running it on 32-bit versions of Windows with most of those being Windows 2003.

For these migrations we have mostly been building new Windows 2008 R2 virtual machines and using the new vCenter Data Migration tool – Migrating an existing vCenter Server database to 4.1 using the Data Migration Tool

The data migration tool is handy for backing up the SSL certificates and registry settings from the vCenter 2.5 server and then restoring those to the new vCenter 4.1 server. The tool will also backup and restore SQL Express databases but most of our clients are running full SQL.

On my most recent upgrade/migration I ran into an SQL issue that I hadn’t seen before. In this case the client was running SQL 2005 as their database server for vCenter 2.5. Since they were already running SQL 2005 we didn’t see the need to move the database or upgrade SQL since 2005 is supported by vCenter 4.1.

During the install of vCenter 4.1 the database upgrade processes started as normal but then about 5 minutes into it we received this error “Exception Thrown while executing SQL script” and the install was halted. We immediately looked at the vCenter install log (vminst.log) to investigate what happened. The log had some additional SQL logging information but nothing specific so I then Googled the exception message and the first link in the search results was just what I was looking for.

Upgrading vCenter Server 4.0 to 4.1 fails with the error: Exception Thrown while executing SQL script

The issue was due to the fact that the vCenter database was set to SQL 2000 compatibility mode. The reason it was set like this is because this server was originally SQL 2000 and then upgraded to SQL 2005.

The fix wasn’t as easy as just changing the database to SQL 2005 compatibility because the failed upgrade left the database in an inconsistent state. To correct the issue we first had to restore the database from the backup we made before the upgrade and then change the compatibility mode to 2005.

The other issue we ran into was more of an annoyance than an issue. After upgrading vCenter to 4.1 and installing vCenter Converter we noticed this error in the vCenter Service Status tool “Unable to retrieve health data from https://<VC servername or IP address>/converter/health.xml”. A quick Google search led us to the fix for this. – vCenter Service Status displays an error for com.vmware.converter

I developed a script for automating the installation of ESXi 4.1 on Cisco UCS boot from SAN. The UDA template I setup uses a subtemplate for the host names, management IP and vMotion IP.

Here is my configuration:

1. 6 GB Boot LUN hosted on an EMC CX4
2. Cisco UCS B200-M1 blades
3. Cisco VIC (Palo) adapters
4. These vNICs

And these vHBAs

Here is the subtemplate I am using

SUBTEMPLATE;IPADDR;HOSTNAME;VMOTIONIP
UCSESX1;10.150.15.11;ucsesx1;10.150.12.11
UCSESX2;10.150.15.12;ucsesx2;10.150.12.12

Here is the script I am using

accepteula
rootpw –iscrypted $1$NvqID7HA$mkw26HiBQgbso6jk1jX014
clearpart –alldrives –overwritevmfs
autopart –firstdisk=fnic –overwritevmfs
reboot
install url http://[UDA_IPADDR]/[OS]/[FLAVOR]
network –bootproto=static –ip=[IPADDR] –gateway=10.150.15.254 –nameserver=10.150.7.3 –netmask=255.255.255.0 –hostname=[HOSTNAME].domain.com –addvmportgroup=0

%firstboot –unsupported –interpreter=busybox

#Set DNS
vim-cmd hostsvc/net/dns_set –ip-addresses=10.150.7.3,10.150.7.2

#Add pNIC vmnic1 to vSwitch0
esxcfg-vswitch -L vmnic1 vSwitch0

#Add new vSwitch for vMotion
esxcfg-vswitch -a vSwitch1

#Add vMotion Portgroup to vSwitch1
esxcfg-vswitch -A vMotion vSwitch1

#Add pNIC vmnic2 to vSwitch1
esxcfg-vswitch -L vmnic2 vSwitch1

#Add pNIC vmnic3 to vSwitch1
esxcfg-vswitch -L vmnic3 vSwitch1

#Assign ip address to vMotion vmk1
esxcfg-vmknic -a -i [VMOTIONIP] -n 255.255.255.0 -p vMotion

#Assign VLAN to vMotion PortGroup
esxcfg-vswitch -v 12 -p vMotion vSwitch1

#Enable CDP listen and advertise
esxcfg-vswitch -B both vSwitch0
esxcfg-vswitch -B both vSwitch1

sleep 5

#Enable vMotion to vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1

#Set NIC order policy for vMotion port groups
vim-cmd hostsvc/net/vswitch_setpolicy –nicorderpolicy-active=vmnic3 –nicorderpolicy-standby=vmnic2 vSwitch1

#enable TechSupportModes
vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm
vim-cmd hostsvc/enable_local_tsm
vim-cmd hostsvc/start_local_tsm
vim-cmd hostsvc/net/refresh

# NTP time config
echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 0.vmware.pool.ntp.org >> /etc/ntp.conf
echo server 2.vmware.pool.net.org >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/sbin/chkconfig –-level 345 ntpd on
/etc/init.d/ntpd stop
/etc/init.d/ntpd start

#One final reboot
reboot

The four additional vNICs will be used as uplinks to the Cisco Nexus 1000v dvSwitch.