Creating NFS Exports on Virtual Data Movers on EMC Celerra

I learned something new about NFS exports on a recent EMC NS480 project.

The client I was working with wanted some CIFS shares that could be accessed by both Windows and NFS clients at the same time. I initially thought that this wouldn’t be possible because the CIFS shares were going to be on a file system that was mounted on a Virtual Data Mover (vdm) and I thought that you could only create NFS exports on file systems mounted on a data mover (server_2).

After some research on EMC Powerlink I figured out how to do it,

1. First I create the desired folder structure using the Window file share.
2. Next I ran this command to create the NFS export on the vdm –
   “server_export server_2 -Protocol nfs /root_vdm_1/production-FS/unix/nfs1 -option root=172.24.102.240,rw=172.24.102.0/255”

To mount this export on an NFS client you would use this path

172.24.102.40:/root_vdm_1/production-FS/unix/nfs1

Cisco UCS vNIC/vHBA Placement policies

I recently have had a few people ask me what Cisco UCS adapter placement policies are used for and how/when to use them. This post will hopefully answer those questions and give a few examples.

First I will start with the Cisco definition of what vNIC/vHBA placement policies are. This definition was copied from the Cisco UCS Manager GUI Configuration guide – Cisco UCS GUI Configuration Guide

vNIC/vHBA placement policies are used to assign vNICs or vHBAs to the physical adapters on a server. Each vNIC/vHBA placement policy contains two virtual network interface connections (vCons) that are virtual representations of the physical adapters. When a vNIC/vHBA placement policy is assigned to a service profile, and the service profile is associated to a server, the vCons in the vNIC/vHBA placement policy are assigned to the physical adapters. For servers with only one adapter, both vCons are assigned to the adapter; for servers with two adapters, one vCon is assigned to each adapter.

You can assign vNICs or vHBAs to either of the two vCons, and they are then assigned to the physical adapters based on the vCon assignment during server association. Additionally, vCons use the following selection preference criteria to assign vHBAs and vNICs:

All

The vCon is used for vNICs or vHBAs assigned to it, vNICs or vHBAs not assigned to either vCon, and dynamic vNICs or vHBAs.

Assigned-Only

The vCon is reserved for only vNICs or vHBAs assigned to it.

Exclude-Dynamic

The vCon is not used for dynamic vNICs or vHBAs.

Exclude-Unassigned

The vCon is not used for vNICs or vHBAs not assigned to the vCon. The vCon is used for dynamic vNICs and vHBAs.

For servers with two adapters, if you do not include a vNIC/vHBA placement policy in a service profile, or you do not configure vCons for a service profile, Cisco UCS equally distributes the vNICs and vHBAs between the two adapters

Usage Scenarios

Half-width Blades (B200-Mx)

If you have half-width blades (B200-Mx) then you will only ever have a single mezzanine card or vCon1. In this case you would only use a vNIC/vHBA placement policy in these two scenarios:

1. In a VN-Link in hardware configuration and you are attaching a Dynamic vNIC Connection Policy to the service profile. In this scenario a vNIC/vHBA Placement Policy is required so that the dynamic vNICs get assigned after the non-dynamic vNIC/vHBAs. This guarantees that the ESX vmnics and HBAs are at the top of the PCI numbering and that some of the dynamic vNICs aren’t intermixed. Here is a screen shot of this configuration, anything not assigned (dynamic vNICs) are placed below the assigned vNICs/vHBAs.

   
   

2. To force the PCI numbering of the NICs/HBAs as seen by the operating system. If you wanted to make sure the HBAs were seen before the NICs or vice versa you could do that with a placement policy.

 

Full-width Blades (B250-Mx, B440-M1)

1. Use a placement policy to evenly distribute vNICs and vHBAs across 2 mezzanine cards (vCon1 and vCon2). Here is a screen shot of this configuration,
   
   
2. You have two different type of mezzanine cards; Cisco UCS VIC M81KR (aka Palo) and Cisco CNA M71KR (aka Menlo). Lets say for compatibility reasons you want all vNICs on the Palo and all vHBAs on the Menlo. In this scenario would create a placement policy to configure this assignment. Here is a screen shot of this configuration,

   

3. In a VN-Link in hardware configuration and you are attaching a Dynamic vNIC Connection Policy to the service profile and you want all of the Dynamic vNICs to be on one adapter and regular vNICs/vHBAs to be on the other.
   
   
4. You have two different types of mezzanine cards; Cisco UCS VIC M81KR (aka Palo) and Cisco CNA M71KR (aka Menlo) and you are configuring VN-Link in hardware. Lets say for compatibility reasons you want all vNICs on the Palo and all vHBAs on the Menlo. In this scenario would create a placement policy to configure this assignment. Here is a screen shot of this configuration,

   

 

It is important to note that only the Cisco UCS VIC M81KR (aka Palo) allows you to have more than 2 vNICs/vHBAs per adapter and is the only card that allows for VN-Link in hardware where you have up to 54 Dynamic vNICs that are dynamically assigned to VMs that are configured to be part of the UCSM Managed Distributed Virtual Switch. – VN-Link in Cisco UCS

Cisco UCS Ethernet Frame Flows

In this post I am going to walk through the northbound frame flows in the Cisco UCS system. Hopefully this information will help you better understand UCS networking.

In other posts I will walk through the southbound and maybe east-west frame flows. I am breaking the posts down into multiple posts to make it easier for the reader to get through and understand.

I learned most of the information in this post from the Cisco Live 2010 session “Network Redundancy and Load Balancing Designs for UCS Blade Servers” presented by Sean McGee of Cisco. Some of the diagrams in this post were also taken from that same session.

Acronyms used in this post:

CNA – Converged Network Adapters on the mezzanine card

FEX – Fabric Extender 2104XP that lives in the chassis. There are usually two of these per chassis, one for Fabric A and on for Fabric B.

FI = Fabric Interconnect, the 6120s/6140s. Usually deployed as a pair in an HA configuration.

IOM – I/O Module, another name for the FEX.

First lets take a look at the physical components within the UCS System. Below is a logical and physical diagram of the components that make up UCS.

 

 

Now lets take a look at northbound frame flows. The following diagram lists the decision points that take place as the frame travels from the OS up through and out of UCS to the LAN.

 

First we will start with the northbound traffic leaving from the Operating System.

1. OS creates the frame – The happens the same way in UCS as is it does in any other hardware.
2. The OS then has to decide which PCIe Ethernet interface to forward the frame out of. If there multiple PCIe Ethernet interfaces the OS NIC teaming software decides which interface to use based on a hashing algorithm or a round robin mechanism. In the case of VMware and the VIC (Palo) this could be from 1 to 58 choices, realistically you probably will have less than 10.
3. If there is just 1 PCIe interface and Fabric Failover is enabled then it is the UCS Menlo ASIC that makes the frame forwarding decision.

After the frame has entered the physical mezzanine CNA another decision must be made as to which one of the 2 physical CNA ports are used to send the frame to the FEX. This decision depends on how the vNIC was configured in UCSM. When configured a vNIC as part of a Service Profile in UCSM you must choose which fabric to place this vNIC on (A or B) and whether or not Fabric Failover is enabled. The screen shot below shows these configuration options.

With this configuration Fabric A will be used unless for some reason it is down and then Fabric B will be used because Fabric Failover is enabled. For the purpose of this post we will assume the frame is exiting CNA Port 1 that is physically pinned to FEX (Fabric Extension Module in the back of the chassis) 1 on the mid-plane of the chassis. FEX 1 is the left hand FEX as you are looking at the back of the chassis, see the diagram below.

 

The FEX have 8 10GE KR back-end mid-plane ports or traces and 4 front-end 10GE SFP+ ports. The picture below shows the 8 back-end ports and the 4 front-end ports on a FEX. The 8 back-end ports are not physically visible like the picture shows this is just a good picture that logically shows the ports.

The back-end FEX port that the frame goes across depends on which blade this particular OS is running on. In our example we will say blade 3. There is a blade CNA port to FEX back-end port pinning that happens. For blade 3 CNA 1 the back-end FEX port will be port 3 or Eth1/1/3.

Eth X/Y/Z where

• X = chassis number
• Y = mezzanine card number or CNA number
• Z = FEX (IOM) port number

Below is a screen shot of the output of the cmd “show platform software redwood sts”. This command must be executed from the fex-1# context. To get here from the UCSM cli prompt first type “connect local-mgmt” then “ucs-A(local-mgmt)# connect iom 1” The 1 in this cmd is the chassis number.

 

Once the frame is in the FEX it has to be forwarded to the Fabric Interconnect across one of the 4 front-end 10GE SFP+ ports. The port the frame goes across depends on how many FEX to Fabric Interconnect cables are connected and which blade it originally came from. There is a blade to IOM uplink pinning that happens based on the number of uplinked SFP+ ports. This pinning is not user controllable.

Here is a table and diagram showing how the pinning works in the three different uplink options.

1 IOM-FI Cable	2 IOM-FI Cables	4 IOM-FI Cables
All blades CNA 1 goes out this one cable, 8:1 oversubscription	slots 1,3,5,7 are pinned to one uplink and slots 2,4,6,8 are pinned to the other, 4:1 oversubscription	slots 1,5 out 1 slots 2,6 out 2 slots 3,7 out 3 slots 4, 8 out 4

 

For our example there are 4 IOM-FI uplinks.

So to summarize the path the frame has traveled up to this point.

1. OS generated frame.
2. OS chose the PCIe Ethernet Interface using either NIC teaming software. In our case there was only one PCIe Ethernet vNIC assigned to the Service Profile on this blade.
3. Frame was forwarded to CNA 1 because we set Fabric A as the preferred fabric unless it is down. The Fabric Failover ASIC chose CNA 1 because Fabric A is up.
4. CNA 1 is pinned to IOM 3 because it is blade 3.
5. The uplink IOM SFP+ port 2 was used to forward the frame to Fabric Interconnect A because it is blade 3 and there are 4 SFP+ uplinks connected from IOM-FI.

The next step is for Fabric Interconnect A to forward this frame to a northbound L2 LAN switch. Before this can happen the FI has to determine which northbound Ethernet port to send it out of. This decision process is as follows

1. Are there any LAN Pin Groups? If manual pinning is configured for this blade’s vNIC 1 to go out a specific uplink or Port Channel then that port or Port Channel is used to forward the frame. If it is Port Channel then the specific interface of the Port Channel that is used is based on the Port Channel hashing algorithm.
2. Are Port Channels in use? If it is Port Channel then the specific interface of the Port Channel that is used is based on the Port Channel hashing algorithm.

If there are no LAN Pin Groups or Port Channels then automatic pinning is used in a round robin fashion. If there are 8 blades in the chassis and 4 northbound Ethernet uplinks then there will be 2 blades pinned to each uplink. This works in much the same way the default VMware vSwitch teaming works if you familiar with that.

So I think that about covers the northbound Ethernet frame flows. I stated earlier I will be posting the corresponding south, west and east flows in follow on posts.

Cisco UCS Enhancements in Firmware 1.3

Cisco UCS firmware 1.3 was released last week. There are lots of new features in this latest update and Dave Alexander (ucs_dave) outlined most of those in his blog post – http://www.unifiedcomputingblog.com/?p=151

What I will be covering here are the features that Dave didn’t cover nor were they mentioned in the release notes for 1.3 – http://www.cisco.com/en/US/docs/unified_computing/ucs/release/notes/ucs_22863.html

The feature I was most excited about was the advanced BIOS settings control from the Service Profile.

I updated our UCS lab this morning and on the the first login to UCSM after I finished the update I was surprised to see the Fabric Interconnects now have listed in their name what their current cluster role is. Before you had to drill down into each one to see this.

The next change I noticed was that the BMC had been renamed to CIMC (Cisco Integrated Management Controller). I personally like the new name better because I think it makes more since than BMC.

I was then looking around in the Admin tab and noticed a new section labeled Capability Catalog. The UCSM GUI Configuration Guide for 1.3 states this about the catalog

“The capability catalog is a set of tunable parameters, strings, and rules. Cisco UCS Manager uses the catalog to update the display and configurability of components such as newly qualified DIMMs and disk drives for servers.”

The catalog also has an Update Catalog function that can be used to update as Cisco updates it.

On the VM tab there is a new “Configure VMware Integration” wizard to assist in configuring VN-Link in hardware. (this requires the VIC adapter in the server blades and VEM installed on the ESX hosts)

Here are a few screen shots of the new BIOS settings that can be configured. These new settings are implemented via a BIOS Policy that is then tied to a Service Profile or Service Profile Template.

I am sure there may be a few more hidden gems in here but these are the ones I have noticed so far.

Cisco UCS Server Pools: Configuration

In my previous post I discussed Cisco UCS Server Pools and use cases. In this post I will be walking through the configuration of Server Pools, Server Pool Policies and Server Pool Policy Qualifications.

As stated in my previous post one of the use cases for Server Pools is in the server farm model where you have varying amounts of RAM and CPUs in your blades and and you want a way to efficiently deploy a farm of web server, ESX hosts and database servers without having to specifically choose a server blade.

In this configuration example I will walk through the configuration on an auto-populated Server Pool using Server Pool Policies and Server Pool Policy Qualifications.

The first step is to create a Server Pool. To do this go to the Servers tab, Pools.

 

 

Right-click on Server Pools to create a new pool

On the add server page temporarily add a blade and then remove it. If you don’t do this the Finish button stays grayed out. This is one of the UCSM nuances.

The next step is to create a Server Pool Policy Qualification

I am going to create a web server qualification that uses a Memory Qualification for 8 GB RAM

Now to tie it all together with a Server Pool Policy

To use this new auto-populating Server Pool associate a Service Profile Template to is so that when you deploy new Service Profiles from the template you don’t have to select a blade, it will be automatically selected if there is one available in the associated pool.

 

One important note is that if your blades have already been discovered you will need to re-acknowledge them before they will be added to a server pool.

Another important note is that if you re-acknowledge a blade it will reboot it without asking so you probably only want to re-acknowledge blades that are not associated with a Service Profile.

Cisco UCS Server Pools: Use Cases

This is a a two part blog post on Cisco UCS Server Pools. This first post will focus on what Server Pools are and use cases and the next post will focus on configuration.

Cisco UCS Server pools are one of the more mysterious features of UCS. I say mysterious because in my opinion the usage and configuration of them is not very intuitive. Let’s start with Cisco’s definition of a Server Pool, this definition was copied from the Cisco UCS Manager GUI Configuration Guide, Release 1.2(1) found here – http://www.cisco.com/en/US/products/ps10281/products_installation_and_configuration_guides_list.html

“A server pool contains a set of servers. These servers typically share the same characteristics. Those characteristics can be their location in the chassis, or an attribute such as server type, amount of memory,local storage, type of CPU, or local drive configuration.”

You are probably thinking that doesn’t sound very mysterious and you are right it doesn’t until you start digging into the use cases and configuration of Server Pools.

Here are some characteristics of a Server Pool:

• Can be manually populated or auto-populated
  • Manual is where you select the blades that you want to be part of a given pool.
  • Auto-populated works by creating Server Pool Qualifications and Server Pool Policies where you define the specific hardware characteristics and based on that the blades are placed into one pool or another.
• A blade can be in multiple pools at the same time – For example lest say you have a multi-tenancy UCS environment and multiple Server Pools have the same blades in them. The blades are not actually tied to a specific organization in UCS and so when a user deploys a new service profile and uses a Server Pool for the assignment they will grab the next available server in their pool. This blade could have also been in any other Server Pool in any other UCS organization that met the qualification. It works on a first come first serve model. Once a blade is associated with a Service Profile is not available to any
• A service profile template or service profile can be associated with a Server Pool for rapid blade deployments
  
• A blade can be located in any chassis that is management by the UCS Cluster

Server Pool Use Cases

Server Farm Model – In this model you have an environment made up of varying applications that have varying hardware requirements.

For example:

1. Web server farm that needs at least 8 GB RAM and 1 quad core CPU.
2. Database server farm that needs at least 32 GB RAM and 2 quad core CPUs.
3. An ESX server farm that needs at least 48 GB RAM, 2 quad core CPUs and a Cisco UCS VIC M81KR adapter.

Rapid Deployments – This would be a new UCS implementation where you are wanting to deploy new blades as fast and efficiently as possible without have to select specific blades to associate Service Profiles with.

Cisco UCS Palo and EMC PowerPath VE Incompatibility

******UPDATE****** There is a new VMware driver out that corrects this incompatibility. You can download it here – http://downloads.vmware.com/d/details/esx40_cisco_cna_v110110a/ZHcqYmRwKmViZHdlZQ

I came across a much unexpected incompatibility this week between Cisco UCS VIC M81KR (Palo) and PowerPath VE.

I was implementing Cisco Nexus 1000v and EMC PowerPath VE on Cisco UCS blades with the new Cisco UCS VIC M81KR Virtual Interface Card (Palo). We did the Nexus 1000v implementation first and that went flawlessly. Being able to present 4 10G vNICs to the UCS blade with Palo makes for a very easy and trouble free Nexus 1000v  install because you don’t have to put the ESX Service Console in the Nexus 1000v vNetwork Distributed Switch.

After the Nexus 1000v was complete we moved on to PowerPath V/E. This environment was already using PowerPath VE on their other UCS blades but those have the Menlo mezzanine cards with the QLogic HBA chip set. We were expecting this piece of the project to be the easiest because with PowerPath V/E you install it on each ESX host, license it and then that is it. There is zero configuration with PowerPath VE on ESX.

So we downloaded the latest PowerPath VE build from Powerlink (5.4 sp1). We then configured an internal vCenter Update Manager patch repository so that we could deploy PowerPath V/E with VUM. After we deployed PowerPath VE to the first host we noticed in the vSphere client that the LUNs were still owned by NMP. At first I thought maybe it was because it wasn’t licensed yet but then I remembered on the other PowerPath VE installs I did that PowerPath should already own the SAN LUNs.

I SSHed into the host and looked at the vmkwarning log file and sure enough there were lots of these warnings and errors.

WARNING: ScsiClaimrule: 709: Path vmhba2:C0:T1:L20 is claimed by plugin NMP, but current claimrule number 250 indicates that it should be claimed by plugin PowerPath.

vmkernel: 0:00:00:50.369 cpu8:4242)ALERT: PowerPath: EmcpEsxLogEvent: Error:emcp:MpxEsxPathClaim: MpxRecognize failed

It took us a few minutes but then we realized it was probably an incompatibility between Palo and PowerPath VE. We opened both a Cisco TAC and EMC support case on the issue and sure enough there is an incompatibility between the current ESX Palo driver and PowerPath VE. Cisco TAC provided us a beta updated fnic ESX driver for us to test but said that it wasn’t production ready.

We tested the new driver and that fixed the issue. PowerPath VE was then able to claim the SAN LUNs. Since the driver is beta and not fully tested by VMware we are going to hold off using it. Cisco didn’t give us date as to when the driver would be released. I imagine that once VMware gives it their blessing they will post it to the vCenter Update manager repository and it can be installed from there. Cisco may even have it out sooner as a single driver download from their UCS downloads page.

Since both the UCS Palo and PowerPath VE are part of vBlock I am very surprised this wasn’t already tested by Cisco, VMware and EMC. Oh well I know Cisco will have this fixed soon so it isn’t that big of a deal.

How to Enable CDP On Cisco UCS vNICs

If you are familiar with managing VMware ESX 3.5/4.x in an environment that includes Cisco LAN switches then you have probably used “CDP listen state” that is enabled by default on an ESX install. To view this information in vCenter select an ESX host, go to the configuration tab in the right pane, select the Networking link and then click on the little blue call out box next to a vmnic that is uplinked to a vSwitch. A pop-up window opens displaying the CDP information. The information can be invaluable when troubleshooting networking issues. You can determine which switch/switch port the NIC is plugged into, native VLAN and other useful information. This is also a great way to verify that your vSwitch uplinks are going to 2 different physical switches (if you have that option).

 

 

 

As I stated earlier the default CDP configuration on an ESX vSwitch is the listen only state. I have found that the network engineers find it very useful if you configure CDP to advertise as well. When you enable this on a vSwitch the network engineer can issue the “show cdp neighbors” command from the IOS command line and witch switch ports each ESX vmnic is plugged into. This can also be very useful when you and the network engineer are troubleshooting network issues with ESX.

To configure CDP to advertise run this command from the ESX console or from an SSH session.

“esxcfg-vswitch -B both vSwitch0”

To check the state of the CDP configuration run this command..

“esxcfg-vswitch -b vSwitch0”

Note – you must enable CDP on all vSwitches if you want to see every vmnic from the switch side.

If you are using a VMware vNetwork Distributed Switch then you can configure the CDP state from the vCenter GUI. To do this go to the edit settings on the dvSwitch and then go to Advanced.

 

 

Ok, now to the point of configuring all of this on Cisco UCS blades.

By default the vNICs in Cisco UCS have CDP listen and advertise turned off. You can see this from an ESX host that is running on a UCS blade by clicking on the little blue call out box. When the pop-up opens it states that Cisco Discovery Protocol is not available.

 

 

To enable CDP the first thing you must do is to create a new Network Control policy. To do this go to the LAN tab in UCSM, expand Policies, right-click Network Control Policies to create a new policy. Name it something like “Enable-CDP” and select the option to enable CDP.

 

 

The next step is to apply the new policy to the ESX vNICs. If you are using updating vNIC templates then all you need to do is go to each vNIC template for your ESX vNICs select the new policy from the Network Control Policy drop down. If you are not using vNIC templates but you are using an updating Service Profile Template then you can enable it there. If you are using one-off Service Profiles are a non-updating Service Profile then you must go to every Service Profile and enable this new policy on every vNIC.

 

 

 

 

 

 

Now when you click the call-out box you should see the CDP information coming from the Fabric Interconnect that you are plugged into.

 

Cisco UCS VN-Link Hardware Implementation Walk Through

Here is another video blog on Cisco UCS. In this video I walk through the implementation of VMware VN-Link in hardware on the Cisco UCS VIC (Palo) adapter.

http://www.screencast.com/t/YWU3ZGVmYzAt

Command line output of the ESX configuration

esxupdate query
—-Bulletin ID—- —–Installed—– ————Summary————-
ESX400-Update01     2010-03-25T22:03:33 VMware ESX 4.0 Complete Update 1
VEM400-200912272-BG 2010-03-25T22:35:03 Cisco Nexus 1000V  4.0(4)SV1(2)

vem version
Package vssnet-esx4.1.0-00000-release
Version 4.0.4.1.2.0.80-1.9.179
Build 179
Date Wed Dec 9 08:13:45 PST 2009

vem status

VEM modules are loaded

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          4           32                1500    vmnic0,vmnic1
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
UCS-dvSwitch0  256         230         256               vmnic3,vmnic2

VEM Agent (vemdpa) is running

esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          4           32                1500    vmnic0,vmnic1

PortGroup Name      VLAN ID  Used Ports  Uplinks
Service Console     0        1           vmnic0,vmnic1

DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
UCS-dvSwitch0  256         230         256               vmnic3,vmnic2

DVPort ID           In Use      Client
1628                1           vmnic2
1629                1           vmnic3
1630                0
1631                0

esxcfg-nics -l
Name    PCI      Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  08:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:a9 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic1  09:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:ba 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic2  0a:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:aa 1500   Cisco Systems Inc 10G Ethernet NIC
vmnic3  0b:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:bb 1500   Cisco Systems Inc 10G Ethernet NIC

lspci | egrep -i “cisco.*pass”
0c:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0d:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0e:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
0f:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
10:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
11:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
12:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
13:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
14:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
15:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
16:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
17:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
18:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
19:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1a:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1b:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1c:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1d:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1e:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
1f:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
20:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
21:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
22:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)
23:00.0 Ethernet controller: Cisco Systems Inc 10G Ethernet pass-thru NIC (rev a2)

Cisco UCS Manager Equipment Tab Video Walkthrough

My first attempt at a video blog demonstrating the UCSM Equipment tab. I hope to do more of these and get better as I go.

Click the following link to view the Screencast video
UCSM Equipment Tab Walkthrough

Cisco UCS vNIC Failover

When you create a vNIC in UCSM with Cisco UCS you have the option to pin that vNIC to Fabric A or Fabric B and an option to Enable Failover. Most of the servers we deploy on UCS blades are ESX 4 and with ESX we always create two vNICs, one on Fabric A and one on Fabric B and then let ESX handle the NIC teaming and failover. With the new Palo adapter we create 4 vNICs (eth0-eth3), assign two for SC/vMotion in a Standard vSwitch and two for VM networking in a Distributed vSwitch or Nexus 1000v.

I was curious about how the UCS level vNIC failover worked so I built a Windows 2008 R2 blade. In the Service Profile I presented only on vNIC to it and checked the Enable Failover option. I was wondering if Windows would see two NICs or one because someone else had told me that Windows will see two NICs but will auto failover between them if the uplink on Fabric A goes down.

After I loaded the Cisco enic Palo drivers in Windows it only sees one NIC,  here is a screen shot of Device Manager and the Network Connections window.

To test the failover I first SSH into our Cisco 3750E switch to see witch 10GB uplink the MAC address was coming across and it was using Te1/0/1.

Cisco3750#show mac address-table | include 0025.b50c.95ab
1    0025.b50c.95ab    DYNAMIC     Te1/0/1

I then started an extended ping to the server and shutdown switch port Te1/0/1, failover happened almost immediately with only one ping missed. I then check where the MAC address was and it was on the other uplink for Fabric B.

Cisco3750#show mac address-table | include 0025.b50c.95ab
1    0025.b50c.95ab    DYNAMIC     Te1/0/2

Wow!! that was cool, NIC failover without having to mess with NIC teaming drivers in Windows. This is a much cleaner solution than using the old Intel ProSet, HP or Broadcom tools to create the third virtual NIC. It has to be more efficient using hardware based failover as well.

I then wanted to see what would happened if I brought switch port Te1/0/1 back online. To my surprise it switched back over to Fabric A almost immediately after enabling the switch port but this time there were 0 pings missed.

I checked the Windows event log and didn’t see any event where Windows detected a loss of network connectivity. This is really cool that Windows never knew anything about the uplink failure.

Very cool stuff.

Presenting 4 vNICs to VMware ESXi 4 with the Cisco UCS VIC (Palo) Adapter

We got our own Cisco UCS lab at the office last week complete with two 6210S, two server blades and the new Cisco VIC (Palo) adapter.

Here is a diagram I copied from the Chassis 1 –> Hybrid Display tab in UCS Manager.

What is not in the diagram are two Cisco MDS 9124s and a Cisco 3750E with two Ten Gigabit uplinks.

After we got our first ESXi 4.1 blade up booting from a CX4-120 LUN I was itching to present more than two 10GB adapters to the ESXi host.

When I initially looked at how to add more than two vNICs to a Service Profile I couldn’t figure out how to do it. I was thinking there was some new configuration screen somewhere where you had to go to and enable the additional vNICs. I was also unable to find any good documentation on how to do it so a posted a question to the Unified Computing Cisco Support Community – https://supportforums.cisco.com/community/netpro/data-center/unified-computing?view=discussions&start=0

If you haven’t checked out this community and you are interested in Cisco UCS you should definitely browse through it. There are some good tips in there.

This is a very active community and the two times I posted a question it was answered within 12 hours.

I posted a question on how to configure the new VIC (Palo) adapter and to my surprise it is a lot easier to configure than what I initially thought.

All that I had to was add two more vNICs to my Service Profile template. I don’t know why I didn’t just try that first.

I went into my updating Service Profile Template and added eth2 to fabric A and eth3 to fabric B, here is a screen shot

Now be careful because when you add vNICs or modify the storage of a Service Profile or an updating Service Profile Template it will power cycle the blade that is bound to the template. I don’t know if there is a way to change this behavior but I think this is dangerous.

After my ESX 4i server rebooted I first checked the vmnic list using the esxcfg-nics –l command. Here is the output

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:08:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:af 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic1  0000:09:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:bf 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic2  0000:0a:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:ae 1500   Cisco Systems Inc VIC Ethernet NIC
vmnic3  0000:0b:00.00 enic        Up   10000Mbps Full   00:25:b5:0c:95:be 1500   Cisco Systems Inc VIC Ethernet NIC

AWESOME!!!

Next I logged into my vSphere Client and checked the Network Adapters and added a new vSwitch for VM traffic.

Now I can keep my Management Network (Service Console) on a standard vSwitch and put my VM networks in a Distributed vSwitch or in Nexus 1000v without worrying about loosing access because a configuration error.

VMware vSphere vCenter Storage Alarms

One of the complaints of vCenter 2.x was the limited number of alarms that could be configured. With vCenter 4.x VMware addressed this by giving admins the ability to create alarms for just about any event on any object.

There are several alarms that are configured out of box and defined at the topmost vCenter server level.

One of the pre-configured alarms that seems to always be in a warnings state is the “Datastore usage on disk” alarm. 95% of the time this alarm is in a warning state because of the local VMFS data store that is required in ESX 4 for the Service Console OS to live in.

I still have yet to understand why VMware decided to put the COS in a VMDK. If anyone knows why this is beneficial I would love to here from you.

Anyway, back to the “Datastore usage on disk” alarm issue.

I have found that the following procedure is the best way to prevent this alarm on the local VMFS datastores.

1. Go to the Home –>Inventory—>Datastores view in the vSphere Client
2. Right-click on your datacenter and create two new folders, one named LOCAL-COS-DATASTORES and one named SAN-DATASTORES.
3. Drag and drop the local VMFS datastores into the LOCAL-COS-DATASTORES folder and the SAN datastores into the SAN-DATASTORES folder.
4. Go to the alarm definition for “Datastore usage on disk” and disable it by clearing the Enable this alarm box.
5. Create a new alarm at the LOCAL-COS-DATASTORES level named “Datastore usage on LOCAL Disks”.
6. Configure the “Datastore usage on LOCAL Disks” trigger to alarm a warning at 90% and alert at 95%.
7. Create a new alarm at the SAN-DATASTORES level named “Datastore usage on SAN Disks”.
8. Configure the “Datastore usage on SAN Disks” trigger to alarm a warning at 70% and alert at 85%.

Now you should have a lot fewer datastore warning alarms.

Windows 2008 Failover Clustering on vSphere with EMC PowerPath VE

VMware/Microsoft doesn’t support third-party multipathing or the Round Robin path policy for VMs setup in a Microsoft Failover Cluster. This fact is stated in the “Setup for Failover Clustering and Microsoft Cluster Service” PDF on pages 11, 25 and 36 http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_mscs.pdf

The reason why this isn’t supported is due to the way the SCSI-3 command sets are changed when the NMP (Native Multipathing Plugin) Round Robin is set or when third-party multipathing software (EMC PowerPath VE) is installed.

If you try to create a cluster on an ESX host with either of these the Microsoft cluster validation check will fail on the SCSI-3 Persistent Reservation check.

If you need to run a Microsoft Cluster on a host with PowerPath VE installed you can modify the claim rules so that PowerPath VE doesn’t claim the LUNs that the cluster will be using.

Lets say for example the LUNs the Microsoft cluster will be using are LUNs 12 – 14. To modify the claim rules using the ESX COS CLI run these commands

To list the current claim rules run

esxcli corestorage claimrule list

To add claim rules that force the Microsoft cluster LUNs to be owned by the NMP (Native Multipathing Plugin) run these commands

esxcli corestorage claimrule add –rule=210 –plugin=”NMP” –lun=12 –type=”location”
esxcli corestorage claimrule add –rule=211 –plugin=”NMP” –lun=13 –type=”location”
esxcli corestorage claimrule add –rule=212 –plugin=”NMP” –lun=14 –type=”location”
esxcli corestorage claimrule load
esxcli corestorage claimrule run

NOTE – The rule number must be between 201 and 249

Run esxcli corestorage claimrule list to verify the new rules were added.

You can also use the VMware vSphere CLI 4 from a remote machine or the VMware vSphere Management Assistant (vMA). When running esxcil from either of these the commands will look like this.

esxcli –server=esxhostname –username=root corestorage claimrule add –rule=210 –plugin=”NMP” –lun=12 –type=”location” corestorage claimrule add –rule=210 –plugin=”NMP” –lun=12 –type=”location”
esxcli –server=esxhostname –username=root corestorage claimrule add –rule=211 –plugin=”NMP” –lun=13 –type=”location” corestorage claimrule add –rule=210 –plugin=”NMP” –lun=12 –type=”location”
esxcli –server=esxhostname –username=root corestorage claimrule add –rule=212 –plugin=”NMP” –lun=14 –type=”location” corestorage claimrule add –rule=210 –plugin=”NMP” –lun=12 –type=”location”
esxcli –server=esxhostname –username=root corestorage claimrule load
esxcli –server=esxhostname –username=root corestorage claimrule run

esxcli –server=esxhostname –username=root corestorage claimrule list

Using vSphere Host Profiles with 10G and CNA Adapters

I am currently working on a VMware vSphere project that is being built on Cisco UCS blades. The UCS blades have a single dual port CNA that acts as both an HBA and a 10G NIC.

When we tried to apply the host profile built from our reference blade we got this error message “Error during the configuration of the host: Invalid argument: Invalid argument”.

The host profiles log “pyVmomiserver.log” didn’t have any useful information and we couldn’t find anything on the web so opened a case with VMware support.

It turns out this is a known issue with 10G adapters and will be fixed in ESX 4 Update 1. The issue is that the host profile tries to set the speed/duplex of the NIC to auto but since auto isn’t an option for 10G adapters it fails.

To workaround this issue you can edit the host profile and set the NIC speed and duplex to 10000 and full. After we set the NIC speed and duplex the host profile successfully applied to the other hosts.

 

 

 

Synchronizing BlackBerry, Outlook and GMail without a BES Server

My wife recently purchased a BlackBerry Storm and wanted a way to keep her Outlook contacts and calendar in sync with her BlackBerry and Gmail. I did some research and found that it isn’t as easy as I originally thought would be.

A quick Google search pointed my to a free Google application called Google Sync that claims to sync BlackBerry contacts and calendar with Gmail.

This sounded exactly like what I was looking for so I went ahead and installed Google Sync on the BlackBerry and performed a sync. After the sync I had a big mess on my hands, there were duplicates, triplicates and quadruplicates of all of the contacts. I did some research and found that a lot people were having this same issue. Back to square 1.

After we spend a few hours cleaning up the mess I did some more research and found a better solution.

After searching the web and looking at a few different products I came up with a solution using two products;  gSyncit – http://www.daveswebsite.com/software/gsync/ and Handheld Contact – http://outlook.handheldcontact.com/support.php#contact-form

gSyncit synchronizes Outlook Contacts and Calendar with Gmail Contacts and Calendar.

Hendheld Contact synchronizes Outlook Contacts and Calendar with BlackBerry Contacts and Calendar.

gSyncit allows one way or two way synchronization and you can have multiple Outlook profiles syncing with multiple Gmail accounts.

gSyncit has a one time $15 dollar license fee and works with Outlook XP – Outlook 2007 and Windows XP – Windows 7 (32 and 64 bit editions).

Setup and configuration of gSyncit was very simple and didn’t require a reboot.

Handheld Contact is a service that has an annual fee of around $60 dollars. There is client that is installed in Outlook and on the BlackBerry.

The combination of these two products works great. I can add, edit a contact or calendar event in Outlook, Gmail or BlackBerry and it will sync over the air with the other applications.

Automating ESX 4 Installs with the Ultimate Deployment Appliance

In preparation for a vSphere 4 upgrade project I was researching the best way to automate the ESX 4 builds. Even though this is an upgrade project from vCenter 2.5 and ESX 3.5 we have found it best to rebuild the hosts instead of upgrading them. The upgrades take longer and are not as clean as fresh installs.

With ESX 3.x for our larger clients we had used a third part tool called VPloyment from Repton. It seams that Repton is no longer around I was unable to find any information on VPloyment support for ESX 4.

Since VPloyment was not an option my first thought was to script the install using a kickstart script so did some research and found out that the install options set during a manual install of ESX 4 are in a kickstart script that is saved to /root/ks.cfg. I considered putting a kickstart script on a floppy disk image or USB key and then mounting the ESX ISO with the Dell DRAC to perform the automated install. I decided against this route because it wasn’t very efficient and I would have to have a different script for every server.

The other goal I wanted to achieve was to provide a better way around preventing the installer from wiping all the VMFS LUNs that the host is connected. For manual installs we pule the fiber cables or removing the LUNs from the host before the rebuild. With this automated approach I wanted to find a more efficient option.

Enter Mike Laverick and the Ultimate Deployment Appliance (UDA). I had heard about UDA a couple years ago but never took the time to try it out because we started using VPloyment and 90% of the ESX installs I do are less than 5 hosts.

I downloaded UDA 2 from http://www.ultimatedeployment.org/uda20beta.html and using Mike Laverick’s excellent guide http://www.rtfm-ed.co.uk/?page_id=366 I tested it out using VMware Workstation for the UDA and my test ESX 4 host.

Within 30 minutes I had configured UDA and performed an automated install of ESX 4.

One of the coolest features of UDA 2 are the subtemplates that let you create your own variables for things like hostnames and IP addresses. These allow you to have one kickstart script file for any number of hosts.

Here is a short video of an install of ESX 4 in a VMware Workstation VM.

http://goview.com/?id=1e5f67a7-4b84-4b56-962a-f4eec218c412

How to copy profiles in Windows 2008 R2 and Windows 7

In Windows 2008 R2 and Windows 7, Microsoft has disabled the Copy To.. button on the User Profiles screen.

This process has been used for years to create a custom default user profiles and now they go and break that with no real workaround.

They want people to move to Windows 2008 and Windows 7 but they keep doing things like this that keep companies away from deploying it.

They say they did this because there are issues with copying user profiles and instead want you to use the methods in the KB – http://support.microsoft.com/default.aspx?scid=kb;EN-US;959753

I have had a lot of questions on this of late so I decided to do some research to figure out a workaround. Thankfully someone else has already did the work and has come up with a good workaround.

http://joeelway.spaces.live.com/blog/cns!2095EAC3772C41DB!2708.entry

Terminal Server/XenApp vs VDI

There is a lot of hype these days around VDI and it seems like every week there is some new VDI related product or company being announced. With all of these options available how is company ever to decide on which VDI solution to go with?

In my opinion you should first decide on whether or not you want to go with Terminal Services/XenApp or VDI. Both technologies offer the same advantages and feature sets but Terminal Services/XenApp has a much cheaper per user cost and is a proven solution that has been around for 12 years. VDI is cool, hip and is in all of the industry news but it is very young and still evolving.

In my opinion VDI is for special use cases and for applications that will not run in Terminal Services.

Brian Madden did a great session on this at VMworld Europe 2009.

Any company currently looking at VDI should watch this session and first decide if Terminal Services would work for them. If so you can save a lot of money in infrastructure costs by using Terminal Services/XenApp.

http://www.brianmadden.com/blogs/videos/archive/2009/03/23/terminal-services-versus-vdi-brian-s-presentation-from-vmworld-europe-2009.aspx

VMworld 2009 Updates

For anyone interested I am posting updates from VMworld 2009 on Twitter. You can access my Twitter account here.

http://twitter.com/jeremywaldrop