Automating ESX 4 Installs with the Ultimate Deployment Appliance

In preparation for a vSphere 4 upgrade project I was researching the best way to automate the ESX 4 builds. Even though this is an upgrade project from vCenter 2.5 and ESX 3.5 we have found it best to rebuild the hosts instead of upgrading them. The upgrades take longer and are not as clean as fresh installs.

With ESX 3.x for our larger clients we had used a third part tool called VPloyment from Repton. It seams that Repton is no longer around I was unable to find any information on VPloyment support for ESX 4.

Since VPloyment was not an option my first thought was to script the install using a kickstart script so did some research and found out that the install options set during a manual install of ESX 4 are in a kickstart script that is saved to /root/ks.cfg. I considered putting a kickstart script on a floppy disk image or USB key and then mounting the ESX ISO with the Dell DRAC to perform the automated install. I decided against this route because it wasn’t very efficient and I would have to have a different script for every server.

The other goal I wanted to achieve was to provide a better way around preventing the installer from wiping all the VMFS LUNs that the host is connected. For manual installs we pule the fiber cables or removing the LUNs from the host before the rebuild. With this automated approach I wanted to find a more efficient option.

Enter Mike Laverick and the Ultimate Deployment Appliance (UDA). I had heard about UDA a couple years ago but never took the time to try it out because we started using VPloyment and 90% of the ESX installs I do are less than 5 hosts.

I downloaded UDA 2 from http://www.ultimatedeployment.org/uda20beta.html and using Mike Laverick’s excellent guide http://www.rtfm-ed.co.uk/?page_id=366 I tested it out using VMware Workstation for the UDA and my test ESX 4 host.

Within 30 minutes I had configured UDA and performed an automated install of ESX 4.

One of the coolest features of UDA 2 are the subtemplates that let you create your own variables for things like hostnames and IP addresses. These allow you to have one kickstart script file for any number of hosts.

Here is a short video of an install of ESX 4 in a VMware Workstation VM.

http://goview.com/?id=1e5f67a7-4b84-4b56-962a-f4eec218c412

Advertisements

Terminal Server/XenApp vs VDI

There is a lot of hype these days around VDI and it seems like every week there is some new VDI related product or company being announced. With all of these options available how is company ever to decide on which VDI solution to go with?

In my opinion you should first decide on whether or not you want to go with Terminal Services/XenApp or VDI. Both technologies offer the same advantages and feature sets but Terminal Services/XenApp has a much cheaper per user cost and is a proven solution that has been around for 12 years. VDI is cool, hip and is in all of the industry news but it is very young and still evolving.

In my opinion VDI is for special use cases and for applications that will not run in Terminal Services.

Brian Madden did a great session on this at VMworld Europe 2009.

Any company currently looking at VDI should watch this session and first decide if Terminal Services would work for them. If so you can save a lot of money in infrastructure costs by using Terminal Services/XenApp.

http://www.brianmadden.com/blogs/videos/archive/2009/03/23/terminal-services-versus-vdi-brian-s-presentation-from-vmworld-europe-2009.aspx

VMware Fault Tolerance Requirements and Limitations

Here are a good list of links on the new VMware FT requirements and limitations. FT is the new HA feature in vSphere 4 that offers VM level continuous availability.

CPU compatibility with FT

VMware Fault Tolerance Requirements and Limitations

Processors and guest operating systems that support VMware Fault Tolerance

Understanding VMware Fault Tolerance

vSphere 4 Availability Guide

vSphere Pre-requisites Checklist

VMware Site Recovery Manager Service Account

You are probably wondering what I am talking about because the VMware Site Recovery Manager Service in the Windows Services manager runs as “Local System”.

VMware SRM Service

The service account I am talking about is the one that is used by the SRM service to pair the sites together and to connect to vCenter. This is the account used for Credential Based Authentication.

I stumbled upon this after the VMware Site Recovery Manager service wouldn’t start. I looked in the log file located in the C:\Documents and Settings\All Users\Application Data\VMware\VMware Site Recovery Manager\Logs” directory. I opened the latest vmware-dr-x.log file and found this.

Failed to login to VC:

Unexpected MethodFault (vim.fault.InvalidLogin) {

dynamicType = <unset>,

msg =
“Login failed due to a bad username or password.”

As soon as I saw that in the log file I new what had happened. The admin that had installed SRM had changed his password.

I looked in the “Site Recovery Manager Administration Guide” and found a command line tool called srm-config.exe on page 86. On page 86 there is a section called “Reinitializing credential-based authentication after a user ID or password change“.

I created a new local administrator on both SRM servers named SRMAdmin, gave the account vCenter Administrator permissions and SRM Administrator permissions.

I then ran this command to change the account in SRM.

C:\Program Files\VMware\VMware Site Recovery Manager\bin>srm-config.exe -cmd updateuser -cfg ..\config\vmware-dr.xml -u SRMAdmin

capture1


After I ran this on both SRM servers I was able to start the service.

I then had to reconfigure the connection between the two SRM servers.

There isn’t really any recomendation in the admin guide about a dedicated account for SRM. The only reference to the fact that it even uses an account is on page 34 when it is talking about using Credential Based Authentication and on page 86 in the srm-config.exe section.

VMware should add this to their guide.

How to automatically upgrade VMware Tools

The following how-to is from this VMware KB  —  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010048

Before you begin

Before you perform the steps in this article, ensure that you have applied these prerequisite patches:

Note: Ensure to choose the patch for the product you are running.

The steps in this article require these components:

Note: VMware recommends that you install NET 2.0 SP1 to avoid slow operations.

Setting all virtual machines to automatically upgrade VMware Tools

To use VI Toolkit and PowerShell to set all virtual machines to automatically upgrade VMware Tools:
  1. Start VI Toolkit from Start > Programs > VMware > VMware VI Toolkit > VMware VI Toolkit.
  2. Connect to the VirtualCenter Server with the command:

    connect-viserver -server <VirtualCenter Server IP address> -user <VirtualCenter User> -password <VirtualCenter password>

  3. Copy the following command into the Windows VI Toolkit window:

    Foreach ($v in (get-vm)) {
    $vm = $v | Get-View
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo
    $vmConfigSpec.Tools.ToolsUpgradePolicy = “UpgradeAtPowerCycle”
    $vm.ReconfigVM($vmConfigSpec)
    }

    The VI Toolkit window does not display any output until the command has completed for each virtual machine. When the command is done running, a reconfigure task displays in VirtualCenter for every virtual machine.

Note: To disable this setting, follow steps 1 and 2, then copy this command into the VI Toolkit window:

Foreach ($v in (get-vm)) {
$vm = $v | Get-View
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo
$vmConfigSpec.Tools.ToolsUpgradePolicy = “manual”
$vm.ReconfigVM($vmConfigSpec)
}

How to build and Optimize a Windows XP Image for XenDesktop

1. Use a volume license Windows XP ISO that already has SP2/SP3 on it.

2. Build a new VM that has at least 512 MB and a 6 – 8 GB drive. You may need more or less depending the applications that will be installed.

3. Attach the Windows XP ISO to the CD/DVD drive and set it to connect at power on.

4. Attach the Buslogic driver floppy image “/vmimages/floppies/vmscsi-1.2.0.2.flp” to the Floppy drive.
clip_image002

5. On the VM Edit Settings window go to the Options tab and in the General section uncheck the Enable logging check box.
clip_image004

6. Go to the Boot Options section and check the box to force going into the BIOS on next boot.
clip_image006

7. Power on the VM and when the BIOS appears go to Advanced, I/O Device Configuration.
clip_image008

8. Disable the Serial and Parallel ports.
clip_image010

9. Go to the Boot menu and set the CD-ROM first in the boot order.
clip_image012

10. Save and exit.

11. When the install starts press the F6 key to load a mass storage driver.

12. Press S to load the Buslogic driver.
clip_image014

13. Press Enter to choose the driver.
clip_image016

14. Continue installing Windows XP.

15. During the install you will get a couple of driver install prompts, choose Yes to both.
clip_image018

16. On first boot after the initial install the startup wizard will run. On the 2nd screen choose “Not right now”
clip_image020

17. Choose, “No” and then choose Skip at the bottom.
clip_image022

18. Choose “No, not at this time.”
clip_image024

19. Type “temp” for your name.
clip_image026

20. Go to VM, Edit settings, disconnect the CD and floppy, set them both to Client device.
clip_image028

21. Install VMware Tools, choose to do a Custom install.
clip_image030

22. Do not install the Shared Folders driver.
clip_image032

23. After the install completes reboot.

24. Right-click on the desktop and go to display properties, set the theme to Windows Classic.
clip_image034

25. Set the Screen Saver to none. Click Apply.

26. Go to My Computer properties, Advanced tab, Performance settings. Choose “Adjust for best performance”, click Apply.
clip_image036

27. Go to Startup and Recovery settings. Set the debugging information to none, uncheck the “Send an administrative alert”. Click OK.
clip_image038

28. Go to Error Reporting and choose Disable error reporting and uncheck the notify box.
clip_image040

29. Go to the Remote tab and uncheck both boxes. Click Apply.
clip_image042

30. Go to the Automatic Updates tab and turn off automatic updates. Click Apply.
clip_image044

31. Go to the System Restore tab and turn off System Restore. Click Apply.
clip_image046

32. Click OK, to exit the System Properties window.

33. Right-click on the taskbar and go to properties. Go to the Start Menu tab and choose Classic Start Menu, click apply, click Customize.
clip_image048

34. Scroll to the bottom of the Advanced Start menu options and check the box for “Show Small Icons in Start menu” and uncheck the “Use Personalized Menus” box. Click OK, and click OK.
clip_image050

35. Right-click on the desktop and go to properties. Go to the Desktop tab and click Customize Desktop, uncheck the Run Desktop Cleanup Wizard every 60 days, click OK, click Apply.
clip_image052

36. Go to the Appearance tab and click Advanced, uncheck all of the boxes and click OK. Click OK.
clip_image054

37. Right-click on the taskbar, go to Toolbars and choose Quick Launch.
clip_image056

38. Open the Control Panel, go to the View menu, choose Status bar and List.
clip_image058

39. Right-click on the Go button and uncheck the Go Button.
clip_image060

40. Right-click on the toolbar and choose Customize, set the Text options to No text labels and set Icon options to Small icons.
clip_image062

41. Go to Tools, Folder Options, View tab and check/uncheck the boxes below.
clip_image064
clip_image066

42. Click Apply, click Apply to All Folders and choose Yes.

43. Go to Add/Remove Programs, Add/Remove Windows Components. Go to Accessories and Utilities, remove Games and remove the optional components shown below.
clip_image068

44. Remove MSN Explorer, Networking Services, Outlook Express and Windows Messenger.
clip_image070

45. Choose No to reboot now.

46. Open User Accounts, click Change the way users log on or off, Uncheck both boxes and click apply.
clip_image072

47. Right-click on the taskbar, go to properties and check the box to Display Log Off.
clip_image074

48. Right-click on the Start button and choose Open All Users. Delete the three shortcuts for Set Program Access and Defaults, Windows Catalog and Window Update.
clip_image076

49. Go to Control Panel, Administrative Tools, Right-click Computer Management and choose Author.
clip_image078

50. Maximize the window, Expand Services and Applications, select Services, expand the Name column, go to File Save, choose Yes to the prompt. Close Computer Management.
clip_image080

51. Open Computer Management again, go to Services, Stop and disable these services.

a. Computer Browser

b. Error Reporting Service

c. Help and Support

d. Security Center

e. Task Scheduler

f. SSDP Discovery Service

g. Themes

h. Windows Firewall/Internet Connection Sharing (ICS)

i. Wireless Zero Configuration

52. Go to Start, Run and type regedit, go to this key, HKEY_USERS\.DEFAULT\Control Panel\Desktop and delete the SCRNSAVE.EXE name.
clip_image082

53. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters, right-click on Parameters and choose New, DWORD value, type DisableTaskOffload and set the Data to 1.
clip_image084

54. Go to HKEY_CURRENT_USER\Control Panel\Desktop and change the MenuShowDelay to 0.
clip_image086

55. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control, right-click on Control and choose New DWORD value, type ServicesPipeTimeout and set the Data to 120000 decimal.
clip_image088

56. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server and add a new DWORD value called AllowRemoteRPC and set it to 1. This will allow shadowing of the XenDesktop session for remote support. Once this key is set you can shadow a session by running this command “shadow 0 /server:VMNAME.
clip_image090

57. Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run and delete the VMware Tools name.
clip_image092

58. Go to Start, Run and type cmd to open the CMD prompt. Type “FSUTIL behavior set disablelastaccess 1” and press Enter.
clip_image094

59. Type “sfc /purgecache”, press enter, type “sfc /cachesize=0” and press enter. Type exit.
clip_image096

60. Reboot and login as temp.

61. Click on the Windows XP Tour in the system tray, click Cancel.
clip_image098

62. Go to Start, Settings, Printers and Faxes. Go to File, Server Properties, Advanced tab and uncheck all of the boxes.
clip_image100

63. Open Internet Explorer, go to Tools, Windows Update. Install the Microsoft Update by clicking the Go button.
clip_image102

64. Install all High Priority updates and under Software, Optional install Root Certificates Update.
clip_image104

65. After the updates complete, reboot.

66. Login as temp.

67. Run Microsoft Update from the Start menu to get any remaining updates.

68. Right-click on the Language bar and choose Settings.
clip_image106

69. Go to the Advanced tab and turn off advanced text services.
clip_image108

70. Go to the cmd prompt and run this command to unregister the language bar.
“Regsvr32.exe /u msutb.dll”
clip_image110

71. Right-click on Internet Explorer on the desktop and go to properties.

72. Set the Home page to blank and click Delete and then click Delete all, check the box to Also delete files.
clip_image112

73. Go to the Advanced tab and check/uncheck the boxes below
clip_image114
clip_image116

74. Open Internet Explorer and go through the first time startup wizard, choose to keep

75. Change the default search provider from Live Search to Google.
clip_image118
clip_image120
clip_image122

76. Go to My Computer properties, Advanced tab, Performance settings, Advanced tab and set the page file on the C drive to No paging file, click Set. Do not reboot yet.
clip_image124

77. Download and install the User Profile Hive Cleanup tool. The fastest way to find this tool is to search Google for uphclean.
clip_image126

78. Reboot and login as Administrator.

79. Open My Computer, go to Tools, Folder options and set the view option to show hidden files and folders.
clip_image128

80. Right-click on My Computer, go to Properties, Advanced, User Profiles Settings, select temp, copy to, browse to c:\documents and settings\default user, change Permitted to use to Authenticated users. Click OK, Click Yes to the confirm copy question.
clip_image130

81. Log out and login as temp.

82. Delete the local administrators profile from the system properties tool.
clip_image132

83. Log out and login as Administrator.

84. Delete the temp user account and user profile.

85. Go to C:\WINDOWS and delete all the folders that start and end with a $
clip_image134

86. Open C:\Documents and Settings\Default User and delete the temp’s Documents folder.
clip_image136

87. Defragment the C drive.
clip_image138

88. Power down the VM and convert it to a template.

89. Use this template to deploy an XP VM for use with any VDI (XenDesktop, VMware) or use it for any purpose that an XM VM is needed.

How to Build a Windows 2008 VMware ESX VM Template

NOTE – VMware Virtual Center 2.5 Update 4 or vCenter 4 is required to use Guest Customizations with Windows 2008.

http://www.vmware.com/support/vi3/doc/vi3_vc25u4_rel_notes.html#whatsnew

Link to the PDF version of thishttp://www.divshare.com/download/9496736-c61

For Windows 2008 R2 if you are installing on vSphere 4 without Update 1  or on ESX 3.5 without update 5 set the Template VM OS to Windows 2008 instead of Windows 2008 R2. After a new VM is deployed from the template and guest customization as finished you can set the OS back to Windows 2008 R2.

1. Use an ISO for faster OS setup.

2. Build a new VM that has at least 1 GB RAM and a 20 GB drive.

3. Attach the Windows 2008 ISO to the CD/DVD drive and set it to connect at power on.

4. On the VM Edit Settings window go to the Options tab and in the General section uncheck the Enable logging check box.
clip_image002

5. Go to the Boot Options section and check the box to force going into the BIOS on next boot.
clip_image004

6. Power on the VM and when the setup screen appears go to Advanced, I/O Device Configuration.
clip_image006

7. Disable the Serial and Parallel ports.
clip_image008

8. Go to the Boot menu and set the CD-ROM first in the boot order.
clip_image010

9. Save, exit and install Windows 2008, during the install choose either Standard or Enterprise.

10. On first boot up after install set the Administrator password.

11. Close the “Initial Configuration Tasks” window, after the VMware Tools have been installed we will configure the server.
clip_image012

12. Close the “Server Manager” window.
clip_image014

13. Disconnect the Windows 2008 ISO and set the device type to Client Device.
clip_image016

14. Install VMware Tools, choose Custom install type.
clip_image018

15. Disable the “Shared Folders” driver. This driver can cause issues with profiles unloading properly.
clip_image020

16. Set the “Hardware acceleration” to Full before reboot.
clip_image022

17. After reboot from the VMware Tools install configure the time zone.
clip_image024

18. Configure Networking. Uninstall these networking components. It is best to remove any components not needed to free up resources.

a. QoS Packet Scheduler – Not needed unless you are doing QoS at the Windows layer instead of the L2/L3 switch layer.

b. Link Layer Topology Discovery components – http://en.wikipedia.org/wiki/Link_Layer_Topology_Discovery

c. Uncheck TCP/IP v6
clip_image026

19. Leave the IP address set to DHCP unless DHCP isn’t available on the network.

20. Set the computer name to W2K8TEMPALTE
clip_image028

21. Download and install updates. Turn on Windows Updates to download and notify before install.
clip_image030

22. Check for updates and install the latest critical and recommended updates. If the updates fail to download restart and try again.

23. Enable Remote Desktop
clip_image032

24. Disable the Windows Firewall.

25. Below is a screen shot of how the server should be configured.
clip_image034

26. Check the box to “Do not show this window at logon” and click Close.

27. When the Server Manager window opens check the box to “Do not show me this console at logon” and close Server Manager.
clip_image036

28. Go to the Taskbar properties and check these boxes.
clip_image038

29. Go to the Start Menu tab and check these boxes.
clip_image040

30. Click Customize and check these boxes.
clip_image042clip_image044

31. Go to the Notification Area tab and check these boxes.
clip_image046

32. Click OK.

33. Go to Computer properties on the desktop and set the Performance options below.
clip_image048

34. Right-click on the desktop and go to Personalization.
clip_image050

35. Go to Screen Saver and set it to blank.
clip_image052

36. Disable the system screen saver, this is the screen saver that starts when no user is logged into the system. Go to start run and type regedit. Browse to this registry path “HKEY_USERS\.DEFAULT\Control Panel\Desktop” and delete the SCRNSAVE.EXE name.
clip_image054

37. Go to Sounds, when prompted choose No.
clip_image056

38. Set the Sound scheme to No Sounds.
clip_image058

39. Click on Change desktop Icons and uncheck the boxes below.
clip_image060
clip_image062

40. Go to Settings Control Panel. Go to View and choose Status Bar and Details
clip_image064

41. Go to Tools, Folder Options, View tab and check/uncheck these boxes.
clip_image066 clip_image068

42. Apply and apply to folders.

43. Delete the Desktop.ini files on the desktop.
clip_image070

44. Right-click on the desktop and arrange icons by Name.

45. Go to Start, Programs, Administrative Tools, Server Manger and click on Configure IE ESC, set it to off for both Administrators and Users.
clip_image072

46. Go to Internet Explorer properties and set the home page to blank.
clip_image074

47. Go to the Advanced tab and check/uncheck these boxes.
clip_image076 clip_image078

48. Clear the C:\Users\Administrator\AppData\Local\Temp folder.
clip_image080

49. Go to the properties of the C drive and uncheck the “Index this drive for faster searching” box, click apply and choose Apply changes to drive c:\ only.
clip_image082

50. Go to the cmd prompt and type powercfg.exe -h off to disable hibernation. This command also deletes the hiberfile.sys from C:\
clip_image084

51. Delete the page file and reboot. This is so the C drive can be fully defragmented.
clip_image086

52. After reboot login and delete the pagefile.sys from the C: drive. Defragment the C: drive. After defragmentation completes reconfigure the page file system managed.

53. Go to Control Panel, Power options and set the Power Plan to High performance.
clip_image088

54. Click the change plan settings and set both options to Never.
clip_image090

55. Stop and disable the Windows Firewall service.

56. Create a new temporary user named ptemp and add the user to the local administrators group.

57. Log out an login as ptemp

58. Go to Computer properties, Advanced system settings, User Profiles, Settings
clip_image092

59. Select Administrator and choose Copy To… Type C:\Users\Default and change Permitted to use to Authenticated Users.

NOTE – FOR WINDOWS 2008 R2 FOLLOW THESE INSTRUCTIONS FOR COPYING THE USER PROFILE.https://jeremywaldrop.wordpress.com/2009/09/18/how-to-copy-profiles-in-windows-2008-r2-and-windows-7/
clip_image094

60. Logout and login as Administrator, delete the ptemp user profile and user account.

61. Go to C:\Users\Default and delete the old NTUSER.DAT{ files.
clip_image096

62. Delete these folders.
clip_image098

63. Shutdown the VM.

64. Convert the VM to a template.

65. Create a Windows 2008 Customization Specification for Windows 2008. You do not need to worry about the sysprep files. In Vista and Windows 2008 sysprep is in C:\Windows\System32\sysprep. When the guest OS type is set to Vista VMware VC knows that the sysprep files are already on the OS.
clip_image102

66. When a new VM gets deployed from this template choose your Windows 2008 specification.

67. On first boot up the VM will boot up to a login screen, do not login the VM will reboot automatically after a minute. On the next boot up you should see this screen appear.
clip_image104

68. And this screen. This process may take a few minutes, so be patient.
clip_image106

69. The VM will automatically reboot again. Login and shutdown the VM.

71. The VM is now ready for use.

VLAN Trunking with VMware ESX and HP ProCurve Switches

I did a VMware ESX project last week at a client that has HP ProCurve switches. I had worked a little with HP ProCurve switches in the past so I knew the terminology that HP uses is different than what Cisco uses.

Cisco vs. HP terminology:

Trunk

  • Cisco – Trunk links provide VLAN identification for frames traveling between switches and Trunks carry traffic from all VLANs to and from the switch by default but can be configured to carry only specified VLAN traffic. Cisco switches have two Ethernet trunking mechanisms: ISL and IEEE 802.1Q. VMware supports only 802.1Q. To configure a switch port as a trunk you would issue the commands
    switchport mode trunk”
    “switchport trunk encapsulation dot1q”

    To allow only specific VLANs to be trunked you would issue the command
    switchport trunk allowed vlan” and type in the VLANs to allow.

  • HP – A trunk is a method of combining (aggregating) 2 or more switch ports to get more bandwidth between two switches. This is what Cisco calls Port Channeling or Etherchannel. To do the equivalent to of the Cisco trunk you would tag the switch ports connected to the ESX host with the VLANs that need to be “trunked”. Here is a screen shot of the VLAN configuration page of an HP ProCurve switch. Ports 1 and 2 are connected to an ESX host. Notice how ports 1 and 2 are tagged for VLAN 100 and untagged for VLAN 1. VLAN 1 is untagged because it is the default VLAN.
    image

VLAN Assignment

  • Cisco – To assign a switch port to a VLAN on a Cisco switch you would use the command “switchport access vlan 10”
  • HP – To assign a switch port to a VLAN on and HP switch you would untagg that port in the specified VLAN.

Here is a link to another blog post from Scott Lowe where he is discussing the same thing.

http://blog.scottlowe.org/2008/09/05/vmware-esx-nic-teaming-and-vlan-trunking-with-hp-procurve/

ISCSI Multipathing with Clariion CX3-10c and VMware ESX 3.5

I recently did a VMware project using an EMC Clariion CX3-10c and VMware 3.5 update 2. The plan was to use the ISCSI front end ports on the CX3-10 for VMware ISCSI storage connectivity. The design included two dedicated Cisco 3650g switches for the ISCSI network and two dedicate gigabit NICs on the ESX host for ISCSI traffic.

The ESX hosts have a total of 6 gigabit NICs split between 3 physical cards; two onboard, one quad port and one dual port. Below is a screen shot of the original vSwitch design.

image

  • Two NICs from two different physical cards for Service Console and Vmotion.
  • Two NICs from two different physical cards for Virtual Machine traffic.
  • Two NICs from two different physical cards for ISCSI storage traffic. The two ISCSI NICs  were each plugged into a different physical Cisco switch.

The ISCSI front end ports on the CX3-10c were also split between the two dedicated Cisco switches. See diagram below.

image

The IP addresses of all four front end ISCSI ports were originally in the same subnet. For example

SPA0: 10.1.210.30

SPA1: 10.1.210.31

SPB0: 10.1.210.32

SPB1: 10.1.210.33

I then tested connectivity from ESX to the ISCSI front end ports using the vmkping tool. I was able to successfully ping SPA0 and SPB0 but not SPA1 or SPB1.

I initially thought I had an incorrect configuration some where so I verified my ESX configuration and switch port configuration. After about 15 minutes of checking configurations I remembered that the VMkernel networking stack does not load balance like VM networking stacks. A VMkernel networking stack will only use the other NIC on a vSwitch if the first one fails.

I then tested this by unplugging the cables for the NIC in switch 1 and was then able to ping SPA1 and SPB1.

I then went back to the drawing board to come up with a way to see all 4 paths and also provide fault tolerance.

I did some searches on Powerlink and found an article (http://csgateway.emc.com/primus.asp?id=emc156408) that states having all both ISCSI NICs on the same subnet is not supported. After reading this I changed the IP addresses on the ISCSI front end ports on the Clariion to these so that the SPs are in different subnets.

SPA0: 10.1.210.30

SPA1: 10.1.215.30

SPB0: 10.1.210.32

SPB1: 10.1.215.32

CX3-10c ISCSI 2

I then changed the ESX configuration to have two ISCSI vSwitches with one NIC in each vSwitch. See screen shot below.

image

With this configuration I as then able to ping all four ISCSI front end ports on the Clariion from ESX using vmkping.

I then configured the ISCSI software initiator on ESX and added all four targets.

image

I did a rescan on the host and then checked the connectivity status on the Clariion and all four paths were registered.

image

With this configuration I am able to use both NICs, both switches and all four SPs for optimal load balancing.

The failover time is very quick as well.