Archive

Archive for the ‘NetApp’ Category

HOWTO: Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

September 17, 2014 Leave a comment

This script is intended to be able to validate and confirm that the NetApp SnapDrive for Windows v7.0.3 upgrade completed as expected.  This can either be run interactively post installation or can be run via automation (eg: via PSEXEC, against a list of known servers) – as it logs to both console and to log file.   These processes should be performed in the following order:

1) HOWTO: Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

2) HOWTO: Silent Installation for NetApp Windows Host Utilities 6.0.2

3) HOWTO: Silent Installation for NetApp SnapDrive for Windows v7.0.3

4) HOWTO: Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

The following processes are verified:

· Data ONTAP DSM for Windows MPIO is installed and displays version 4.1.4348.1209

· NetApp Windows Host Utilities 6.0.2 is installed and displays version 6.0.4649.1236

o NOTE: Yes, the Program NAME includes the string “…6.0.2” even though the Version is “6.0.4…”

· SnapDrive is installed and displays version 7.0.3.6341

· REG key HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters /v PDORemovePeriod is set to 190 seconds

· REG key HKLM\SYSTEM\CurrentControlSet\Services\Disk /v TimeoutValue is set to 190 seconds

o NOTE: this is the Windows MPIO setting, and is typically only found if NetApp DSM is not installed.  Setting both options ensures that in either case, 190 seconds is the corporate default.

Potential improvements could include:

· Validate or test MPIO Failover Policy matches (Previously recorded version?  Preferred version?  Known required version for this host and is recorded somewhere?)

· INSTALL_*.BAT, INSTALL_*.LOG, and this VERIFY*.BAT/LOG location could be better standardized.  For now, I am utilizing the default software share locations. 

This process has been tested and confirmed on:

· Windows 2012

· Windows 2008 R2 SP1

Other flavours may result in an update to this document, if required.

=== VERIFY_NETAPP_SDW_703.BAT ===

@ECHO OFF

set INSTALL_FOLDER=\\NETWISE.CA\DFS\INSTALLLS\NetApp\

REM %~n0 is a special variable that means “CALLING BATCH FILE NAME”

set LOG_FILE="%INSTALL_FOLDER%\%~n0.LOG"

ECHO.

ECHO. >>%LOG_FILE%

ECHO This should display:     Data ONTAP DSM for Windows MPIO  4.1.4348.1209

wmic product where "Name like ‘%%DSM%%’ and Version like ‘4.1.4348.1209’" get Name, Version | find /i "4.1.4348.1209"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – Data ONTAP DSM for Windows MPIO  4.1.4348.1209 – NOT PRESENT! >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – Data ONTAP DSM for Windows MPIO  4.1.4348.1209 – SUCCESS! >>%LOG_FILE%

ECHO.

ECHO. >>%LOG_FILE%

ECHO This should display:     NetApp Windows Host Utilities 6.0.2  6.0.4649.1236

wmic product where "Name like ‘%%NetApp Windows Host Utilities%%’ and Version like ‘6.0.4649.1236’" get Name, Version | find /i "6.0.4649.1236"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – NetApp Windows Host Utilities 6.0.2  6.0.4649.1236 – NOT PRESENT! >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – NetApp Windows Host Utilities 6.0.2  6.0.4649.1236 – SUCCESS! >>%LOG_FILE%

ECHO.

ECHO. >>%LOG_FILE%

ECHO This should display:     SnapDrive  7.0.3.6341

wmic product where "Name like ‘%%SnapDrive%%’ and Version like ‘7.0.3.6341’" get Name, Version | find /i "7.0.3"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – SnapDrive  7.0.3.6341 – NOT PRESENT! >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – SnapDrive  7.0.3.6341 – SUCCESS! >>%LOG_FILE%

ECHO.

ECHO. >>%LOG_FILE%

ECHO This should display:     PDORemovePeriod    REG_DWORD    0xbe

REG query HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters /v PDORemovePeriod | find /i "0xbe"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – Registry HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters PDORemovePeriod at 190 seconds – NOT PRESENT! >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – Registry HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters PDORemovePeriod at 190 seconds – SUCCESS! >>%LOG_FILE%

ECHO.

ECHO. >>%LOG_FILE%

ECHO This should display:     TimeoutValue    REG_DWORD    0xbe

REG query HKLM\SYSTEM\CurrentControlSet\Services\Disk /v TimeoutValue | find /i "0xbe"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – Registry HKLM\SYSTEM\CurrentControlSet\Services\Disk\TimeoutValue at 190 seconds – NOT PRESENT! >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – Registry HKLM\SYSTEM\CurrentControlSet\Services\Disk\TimeoutValue at 190 seconds – SUCCESS! >>%LOG_FILE%

ECHO.

ECHO. >>%LOG_FILE%

=== VERIFY_NETAPP_SDW_703.BAT ===

=== OUTPUT – SCREEN/CONSOLE ===

This should display:    Data ONTAP DSM for Windows MPIO  4.1.4348.1209

Data ONTAP DSM for Windows MPIO  4.1.4348.1209

This should display:    NetApp Windows Host Utilities 6.0.2  6.0.4649.1236

NetApp Windows Host Utilities 6.0.2  6.0.4649.1236

This should display:    SnapDrive  7.0.3.6341

SnapDrive  7.0.3.6341

This should display:    PDORemovePeriod    REG_DWORD    0xbe

    PDORemovePeriod    REG_DWORD    0xbe

This should display:    TimeoutValue    REG_DWORD    0xbe

    TimeoutValue    REG_DWORD    0xbe

=== OUTPUT – SCREEN/CONSOLE ===

=== OUTPUT – LOG FILE ===

SRVD511 Thu 09/11/2014 17:16:07.26 – Data ONTAP DSM for Windows MPIO  4.1.4348.1209 – SUCCESS!

SRVD511 Thu 09/11/2014 17:16:24.00 – NetApp Windows Host Utilities 6.0.2  6.0.4649.1236 – SUCCESS!

SRVD511 Thu 09/11/2014 17:16:40.28 – SnapDrive  7.0.3.6341 – SUCCESS!

SRVD511 Thu 09/11/2014 17:16:40.30 – Registry HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters PDORemovePeriod at 190 seconds – SUCCESS!

SRVD511 Thu 09/11/2014 17:16:40.31 – Registry HKLM\SYSTEM\CurrentControlSet\Services\Disk\TimeoutValue at 190 seconds – SUCCESS!

SRVD513 Thu 09/11/2014 17:33:57.96 – Data ONTAP DSM for Windows MPIO  4.1.4348.1209 – NOT PRESENT!

SRVD513 Thu 09/11/2014 17:34:11.03 – NetApp Windows Host Utilities 6.0.2  6.0.4649.1236 – NOT PRESENT!

SRVD513 Thu 09/11/2014 17:34:24.07 – SnapDrive  7.0.3.6341 – NOT PRESENT!

SRVD513 Thu 09/11/2014 17:34:24.12 – Registry HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters PDORemovePeriod at 190 seconds – SUCCESS!

SRVD513 Thu 09/11/2014 17:34:24.14 – Registry HKLM\SYSTEM\CurrentControlSet\Services\Disk\TimeoutValue at 190 seconds – NOT PRESENT!

=== OUTPUT – LOG FILE ===

Advertisements
Categories: ISCSI, NetApp, SnapDrive

HOWTO: Silent Installation for NetApp SnapDrive for Windows v7.0.3

September 16, 2014 Leave a comment

This document will assist with silently installing NetApp SnapDrive for Windows v7.0.3.  These processes should be performed in the following order:

1) HOWTO: Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

2) HOWTO: Silent Installation for NetApp Windows Host Utilities 6.0.2

3) HOWTO: Silent Installation for NetApp SnapDrive for Windows v7.0.3

4) HOWTO: Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

The following pre-requisites and requirements are assumed:

· SnapDrive7.0.3_x64.exe and/or SnapDrive7.0.3_x86.exe downloads from the NetApp NOW site

o NOTE: NO testing has been done for 32 bit OS, and no provisions exist in the current batch file.  It is assumed that you are on an actual modern server OS, and it is x64.  If not… perform the installation manually.

· 215-07929_B0 – SnapDrive® 7.0 for Windows® Installation Guide.pdf from NetApp NOW site

· .NET 3.5 SP1 is *ALREADY* installed.  This could be expanded to put in scope tests for .NET 3.5 and/or installs, but the reality is this is going to be used for upgrades of already installed SDW installations, so we’re safe to assume .NET 3.5 is installed.  This process will require updating and testing to validate and test for .NET 3.5 presence and install it

· This is complicated by the fact that Windows Updates to update .NET must be installed after.  For sites that utilize Microsoft’s Windows Update, or a local WSUS this is fine.  Those that utilize SCCM would require a re-push/re-detection of current updates post-.NET installation, updates, reboot, and then continuation of the installation for SDW.  Understandably, this is out of scope for today…

· As this was intended for UPGRADES of systems with at least SnapDrive for Windows, we are assuming that the .NET 3.5 pre-requisite has been met.  Additional work will be required in order to provide the error checking and installation processes for .NET 3.5, as each OS has a slightly different method of doing so.  I’d recommend creating a standardized “INSTALL_DOTNET_35.BAT” that could be called by any other installation that requires it, to keep the process modular.

· As SDW gets upgraded AFTER DSM MPIO and Host Utilities, which both check for, and install, the associated HotFixes, we do not also check for it here.  The portion of the script is still present though, for uniformity.  Also note there is a command line parameter for the actual installer to “SKIP_HOTFIX_CHECK=1”

The general process in the installation batch file is:

· LOG output is in the PRODUCT_FOLDER as %BATCH%.LOG

· MSI LOG is to %TEMP% as %BATCH%.LOG, and is NOT cleaned up after installation.  Use for troubleshooting.

· Check if the product is already installed, using WMIC.  If installed, jumps to CLEANUP

· Check for OS version to determine what hotfixes are required

· Copy hotfixes locally – they will not run properly via a UNC path as they update the system via DISM and the machine account likely has no rights to the share

· Run the installers for the hotfixes and wait for each to complete – it does not check if they are missing or already present

· Copy installation MSI locally

· Run the installer for the MSI and wait for complete

· Cleans up local files

=== INSTALL_NETAPP_SDW_703.BAT ===

@echo off

REM

REM Enter the path to the installation folder – does not require quotes, quotes are assumed, with or without spaces – MUST not contain training "\"

REM

set INSTALL_FOLDER=\\NETWISE.CA\DFS\INSTALLS\NetApp\SnapDrive for Windows v7.0.3

set PRODUCT_NAME=NetApp SnapDrive for Windows v7.0.3

set LOG_FILE="%INSTALL_FOLDER%\%~n0.LOG"

REM

REM MUST UPDATE THE SERVICE ACCOUNT and PASSWORD!

REM

set SVCUSERNAME=<DOMAIN>\<USERNAME>

set SVCUSERPASSWORD=XXXXXXXXXXXXX

set ESXUSERNAME=

set ESXUSERPASSWORD=

set TRANSPORTUSERNAME=

set TRANSPORTPASSWORD=

set ESXIPADDRESS=

set SMVIIPADDRESS=

ECHO. >>%LOG_FILE%

:CHECK_ALREADY_INSTALLED

REM

REM Use a Product Name substring search and the exact version number.  Note the double percent characters are correct

REM

wmic product where "Name like ‘%%SnapDrive%%’ and Version like ‘7.0.3.6341’" get Name, Version | find /i "7.0.3"

if %ERRORLEVEL%==1 goto CHECK_OS_VERSION

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% already installed! >>%LOG_FILE%

goto CLEANUP

:CHECK_OS_VERSION

REM

REM Check for OS Version

REM

set OS_VER=0

set OS_NAME=0

for /f "tokens=1-5* delims=[] " %%A in (‘ver’) do set OS_VER=%%D

if %OS_VER%==6.3.9200 set OS_NAME=2012R2

if %OS_VER%==6.2.9200 set OS_NAME=2012

if %OS_VER%==6.0.6001 set OS_NAME=2008

if %OS_VER%==6.0.6002 set OS_NAME=2008SP2

if %OS_VER%==6.1.7600 set OS_NAME=2008R2

if %OS_VER%==6.1.7601 set OS_NAME=2008R2SP1

goto INSTALL_SDW

:INSTALL_PREREQ_HOTFIXES

REM

REM Will copy ALL *.MSU files locally from matching folder name to OS_NAME list above, and then install ALL *.MSU files from the local drive,

REM waiting for each to complete before continuing. 

REM

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying pre-requisite hotfixes locally >>%LOG_FILE%

xcopy "%INSTALL_FOLDER%"\WINDOWS_HOTFIXES\%OS_NAME%\*.msu %TEMP%\SDW.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing pre-requisite hotfixes >>%LOG_FILE%

for /f "tokens=1* delims=" %%A in (‘dir %temp%\SDW.INST /b’) do start /w wusa.exe %temp%\%%A /quiet /norestart

goto INSTALL_SDW

:INSTALL_SDW

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying installation files locally >>%LOG_FILE%

mkdir %TEMP%\SDW.INST

xcopy "%INSTALL_FOLDER%"\"SnapDrive7.0.3_x64.exe" %TEMP%\SDW.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing main application >>%LOG_FILE%

start /w "SDW_INSTALL" "%TEMP%\SDW.INST\SnapDrive7.0.3_x64.exe" /s /v"/qb SILENT_MODE=1 /Li %TEMP%\%~n0.LOG INSTALLDIR=\"c:\Program Files\NetApp\SnapDrive\" SVCUSERNAME=%SVCUSERNAME% SVCUSERPASSWORD=%SVCUSERPASSWORD% SVCCONFIRMUSERPASSWORD=%SVCUSERPASSWORD% IGNORE_COMPMGMT_RUNNING=1 SDW_WEBSRV_TCP_PORT=808 SDW_WEBSRV_HTTP_PORT=4094 SDW_WEBSRV_HTTPS_PORT=4095 TRANSPORT_SETTING_ENABLE=1 TRANSPORT_PRT_SELECTION=1 SKIP_HOTFIX_CHECK=1 ADD_WINDOWS_FIREWALL=1"

wmic product where "Name like ‘%%SnapDrive%%’ and Version like ‘7.0.3.6341’" get Name, Version | find /i "7.0.3"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as NOT INSTALLED >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as INSTALLED >>%LOG_FILE%

goto CLEANUP

:CLEANUP

REG QUERY "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager" /v PendingFileRenameOperations | find /i "C:"

if %ERRORLEVEL%==0 ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% HAS a reboot pending… >>%LOG_FILE%

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% cleaning up installation files locally >>%LOG_FILE%

if exist %TEMP%\SDW.INST rmdir %TEMP%\SDW.INST /q /s

ECHO. >>%LOG_FILE%

=== INSTALL_NETAPP_SDW_703.BAT ===

=== SAMPLE LOG OUTPUT ===

SRVDC113B 11/09/2014 16:52:26.38 – NetApp SnapDrive for Windows v7.0.3 already installed!

SRVDC113B 11/09/2014 16:52:26.38 – NetApp SnapDrive for Windows v7.0.3 cleaning up installation files locally

SRVD511 Thu 09/11/2014 16:53:55.57 – NetApp SnapDrive for Windows v7.0.3 reports as INSTALLED

SRVD511 Thu 09/11/2014 16:53:55.59 – NetApp SnapDrive for Windows v7.0.3 HAS a reboot pending…

SRVD511 Thu 09/11/2014 16:53:55.59 – NetApp SnapDrive for Windows v7.0.3 cleaning up installation files locally

SRVD513 Thu 09/11/2014 17:47:28.82 – NetApp SnapDrive for Windows v7.0.3 copying installation files locally

SRVD513 Thu 09/11/2014 17:47:29.00 – NetApp SnapDrive for Windows v7.0.3 installing main application

SRVD513 Thu 09/11/2014 17:50:23.21 – NetApp SnapDrive for Windows v7.0.3 reports as INSTALLED

SRVD513 Thu 09/11/2014 17:50:23.21 – NetApp SnapDrive for Windows v7.0.3 HAS a reboot pending…

SRVD513 Thu 09/11/2014 17:50:23.21 – NetApp SnapDrive for Windows v7.0.3 cleaning up installation files locally

=== SAMPLE LOG OUTPUT ===

Categories: ISCSI, NetApp, SnapDrive

HOWTO: Silent Installation for NetApp Windows Host Utilities 6.0.2

September 15, 2014 Leave a comment

This document will assist with silently installing NetApp Windows Host Utilities 6.0.2.  These processes should be performed in the following order:

1) HOWTO: Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

2) HOWTO: Silent Installation for NetApp Windows Host Utilities 6.0.2

3) HOWTO: Silent Installation for NetApp SnapDrive for Windows v7.0.3

4) HOWTO: Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

The following pre-requisites and requirements are assumed:

· netapp_windows_host_utilities_6.0.2_x64.msi and/or netapp_windows_host_utilities_6.0.2_x86.msi downloads from the NetApp NOW site

o NOTE: NO testing has been done for 32 bit OS, and no provisions exist in the current batch file.  It is assumed that you are on an actual modern server OS, and it is x64.  If not… perform the installation manually.

· 215-07794_A0 – Installation and Setup Guide.pdf from NetApp NOW site

· As this was intended for UPGRADES of systems with at least SnapDrive for Windows, we are assuming that the .NET 3.5 pre-requisite has been met.  Additional work will be required in order to provide the error checking and installation processes for .NET 3.5, as each OS has a slightly different method of doing so.  I’d recommend creating a standardized “INSTALL_DOTNET_35.BAT” that could be called by any other installation that requires it, to keep the process modular.

· The following Windows HotFixes:

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008\Windows6.0-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008\Windows6.0-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2520235-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2528357-v2-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB979711-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2528357-v2-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB968675-x64.msu

I:\INSTALLS\NETAPP\Windows Host Utilities 6.0.2\WINDOWS_HOTFIXES\2012\Windows8-RT-KB2796995-x64.msu

o NOTE: the Folder path for the HotFixes from Microsoft MUST be:

§ IN the installation folder

§ IN a folder called “WINDOWS_HOTFIXES”

§ IN folders based on OS of 2008, 2008SP2, 2008R2, 2008R2SP1, 2012 (there are no 2012R2 hotfixes required)

· Links to obtain the hotfixes are located in the Administration Guide, on Page 56.  Links will have a HF # only, eg: “2796995”.  These links may include downloads for multiple OS’s (eg: Vista/Windows 7 – equal to 2008 and 2008 R2), and each must be downloaded and located separately.  These hotfixes will unfortunately be named such as “457359_intl_x64_zip.exe” for KB2796995.  This is unfortunate as it is near impossible to correlate the versions and the files.  I would recommend saving the files as “KB2796995-457359_intl_x64_zip.exe” with the KB######## as the prefix.  Upon downloading, UNZIP the files in the same folder/folder structure, and you will note the final output of “Windows8-RT-KB2796995.msu”.  You will quickly be able to tell if you have only downloaded one version if you see “Windows6.1” for a 2008 file and “Windows6.0 for a 2008R2” file, etc.

The general process in the installation batch file is:

· LOG output is in the PRODUCT_FOLDER as %BATCH%.LOG

· MSI LOG is to %TEMP% as %BATCH%.LOG, and is NOT cleaned up after installation.  Use for troubleshooting.

· Check if the product is already installed, using WMIC.  If installed, jumps to CLEANUP

· Check for OS version to determine what hotfixes are required

· Copy hotfixes locally – they will not run properly via a UNC path as they update the system via DISM and the machine account likely has no rights to the share

· Run the installers for the hotfixes and wait for each to complete – it does not check if they are missing or already present

· Copy installation MSI locally

· Run the installer for the MSI and wait for complete

· Cleans up local files

This process has been tested and confirmed on:

· Windows 2012

· Windows 2008 R2 SP1

Other flavours may result in an update to this document, if required.

=== INSTALL_NETAPP_HOST_UTILS602.BAT ===

@echo off

REM

REM Enter the path to the installation folder – does not require quotes, quotes are assumed, with or without spaces – MUST not contain training "\"

REM

set INSTALL_FOLDER=\\NETWISE.CA\DFS\INSTALLS\NetApp\Windows Host Utilities v6.0.2

set PRODUCT_NAME=NetApp Windows Host Utilities 6.0.2

set LOG_FILE="%INSTALL_FOLDER%\%~n0.LOG"

ECHO. >>%LOG_FILE%

:CHECK_ALREADY_INSTALLED

REM

REM Use a Product Name substring search and the exact version number.  Note the double percent characters are correct

REM NOTE: The PRODUCT is “6.0.2” but the VERSION is “6.0.4649.1236” – fun stuff…

REM

wmic product where "Name like ‘%%NetApp Windows Host Utilities%%’ and Version like ‘6.0.4649.1236’" get Name, Version | find /i "6.0.4649.1236"

if %ERRORLEVEL%==1 goto CHECK_PREVIOUS_VERSION

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% already installed! >>%LOG_FILE%

goto CLEANUP

:CHECK_PREVIOUS_VERSION

wmic product where "Name like ‘%%NetApp Windows Host Utilities%%’ and Version like ‘6.0.4424.1239’" get Name, Version | find /i "6.0.4424.1239"

if %ERRORLEVEL%==1 goto CHECK_OS_VERSION

if %ERRORLEVEL%==0 goto REMOVE_PREVIOUS_VERSION

goto CHECK_OS_VERSION

:REMOVE_PREVIOUS_VERSION

echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% removing v6.0.4424.1239 version >>%LOG_FILE%

start /w MsiExec.exe /qb /X{203B48A2-023D-4E77-B586-E5FCB2E4A382}

goto CHECK_OS_VERSION

:CHECK_OS_VERSION

echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% has no previous version installed. >>%LOG_FILE%

REM

REM Check for OS Version

REM

set OS_VER=0

set OS_NAME=0

for /f "tokens=1-5* delims=[] " %%A in (‘ver’) do set OS_VER=%%D

if %OS_VER%==6.3.9200 set OS_NAME=2012R2

if %OS_VER%==6.2.9200 set OS_NAME=2012

if %OS_VER%==6.0.6001 set OS_NAME=2008

if %OS_VER%==6.0.6002 set OS_NAME=2008SP2

if %OS_VER%==6.1.7600 set OS_NAME=2008R2

if %OS_VER%==6.1.7601 set OS_NAME=2008R2SP1

goto INSTALL_PREREQ_HOTFIXES

:INSTALL_PREREQ_HOTFIXES

REM

REM Will copy ALL *.MSU files locally from matching folder name to OS_NAME list above, and then install ALL *.MSU files from the local drive,

REM waiting for each to complete before continuing. 

REM

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying pre-requisite hotfixes locally >>%LOG_FILE%

xcopy "%INSTALL_FOLDER%"\WINDOWS_HOTFIXES\%OS_NAME%\*.msu %TEMP%\HOSTUTIL.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing pre-requisite hotfixes >>%LOG_FILE%

for /f "tokens=1* delims=" %%A in (‘dir %temp%\HOSTUTIL.INST /b’) do start /w wusa.exe %temp%\HOSTUTIL.INST\%%A /quiet /norestart

goto INSTALL_HOST_UTIL

:INSTALL_HOST_UTIL

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying installation files locally >>%LOG_FILE%

mkdir %TEMP%\HOSTUTIL.INST

xcopy "%INSTALL_FOLDER%"\"netapp_windows_host_utilities_6.0.2_x64.msi" %TEMP%\HOSTUTIL.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing main application >>%LOG_FILE%

start /w "HOST_UTIL" msiexec.exe /quiet /i"%TEMP%\HOSTUTIL.INST\netapp_windows_host_utilities_6.0.2_x64.msi" /l*v %TEMP%\%~n0.LOG MULTIPATHING=1 /NORESTART

wmic product where "Name like ‘%%NetApp Windows Host Utilities%%’ and Version like ‘6.0.4649.1236’" get Name, Version | find /i "6.0.4649.1236"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as NOT INSTALLED >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as INSTALLED >>%LOG_FILE%

goto CLEANUP

:CLEANUP

REG QUERY "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager" /v PendingFileRenameOperations | find /i "C:"

if %ERRORLEVEL%==0 ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% HAS a reboot pending… >>%LOG_FILE%

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% cleaning up installation files locally >>%LOG_FILE%

if exist %TEMP%\HOSTUTIL.INST rmdir %TEMP%\HOSTUTIL.INST /q /s

ECHO. >>%LOG_FILE%

=== INSTALL_NETAPP_HOST_UTILS602.BAT ===

=== OUTPUT – LOG FILE ===

SRVDC217B Thu 09/11/2014 16:30:02.27 – NetApp Windows Host Utilities 6.0.2 already installed!

SRVDC217B Thu 09/11/2014 16:30:02.27 – NetApp Windows Host Utilities 6.0.2 cleaning up installation files locally

SRVDC113B 11/09/2014 16:47:14.10 – NetApp Windows Host Utilities 6.0.2 removing v6.0.4424.1239 version

SRVDC113B 11/09/2014 16:47:42.27 – NetApp Windows Host Utilities 6.0.2 has no previous version installed.

SRVDC113B 11/09/2014 16:47:42.27 – NetApp Windows Host Utilities 6.0.2 copying pre-requisite hotfixes locally

SRVDC113B 11/09/2014 16:47:42.36 – NetApp Windows Host Utilities 6.0.2 installing pre-requisite hotfixes

SRVDC113B 11/09/2014 16:47:57.42 – NetApp Windows Host Utilities 6.0.2 copying installation files locally

SRVDC113B 11/09/2014 16:47:57.57 – NetApp Windows Host Utilities 6.0.2 installing main application

SRVDC113B 11/09/2014 16:50:15.93 – NetApp Windows Host Utilities 6.0.2 reports as INSTALLED

SRVDC113B 11/09/2014 16:50:15.93 – NetApp Windows Host Utilities 6.0.2 cleaning up installation files locally

=== OUTPUT – LOG FILE ===

Categories: ISCSI, NetApp, SnapDrive

HOWTO: Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

September 12, 2014 Leave a comment

This document will assist with silently installing NetApp Data ONTAP DSM for Windows MPIO v4.1.   These processes should be performed in the following order:

1) HOWTO: Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

2) HOWTO: Silent Installation for NetApp Windows Host Utilities 6.0.2

3) HOWTO: Silent Installation for NetApp SnapDrive for Windows v7.0.3

4) HOWTO: Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

The following pre-requisites and requirements are assumed:

· ntap_win_mpio_4.1P1_setup_x64.msi and/or ntap_win_mpio_4.1P1_setup_x86.msi downloads from the NetApp NOW site

o NOTE: NO testing has been done for 32 bit OS, and no provisions exist in the current batch file.  It is assumed that you are on an actual modern server OS, and it is x64.  If not… perform the installation manually.

· 215-08396_A0 – Installation and Administration Guide.pdf from NetApp NOW site

· As this was intended for UPGRADES of systems with at least SnapDrive for Windows, we are assuming that the .NET 3.5 pre-requisite has been met.  Additional work will be required in order to provide the error checking and installation processes for .NET 3.5, as each OS has a slightly different method of doing so.  I’d recommend creating a standardized “INSTALL_DOTNET_35.BAT” that could be called by any other installation that requires it, to keep the process modular.

· The following Windows HotFixes:

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008\Windows6.0-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008\Windows6.0-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2520235-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2528357-v2-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2\Windows6.1-KB979711-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2528357-v2-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008R2SP1\Windows6.1-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB2684681-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB2754704-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2008SP2\Windows6.0-KB968675-x64.msu

I:\INSTALLS\NETAPP\Data ONTAP DSM for Windows v4.1\WINDOWS_HOTFIXES\2012\Windows8-RT-KB2796995-x64.msu

o NOTE: the Folder path for the HotFixes from Microsoft MUST be:

§ IN the installation folder

§ IN a folder called “WINDOWS_HOTFIXES”

§ IN folders based on OS of 2008, 2008SP2, 2008R2, 2008R2SP1, 2012 (there are no 2012R2 hotfixes required)

o Links to obtain the hotfixes are located in the Administration Guide, on Page 56.  Links will have a HF # only, eg: “2796995”.  These links may include downloads for multiple OS’s (eg: Vista/Windows 7 – equal to 2008 and 2008 R2), and each must be downloaded and located separately.  These hotfixes will unfortunately be named such as “457359_intl_x64_zip.exe” for KB2796995.  This is unfortunate as it is near impossible to correlate the versions and the files.  I would recommend saving the files as “KB2796995-457359_intl_x64_zip.exe” with the KB######## as the prefix.  Upon downloading, UNZIP the files in the same folder/folder structure, and you will note the final output of “Windows8-RT-KB2796995.msu”.  You will quickly be able to tell if you have only downloaded one version if you see “Windows6.1” for a 2008 file and “Windows6.0 for a 2008R2” file, etc.

· You are installing as “Local System” account vs a named service account – there is no interaction with the NetApp Filer, so there is no functional reason to use a named service account

· You have been given the MPIO-WIN 14 digit product key from the NetApp NOW Licencing portal. 

The general process in the installation batch file is:

· LOG output is in the PRODUCT_FOLDER as %BATCH%.LOG

· MSI LOG is to %TEMP% as %BATCH%.LOG, and is NOT cleaned up after installation.  Use for troubleshooting.

· Check if the product is already installed, using WMIC.  If installed, jumps to CLEANUP

· Check for OS version to determine what hotfixes are required

· Copy hotfixes locally – they will not run properly via a UNC path as they update the system via DISM and the machine account likely has no rights to the share

· Run the installers for the hotfixes and wait for each to complete – it does not check if they are missing or already present

· Copy installation MSI locally

· Run the installer for the MSI and wait for complete

· Updates the registry with 190 second MPIO timeouts in two places

· Cleans up local files

=== INSTALL_NETAPP_DSM_MPIO_41.BAT ===

@echo off

REM

REM Enter the path to the installation folder – does not require quotes, quotes are assumed, with or without spaces – MUST not contain training "\"

REM

set INSTALL_FOLDER=\\netwise.ca\DFS\INSTALLS\NetApp\Data ONTAP DSM for Windows v4.1

set PRODUCT_NAME=Data ONTAP DSM for Windows MPIO

set LOG_FILE="%INSTALL_FOLDER%\%~n0.LOG"

ECHO. >>%LOG_FILE%

REM Obtain Licence Code and update

set LICENSECODE=XXXXXXXXXXX

:CHECK_ALREADY_INSTALLED

REM

REM Use a Product Name substring search and the exact version number.  Note the double percent characters are correct

REM

wmic product where "Name like ‘%%DSM%%’ and Version like ‘4.1.4348.1209’" get Name, Version | find /i "4.1.4348.1209"

if %ERRORLEVEL%==1 goto CHECK_OS_VERSION

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% already installed! >>%LOG_FILE%

goto CLEANUP

:CHECK_OS_VERSION

REM

REM Check for OS Version

REM

set OS_VER=0

set OS_NAME=0

for /f "tokens=1-5* delims=[] " %%A in (‘ver’) do set OS_VER=%%D

if %OS_VER%==6.3.9200 set OS_NAME=2012R2

if %OS_VER%==6.2.9200 set OS_NAME=2012

if %OS_VER%==6.0.6001 set OS_NAME=2008

if %OS_VER%==6.0.6002 set OS_NAME=2008SP2

if %OS_VER%==6.1.7600 set OS_NAME=2008R2

if %OS_VER%==6.1.7601 set OS_NAME=2008R2SP1

goto INSTALL_PREREQ_HOTFIXES

:INSTALL_PREREQ_HOTFIXES

REM

REM Will copy ALL *.MSU files locally from matching folder name to OS_NAME list above, and then install ALL *.MSU files from the local drive,

REM waiting for each to complete before continuing. 

REM

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying pre-requisite hotfixes locally >>%LOG_FILE%

xcopy "%INSTALL_FOLDER%"\WINDOWS_HOTFIXES\%OS_NAME%\*.msu %TEMP%\DSM.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing pre-requisite hotfixes >>%LOG_FILE%

for /f "tokens=1* delims=" %%A in (‘dir %temp%\DSM.INST /b’) do start /w wusa.exe %temp%\%%A /quiet /norestart

goto INSTALL_DSM_MPIO

:INSTALL_DSM_MPIO

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% copying installation files locally >>%LOG_FILE%

xcopy "%INSTALL_FOLDER%"\"ntap_win_mpio_4.1P1_setup_x64.msi" %TEMP%\DSM.INST /s /e /c /k /i /y

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% installing main application >>%LOG_FILE%

msiexec.exe /quiet /i"%TEMP%\DSM.INST\ntap_win_mpio_4.1P1_setup_x64.msi" /l*v %TEMP%\%~n0.LOG LICENSECODE=%LICENSECODE% USESYSTEMACCOUNT=1 /norestart

wmic product where "Name like ‘%%DSM%%’ and Version like ‘4.1.4348.1209’" get Name, Version | find /i "4.1.4348.1209"

if %ERRORLEVEL%==1 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as NOT INSTALLED >>%LOG_FILE%

if %ERRORLEVEL%==0 echo %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% reports as INSTALLED >>%LOG_FILE%

goto CLEANUP

:CLEANUP

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% cleaning up installation files locally >>%LOG_FILE%

if exist %TEMP%\DSM.INST rmdir %TEMP%\DSM.INST /q /s

REM

REM Set Registry keys for both NetApp MPIO and Windows MPIO to 190 second timeouts

REM

ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% updating registry with 190 second MPIO timeouts >>%LOG_FILE%

REG add HKLM\System\CurrentControlSet\Services\ontapdsm\Parameters /v PDORemovePeriod /t REG_DWORD /d 190 /f

REG add HKLM\SYSTEM\CurrentControlSet\Services\Disk /v TimeoutValue /t REG_DWORD /d 190 /f

REG QUERY "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager" /v PendingFileRenameOperations | find /i "C:"

if %ERRORLEVEL%==0 ECHO %COMPUTERNAME% %DATE% %TIME% – %PRODUCT_NAME% HAS a reboot pending… >>%LOG_FILE%

ECHO. >>%LOG_FILE%

=== INSTALL_NETAPP_DSM_MPIO_41.BAT ===

=== SAMPLE LOG OUTPUT ===

SRVDC113B 11/09/2014 16:46:13.13 – Data ONTAP DSM for Windows MPIO already installed!

SRVDC113B 11/09/2014 16:46:13.13 – Data ONTAP DSM for Windows MPIO cleaning up installation files locally

SRVDC113B 11/09/2014 16:46:13.13 – Data ONTAP DSM for Windows MPIO updating registry with 190 second MPIO timeouts

SRVD511 Thu 09/11/2014 16:49:06.95 – Data ONTAP DSM for Windows MPIO copying pre-requisite hotfixes locally

SRVD511 Thu 09/11/2014 16:49:07.08 – Data ONTAP DSM for Windows MPIO installing pre-requisite hotfixes

SRVD511 Thu 09/11/2014 16:49:07.25 – Data ONTAP DSM for Windows MPIO copying installation files locally

SRVD511 Thu 09/11/2014 16:49:07.36 – Data ONTAP DSM for Windows MPIO installing main application

SRVD511 Thu 09/11/2014 16:49:46.44 – Data ONTAP DSM for Windows MPIO reports as INSTALLED

SRVD511 Thu 09/11/2014 16:49:46.44 – Data ONTAP DSM for Windows MPIO cleaning up installation files locally

SRVD511 Thu 09/11/2014 16:49:46.45 – Data ONTAP DSM for Windows MPIO updating registry with 190 second MPIO timeouts

=== SAMPLE LOG OUTPUT ===

Categories: ISCSI, NetApp, SnapDrive

Upgrading NetApp SnapDrive for Windows

September 11, 2014 Leave a comment

I was recently asked to perform SnapDrive upgrades against systems in preparation for a NetApp Data ONTAP upgrade.  As I’ve been through this a time or two before, I knew it wasn’t quite as simple as “upgrade” SnapDrive.  I figured I’d share some information that might help someone have a smooth transition when they do it as well. 

The first thing to do is check out the NetApp Interoperability Matrix Tool (IMT).  This will help you determine all of the supported and required versions. 

The next thing you’ll need to know is what versions are supported.  As of Sep 2014, you’ll want to also look at:

  • SnapDrive for Windows – v7.0.3
  • DSM MPIO – v4.1
  • ISCSI Host Utilities – v6.0.2
  • These are the versions you’ll need to support Windows 2012 R2 and ONTAP v8.2.x 7-Mode.

Next, you’ll need to perform an inventory against your systems to determine what versions or even if some of the above tools are installed.  It’s not uncommon to find situations where SnapDrive might be installed, but the other tools may or may not be.  Equally, the versions may or may not be what you’re expecting to find or standardized.  Some of this will depend on who installed what, and when.  Also, you’re likely to run into some older systems such as Windows 2003 or 2003 R2 that are running older versions, and have no upgrade path.   So you may need to come up with a migration or lifecycle strategy for some of your systems.

One of the most frustrating things I’ve found from these events in the past, is that NetApp doesn’t have any log of what is installed.  That’s understandable for the Windows systems, but these tools interact with a NetApp controller.  For systems that don’t – perhaps they don’t any longer, they’re orphaned, they’ve been P2V’d, etc – that’s understandable.  But it would be GREAT if NetApp could be so kind as to log connections from systems and what version of the software is involved.  Something akin to WSUS, but at the controller level – even if just logged to the MESSAGES file.  Either way, that’s just dreaming Smile

The above software is going to have some pre-requisites and requirements.  These include:

  • .NET 3.5 is required – likely, this already is, if you’re upgrading.  If you’re installing clean, then it may not be present. 
  • Various Windows HotFixes, as identified by the installation guides
  • Knowing what your ISCSI timeout settings need to be – my environment has been tested to prefer 190 seconds.
  • I’ve created silent installer batch files for each of the three application installations and a script to verify the installation.  It likely can’t be distributed via SCCM or similar tools at this point, but it’s pretty close.  Expect to see these posted shortly:

HOWTO- Silent Installation for NetApp Data ONTAP DSM for Windows MPIO v4.1

HOWTO- Silent Installation for NetApp Windows Host Utilities 6.0.2

HOWTO- Silent Installation for NetApp SnapDrive for Windows v7.0.3

HOWTO- Verify NetApp SnapDrive for Windows 7.0.3 Installations were successful

I’ve tested these against Windows 2008 R2 SP1 and 2012 thus far, with no issues.

With luck, I can help someone else’s upgrades go smoothly. 

For those that know me – this would be a GREAT place for an obligatory comment like “Ditch the LUN’s, VMDK’s don’t have this issue, and it’s time to stop living in a world with physical constructs, lack of portability, and vendor lock in.”  I like NetApp products, but I like it better when it has VMware on top of it, and the Windows systems don’t have a clue!

Categories: ISCSI, NetApp, SnapDrive, Storage

Lab Learning Lessons–01

July 27, 2014 Leave a comment

So I figured I’d start a new theme, which the title represents.  This is “Lab Learning Lessons” or things you learn in the lab, that are better learned here than in Production somewhere.  Hopefully this will help you out in the future, or if nothing else will reinforce for me that I’ve done this before.  So with that in mind – this week’s lessons!

 

1) I can’t find the stupid ISCSI Target!

Ever have one of those days?  Setup a new SAN, configure the NIC’s, configure ISCSI, make some LUN’s, configure your Initiator Groups, and… nothing?  Add the ISCSI target to the Dynamic Initiators in the ESXi Software ISCSI Initiator and…. it never finds any Static Initiators like it should?  So you try to do a “vmkping <target_IP>” and sure enough, THAT works.  Worse yet, you do the SAME thing on the secondary NetApp (in this case) controller in that chassis, and IT works.  So you’re doing the right thing.  So you check against the OLD controllers – and your settings are similar as they should be.

So you change the IP addresses on the Targets and… boom.  It works.

Lesson Learned:  IP Address conflicts don’t tell you if the thing that is responding to your test pings is the device you WANT it to be.

 

2) Can’t vMotion VM’s.  Or create a new one.  Or create objects on a new datastore.

Sounds strange right?  The error includes “pbm.fault.pbm fault.summary” for everything you do.  VM’s are otherwise working and doing what they should be.  You can start, restart, reboot, etc.  You just can’t move them around.  A little Google-fu will suggest that you restart the vCenter Inventory and/or Profile Driven Storage service(s).  Sounds reasonable.  Except those take forever to do so.  So you reboot the vCenter server, hoping that’ll help.  No go.

Then you open Explorer…. and realize your vCenter is out of space.  Except all the services are quite happily started.  No “Automatic” services are unstarted or failing to start.  Nothing is tripping an error.  It’s just “not working”.

Lesson Learned:  Maybe if you’re not going to use a 3rd party monitoring solution (eg: Nagios, ZenOS, PRTG, SolarWinds, etc), then you should configure basic Windows Scheduled Tasks to send e-mails when a drive gets to a certain used amount.  Might save some stress.

 

3) IP Address Planning.

I’m big on having “predictable” IP Address standards.  If you can, have “Primary” addresses be a x.y.z.1# and “Secondary” be x.y.z.2#, or some other system that works for you.  Also if you have 4 NIC’s maybe the #’s in the previous examples should be the NIC #.  So on a NetApp, e0c and e0d would be 3 and 4, so your IP would be x.y.z.13, x.y.z.14, x.y.z.23, x.y.z.24, or something else.

The downside is you really need to be able to look at the final configuration, and work backwards.  Are you going to do one IP per NIC?  One per LACP/PortChannel?  (Not so much for ISCSI, but for NFS/CIFS).  if you do one for a virtual interface like an LACP vif – what # is it?  It’s none of NIC 1-4 (e0a/e0b/e0c/e0d).  Would you make it .10 and .20?  Maybe.  Or maybe .19 and .29, as it’s ‘odd’.

What if you have a second unit in the same place?   Is your solution scalable?  The if your first pair of controllers is NW-SAN1 and NW-SAN2, and is .1x and .2x, then NW-SAN3 and NW-SAN4 could easily be .3x and .4x – but are you chewing up a lot of IP’s?  Maybe.  But in my opinion, it’s so worth it.  Reading logs and troubleshooting becomes amazingly simpler, as you can now logically map one device to another by IP, hostname, port and NIC.

Lesson Learned:  Plan out as much as you can in advance.   But if you can’t, try it in a simulator and work through your options.  This is why they exist, and why we have labs.

HOWTO: Troubleshoot high CIFS IOPS on NetApp

October 15, 2013 Leave a comment

In my office, we’ve been keeping an eye on some high usage on our EDM based NetApp, particularly due to backup times with NDMP and NetBackup to tape.  In doing so, from time to time, I see other issues.  One that comes up is an unusually high CIFS (Windows File Shares) usage.  In order to troubleshoot this, and get to the root of the issue – much like one might with SFLOW or a sniffer on the network – these are the steps you can take.

1) Obtain the system stats on the NetApp from the console, by running “sysstat –c 12 –x 10”.  –c 12 indicates 12 iterations or “counts”, and –x 10 indicates “every 10 seconds”

clip_image001

I’m sure that at first, these results look a little confusing – so I’ll break them down:

· GREEN shows the NFS OPS per Second.  Because VM backups are occurring at night (during this example), it is expected that there will be high NFS load on the system.

· ORANGE shows the NET KB/Sec OUT – the activity that would be used by VM’s on NFS performing READS FROM the SAN – thus, NET OUT traffic.

· PURPLES shows the DISK KB/Sec READ.  We expect this to be high both because of NFS OPS, but also because of…..

· GRAY shows the TAPE KB/sec WRITE, which is NDMP traffic OUT, WRITING to the tapes.

So all of this is largely expected and accounted for.  Except why is CIFS, shown in RED, so high?  CIFS is used by Windows users and not NFS, not part of internal SAN activity (eg: “aggr scrub”, “vol move”, “reallocate –p”, SnapMirror/Vault, etc), and is not part of NDMP or the backups.  So – what is the activity?

(Also worth noting – Cache Age is 0s – indicating we are massively overrunning the available system cache, or it would show how many seconds or minutes of cache exists, and 0s is none)

2) From a Windows server where you are logged in with an Administrator account (to make pass-thru authentication to the SAN work better), open Computer Management:

clip_image002

Right click on COMPUTER MANAGEMENT in the MMC and choose “CONNECT TO ANOTHER COMPUTER”.  Enter the name of the NetApp controller – eg: NETAPP1.  Because NetApp has licenced API’s and functionality from Microsoft for their Domain participation and CIFS sharing, much of the CIFS portion of the controller can be managed as if it were a remote Windows Server.

3) Expand SYSTEM TOOLS -> SHARED FOLDERS -> SESSIONS:

clip_image003

Click on the # OPEN FILES header to sort by the highest number.

As you can see, we now know the USERNAME and COMPUTERNAME that has the highest number of files open.  The immediate questions that come to mind here are:

· Why so many users with more than one connection?

· Why so many open files?

· What might the activity be that is causing high CIFS IO, as an open file doesn’t cause IO, it simply has a lock.

4) Expand SYSTEM TOOLS -> SHARED FOLDERS -> OPEN FILES:

clip_image004

If we sort by ACCESSED BY, and look for the highest users by # of open files in the previous step, we know to look for username “saxxx.xxxxx”.  Looking at the files opened, we can reasonably assume that there is something going on with an ArcGIS GDB (GeoDatabase), by the number of GDB files open (had to be obfuscated, sorry).  Likely this user is either a) active or b) running some long-running task overnight.

User “pnxxxx” however, just appears to have open files.

The difficulty here is that none of this tells you WHAT the users are doing.  But it gives you a reasonable place to go and look and investigate.

HOWEVER – something to keep in mind – if a user is say copying a large number of files (eg: robocopy backup, zip archive, etc), the above method MAY NOT find it.  That activity will look like the user opening one file after the other, closing the first, and may never appear as the user having more than one file open at any given time.  These methods above are intended to be a guide, not a solution.

The downside to all of this?  Users who hammer on the system over the weekend, affect the performance of backups.  We try not to do backups during the week so their user experience is good – but the reverse is not always true, when we need backups to get the highest priority and performance.  All of that said – the system is there for the users to perform work for the company, so it is an understandably necessary evil.

5) There IS an option that one can set to get more information though:

NETAPP1> options cifs.per_cifs.per_client_stats.enable off

However, it is recommended by NetApp to leave the “per_client_stats” set to disabled, unless needed as the tracking of these stats …. Puts more load on the system, and thus can slow it down in a situation where you are already troubleshooting poor performance.  It is worth knowing it exists, in case asked by NetApp Support to enable it for troubleshooting.

To enable the option, simply run:

NETAPP1> cifs top
The cifs.per_client_stats.enable option must be on to use “cifs top”
NETAPP1> options cifs.per_client_stats.enable on

As you can see, “cifs top” will not provide any useful information until “per_client_stats” are enabled.  You can safely disable them when you’re done troubleshooting.

NETAPP1> cifs top
ops/s  reads(n, KB/s) writes(n, KB/s) suspect/s   IP              Name
553 |      0     0 |       0     0 |        0 | 172.21.250.45     DOMAIN\svcspaceobserver
    19 |      0     0 |       0     0 |        0 | 172.23.0.67       DOMAIN\jasxxxxx
    10 |      0     0 |       0     0 |        0 | 172.23.0.67       DOMAIN\jasxxxxx
     2 |      1   145 |       0     0 |        0 | 172.21.250.30     DOMAIN\cmxxxx
     0 |      0     0 |       0     0 |        0 | 172.21.1.48       DOMAIN\lawxxxx
     0 |      0     0 |       0     0 |        0 | 172.22.0.65       DOMAIN\saxxxxx
     0 |      0     0 |       0     0 |        0 | 172.22.17.76      DOMAIN\derxxxx
     0 |      0     0 |       0     0 |        0 | 172.21.61.133   DOMAIN\nexexxxxxx
     0 |      0     0 |       0     0 |        0 | 192.168.52.66   DOMAIN\aroxxxx
     0 |      0     0 |       0     0 |        0 | 172.22.17.56      DOMAIN\armaxxxx

When you run “cifs top” you may need to give the system some time to collect those “per_client_stats”, perhaps 60-120 seconds.  But then what you see is shown above.   Clearly, “DOMAIN\svcspaceobserver” is the biggest cultprit here – at 553 OPS/second.  You can see it is NOT doing a lot of KB/sec read or write, but simply crawling the file system results in a lot of “operations”.  This would be one of those situations that would not show up as a high number of open files in Computer Management, as it is massively sequential access, one operation at a time.

Don’t forget to DISABLE the “per_client_stats” once you’re done troubleshooting, as there is no point collecting this information if it will not be used.

NETAPP1> options cifs.per_client_stats.enable off

So the short moral of the story?  Don’t run Space Observer on a share on a NetApp during backups – or it will just compound poor backup performance.   Hopefully this information might help you troubleshoot high CIFS activity in the future.

Categories: CIFS, NetApp, Windows2008R2