Archive

Archive for the ‘VMware’ Category

Fighting ESXi 6.0 Network Disconnects–might be KB2124669?

October 7, 2015 2 comments

Recently I’ve been having some issues in my home lab where hosts seem to stop responding.  I’ve been busy, so had a tough time finding the time to verify if the hosts crashed, locked up, or if it was just a networking issue.  So quite unfortunately, I’ve been rebooting them to bring them back and keeping it on my To Do list.

Until it started happening at work.  We have a standalone box that is non clustered that has internal disks and functions as a NAS.  So it’s “very important”, but “not critical” – or it would be HA, etc.  And the same symptoms occurred there – where we of course HAD to troubleshoot it. 

Tried the following:

* Disconnect the NIC’s and reconnect

* Shut/no-shut the ports on the switch

* Access via IPMI/IDRAC – works, could access DCUI to verify host was up.  Could ping VM’s on the host, so verified those were working.

* Tried removing or adding other NIC’s via the DCUI to the vSwitch that has the management network.  No go.  Didn’t matter if the two onboard NIC’s, two add in, split across two controllers, one NIC only, etc. 

Understandably we’re pretty sure we have a networking issue – but only on the one host at this time.  The Dell R610 and R620 have no similar issues, but this is an in-house built Supermicro we use for Dev.  So the issue appears related to the host, and having only one, it’s tough to troubleshoot.

ESXi NIC’s are not the same – the host has 82574L onboard and 82571EB 4 port.  The C6100 I have is 82576 – and I haven’t confirmed it is having the same issue.  But in any case, it doesn’t seem like an issue with a certain chipset or model of NIC. 

So we’ve started looking for others having similar issues and found:

https://communities.vmware.com/thread/517399

After upgrade of ESXi 5.5 to 6.0 server loses every few days the network connection

Reach out to VMware support on this issue to see if it is related to a known bug in ESXi 6.  The event logged in my experience with this same situation is "netdev_watchdog:3678: NETDEV WATCHDOG".  I did not see this in your logs but the failure scenario I have experienced is the same as you described.

Okay, not great, but sounds like the same sort of issue.  Someone then suggested that the bug is known.  Okay.   Check the release notes and:

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-esxi-60u1-release-notes.html

New Network connectivity issues after upgrade from ESXi 5.x to ESXi 6.0
After you upgrade from ESXi 5.x to ESXi 6.0, you might encounter the following issues
The ESXi 6.0 host might randomly lose network connectivity
The ESXi 6.0 host becomes non-responsive and unmanageable until reboot
After reboot, the issue is temporarily resolved for a period of time, but occurs again after a random interval
Transmit timeouts are often logged by the NETDEV WATCHDOG service in the ESXi host. You may see entries similar to the following in the/var/log/vmkernel.log file:
cpu0:33245)WARNING: LinNet: netdev_watchdog:3678: NETDEV WATCHDOG: vmnic0: transmit timed out
cpu0:33245)WARNING: at vmkdrivers/src_92/vmklinux_92/vmware/linux_net.c:3707/netdev_watchdog() (inside vmklinux)
The issue can impact multiple network adapter types across multiple hardware vendors. The exact logging that occurs during a transmit timeout may vary from card to card.

That sounds similar.  We’re still digging through logs and also waiting for it to occur again to catch it in the act.

The next hit we found was:

https://communities.vmware.com/message/2525461#2525461

If you’re seeing this in your vmkernel.log at the time of the disconnect it could be related to an issue that will one day be described at the below link (it is not live at this time). We see this after a random amount of time and nothing VMware technical support could do except reboot the host helped.

http://kb.vmware.com/kb/2124669

KB2124669 definitely covers the observed symptoms.  Note the date:

  • Updated: Oct 6, 2015

That’s yesterday, maybe we’re on the right track!  Now also look at the RESOLUTION:

This issue is resolved in ESXi 6.0 Update 1a, available at VMware Downloads. For more information, see the VMware ESXi 6.0 Update 1a Release Notes.

There’s a 1*A* now?

https://my.vmware.com/group/vmware/details?downloadGroup=ESXI600U1&productId=491&rPId=8764#errorCheckDiv

image

There is in fact, a 6.0U1a out.  Who knew!  Released _yesterday_.

Now, watch the top of your screen, because VMware DOES attempt to make sure you know there’s some issues (as there always are, no new release is perfect):

image

http://kb.vmware.com/kb/2131738

And in that list, includes:

Also if you haven’t bumped into it before, there’s great links here for Upgrade Best Practices, Update Sequences, etc.  DO give this a read.

Understandably, we won’t get this fix in TODAY.  We do have a maintenance cycle coming up soon, where we’ll get vCenter Server 6.0U1 in and can go to ESXi v6.0U1a – and hopefully this will fix the network issue.  If not, back to troubleshooting I guess.  Fingers crossed. 

I’ll post an update next week if we get the upgrade in and see a resolution to this. 

HOWTO: Dell vSphere 6.0 Integration Bits for Servers

October 5, 2015 3 comments

When I do a review of a vSphere site, I typically start by looking to see if best practices are being followed – then look to see if any of the 3rd party bits are installed. This post picks on Dell environments a little, but the same general overview holds true of HP, or IBM/Lenovo, or Cisco, or…. Everyone has their own 3rd party integration bits to be aware of. Perhaps this is the part where everyone likes Hyper Converged because you don’t have to know about this stuff. But as an expert administering it, you should at least be aware of it, if not an expert.

I’m not going to into details as to how to install or integrate these components. I just wanted to make a cheat sheet for myself, and maybe remind some folks that regardless of your vendor, make sure you check for the extra’s – its part of why you’re not buying white boxes, so take advantage of it. Most if it is free!

The links:

I’ve picked on a Dell PowerEdge R630 server, but realistically any 13G box would have the same requirements. Even older 11/12G boxes such as an R610 or R620 would. So first we start with the overview page for the R630 – remember to change that OS selection to “VMware ESXi v6.0”
http://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r630/drivers

 

Dell iDRAC Service Module (VIB) for ESXi 6.0, v2.2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=2XHPY

You’re going to want to be able to talk to and manage the iDRAC from inside of ESXi, so get the VIB to allow you to do so. This installs via VUM incredibly easy.

 

Dell OpenManage Server Administrator vSphere Installation Bundle (VIB) for ESXi 6.0, v8.2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=VV2P2

Next, you’ll want to be able to handle talking to OMSA on the ESXi box itself, to get health, management, inventory, and other features. Again, this installs with VUM.

 

OpenManage™ Integration for VMware vCenter, v3.0
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=8V0JG

This will let your vCenter present you with various tools to manage your Dell infrastructure right from within vCenter. Installs as an OVF and is a virtual appliance, so no server required.

 

VMware ESXi 6.0
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=CG9FP 

Your customized ESXi installation ISO. Note the file name – VMware-VMvisor-Installer-6.0.0-2809209.x86_64-Dell_Customized-A02.iso – based on the -2809209 and the –A02 and a quick Google search, you can see that this is v6.0.0b (https://www.vmware.com/support/vsphere6/doc/vsphere-esxi-600b-release-notes.html) vs v6.0U1.

 

Dell Systems Management Tools and Documentation DVD ISO, v.8.2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=4HHMH

You likely will not need this for a smaller installation, but it can help out if you need to standardize, by allowing you to configure and export/import things like BIOS/UEFI, Firmware, iDRAC, LCC, etc settings. Can’t hurt to have around.

 

There is no longer a need for a “SUU” – Systems Update Utility, as the Lifecycle Controller built into ever iDRAC, even the Express will allow you to do updates from that device. I recommend doing them via the network as it is significantly less hassle than going through the Dell Repository Builder, and downloading your copies to a USB/ISO/DVD media and doing it that way.

Now, the above covers what you’ll require for vSphere. What is NOT immediately obvious is the tools you may want to use in Windows. Even though you now have management capability on the hosts and can see things in vCenter, you’re still missing the ability to talk to devices and manage them from Windows – which is where I spend all of my actual time. Things like monitoring, control, management, etc, all are done from within Windows. So let’s go ahead and change that OS to “Windows Server 2012 R2 SP1” and get some additional tools:

 

Dell Lifecycle Controller Integration 3.1 for Microsoft System Center Configuration Manager 2012, 2012 SP1 and 2012 R2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=CKHYR

If you are a SCCM shop, you may very much want to be able to control the LCC via SCCM to handle hardware updates.

 

Dell OpenManage Server Administrator Managed Node(windows – 64 bit) v.8.2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=6J8T3

Even though you’ve installed the OMSA VIB’s on ESXi, there is no actual web server there. So you’ll need to install the OMSA Web Server tools somewhere – could even be your workstation – and use that. You’ll then select “connect to remote node” and specify the target ESXi system and credentials.

 

Dell OpenManage Essentials 2.1.0
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=JW22C

If you’re managing many Dell systems and not just servers, you may want to go with OME if you do not have SCCM or similar. It’s a pretty good 3rd party SNMP/WMI monitoring solution as well. But will also allow you to handle remote updates of firmware, BIOS, settings, etc, on various systems – network, storage, client, thin client, etc.

 

Dell OpenManage DRAC Tools, includes Racadm (64bit),v8.2
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=9RMKR 

RACADM is a tool I’ve used before and have some links on how to use remotely. But this tool can grandly help you standardize your BIOS/IDRAC settings via a script.

 

Dell Repository Manager ,v2.1
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=2RXX2

The repository manager as mentioned is a tool you can use to download only the updates required for your systems. Think of it like WSUS (ish).

 

Dell License Manager
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=68RMC

The iDRAC is the same on every system it is only the licence that changes. To apply the Enterprise license, you’ll need the License Manager.

 

Hopefully this will help someone keep their Dell Environment up to date. Note that I have NOT called out any Dell Storage items such as MD3xxx, Equallogic PSxxxx, or Compellent SCxxxx products. If I tried to, the list would be significantly longer. Also worth noting is that some vendors _networking_ products have similar add ins, so don’t forget to look for those as well.

HOWTO: Fix Veeam v8.0 NFC errors with vSphere v6.0U1 using SSLv3

September 28, 2015 Leave a comment

Recently updated vSphere to v6.0U1 from v6.0?  Using Veeam Backup & Recovery v8.0.0.2030?  Getting NFC storage issues like those below?

clip_image002

Specifically:         ERR |SSL error, code: [336151568].error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

You can find out more about this on the Veeam Forums at: http://forums.veeam.com/vmware-vsphere-f24/vsphere-6-0-u1-t30210.html

The high level explanation is that Veeam is using both TLS and SSLv3 components to connect to VMware – and VMware has disabled the SSL method in v6.0U1.  There is a bug in how Veeam is auto-detecting SSL or TLS connectivity, causing this issue.  Other VMware products are having similar issues talking to their own products, from what I understand.

Veeam has a KB2063 on the issue here: http://www.veeam.com/kb2063   You have two options – call in and request a private out of band hotfix from Veeam, or make changes on the VMware side. 

The VMware KB20121021 discusses how you can make these changes:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2121021

The high level gist is:

· Add "enableSSLv3: true" to /etc/sfcb/sfcb.cfg and then: /etc/init.d/sfcbd-watchdog restart

· Add "vmauthd.ssl.noSSLv3 = false" to /etc/vmware/config and then: /etc/init.d/rhttpproxy restart

I’ve whipped up a quick BASH script that seems to work in my testing.  It will:

· see if the desired option exists and exits

· if the options exists, but is the opposite setting (true vs false, etc) it will flip the setting

· if the option does not exist, it will add it

TEST IT BEFORE YOU RUN IT IN YOUR ENVIRONMENT, I’m not responsible if it does wonky things to you. 

Applying the changes does NOT require Maintenance Mode on the hosts, or any Veeam service restarting.  You can simply “retry” the job on the Veeam server, and “It Just Works”

This will likely be resolved by end of September when Veeam releases the next update to Veeam B&R – or there may be a vSphere v6.0U1a released.  Once the Veeam fix is released, it may be prudent to reverse or disable these changes on your hosts so you can use TLS vs SSL.

==== BEGIN VeeamNFCFix.sh =====

#

# Actions recommendations in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2121021

#

#

# Look for "enableSSLv3: true" in /etc/sfcb/sfcb.cfg

#

cp /etc/sfcb/sfcb.cfg /etc/sfcb/sfcb.cfg.old

if grep -q -i "enableSSLv3: true" /etc/sfcb/sfcb.cfg; then

echo "true found – exiting"

else

if grep -q -i "enableSSLv3: false" /etc/sfcb/sfcb.cfg; then

  echo "false found – modifying"

  sed -i ‘s/enableSSLv3: false/enableSSLv3: true/g’ /etc/sfcb/sfcb.cfg

else

  echo "false not found – adding true"

  grep -i "enableSSLv3: true" /etc/sfcb/sfcb.cfg || echo "enableSSLv3: true" >> /etc/sfcb/sfcb.cfg

fi

fi

/etc/init.d/sfcbd-watchdog restart

#

# Look for "vmauthd.ssl.noSSLv3 = false" in /etc/vmware/config

#

cp /etc/vmware/config /etc/vmware/config.old

if grep -q -i "vmauthd.ssl.noSSLv3 = false" /etc/vmware/config; then

echo "false found – exiting"

else

if grep -q -i "vmauthd.ssl.noSSLv3 = true" /etc/vmware/config; then

  echo "true found – modifying"

  sed -i ‘s/vmauthd.ssl.noSSLv3 = true/vmauthd.ssl.noSSLv3 = false/g’ /etc/vmware/config

else

  echo "true not found – adding false"

  grep -i "vmauthd.ssl.noSSLv3 = false" /etc/vmware/config || echo "vmauthd.ssl.noSSLv3 = false" >> /etc/vmware/config

fi

fi

/etc/init.d/rhttpproxy restart

==== END VeeamNFCFix.sh =====

Categories: Veeam B&R, VMware, vSphere

EVALExperience now includes vSphere 6!

June 3, 2015 Leave a comment

I know I’ve had both a lot of local VMUG members as well as forum members where I frequent, asking about when vSphere 6, vCenter 6, and ESXi 6 would be available as part of EVALExperience – as understandably, people are anxious to get their learning, labbing, and testing on. 

I’m happy to announce that it looks like it’s up.  If you head over to the VMUG page found at http://www.vmug.com/p/cm/ld/fid=8792 you’ll note that they show:

NEW! vSphere 6 and Virtual SAN 6 Now Available!

Of course, if you’ve signed up with VMUG, you should be getting the e-mail I just received as well.  I’m not certain if it would go to all VMUG Members, only those that are already EVALExperience subscribers, or what. 

What is now included, as per the e-mail blast is:

NOW AVAILABLE! VMware recently announced the general availability of VMware vSphere 6, VMware Integrated OpenStack and VMware Virtual SAN 6 – the industry’s first unified platform for the hybrid cloud! EVALExperience will be releasing the new products and VMUG Advantage subscribers will be able to download the latest versions of:

  • vCenter Server Standard for vSphere 6
  • vSphere with Operations Management Enterprise Plus
  • vCloud Suite Standard
  • Virtual SAN 6
  • *New* Virtual SAN 6 All Flash Add-On
    It is worth noting that the product download has been updated and upgraded.  They do call out that the old product and keys will no longer be available.  I can understand why, as part of this will be to help members stay current.  But it would be nice if you could use the N-1 version for a year of transition, etc.  Not everyone can cut over immediately and some people use their home labs to mirror the production environment at work so they can come home and try something they couldn’t at the office.
    Some questions I’ve had asked, and the answers I’m aware of:
  1. How many sockets are included?  The package includes 6 sockets for 3 hosts.
  2. Are the keys 365 days or dated expiry?  I understand they’re dated expiry, so if you install a new lab 2 weeks before the end of your subscription, you’ll see 14 days left, not 364.
  3. What about VSAN?  There had previously been a glitch which gave only one host worth of licences – which clearly does not work.  This has been corrected. 
    Just a friendly reminder, as a VMUG leader to look into the VMUG Advantage membership.  As always, VMUG membership itself is free, come on down and attend a local meeting (the next Edmonton VMUG is June 16 and you can sign up here – http://www.vmug.com/p/cm/ld/fid=10777). 

In addition, your VMUG Advantage subscriber benefits include:  

  • FREE Access to VMworld 2015 Content
  • 20% discount on VMware Certification Exams & Training Courses (If you have a $3500 course you need/want, plus a $225 exam, for $3725 total, spending $200 or so on a VMUG Advantage to make your costs $2800+$180=$2980 is a great way to get $745 off.  This is the sell you should be giving your employer Smile)
  • $100 discount on VMworld 2015 registration (This is the only “stackable” discount for VMworld.  Pre-registration/early-bird ends on June 8th I believe)
  • 35% discount on VMware’s Lab Connect
  • 50% discount on Fusion 7 Professional
  • 50% discount on VMware Player 7 Pro
  • 50% discount on VMware Workstation 11
    Happy labbing, and if you’re local, hope to see you on June 16!

    vSphere 6 CBT Bug–and patch now available.

    May 19, 2015 Leave a comment

    Recently a CBT bug was identified with vSphere (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2090639) which involved VMDK disks extended beyond a 128GB boundary.  I originally found out about it via a Veeam weekly newsletter, and in Veeam Backup & Recovery v8.0 Update 2, a workaround was available. 

    It seems there is also a specific bug related to vSphere 6 specifically.  (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2114076)  This bug doesn’t require extending the VMDK, as it relates only to CBT.  The workaround here seemed to be to disable CBT – which of course affected backup windows in a big way.

    This bug is now fixed in http://kb.vmware.com/kb/2116125, and should be downloaded, tested, and verified in your environments if you’re an early adopter of vSphere 6. 

    You should be able to find patch in VUM, so apply soon. 

    Categories: Veeam B&R, VMware, vSphere

    vSphere 6.0 GA is out!

    March 12, 2015 2 comments

    Get your download accelerators warmed up folks, looks like you can download vSphere 6.0 today!

    image

    Don’t forget if you’re looking to get Customized OEM ISO’s, to check for those.  So far, it looks like only HP is listed at the VMware download site:

    image

    I’ve done some looking, and no Dell Customized ESXi v6.0 yet. 

    Remember to do your testing and pre-requisite checks.  Not only should you be using the OEM Customized ISO’s if appropriate, but also check your 3rd party support.  Things like Dell OMSA, IBM System Director, Equallogic, NetApp VSC and other storage related plug ins etc.  Also probably most important is your backup products – Veeam Backup & Recovery v8.0, etc.  Don’t jump in with both feet until you’ve done a little testing. 

    But DO download the bits, get them ready, and give them a shot in a lab.  Remember you can do Nested ESXi, which might be a good way to give a vCenter Appliance a test and see what’s new before you try to integrate your entire infrastructure with it.  I highly recommend giving the new vCSA Appliance a try, as well as improvements to the Web Client – it’s still no C# client, but you’re going to find it to actually be very much improved!

    I understand that the VMUG EVALExperience will have keys in about 3 weeks from GA, so you may need to use a no-key 60 day trial at this point to get started. 

    As I get updates to links for OEM Customized ISO’s and more information I’ll update this post accordingly.

    Happy Downloading!

    Categories: VMware, vSphere

    VCP Recertification Deadline Extended!

    March 9, 2015 Leave a comment

    Some breaking news just announced regarding VMware certifications, and I’m excited to share this time!

    http://blogs.vmware.com/education/2015/03/vcp-recertification-deadline-extended.html

    It looks like EVERYONE will be getting a short extension to the grace period up until May 8, 2015 from March 10, 2015.  So an extra 2 months.  This is certainly good news.  A little disappointing still that some of the VCAP exams got abruptly pulled, but all the same, 2 months extra study time is a great thing. 

    Also if you have ALREADY recertified – you get a reward.  You can do the VCP6 migration exam for 65% off the normal retail price of $225 USD or $79.  You’ll need to take this exam by August 31 2015.

    I still haven’t seen any official courseware or anything on the VCP6/vSphere 6 track.  So if you’re aware of something, please let me know.  I’d like to start studying for the VCP6 Migration Exam ASAP, so anything is better than nothing. 

    If you haven’t booked your recertification yet – DO IT! 

    Categories: Certification, VMware, vSphere