Registration Now Open: Edmonton VMUG Meeting June 16

May 21, 2015 Leave a comment

Hey all my local Edmonton VMUG people!  Registration is now open for our next VMUG, on Tuesday June 16. 

Registration Now Open: Edmonton VMUG Meeting

Sponsors will be:

  • Scalar with Nutanx – and I’m sure there will be many questions about Nutanix Community Edition!
  • Zerto

We’ll also be dragging someone kicking and screaming (sounds like me) to do some talking about some Tales from the Trenches again and things we’ve run into and seen.   Maybe you’ve run into them, maybe you haven’t.  Maybe you have some to share.  If you have some notes that you’re not willing to present, do reach out to me, I’m happy to present them to the crowd on your behalf. 

Hope to see you there!

Categories: VMUG

IBM RackSwitch–40GbE comes to the lab!

May 20, 2015 3 comments

Last year, I had a post about 10GbE coming to my home lab (  This year, 40GbE comes! 

This definitely falls into the traditional “too good to pass up” category.  A company I’m doing work for picked up a couple of these, and there was enough of a supply that I was able to get my hands on a pair for a reasonable price.  Reasonable at least after liquidating the G8124’s from last year.  (Drop me a line, they’re available for sale! Smile)

Some quick high level on these switches, summarized from the IBM/Lenovo RedBooks (

  • 1U Fully Layer 2 and Layer 3 capable
  • 4x 40Gbe QSFP+ and 48x 10GbE SFP+
  • 2x power supply, fully redundant
  • 4x fan modules, also hot swappable.
  • Mini-USB to serial console cable (dear god, how much I hate this non-standard part)
  • Supports 1GbE Copper Transceiver – no issues with Cisco GLC-T= units so far
  • Supports Cisco Copper TwinAx DAC cabling at 10GbE
  • Supports 40GbE QSFP+ cables from 10GTek
  • Supports virtual stacking, allowing for a single management unit

Front panel of the RackSwitch G8264

Everything else generally falls into line with the G8124.  Where those are listed as “Access” switches, these are listed as “Aggregation” switches.  Truly, I’ll probably NEVER have any need for this many 10GbE ports in my home lab, but I’ll also never run out.  Equally, I now have switches that match production in one of my largest environments, so I can get good and familiar with them.

I’m still on the fence about the value of the stacking.  While these are largely going to be used for ISCSI or NFS based storage, stacking may not even be required.  In fact there’s an argument to be made about having them be completely segregated other than port-channels between them, so as to ensure that a bad stack command doesn’t take out both.  Also the Implementing IBM System Networking 10Gb Ethernet Switches guide, it shows the following limitations:

When in stacking mode, the following stand-alone features are not supported:
Active Multi-Path Protocol (AMP)
BCM rate control
Border Gateway Protocol (BGP)
Converge Enhanced Ethernet (CEE)
Fibre Channel over Ethernet (FCoE)
IGMP Relay and IGMPv3
Link Layer Detection Protocol (LLDP)
Loopback Interfaces
MAC address notification
Port flood blocking
Protocol-based VLANs
Router IDs
Route maps
sFlow port monitoring
Static MAC address addition
Static multicast
Uni-Directional Link Detection (UDLD)
Virtual NICs
Virtual Router Redundancy Protocol (VRRP)

That sure seems like a lot of limitations.  At a glance, I’m not sure anything there is end of the world, but it sure is a lot to give up. 

At this point, I’m actually considering filling a number of ports with GLC-T’s and using that for 1GbE.  A ‘waste’, perhaps, but if it means I can recycle my 1GbE switches, that’s an additional savings.  If anyone has a box of them they’ve been meaning to get rid of, I’d be happy to work something out. 

Some questions that will likely get asked, that I’ll tackle in advance:

  • Come on, seriously – they’re data center 10/40GbE switches.  YES, they’re loud.  They’re not, however, unliveable.  They do quite down a bit after warm up, where they run everything at 100% cycle to POST.  But make no mistake, you’re not going to put one of these under the OfficeJet in your office and hook up your NAS to it, and not shoot yourself. 
  • Power is actually not that bad.  These are pretty green, and drop power to unlit ports.  I haven’t hooked up a Kill-a-Watt to them, but will tomorrow.  They’re on par with the G8124’s based on the amp display on the PDU’s I have them on right now. 
  • Yes, there are a couple more Winking smile  To give you a ballpark, if you check eBay for a Dell PowerConnect 8024F and think that’s doable – then you’re probably going to be interested.  You’d lose the 4x10GBaseT combo ports, but you’d gain 24x10GbE and 4x 40GbE.
  • I’m not sure yet if there are any 40GbE compatible HBA – just haven’t looked into it.  I’m guessing Mellanox ConnectX-3 might do it.  Really though, even at 10GbE, you’re not saturating that without a ton of disk IO. 

More to come as I build out various configurations for these and come up with what seems to be the best option for a couple of C6100 hosts. 

Wish me luck!

Categories: Hardware, Home Lab, IBM, RackSwitch

vSphere 6 CBT Bug–and patch now available.

May 19, 2015 Leave a comment

Recently a CBT bug was identified with vSphere ( which involved VMDK disks extended beyond a 128GB boundary.  I originally found out about it via a Veeam weekly newsletter, and in Veeam Backup & Recovery v8.0 Update 2, a workaround was available. 

It seems there is also a specific bug related to vSphere 6 specifically.  (  This bug doesn’t require extending the VMDK, as it relates only to CBT.  The workaround here seemed to be to disable CBT – which of course affected backup windows in a big way.

This bug is now fixed in, and should be downloaded, tested, and verified in your environments if you’re an early adopter of vSphere 6. 

You should be able to find patch in VUM, so apply soon. 

Categories: Veeam B&R, VMware, vSphere

HOWTO: Fix pbm.fault.pbmfault.summary when cloning

May 2, 2015 3 comments

Let’s suppose you’re cloning some VM’s from a template, and you receive a “pbm.fault.pbmfault.summary” error and it fails. 

A very quick google search will find you the following VMware KB article.

While it references the vCenter Server Appliance, and a DNS issue, the key portion is that it comments on the vCenter Server Inventory Service.


Which should certainly not be STOPPED, never mind what type of server it’s on or if DNS is working. 

Get that started, and there’s a good chance you won’t have to worry about if you’re on the appliance or have DNS issues. 

This is also probably a good time to look at your service failure handling:


Perhaps you want to automatically restart the service, send an e-mail, or restart the computer.  In any event, give some thought to how you’ll make this more self-healing in the future!

Categories: vCenter Server, vSphere

HOWTO: Set Logon as a Service Dynamically via GPO

March 16, 2015 Leave a comment

I recently ran into a situation where a client has a group per server for Administrators, Remote Desktop Users, and hopefully, Service Accounts.  This may or may not be the best way of dealing with this, but it does solve a need by moving user access to AD vs configuration on local servers.  It’s a little easier to centralize and manage by administrators that may have access to AD but not the servers themselves (eg: HelpDesk users).  The problem, as indicated below, is that setting the rights for the service account/groups has been getting done manually to the systems as they are built or needed.  This has resulted in inconsistencies, as one might expect.  So I found a way to standardize and bring it all “back up to code”, as it were.



You have a need to set a user or group to have “Log on as a Service” or “Log on as a Batch Job” rights.  This can be done via the Local Security Policy (secpol.msc) or via GPO.  However, there are two obvious issues with this:

1) Using SECPOL.MSC means you’re editing the local security policy.  While this may be the only way to accomplish this, it is decentralized and uncertain to maintain. 

2) Using the GPO method only allows you to set a particular set of user(s) or group(s) to the affected machines

However, if you have a need to set a 1:1 relationship with a dynamic name to the system, GPO’s and the Local Security Policy leave something to be desired.  There is no functionality within the GPO to say “Apply GRP-%SERVERNAME%-SVC” to have this rights, and have it apply as needed – at least for the Logon As a Service right.  Using other methods you can allocate to existing groups with existing rights, but you cannot either dynamically specify a group in THIS GPO location, affect the Local Security Policy, or set the rights for this local group. 


  • Have each server/system have a group such as GRP-SERVER01-SVC group identifying service accounts.  This would be a company policy scenario, and would ensure that administration and auditing of local group memberships was ONLY done via Active Directory, and could be done via delegated rights by users who may not have rights to login to the server. 
  • Have the group apply only to the named server.  Eg: GRP-SERVER01-SVC should have rights on SERVER01, but not SERVER02 or SERVER03
  • If possible, one should also be able to add to the local group a GRP-ALLSERVERS-SVC for a service account that might be globally allowed. Eg: DOMAIN\svcAutomation, DOMAIN\svcBackup, etc. 
  • Centrally manageable
  • Automatic, dynamic, updates and standardizes over time. 
  • OPTIONAL – also do similar for the pre-existing local groups of “Administrators” and “Remote Desktop Users” for a corresponding GRP-%COMPUTERNAME%-ADM and GRP-%COMPUTERNAME%-RDP as appropriate.


1) Obtain the file “NTRIGHTS.EXE” from the Windows 2003 Resource Kit found at

Unpack/install the Resource Kit and copy the file where appropriate. 

2) Copy the file centrally to a location that is accessible by the MACHINE account, not a user.  A great example would be to place the file in \\DOMAIN\NETLOGON, as this allows Read/Execute.

3) Create a script that will run in that location that contains the following:


@echo off 

net localgroup "Service Accounts" /add /Comment:"Used for allowing Service Accounts local rights" >> \\SERVER\INSTALLS\BIN\logs\SET_LOGONASSERVICE.LOG

\\SERVER\INSTALLS\BIN\ntrights +r SeServiceLogonRight -u "Service Accounts" -m \\%COMPUTERNAME% >> \\SERVER\INSTALLS\BIN\logs\SET_LOGONASSERVICE.LOG 


4) If required, this script can be called via PSEXEC and executed against a list of computers:


This MUST be run with the –u / -p switch to specify the user to use with the –h “highest privileges”.  The –C must also be used to copy the batch file to the local system so it can run. 

You will see entries in the log similar to:

Granting SeServiceLogonRight to Service Accounts on \\NW-ADCS1... successful 

Granting SeServiceLogonRight to Service Accounts on \\NW-DC1... successful 

Granting SeServiceLogonRight to Service Accounts on \\NW-DC2... successful 

5) We now have a local group called “Service Accounts” and this local group has the rights “Logon as a Service”. 

We can verify this by running “SECPOL.MSC” on one of the servers and checking the rights assignments:


Sure enough, the local “Service Accounts” group is listed.

6) We can now handle the remainder of this via normal GPO’s for Restricted Groups, using DYNAMIC naming. 

Open the GPO editor and create a new GPO and name it something obvious such as “LOCAL_RESTRICTED_GROUPS”, and then edit it.



Right click and select NEW -> LOCAL GROUP

8) Now we modify the properties for this group:


We will choose UPDATE for an action, as the group should already exist based on our previous work. 

The group name will be “SERVICE ACCOUNTS”. 

Click ADD to add members


This is where the magic comes in.  If you press the “…” beside the NAME, you can search for the group/user based on a traditional ADUC type search.  But we don’t want that.  Instead, place your cursor in the NAME field.  Press the F3 key:


We get a list of VARIABLES!  We want to use ComputerName so that we can reference the group as GRP-%COMPUTERNAME%-SVC and each computer will get its own group.  Click SELECT.


Note the variable shows %ComputerName% as expected.  Modify that as needed to have the GRP- and -SVC prefix and suffix.


Click OK to close this window.

I’ve chosen to also add an -ADM and –RDP group for Administrators and Remote Desktop Users as this is another use case.


Close and save the GPO

9) Link your GPO appropriately:


Here I have a GROUPS-TEST OU and I have placed my NW-VEEAM01 server in this OU, along with the 3 associated groups.   This will limit impact during testing.

10) On the system in question, check the current group memberships:


11) On the system in question, run a “gpupdate /force”

12) Again on the system in question, confirm the updated group membership:


There you have it.  The ADM/RDP groups were easy as they not only pre-exist, but are pre-defined.  The complication really was the “Service Accounts” group, which both does not pre-exist, and has no special rights by default or built in direct way of adding them via the command line. 

The recommendation would be to run the SET_LOGONASSERVICE.BAT as part of the server build process/scripts, or have it pre-done in your deployment image/WIM/VM Template.  Equally, a PSEXEC run against all servers in the domain could force set this group on a periodic basis to ensure the rights existed.  Additional error checking could be built in to check if the command was successful, check if the domain group exists, create it if required, etc. 

Some post comments:

  • Remember that the local account has a SID.  If it is deleted, and recreated with the same name, that won’t be enough as the Log on as a Service right will be assigned to the old SID
  • As the batch file creates the account with a description and we didn’t tell the GPO to do so, it’ll create a new group if required, but with no description.  This is your identifier that something is off, and hopefully that helps you troubleshoot.

vSphere 6.0 GA is out!

March 12, 2015 2 comments

Get your download accelerators warmed up folks, looks like you can download vSphere 6.0 today!


Don’t forget if you’re looking to get Customized OEM ISO’s, to check for those.  So far, it looks like only HP is listed at the VMware download site:


I’ve done some looking, and no Dell Customized ESXi v6.0 yet. 

Remember to do your testing and pre-requisite checks.  Not only should you be using the OEM Customized ISO’s if appropriate, but also check your 3rd party support.  Things like Dell OMSA, IBM System Director, Equallogic, NetApp VSC and other storage related plug ins etc.  Also probably most important is your backup products – Veeam Backup & Recovery v8.0, etc.  Don’t jump in with both feet until you’ve done a little testing. 

But DO download the bits, get them ready, and give them a shot in a lab.  Remember you can do Nested ESXi, which might be a good way to give a vCenter Appliance a test and see what’s new before you try to integrate your entire infrastructure with it.  I highly recommend giving the new vCSA Appliance a try, as well as improvements to the Web Client – it’s still no C# client, but you’re going to find it to actually be very much improved!

I understand that the VMUG EVALExperience will have keys in about 3 weeks from GA, so you may need to use a no-key 60 day trial at this point to get started. 

As I get updates to links for OEM Customized ISO’s and more information I’ll update this post accordingly.

Happy Downloading!

Categories: VMware, vSphere

VCP Recertification Deadline Extended!

March 9, 2015 Leave a comment

Some breaking news just announced regarding VMware certifications, and I’m excited to share this time!

It looks like EVERYONE will be getting a short extension to the grace period up until May 8, 2015 from March 10, 2015.  So an extra 2 months.  This is certainly good news.  A little disappointing still that some of the VCAP exams got abruptly pulled, but all the same, 2 months extra study time is a great thing. 

Also if you have ALREADY recertified – you get a reward.  You can do the VCP6 migration exam for 65% off the normal retail price of $225 USD or $79.  You’ll need to take this exam by August 31 2015.

I still haven’t seen any official courseware or anything on the VCP6/vSphere 6 track.  So if you’re aware of something, please let me know.  I’d like to start studying for the VCP6 Migration Exam ASAP, so anything is better than nothing. 

If you haven’t booked your recertification yet – DO IT! 

Categories: Certification, VMware, vSphere