Archive

Archive for the ‘Veeam B&R’ Category

HOWTO: Fix Veeam v8.0 NFC errors with vSphere v6.0U1 using SSLv3

September 28, 2015 Leave a comment

Recently updated vSphere to v6.0U1 from v6.0?  Using Veeam Backup & Recovery v8.0.0.2030?  Getting NFC storage issues like those below?

clip_image002

Specifically:         ERR |SSL error, code: [336151568].error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

You can find out more about this on the Veeam Forums at: http://forums.veeam.com/vmware-vsphere-f24/vsphere-6-0-u1-t30210.html

The high level explanation is that Veeam is using both TLS and SSLv3 components to connect to VMware – and VMware has disabled the SSL method in v6.0U1.  There is a bug in how Veeam is auto-detecting SSL or TLS connectivity, causing this issue.  Other VMware products are having similar issues talking to their own products, from what I understand.

Veeam has a KB2063 on the issue here: http://www.veeam.com/kb2063   You have two options – call in and request a private out of band hotfix from Veeam, or make changes on the VMware side. 

The VMware KB20121021 discusses how you can make these changes:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2121021

The high level gist is:

· Add "enableSSLv3: true" to /etc/sfcb/sfcb.cfg and then: /etc/init.d/sfcbd-watchdog restart

· Add "vmauthd.ssl.noSSLv3 = false" to /etc/vmware/config and then: /etc/init.d/rhttpproxy restart

I’ve whipped up a quick BASH script that seems to work in my testing.  It will:

· see if the desired option exists and exits

· if the options exists, but is the opposite setting (true vs false, etc) it will flip the setting

· if the option does not exist, it will add it

TEST IT BEFORE YOU RUN IT IN YOUR ENVIRONMENT, I’m not responsible if it does wonky things to you. 

Applying the changes does NOT require Maintenance Mode on the hosts, or any Veeam service restarting.  You can simply “retry” the job on the Veeam server, and “It Just Works”

This will likely be resolved by end of September when Veeam releases the next update to Veeam B&R – or there may be a vSphere v6.0U1a released.  Once the Veeam fix is released, it may be prudent to reverse or disable these changes on your hosts so you can use TLS vs SSL.

==== BEGIN VeeamNFCFix.sh =====

#

# Actions recommendations in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2121021

#

#

# Look for "enableSSLv3: true" in /etc/sfcb/sfcb.cfg

#

cp /etc/sfcb/sfcb.cfg /etc/sfcb/sfcb.cfg.old

if grep -q -i "enableSSLv3: true" /etc/sfcb/sfcb.cfg; then

echo "true found – exiting"

else

if grep -q -i "enableSSLv3: false" /etc/sfcb/sfcb.cfg; then

  echo "false found – modifying"

  sed -i ‘s/enableSSLv3: false/enableSSLv3: true/g’ /etc/sfcb/sfcb.cfg

else

  echo "false not found – adding true"

  grep -i "enableSSLv3: true" /etc/sfcb/sfcb.cfg || echo "enableSSLv3: true" >> /etc/sfcb/sfcb.cfg

fi

fi

/etc/init.d/sfcbd-watchdog restart

#

# Look for "vmauthd.ssl.noSSLv3 = false" in /etc/vmware/config

#

cp /etc/vmware/config /etc/vmware/config.old

if grep -q -i "vmauthd.ssl.noSSLv3 = false" /etc/vmware/config; then

echo "false found – exiting"

else

if grep -q -i "vmauthd.ssl.noSSLv3 = true" /etc/vmware/config; then

  echo "true found – modifying"

  sed -i ‘s/vmauthd.ssl.noSSLv3 = true/vmauthd.ssl.noSSLv3 = false/g’ /etc/vmware/config

else

  echo "true not found – adding false"

  grep -i "vmauthd.ssl.noSSLv3 = false" /etc/vmware/config || echo "vmauthd.ssl.noSSLv3 = false" >> /etc/vmware/config

fi

fi

/etc/init.d/rhttpproxy restart

==== END VeeamNFCFix.sh =====

Categories: Veeam B&R, VMware, vSphere

vSphere 6 CBT Bug–and patch now available.

May 19, 2015 Leave a comment

Recently a CBT bug was identified with vSphere (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2090639) which involved VMDK disks extended beyond a 128GB boundary.  I originally found out about it via a Veeam weekly newsletter, and in Veeam Backup & Recovery v8.0 Update 2, a workaround was available. 

It seems there is also a specific bug related to vSphere 6 specifically.  (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2114076)  This bug doesn’t require extending the VMDK, as it relates only to CBT.  The workaround here seemed to be to disable CBT – which of course affected backup windows in a big way.

This bug is now fixed in http://kb.vmware.com/kb/2116125, and should be downloaded, tested, and verified in your environments if you’re an early adopter of vSphere 6. 

You should be able to find patch in VUM, so apply soon. 

Categories: Veeam B&R, VMware, vSphere

Veeam: Backup & Recovery v7.0 – Installation

August 16, 2013 Leave a comment

A few days ago, Veeam announced the General Availability of Veeam Backup & Recovery 7.0, which was highly anticipated for the number of features and improvements that it brings.  I’m going to do a few posts on the installation and usage of the product to get used to it myself.

1) Download the installation files from Veeam’s web site.  You’ll need to register, which will e-mail you a trial key.   The installation file comes as an ISO, so you can unpack it or mount it to your VM.  When you do, launch the SETUP.EXE

2) Click on the Install for Backup & Recovery

image

3) Click through the licence agreement, and then select the licence file.  The trial comes with 32 sockets.

image

4) Select the program features to install.  PowerShell SDK is not selected by default.

image

5) Veeam will identify missing pre-requisites, and install them with a single click:

image

image

6) Enter the account for the service to run with.  I’d recommend a Service Account of some sort.

image

7) Install the bundled MSDE SQL 2008 R2 instance.

image

8) Select the default ports:

image

9) vPower NFS is used to mount backups as an NFS datastore – and your system will need a root folder to mount this in.  You also need to select a System Catalog folder for VM’s you will be indexing the contents of – useful if you need to search for files within VM’s in your backup.

image

10) You’ll get a chance to review the installation summary screen and click INSTALL.

image

11) That’s it.  You didn’t think it was that hard, did you? 🙂  It’s Veeam – it’s that good!

image

Categories: Veeam B&R

Veeam: Backup & Recovery v7.0 – Basic Setup

August 16, 2013 Leave a comment

In my previous post, I covered the installation of Veeam B&R v7.0.  Now I’ll review the basic setup

1) Launch the program via the shortcut.

image

2) Click on the BACKUP INFRASTRUCTURE button.

image

Click on ADD SERVER and then choose VMWARE VSPHERE.

3) Enter the FQDN of your vCenter Server.

image

4) Enter credentials for the vCenter Server.

image

I create a service account, “svcVeeamAdmin” that has rights both to vCenter and Domain Admins for internal VM tasks such as VSS, Indexing, etc.  Follow whatever best practices apply for your environment.

5) On the summary screen, click FINISH.

image

6) If you expand Managed Servers -> VMware vSphere -> vCenter Servers -> <vCenter Server>, you should see your cluster hosts on the right.

image

7) Click Backup Repositories and then click ADD REPOSITORY

image

Name the repository appropriately so you can easily identify it later.

8) Choose the type of Backup Repository.  While Microsoft Windows Server and Shared Folder (CIFS/SMB) may seem similar, the Windows Server option allows for the installation of an agent to help with efficiency and a Shared Folder would be more like a Linux server or NAS that is exposing a CIFS share but is not otherwise running Windows.

image

9) Enter the FQDN of your repository server:

image

10) Choose the credentials you added previously:

image

11) When you click NEXT, it will check for the presence of an agent on the destination, and then allow you to install or upgrade.

image

12) Once the installation completes, you’ll see the summary.

image

Click Next

13) On the summary screen, click FINISH to complete creating the repository.

image

14) Click POPULATE to show the drives on the system with the agent installed.

image

Select the drive you want to use, and click Next.

15) Click BROWSE or type in the PATH TO FOLDER, then click POPULATE.

image

You can also control the load – the number of concurrent backups or limit the MB/sec (so as to not saturate the network).

If you click ADVANCED:

image

You get options to align backup blocks and decompress the backups – largely for when writing to a repository that supports its own deduplication.  Different use cases will have the remote path obtaining better results than the Veeam deduplication.

16) Next, we configure the vPower NFS folder.

image

We can select to enable it – or not.  Also you can select a Write Cache folder – such as an SSD path, to speed up the writes.  Click Next.

17) On the REVIEW page, you’ll see a summary:

image

Click NEXT

18) And then on the APPLY screen, click FINISH.

image

 

19) Click on the Backup & Replication button, and then click Backup Job.

image

Name your backup job, and click Next.

20) Click ADD to add VM’s to your backup:

image

21) The default view, when you expand the tree shows your Hosts & Clusters view, with your VM’s – just like the vCenter Client.

image

I’ll pick my running VM’s, but an alternative might be to use a VM’s & Template’s view, and pick VM folders.

image

22) Next, we need to pick a Backup Proxy (Used to speed up backups if the Veeam B&R VM cannot attach to them locally) and Backup Repository (to store the backup files).  The Backup Repository can be off host, such as a DAS or a NAS that is not part of your Shared Storage – remember not to keep your backups where your primary data is!

image

Select the REPOSITORY we create earlier.

Note the “Configure Secondary Destination for this job” – this will let you store a second copy on a second repository.  We’ll cover that another day, after we make our first backup job.

23) Click the ADVANCED button to get access to the good options:

image

I personally prefer “Reverse Incremental” which makes the previous night’s backup file the Full.  This does require a higher IOPS rate, and there are reasons why other options are better.

I would recommend clicking the check box for PERFORM ACTIVE FULL BACKUPS PERIODICALLY if you want to create a new FULL backup every now and again.  It’s up to you if you wish to choose “forever incremental” or have occasional full’s.  Its really a matter of confidence.

24) Click on the STORAGE tab:

image

Here you set the Deduplication, Compression, and optimize the storage method.

25) Click on the NOTIFICATIONS tab:

image

Put in an e-mail address to send your alerts to.  I recommend checking the VM NOTES box and choose a field other than “NOTES”.  If you use NOTES the default NOTES field in vCenter Client will get overwritten – and you might be using that for something.  Veeam is nice enough to let you pick the field you want to use, and if it does not exist, it will create it.

26) Click on the vSphere tab:

image

Here, you choose if you want to use VMware Tools quiescing – this will get you “Application Consistent” backups vs “Crash Consistent”.  Not using CBT (Change Block Tracking), which only backs up new/changed blocks from the previous backups, would be just silly.

27) Click on the ADVANCED tab:

image

The defaults are pretty good, but look into the manual to decide if you want to fiddle with this.

28) The STORAGE INTEGRATION tab:

image

Will allow you to use the SAN snapshot to do the backup.  I understand this is limited to just the HP VSA currently, but let’s the SAN snapshots be used, with the assumption that the SAN tools created clean, application consistent quiesced backups.

Click OK.

29) You’ll get a warning that you need to set up e-mail options:

image

So we’ll go do that in a bit.

30) Back on the NEW BACKUP JOB screen, we go to the GUEST PROCESSING option:

image

If you’re just looking for the VM’s and crash-consistent backups, then you don’t have to change anything.  There are many reasons why you might want this.  However, I would check the boxes for Enable application-aware image processing (for VSS to truncate things like Exchange and SQL logs) and then Enable Guest File system indexing (if you want to search backups for particular files).  Choose the credentials we created earlier:

image

If you click the ADVANCED button, you can modify the previous options on a per-VM basis.

Click NEXT.

31) On the SCHEDULE window:

image

Set up your schedule to run on particular days and times.  Veeam has an excellent Automatic Retry option where it will retry failed VM’s at the end of the job.  Often this is enough to bypass whatever made the VM fail in the first place – maybe it was being backed up by something else, it was too busy to quiesce, etc.  I’ve seen other backup products fail an entire job of 50 VM’s if only a small error occurred and not retry – or retry the WHOLE job.

You also have the option to automatically terminate the job if it exceeds the backup window – such as 8AM in the morning, when staff start showing up.

Click CREATE!

32) On the summary screen, click RUN THE JOB if you wish, and click FINISH.

image

33) Now let the job run.

image

34) Let’s go configure those mail options for backup notifications.  Click the upper left hand corner icon and choose OPTIONS:

image

35) On the E-Mail Settings tab:

image

Check the ENABLE box and enter your SMTP server name, and TO/FROM accounts.  Click TEST MESSAGE and verify it works, then click OK.

 

That’s really about it.  I realize at 35 steps, it *seems* like a lot, but that’s not just creating the first job, but setting up the entire system, the repository, the proxy, the notifications, etc.  They’re also very easy steps.  At this point you have a fully functioning daily backup of your VM infrastructure.

 

When you click on the job, it will show you details in the lower pane.  Depending on available resources (number of CPU’s), Veeam will choose the number of parallel VM’s it can back up.  More vCPU’s = More simultaneous VM’s.

image

Here you can see that I’m getting 81.9MB – a decent rate considering I only have 1GbE link to my media server.  More interesting is that it has processed 27.5GB, but only had to transfer 13.8GB – a 2x savings in bandwidth.  You can see the deduplication and compression savings live.  This rate tends to get increasingly better as additional VM’s are processed, as they likely contain very similar data if they’re all similar OS’s with similar roles or built from similar templates.

image

And now that the job is done (A different job, so the times are off), you can see that my 8 VM’s took 93 minutes to backup, the bottleneck was actually reading from the source (1GbE NFS, who knew!), processed 325GB, but only transferred 79.2GB – a 3.9x dedupe ratio.

image

An incremental, run immediately after shows a 38 minute duration that read 245GB of VM’s, only had to read 48.9GB of actual data, and only transferred 1.3GB of it due to deduplication and compression.  Imagine what that would be like across the WAN compared to a ‘normal’ backup!

I’ll have more posts on the new Veeam B&R v7.0 coming, but head on over to our local Veeam SE, Rick Byrne for more advanced options – http://rickrbyrne.wordpress.com/

Categories: Veeam B&R

Veeam B&R v6.1 and vSphere v5.1–hold off for a bit

September 16, 2012 2 comments

Just saw an e-mail come through with the Veeam Community digest.  While I don’t have a link directly to that as a post to reference, there are two important bits of information:

Quoted from Gostev directly:

Our QC has finished smoke testing of B&R 6.1 against vSphere 5.1 RTM code, and based on the first results, version 6.1 will NOT officially support vSphere 5.1. In case you are wondering what B&R functionality specifically brakes after vSphere 5.1 upgrade, our quick testing uncovered at least the following problems. Note that there could be more issues – even in functionality that “seemed” to work – as this was not full regressive testing. Thus, please avoid upgrading to vSphere 5.1 until we release B&R 6.5, which we are aiming to make generally available later this year – really, as soon as we can.

Critical issue warning for those still on VMware ESX 4.1! You must update B&R with the latest build (6.1 patch 1b) before upgrading to the recently released ESX 4.1 Update 3, or all your jobs will start failing (refer to the Known Issues sticky for more details). There are no similar fixes available for previous B&R versions, meaning that if you are not on B&R 6.1 yet, you must upgrade B&R first. This is really something extraordinary – usually, VMware does great job with the maintenance updates – I don’t remember a single one that would break our products. Nevertheless, this once again shows how important it is to test EVERY update before applying into production (or, at least, not jump the update as soon as it is released – but let other people validate it, or run into any issues first).

So the key takeaways are:

If you are on Veeam B&R v6.1 and considering moving to vSphere v5.1 – hold off for a bit.  Likely until Veeam B&R v6.5 later this year.
http://forums.veeam.com/viewtopic.php?f=24&t=13397&start=15#p61987 (at least the following problems link from above)

If you are on Veeam B&R v6.1 and considering deploying vSphere v4.1 Update 3 which was just recently released, also hold off for a bit.
http://forums.veeam.com/viewtopic.php?f=2&t=12207#p61785 (Known Issues link from above)

I have to say, Veeam’s communications and openness continues to impress me.  A lot of vendors would keep this information close, or tell folks that it must be something they’re doing that’s wrong, and pass the buck.  While some could read the above as Veeam having issues, I see it as them being a part of the community and sharing issues as soon as they’re aware of them.  This is part of the reason I really like dealing with them as a vendor!

It’s also worth sharing the additional detail shared, related to Synology:

I keep getting occasional inquiries on how my new and shiny Synology DS412+ is doing, so here it goes. In short, I have been extremely happy with performance, stability, noise level and functionality. But a few days ago, I became even happier, because Synology has released DSM 4.1 firmware update. Apart of many cool new features, they have added support for multiple additional VAAI primitives. Note how 412+ is the only mid-range NAS to get the full VMware VAAI support. I cannot be more happier with my choice! Really hoping they will add Microsoft ODX support soon, too. Heads up though – just like B&R 6.1, looks like DSM 4.1 does NOT support vSphere 5.1 yet.

This has been pretty big on the Twitter this last week, as many bloggers, vExperts, home labbers, etc, have this kind of equipment in their labs, and obviously, are affected by it.  Compounded moreso by how popular the equipment became when it was one of the first (only?) in this tier to have VAAI support at such a low entry price.

If anyone knows how to find a link to the Veeam Community Digest online, let me know as I’d really like to share a link directly to that, rather than cut/paste from e-mail.  Give credit directly to the source and all that.

Categories: Veeam B&R, vSphere