Archive

Archive for the ‘WSUS’ Category

Windows Patching – What happens when you aren’t paying attention.

November 19, 2014 Leave a comment

Yesterday, I posted some details about MS14-068 and MS14-066 (https://vnetwise.wordpress.com/2014/11/19/cve-2014-6324-ms14-068-and-you/) and of course today, have had to do some investigating into a few sites that have a variety of patching systems.  Some are using SCCM, some WSUS, some have policies and procedures, some don’t.  But I noticed a potential ‘perfect storm’(?) of situations that could cause some of them grief – and it was more than just one.

Let me draw you a picture of what is a pretty common environment:

  • WSUS exists for updates, because that’s “the responsible thing to do”
  • WSUS was likely configured some time ago, and no one likes it because it’s not sexy or fancy, so it doesn’t get any love.  Thus, it is probably running on Windows 2008 or 2008 R2.
  • Someone at some point *did* ensure that WSUS was upgraded or installed with WSUS 3.0 SP2

This all sounds pretty good, on the face of it.  Now let’s introduce some real world into this environment….

  • Someone decreed that they shall “only install Critical and Security Updates” – Updates, Update Rollups, Feature Packs, etc, would not be installed.
  • Procedures state that you will install updates that are previous month or older – so  you’re staying 30 days out, which is reasonable – let someone else go on Day0.
  • Those same procedures state that you will look at the list, and select the Critical and Security Updates from the last month, and approve them.
  • Nothing is stated for what to do about the current month’s patches – they are left as “unapproved” – but also not “declined”

Alright, so still pretty “common” and at face value, not that bad.  A year or two goes by, and now you introduce Windows 2012 and Windows 2012 R2 to the mix.  This itself is not a problem, but it’s where you start to see the cracks.  Without even having to look at the environment, I know already the things I want to be looking for….

  • Because the current month’s updates are not being “Declined”, they’re showing up in the list as “missing”.  If you have 10 updates, and 8 are approved and 2 are not, you will only ever possibly show 90% patched.  The remaining two WSUS/WU knows are “available”, but “I don’t have them.  You want to decline those so they only show up as 8 updates and 100% success.  Otherwise, how do you know at a glance if the missing update is the approved one that SHOULD be there, or one from this month?  Your reporting is bad.  See: https://vnetwise.wordpress.com/2014/03/24/howto-tweaking-wsus-so-it-only-reports-on-updates-you-care-about/

 

  • Because the process counts on someone approving “last months” updates and not “all previous updates”, there’s almost certainly going to be some weird “gap” where there is a period of a few months that isn’t approved and isn’t installed for some reason.  But the “assumption” is that they’re all healthy.  Because the previous point doesn’t “decline” any updates, the reports for completion are untrustworthy – and/or never reviewed anyways.

 

  • Next, Windows 2012+ has been introduced.  There’s a KB that is required to be installed on the WSUS server *and* rebuild of the WSUS package on the client to ensure compatibility.  See MS KB2734608 (http://support.microsoft.com/kb/2734608).  Because this is an “Update” and neither Critical nor Security, it is not applied to either the WSUS server or the clients.

 

 

  • In order for the Windows 2012/2012R2 WU/WSUS behavior to actually be changed, you need GPO’s that Windows 2012/2012R2 understands.  In order for that to be true, you need 2012+ ADMX files in your GPO environment.  Preferably in your GPO “Central Store” (again – https://vnetwise.wordpress.com/2014/03/20/howto-dealing-with-windows-2012-and-2012-r2-windows-update-behavior-and-the-3-day-delay/).  But because Windows 2012 and 2012 R2 were likely “added to the domain” with no testing, studying, certification, or reading, this wasn’t done.  Equally, even if it WAS done, most likely someone is still editing the GPO’s on the 2008/2008R2 based Domain Controller – which wipes out the ADMX based changes and replaces them with ADM files and the subset of options that they understand.  You’ll never know this happened though, and even if you jump up and down and tell people not to do it, they will.

 

  • No one is ever doing a WSUS cleanup, so Expired, Superceded, etc updates are still present.  Which isn’t helping anyone.

 

So to make that detail a little shorter:

  • Choosing Critical and Security Updates only is causing you to miss out on *required* updates.  Stop being “fancy” – just select them all please.
  • Because you’re choosing “date ranges” of updates, you’re missing some from time to time.  Stop being “fancy” – select “from TODAY-## to END”
  • If you introduce a new OS to your environment, you need to ensure your AD and GPO’s support them.

On top of the Updates and Update Rollups above that cause those issues, let’s take a quick look at some of the other things that are NOT considered Critical or Security Updates:

November 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2:

    That’s just ONE Update Rollup.  None of those look like ANYTHING I’d want to happen to my servers.  </Sarcasm> So why WOULDN’T I want to install those?  Yes, there may be features you’re not using.  Perhaps you don’t use DeDuplication or DFS-R.  Won’t it be fun later when you install those Roles/Features, and WSUS scans that server, and says “all good, nothing to update” for you?  Tons of fun!
    So, long story short – please stop being fancy.  You’re introducing complexity and gaps into your environment, and actually making things harder.  This means more work for you and your staff and co-workers.  That likely don’t have enough time and resources as it is.
    Don’t pay technical debt….

CVE-2014-6324, MS14-068, and you!

November 19, 2014 4 comments

By now, you’ve almost certainly heard of the Microsoft Update being released out of band, MS14-068 related to CVE-2014-068, for an in-the-wild Kerberos exploit with some pretty serious ramifications.

Definitely check out this Microsoft Technet Blog post: http://blogs.technet.com/b/srd/archive/2014/11/18/additional-information-about-cve-2014-6324.aspx

The relevant portions to me are:

Today Microsoft released update MS14-068 to address CVE-2014-6324, a Windows Kerberos implementation elevation of privilege vulnerability that is being exploited in-the-wild in limited, targeted attacks. The goal of this blog post is to provide additional information about the vulnerability, update priority, and detection guidance for defenders. Microsoft recommends customers apply this update to their domain controllers as quickly as possible.

And:

The exploit found in-the-wild targeted a vulnerable code path in domain controllers running on Windows Server 2008R2 and below. Microsoft has determined that domain controllers running 2012 and above are vulnerable to a related attack, but it would be significantly more difficult to exploit. Non-domain controllers running all versions of Windows are receiving a “defense in depth” update but are not vulnerable to this issue.

Now, don’t take that to mean my stance is “Meh, don’t patch!”.  Quite the opposite.  As per the article:

Update Priority

  1. Domain controllers running Windows Server 2008R2 and below
  2. Domain controllers running Windows Server 2012 and higher
  3. All other systems running any version of Windows

So get those DC’s patched _now_, and calmly plan to update the remaining servers.

 

But I’ve heard from a number of colleagues/twitter/posts today that this introduces chaos, makes a busy week worse, etc.  Certainly it is critical and important, but I’m not getting the frustration:

  • It immediately only applies to 2008R2 DC’s and lower.  Most Small to Mid size enterprises I know don’t have more than a couple dozen at best, and often many less.  So patch them.
  • You likely don’t have 2012R2 DC’s – for many reasons.  Too many legacy systems that don’t like 2012/2012R2 DC’s, you haven’t had time to get around to it, you haven’t tested, you’re afraid of them, whatever. 
  • They’re DC’s, they’re redundant.  Just patch the bloody things.

But I think it’s that last part that makes people lose their minds.  Folks, if you can’t reboot a DC in your environment, you’ve built a very poor system (or “have” one – maybe you inherited it – it’s still your job to make it better!).  Yes, you should minimize the downtime, so do it in a period of lower activity if you can, but if you have to wait for… 2:00AM on a Sunday, there’s a problem with what you’ve built.  I can probably even guess what these problems are:

  • Even though you likely have Windows Server Datacenter and virtualization (Hyper-V or VMware) for unlimited VM’s, someone is probably all freaked out about “server sprawl” – so you have fewer servers that you could have.
  • Which means you likely aren’t separating out roles
  • So your DC’s are likely serving double exponential duty also serving DNS, and DHCP, and PKI, and RADIUS, and, and, and. 
  • Failover/maintenance has never been tested.  So you have “redundant systems” and maybe tested the failover, in a CONTROLLED fashion – but never tested the equivalent of a “power cord yank”
    Stop doing this. 
    It doesn’t require a $5000 1U server to run a role any more.  Stop building like its 2003.  Server Sprawl is only a problem if you have lousy automation and processes for consistency.  Managing 53 or 153 servers shouldn’t be significantly different.  You SHOULD be able to reboot servers and services at any point in time without concern.  If you cannot, then even if you have multiple, you DO realize you have identified a failure point, right?

If your answer is something along the lines of “But we don’t know the impact it will have…” – seriously?  Why not?  You tested, right?  Your monitoring software will alert you of services or functions that fail when a dependent service fails?  You might have even built in rules to self-heal or scripts to try “the obvious fix”?

Probably not though.  Everyone’s too busy paying 28% “Technical Debt” on the big fancy expensive toys and software they bought that they didn’t get enough people to install completely or got button mashed until it “kinda worked” then the next fire stole the body away.  You know that “Cloud” thing everyone’s talking about and how all the CEO/CIO/Directors/Management “want it” but “don’t know what it does”?  It’s about automation, scale, and self-healing, with growth and shrinking elasticity.  Instead of “wanting it”, it’s time to “build it”. 

Or, we can just keep doing like we’ve always done – chasing the next hot thing, and killing symptoms instead of root causes.  That’s probably what will happen…

 

All that said, MS14-066, which addresses the SChannel issues, that needs to be updated for as well.  But as per many online sources (http://windowsitpro.com/security/ms14-066-months-problem-patch, KB 2992611, http://www.infoworld.com/article/2849292/operating-systems/more-patch-problems-reported-with-the-ms14-066-kb-2992611-winshock-mess.html) there are issues with this update, that have resulted in it getting a re-issue.  Microsoft has a blog post about this as well:

http://blogs.technet.com/b/rmilne/archive/2014/11/13/critical-schannel-vulnerability-ms14-066.aspx

Specific details you care about:

Update 16-11-2014: KB 2992611 has information on known issues.

Update 18-11-2014: V2 of the bulletin was released.  Details from the update:

Reason for Revision: V2.0 (November 18, 2014): Bulletin revised to announce the reoffering of the 2992611 update to systems running Windows Server 2008 R2 and Windows Server 2012. The reoffering addresses known issues that a small number of customers experienced with the new TLS cipher suites that were included in the original release. Customers running Windows Server 2008 R2 or Windows Server 2012 who installed the 2992611 update prior to the November 18 reoffering should reapply the update. See Microsoft Knowledge Base Article 2992611 for more information

So if you’ve already patched, you’ll need to re-patch. 

I wonder if this can be taken to be true:

As of writing, the MSRC and other security assets do not report that there attacks in the wild since the issue was responsibly disclosed to Microsoft. However it is only a matter of time….

Given the issues, and how this is introducing interoperability issues, it may be advisable to give some thought to how fast this update gets rushed into production.

Hope the above information helps, and sorry for my little detour into rant-ville.  I feel better now though, if it matters.

Categories: AD, WSUS

PoSH: Get-PatchDate for SCCM

November 18, 2014 Leave a comment

Anyone who knows me, knows that if I have to do something 3 times, I’m going to do two things:

1) Try to automate it

2) Get angry at you

Lucky for me, anger leads to productivity Smile

The PowerShell that follows allows me to get the dates for patching for a site I’m doing work for.  It would also work for at least two others that I’ve done similar work for, so it’ll definitely be of some broader use than just one site.

The general gist of the script is to find the dates for updating.  This site does their updates with the following schedule:

  • DEV1 group happens on the First Thursday of the month – this covers a middle of week testing during business hours
  • PROD1 group happens on the Second Saturday following the DEV1 group – 9 days later. This handles systems that can be updated in the evening on a weekend.
  • PROD2 group happens on the Sunday following PROD1 – this handles systems that can be done on a weekend, but might be doing some manner of processing at night – batch updates, backup servers, etc.
  • PROD3 group happens on the Monday following PROD2 – this handles systems that could not  be updated at night or during the weekend.

The problem is that the 9 days after DEV is not always “2nd Saturday”, sometimes it is “3rd Saturday” – if the first of the month occurs on a Fri/Sat/Sun.  Equally, #nd Saturday may be #nd+1 Sunday.  So to try to figure this out, I found a script that gets “WeekDayInMonth”.  That got me the basics, but then I still needed to get MY dates from it.

#
# Created By: Avram Woroch
# Purpose: 
#   To obtain the dates in the month for performing Windows Updates.
#   Currently assumes &quot;First Thursday&quot; for DEV, then PROD1 occurs the second Saturday following
#   Followed by PROD2 the next Sunday and PROD3 the next Monday.  
#   We aren't trying to figure out which Sat/Sun/Mon of the month it is, as we can count forward 
#   from the DEV date.
# Usage:
#    Get-PatchDates.ps1 
#       Loads the script
#    Get-PatchDate &lt;optional MONTH in ##&gt; &lt;optional YEAR in ####&gt;
#      If not MM YYYY are provided, the script will assume CurrentMonth and CurrentYear
#      eg: Get-PatchDate 12 2014 - will find Dec 2014
#          Get-PatchDate - will find Nov 2014 (when the script was written)
# 4 Variables are populated, to be leveraged by SCCM scripts
#
# Get-WeekDayInMonth portion borrowed from 
# http://blog.tyang.org/2012/09/03/powershell-function-get-weekdayinmonth/
#

Function Get-WeekDayInMonth ([int]$Month, [int]$year, [int]$WeekNumber, [int]$WeekDay)
{
    $FirstDayOfMonth = Get-Date -Year $year -Month $Month -Day 1 -Hour 0 -Minute 0 -Second 0
    #First week day of the month (i.e. first monday of the month)
    [int]$FirstDayofMonthDay = $FirstDayOfMonth.DayOfWeek
    $Difference = $WeekDay - $FirstDayofMonthDay
    If ($Difference -lt 0)
    {
        $DaysToAdd = 7 - ($FirstDayofMonthDay - $WeekDay)
    } elseif ($difference -eq 0 )
    {
        $DaysToAdd = 0
    }else {
        $DaysToAdd = $Difference
    }
    
    $FirstWeekDayofMonth = $FirstDayOfMonth.AddDays($DaysToAdd)
    Remove-Variable DaysToAdd
    #Add Weeks
    $DaysToAdd = ($WeekNumber -1)*7
    $TheDay = $FirstWeekDayofMonth.AddDays($DaysToAdd)
    If (!($TheDay.Month -eq $Month -and $TheDay.Year -eq $Year))
    {
        $TheDay = $null
    }
    $TheDay
}


Function Get-PatchDate ([int]$Month, [int]$Year)
{
  $DayMonth = $Month
  $DayYear = $Year
  If ($Month -eq &quot;&quot;)
  { 
    $DayMonth=(Get-Date).Month
  }
  If ($Year -eq &quot;&quot;)
  {
    $DayYear=(Get-Date).Year
  }
$PatchDay=(Get-WeekDayInMonth -month $DayMonth -year $DayYear -weeknumber 1 -weekday 4)
$PatchDayDEV1 = (Get-Date $PatchDay).AddDays(+0)
$PatchDayPROD1 = (Get-Date $PatchDay).AddDays(+9)
$PatchDayPROD2 = (Get-Date $PatchDay).AddDays(+10)
$PatchDayPROD3 = (Get-Date $PatchDay).AddDays(+11)

Write-Host &quot;&quot;
Write-Host &quot;In the year&quot;$DayYear&quot; and the month of &quot;$DayMonth&quot;:&quot;
Write-Host &quot;  Group DEV1 will be patched on: &quot;$PatchDayDEV1 
Write-Host &quot;  Group PROD1 will be patched on:&quot;$PatchDayPROD1 &quot;- 9 days later&quot;
Write-Host &quot;  Group PROD2 will be patched on:&quot;$PatchDayPROD2 &quot;- 10 days later&quot;
Write-Host &quot;  Group PROD3 will be patched on:&quot;$PatchDayPROD3 &quot;- 11 days later&quot;
}

I’m still working on how to make this “Better”, and I’ll likely seek input from my resident PowerShell guru (http://pleasework.robbievance.net/) but until then I’m trying it on my own.  The usage is:

“Get-PatchDates.ps1” to load the module

“Get-PatchDate” with no parameters to get the dates for the current month.  Or we can specify the MM YYYY on the command line to override.  But this way allows me to set the script up to run on the first of the month, and get the dates for that month.

So what we end up with is output of:

PS C:\WINDOWS\system32&gt; Get-PatchDate 

In the year 2014 and the month of 11:
  Group DEV1 will be patched on:  11/6/2014 12:00:00 AM
  Group PROD1 will be patched on: 11/15/2014 12:00:00 AM - 9 days later
  Group PROD2 will be patched on: 11/16/2014 12:00:00 AM - 10 days later
  Group PROD3 will be patched on: 11/17/2014 12:00:00 AM - 11 days later

I’m going to have a bunch of posts coming up for some SCCM 2012 Windows Server Windows Updates scripting, that I hope will help someone avoid having to deal with a situation where you hear “So every month, we do this process, and it’s currently manual….”

These would also be able to be workable with some WSUS general scripting, as long as modified GPO’s accordingly in a script and/or reconfigured groups of servers WSUS registry settings.

Here’s hoping this all works…

UPDATE:

I came back realizing I was going to probably want to store the various dates as variables in memory to call in later scripts or functions in this process.  Turns out, I thought of that (or perhaps, just got lucky…) when I used the variable names $PatchDayXXXXXX – which does exactly that.  Later on, I’m going to use the PatchGroup Descriptions to designate a delimited Start and Stop time, so I can use those variables in my maintenance window and deadline designations. 

Categories: PowerShell, SCCM2012, WSUS

HOWTO: Tweaking WSUS so it only reports on updates you care about

March 24, 2014 Leave a comment

WSUS is a great built in tool for working with Windows Updates, but sometimes it takes a bit of effort to find the best way to use that tool. Here are a few things to help make the system run smoother.

The following assumptions are made:

  • You deploy updates during a Quarterly Outage, every 3 months – eg: March, June, Sept, Dec month end weekend.
  • You must validate the patches in advance, including a DEV and TEST domain or environment.
  • There isn’t enough time from “Patch Tuesday” to deploy in DEV, test for a week or two, deploy in TEST, test for a week or two, then approve for Production – which might only be two weeks from Patch Tuesday
  • To accommodate the above schedule, you then install “Current Month -1” for all updates. Thus in March, you would deploy and approve Dec/Jan/Feb updates, but NOT Mar.
  • This allows you to install in DEV the week after Feb Patch Tuesday. You can then install in TEST two weeks later, or about the beginning of Mar. TEST can then be run for 2-4 weeks depending on Quarterly Outage window, to validate and be certain of updates in Production.
  • It is acceptable for TEST and PRODUCTION to be out of sync for this period. There needs to be a balance between TEST/PRODUCTION being identical and being able to pre-validate updates.

1) Approving Updates

In the WSUS console, click on SERVER -> UPDATES -> ALL UPDATES, and then click in the main window.

clip_image002

Right click on the headers and choose to show “RELEASE DATE”. Sort by RELEASE DATE.

NOTE: In my example here I’m showing “APPROVAL=DECLINED”. You would be choosing “APPROVAL=UNAPPROVED” but currently I have none to use as an example.

clip_image004

Sort by the RELEASE DATE column. Remember that as we are in MAR of 2014, we do NOT want to select any ##/03/2014’s as they are “too new”. Select ALL PREVIOUS updates from “Month -1” or 02/2014 in this case. Right click and choose APPROVE.

clip_image006

You want to click on the parent level and choose APPROVED (which has already been done here, as indicated by the GREEN CHECK that is shaed out). Repeat this but then choose APPLY TO CHILDREN – if this is appropriate for all of your WSUS Groups. In this environment, WSUS is only used for Windows Server OS groups, and they’re grouped by “Automatic”, “Manual”, and “Primary/Secondary” groupings. As such, they all GET the same updates, it’s just to have different schedules and methods for installation. Click OK. A new dialog will pop up as it sets each update to APPROVED and will take some time to complete.

Until you perform this step, you will see the updates in reports showing computers that require the update, but they’re not allowed to install it. Thus, even if you go and perform a manual Windows Update check, you’ll never see the updates to be able to select them. A sample update report would look like:

clip_image008

The APPROVAL column for the update(s) would say “Not approved”. The STATUS column will know if the system has already downloaded and staged the update.

2) Declining Updates.

The above all seems well enough, except for the non-obvious results. For this month you’ll have Mar/2014 updates not approved and as the months go buy you’ll have downloaded the updates for Apr/May/Jun. Your reports are now going to show that your systems aren’t 100% compliant, even if you install all current updates available. You’ll spin your wheels trying to figure out why WSUS says you have 2 updates outstanding, but the Windows Update client says “no updates found”. This is because WSUS knows about the updates, and will indicate they’re available but not approved. So your system DOES require them, but you haven’t let them off the leash yet. So the report is in fact, valid. But what it’s really showing you is “next time you do updates, you’ll need to install these updates”. That’s great for the week AFTER quarterly outages, but it does nothing to help you DURING or just after the outage to measure success.

To fix this issue, what you want to do is DECLINE the updates.

clip_image010

Change the APPROVAL drop down to show “ANY EXCEPT DECLINED”, which will not show all previously declined updates. Sort by the RELEASE DATE column. Remember that as we are in MAR of 2014, we DO want to select ONLY ##/03/2014’s as they are “too new” to be Approved Select ALL updates from the last Approved date or 03/2014 in this case. Right click and choose APPROVE. (this is counter-intuitive)

clip_image012

Choose “NOT APPROVED” (still not intuitive – you’re going to want to try looking for a “DECLINE” option, and it’s not an option – you need “NOT APPROVED”) from the top level drop down. Then click again and choose APPLY TO CHILDREN. Then click OK.

Now when you pull reports on your system, you’ll actually see 100%:

clip_image014

You now want to keep performing updates on your servers until everything shows 100%. That will then be:

    • All KNOWN updates
    • Including APPROVED, which will actually allow installation of a KNOWN update
    • NOT including DECLINED, which will not show them as “needed” in your reports of % columns.

3) Each month between “now” and “Next Quarterly Update”

This will now make you fine for “Today” assuming today is “March 2014, after Patch Tuesday, but before April 2014, Patch Tuesday”. However, come April/May/June Patch Tuesday, new updates will get downloaded to the WSUS server. For your reports to remain accurate, you’ll need to come into WSUS and set all the new updates to “DECLINED”. Follow the same process you did in Step #2, only of course you’ll see more than just 03/2014 to select. Just select from the first date of ##/03/2014 and go to the bottom and repeat the DECLINE option.

4) NEXT Quarterly Update cycle:

Steps #1 and #2 above assume you have a net-new WSUS installation. If you’ve done this process before, then come Jun/2014 when you need to select Mar/Apr/May months for approval, you’re going to have all of Mar/Apr/May/June of 2014 set as “DECLINED”. You need to now set them to approved, as well as the now downloaded Apr/May.

Similar to Step #3, you’re now going to take all your Mar/Apr/May updates and set them to “APPROVED”. You’ll want to do this immediately following the May Patch Tuesday, as this will then let your reports be accurate to reflect the number of updates and systems required. You can now provide accurate details on how long and how many updates you will need to perform.

5) Just BEFORE NEXT Quarterly Update cycle:

Understandably, you’ll now show accurate reports for May 2014 and you’ll no longer show 100% up to date, as of course you are not. However, as soon as Jun 2014 rolls around, your numbers will be inflated again because of updates that are now known after June’s Patch Tuesday but are not approved. This will, as per Step 2, skew your numbers and prevent you from hitting 100% success in your maintenance window. So ensure you set then all June updates to “DECLINED”.

A general rule of thumb might be that following a Patch Tuesday you should:

  • Go in and APPROVE all previous month updates
  • Go in and DECLINE all current month updates

This would allow non-critical servers that are set to update automatically on some schedule, to keep up to date on a monthly basis vs waiting for quarterly. This provides two benefits:

1) You get the new updates tested (albeit in limited fashion) on existing servers up to 3 months prior to quarterly outages

2) There is far less load and number of systems to be manually or brute force updated during your maintenance window. Less load, means less IOPS on shared storage, which means updates perform quicker, which means you can do more/other maintenance in the same outage window.

HOWTO: Force WSUS Client to Update using PSEXEC

March 21, 2014 2 comments

WSUS is a great tool for automating and managing Windows Updates to various systems in a domain. However, it’s not really all that granular, which is a problem. While you could say “install all updates at 03:00 on Saturday”, you can’t say “and after rebooting, check again, because you’re still in the maintenance window”. You also can’t specify “do it RIGHT NOW, don’t wait for a random period” and there are some difficulties with “reboot when complete, don’t want 5-15 minutes, don’t wait 3 days, do it now”.

It turns out there some undocumented switches for the Windows Update Client (wuauclt.exe). Various lists can be found all over, I’ve found one at: http://kickthatcomputer.wordpress.com/2013/03/06/windows-update-command-line-options/

If you use these methods it might take you a bit of tweaking and fighting to make it work. Specifically if you’re having issues with Windows 2012/2012R2 systems, check: HOWTO: Dealing with Windows 2012 and 2012 R2 Windows Update Behavior and the 3 Day Delay.

This method can be pushed out to all systems via PSexec. Note though that there are some things to watch for:

· The GPO must be set to “4 – Download and Install Updates”. If it is set to “3 – Download and Notify” then all the “wuauclt /UpdateNow” in the world won’t make it do what it’s not allowed

· Except for maybe on Windows 2012/2012R2 systems, where it will think it’s in a maintenance window, and well, you said “UpdateNow”, so let’s do that.

· I’ve found it to be intermittent if the Day/Hour for the option to install in the GPO is not set near the time you’re pushing out. This doesn’t matter so much if you’re doing a scheduled restart such as “Sunday @ 03:00”. But if you have a manual maintenance window where you’re trying to brute force blast out and confirm all the updates that starts at Friday @ 20:00 – you should probably ensure that the GPO is set to the same, especially given that this batch file will refresh the GPupdate.

· As time goes on through that maintenance window, update the GPO time as well. They must go hand in hand.

clip_image002

What you’ll see is that it will schedule the installation for the next day. In the above example, C:\WINDOWS\WINDOWSUPDATE.LOG is showing that on 2014-03-20 at 2:20AM it says it is scheduling the installation to occur at March 21 2014 at 12:00AM. This is because the first line indicates the GPO setting is “Every day” @ “00:00”. So if anything, you’d like that to be “the next hour, of the same or following day”. Watch things like running Friday at 11:45PM and not changing your “Install Day” from Friday to Saturday to accommodate the 00:00 or 01:00 next time.

· There doesn’t seem to be any harm in pushing out the batch file to a system that’s already updating, other than it will restart the Windows Update service. Where possible though, you want to push it to systems that are not otherwise installing. I don’t yet have a method for knowing if a current update process is occurring. Perhaps if you took the “ping” process that is the timer, and made it a “start /wait” with a title, then looked to see if a process was running with that title, don’t run…. But this is as far as I’ve gotten for now.

· Periodically check the WSUS console for “Last Updated” and “Last Reported” to get an idea for what systems need checking. Also look at the % complete column to know which systems are done.

With all that said, the batch file itself:

===== WSUS_FORCE.BAT =====

@echo off

SET WSUSSERVER=FSRVCLOWSUS1

SET WSUSSHARE=WSUSLOGS

SET WSUSLOG=WSUS_FORCED.LOG

REM

REM PSEXEC Usage

REM psexec @SERVER.LST -u svcautomation -H -f -c -D \\FSRVCLOWSUS1\E$\WSUS\bin\WSUS_FORCE.bat

REM

REM

REM Run a GPUPDATE

REM

gpupdate /force

REM

REM Restart services and refresh Windows Update

REM

net stop wuauserv

REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v LastWaitTimeout /f

REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v DetectionStartTime /f

REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v NextDetectionTime /f

net start wuauserv

wuauclt /scannow

wuauclt /resetauthorization /detectnow

echo %COMPUTERNAME% Checking for WSUS Update at %DATE% %TIME% >>\\%WSUSSERVER%\%WSUSSHARE%\%WSUSLOG%

wuauclt /r /ReportNow

echo %COMPUTERNAME% Installing WSUS Update at %DATE% %TIME% >>\\%WSUSSERVER%\%WSUSSHARE%\%WSUSLOG%

wuauclt /UpdateNow

:CHECK_REBOOT_REQUIRED

REM

REM This registry key only exists if WSUS indicates a reboot is required. Thus, keep checking for it to appear, and then reboot

REM

ping 127.0.0.1 -n 61 > nul

reg query "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired" >nul

if %ERRORLEVEL%==1 goto CHECK_REBOOT_REQUIRED

if %ERRORLEVEL%==0 goto REBOOT

GOTO CHECK_REBOOT_REQUIRED

GOTO END

:REBOOT

echo %COMPUTERNAME% Rebooting after WSUS Update at %DATE% %TIME% >>\\%WSUSSERVER%\%WSUSSHARE%\%WSUSLOG%

shutdown -r -t 0

GOTO END

:END

===== WSUS_FORCE.BAT =====

HOWTO: Dealing with Windows 2012 and 2012 R2 Windows Update Behavior and the 3 Day Delay

March 20, 2014 13 comments

So let’s assume for the moment, that you’re a guy trying to work on some pretty detailed WSUS update deployments for a mixed OS network. This network will included some Windows 2012 and Windows 2012 R2 servers. You configure your WSUS server to auto deploy updates at some time – let’s say Sunday at 3:00AM. Then you check on Monday and realize that some of the servers still have a > 24 hour uptime, so updates must not have installed. You get busy, forget about it thinking the updates failed, and go about your day.

Then Wednesday morning you identify that some of the servers restarted at 3AM. Which is odd, because you didn’t do any maintenance. So something must be broken, now we need to investigate.

http://blogs.technet.com/b/wsus/archive/2013/10/08/enabling-a-more-predictable-windows-update-experience-for-windows-8-and-windows-server-2012-kb-2885694.aspx

Hello, Microsoft KB2885694. It seems in this KB, they’ve released an update to allow 2012/2012R2/8/8.1 to have a more controlled update experience. Gone are the “we’re restarting in 15 minutes” that is non-cancellable. Now what you can do is create a schedule maintenance window, and allow things to happen IN that window. So you’ll install updates and wait for that window to reboot. That might be when you deploy other software as well. Or do weekly reboots. Or just stop monitoring the system to send alarms. Either way, it’s a quasi-good thing. Unless you’re EXPECTING it to do what it has done in Windows 2003/2003R2/2008/2008R2.  If you’re not in an environment that does a lot of patches, or doesn’t do them regularly, and have only recently added 2012/2012R2 systems, this may catch you off guard. 

The body of the text on this is:

KB 2885694 introduces two main changes that define how Windows Update on Windows 8 and Windows Server 2012 computers can be configured using group policy. All policies mentioned are located at this path:

Computer Configuration / Administrative Templates / Windows Components / Windows Update

When enabled with a value of 4…

The Configure Automatic Updates group policy works identically to the Windows 7 / Windows Server 2008 R2 and earlier behavior.

On Windows 8 and Windows Server 2012 without KB 2885694 installed, that policy could configure the main automatic updating setting, but configuring the scheduled install day and time had no effect. After installing KB 2885694, the policy will enable you to configure machines to:

  • Install updates during automatic maintenance, the default behavior, or
  • Install updates at the scheduled day and time defined in the policy

A new group policy called Always automatically restart at the scheduled time enables restarts soon after updates are installed, instead of 3 days later

Also, in the link for http://support.microsoft.com/kb/2885694 is the following important nugget:

However, the scheduled day and time in this policy setting has no effect. The computer still installs downloaded updates during automatic maintenance, which defaults to 2:00 AM. Additionally, the computer no longer restarts 15 minutes after installing updates. To help avoid unintended data loss, a restart timer instead begins 3 days after updates are installed, and only when you are actively using the computer and are able to see and react to the restart timer.

Let’s talk about that last bit “only when you are actively using the computer”. This means the timer MAY NOT START, until an interactive login happens. So no one logs in until Tuesday afternoon at 2PM. All well and good. But THEY don’t know about this process, and they ignore the message. Guess what happens on Friday at 2PM? That *IS* 3 days later after all…

That would be good to know. So you have a few options:

1) Install the update(s) and get understanding of the options, making it more “Windows 2008R2” ish.

2) Deal with the new method, and create some process that automatically restarts during maintenance

3) Create a process that will watch for a “RebootPending” status and when found, reboot. Obviously this has some danger to it, if you try to do this anytime. But if you do it just say “Sunday from 3:00-5:00AM” you’d probably be fine. Until you tried to do mass updates on say a Quarterly Outage on a Friday night at 8:00PM and couldn’t figure out why 30% of your servers aren’t updating.

So for now, my suggestion would be to move towards the Windows 2008R2 method. Likely, you need to do that anyways, as you have a mixed OS fleet to support, so accept the lowest common method. Note though, that’s not as simple as you’d think So – how do you DO that? The same link (http://support.microsoft.com/kb/2885694) indicates how – but you have to read it carefully:

An update is available that lets you control how the Automatic Updates client applies updates in Windows 8 and Windows Server 2012. After you install this update, the "Configure Automatic Updates" policy setting will work again as it used to in previous Windows operating systems. Instead of installing updates during the daily maintenance cycle, Windows updates will be installed at the time and day that are scheduled in the "Configure automatic updates" policy setting.  
This update also introduces the following Group Policy setting that you can configure to force the computer to automatically restart sooner in order to finish installing important updates:

Computer Configuration\Administrative Templates\Windows Components\Windows Update\Always automatically restart at the scheduled time

Update information

To obtain this update, install update rollup 2883201. For more information about how to obtain this update rollup package, click the following article number to view the article in the Microsoft Knowledge Base:

2883201 Windows RT, Windows 8, and Windows Server 2012 update rollup: October 2013

What’s that you say? Your WSUS server is 2008 R2? And your domain/forest is 2008 R2? And you edit your GPO’s from a DC or the WSUS server? Guess what – you’re never going to see the 2012/2012R2 based GPO settings to enable them!

So what you need to do:

· Start modifying GPO’s on the HIGHEST level OS you have. A generally good practice

· Apply update 2883201 to that system

· Modify the GPO

· Apply update 2883201 to ALL your 2012/2012R2 based servers. (What you thought it would be as simple as updating the GPO editing machine? They won’t know what to DO with the updated GPO until they get the update that says that they should use that information to do it the Windows 7/2008R2 way….)

So there’s a fun little catch-22. In order to get control over your updates on your Windows 2012/2012 R2 systems via WSUS/GPO, you need to deploy an update that you’re having issues deploying, to tell it how to deploy updates.

On the plus side, looking at WSUS reports in this environment:

clip_image002

It looks like all the systems have it. So now we just need to go edit the GPO’s on a 2012 R2 system.

So let’s look at how this appears different:

2008R2 based Group Policy Editor:

clip_image004

2012R2 based Group Policy Editor:

clip_image006

2012R2 based Group Policy Editor:

clip_image008

Well that doesn’t make any sense, no new settings. But why?

clip_image010

“Administrative Templates: Policy Definitions (ADMX files) retrieved from the central store”. That means the ADMX files are being loaded from AD, and NOT from this machine. If you haven’t PUT the updated ADMX files on the Central Store… then it’s not going to work.

clip_image012

Policy Definitions (ADMX files) from 2010 likely isn’t going to help us much in 2014… If you go ahead and get: http://www.microsoft.com/en-US/download/details.aspx?id=41193 for the Windows 8.x/2012 ADMX templates, you can update the central store. Note that sometimes some of the ADMX files remove options that used to exist, so consider this when you’re blindly copying in ADMX files that might overwrite others. Unpack the files to a location – on a Windows 7 system this will go to C:\Program Files (x86)\Microsoft Group Policy\Windows 8.1-Windows Server 2012 R2\PolicyDefinitions. Then you want to take the WindowsUpdate.ADMX file and copy it to \sysvol\\Policies\PolicyDefinitions\">\\<domain>\sysvol\<domainname>\Policies\PolicyDefinitions\ Do the same for the ADML (language) file, but put it in \sysvol\\Policies\PolicyDefinitions\en-US">\\<domain>\sysvol\<domainname>\Policies\PolicyDefinitions\en-US. Close and re-open the Group Policy you were editing, and go back to the Windows Update node:

clip_image014

Look who decided to come to the party. Also note there is now an option for completely blocking the “Check for updates directly from Microsoft Update”. This isn’t relevant to our topic, but worth calling it out as an additional update. Set this to ENABLED and set a time (Default and minimum is 15 minutes), and save your Group Policy.

Now, if you’re looking to identify if this is working, you’ll want to look at the C:\WINDOWS\WINDOWSUPDATE.LOG on your Windows 2012 servers:

clip_image016

Here you can see we have a server that has installed updates, and is setting the timer for a few days from today (2014-03-20). That’s no good. So let’s do a GPUpdate /force on the system, and a WUAUCLT /UpdateNow and see what it says. If you’re using something like my WSUS_FORCE.BAT to push the updates and restart when the registry key indicates there is a restart pending, you’ll see:

clip_image018clip_image020

Here you can see right before that, at 1:40:21AM, that Windows Update indicated a restart was required. The reboot in question only took about 15 seconds to complete. This allowed the post restart updates to install and finish. We now show no required updates on the VM.

clip_image022

This matches with what WSUS indicates as well for FSRVCLOTEST6.

So the key takeaways here:

· When introducing a new OS to your environment, such as Windows 2012 or 2012R2, make sure your GPO’s are updated to allow for it.

· Even if they are, be aware that updates can change the behavior of previously known systems.

· Only ever update/modify/edit GPO’s with the MOST RECENT OS in your environment, using the MOST RECENT ADMX files.

· Before blindly copying over the entirety of the ADMX/ADML files, test and confirm you’re not modifying any required settings. Remember you might be support Windows 2003 or 2000 still or Windows XP on the client side.

· Be careful selecting any new features from the latest GPO’s and ensure you check for backwards compatibility. Most GPO settings will tell you the minimum/oldest OS that is supported for that setting.

· Using the above modifications, you can not only be certain that Windows 2012/2012R2 systems will reboot during scheduled/automated maintenance, but also during situations where you are brute force pushing out something like a WSUS_FORCE.BAT which will look for new updates and install them immediately.

HOWTO: Force client to clean up WSUS and rescan

March 18, 2014 Leave a comment

In a previous post (HOWTO- Resolve missing systems from WSUS console) I documented how to reset WSUS SusClientID for missing systems that are duplicated in WSUS.  However, you may not want to remove or reset this ID – it’s a good idea to periodically clean up the C:\Windows\SoftwareDistribution\Download folder.  If you have 400MB of updates on 100 systems, you’re keeping around 40GB of downloads – likely on your SAN, in your backups, as part of your replication, etc. 

It’s a pretty basic script but it’s handy to have around.  It’s nice to run after any sort of WSUS installations – be they automatic or manual. 

===== CLEAN_WSUS.BAT BEGIN =====

@echo off
REM
REM PSEXEC Usage
REM psexec @server.lst -u svcautomation -H -f -c –D \\wsusserver>\E$\admin\bin\clean_wsus.bat
REM

net stop wuauserv
rmdir /q /s C:\WINDOWS\SOFTWAREDISTRIBUTION\DOWNLOAD
net start wuauserv
wuauclt /scannow
wuauclt /detectnow
wuauclt /r /ReportNow

===== CLEAN_WSUS.BAT END =====

Categories: WSUS