Hey! What’s up with my Patch Report?

We’ve recently noticed that many of our clients’ update delivery numbers weren’t up to our usual awesome standards, so we did some digging to discover what was happening.

Q: So… what’s broken?

A: On each Windows system, the Windows Update Agent gets its health information from a variety of Microsoft’s public servers; OS updates might come from one cluster, non-OS updates (like Office patches) might come from another, and Definition updates might come from a third. Inventory information for non-OS updates is sporadically coming back empty from Microsoft’s sites, leading to an inaccurate picture of a system’s health and fooling our ManageSuite software into thinking that fewer updates need to be installed than are actually required by the system.

This problem also is sporadic; sometimes the inventory works as expected, other times it’s incomplete. Further, in investigating within peers in the community, similar inventory inconsistencies are happening to Windows systems regardless of who is in charge of patching them.

Q: Why do you need a patch inventory to deliver updates? Why can’t you just patch our systems?

A: Historically and functionally, the Windows OS and Microsoft in general have always been the experts on the health of the OS. The Windows Update Agent (WUA) communicates with Microsoft to see if any available updates have been superseded, altered, or removed from distribution due to functionality or other issues. Our ManageSuite agents take what the WUA provides and push updates based on that information to make sure we’re always taking the most appropriate action for each system.

Q: So, are we being patched at all?

A: Yes. This problem seems to affect only non-OS updates, like Office patches and Runtime Redistributable updates. Monthly OS security updates, Critical category updates, and monthly quality updates are still being applied normally, as are updates to 3rd-party, non-Microsoft products.

Q: How did you detect this problem?

A: Since this was a silent failure, it took client-by-client analysis to pickup on the symptoms of patch health inconsistencies. Normally, the Windows Update Agent is quite vocal whenever it encounters problems, throwing all sorts of codes and alerts when something goes wrong. We trap those so we can find and fix problems. But these incomplete inventory results claimed to be completely successful and returned no alerts or evidence of malfunction, even though the results were actually incomplete.

Q: Is this Microsoft’s fault?

A: Yes.

Q: What are you doing about it?

A: We brought the developers of our RMM software (part of ManageSuite) into the conversation and sent them everything we have on this problem.  Since they’re part of Microsoft’s development network, they have a larger and more specific list of people at Microsoft to poke and bug about this problem until it gets resolved. As of this writing, the investigation is still in progress.  Stay tuned to our blog for more information as it’s made available!

Q: What else should I know? Is there anything I can do?

A: Nothing, and not much. We’ll post updates here as we get them, so stay tuned for more information.

UPDATE 12/11/18:

RMM Support has confirmed there to be inventory issues with Microsoft’s update sites for at least (but not necessarily limited to) Office 2016 products. We’ve additionally confirmed problems with Office 2013 products; we’ve tallied and submitted additional data to support for analysis. We are continuing to see patch reports improve in inventory quality and delivery effectiveness. Stay tuned for more info!

UPDATE 1/3/19:

We continue to see normalization in inventory quality and delivery effectiveness.  Our related support ticket remains in a “Dev-pending fix” status, meaning new code for additional improvements relating to this behavior may be forthcoming. We will continue to monitor this ticket and post a final update on this issue once our ticket has been closed.