This is a contributed article by Tom Bridge, Principal Product Manager, JumpCloud.
Every month, sysadmins know they will have patches to deal with. Microsoft’s Patch Tuesday is a predictable event that sees the release of multiple updates to deal with various security and performance issues. Alongside Microsoft, other vendors release their updates on the same day. Apple releases updates for its operating systems on a regular cadence too.
This regular flow of updates will be supplemented by urgent patches for specific issues that need to be addressed for security reasons, either because there are exploits in the wild or they are simple to attack.
What are the problems around patching?
There are several reasons why patching is hard and getting harder. To some extent, it is a problem of volume. If we look back at previous years, more than 160,000 patches have been released since 1999. In 2017, the volume of security updates more than doubled compared to the previous year - from 6,454 to 14,714 - marking a huge increase in workload. That workload has not gone down - in 2021, there were more than 20,000 software vulnerabilities released.
This volume of updates is problematic in its own right, but the other problem is how quickly patches have to be deployed. For example, the UK Cyber Essentials program asks for critical and high severity patches to be deployed within 14 days of the patch being released. Similarly, the US Government’s Department of Homeland Security has implemented rules that critical patches are in place within 15 days. On average, it takes around 43 days to roll out patches according to the Ponemon Institute.
So sysadmins have more patches to evaluate, understand and deploy, and they have less time to carry this work out. However, the third facet for this problem is that patching can be tedious and time-consuming work.
It begins with decisions around whether to push that patch through to users immediately, or to carry out testing to check that the update is OK to release. This activity should prevent updates to operating systems or applications from breaking other services. Following that testing stage, sysadmins have to manage communication with users and track how those patches are deployed.
For macOS devices, users are wholly responsible for when those updates get implemented, so sysadmins need good data on how those updates are made. For Windows machines, sysadmins have more control on how and when those patches are implemented. Working with users is still an essential part of patching effectively, and this takes time.
What has to change around patching?
Patch management is one of those basic IT tasks that does not get the respect and support that it should. To make improvements around patching means looking at how you make best use of your resources, and in particular your sysadmins’ time.
For example, many organisations still have different processes and tools in place for each platform that they run. This makes it harder to see how well you are doing. Consolidating your approach, so you have all your platforms covered from one place, makes it easier to track your patching efforts and see where problems exist.
Alongside this, putting a smarter patching process in place can help. Taking a progressive delivery approach to rollouts can help. For example, you can carry out rapid patching to check that patches work and don’t break machines, then roll out to a small set of users that are based in specific departments and act as subject matter experts around business applications. These users can carry out that functional testing to check application compatibility.
Following testing with these groups, you can look at a wider rollout to the majority of employees. This will be where the most effort is needed, but you should already have good information from your previous testing efforts. This stage is more about good internal communication and having accurate data on patching status.
Alongside this main group, you should also bracket together all the other machines that you are responsible for, such as unattended machines used in shared environments, for kiosks or for point of sale machines. These assets may need more effort to support, and they may have stricter security and application compatibility requirements too. Treating them as a separate group can help you concentrate on user needs.
Lastly, looking at your application landscape can help with your patching over time. Older installed applications will tend to need more support compared to newer services that are often delivered through the browser, or that use more modern application design approaches. These applications can be set to update automatically, as they are less likely to break others. Over time, you can migrate to apps that need less support.
Changing the game around patching
If it was a game, patching would be one of those titles that was procedurally generated and that you cannot finish. For those of us that need a story to follow, or that like to be able to tick the box marked ‘done,’ this kind of title can be problematic.
Patching is the same. Instead of thinking of patches individually, we should measure success around the ongoing process being as effective as possible. At the same time, we can work on ways to take out the pain around patching individual problems and help sysadmins use their time and resources in the most effective way possible.
A good patching process can stop security vulnerabilities from being exploited, and it can reduce long term support costs by keeping things up to date and performing well.
Tom Bridge is the Principal Product Manager, Apple at JumpCloud as well as being the Producer of the Mac Admins Podcast. Bridge spends his days working with the Product teams, handling research and development.