Q&A: How do good containers go bad?

Tim Mackey, technology evangelist at Black Duck Software talks security in containers

Recently Tim Mackey, technical evangelist for open source security company Black Duck Software, spoke at London’s DevSecCon about “When Good Containers Go Bad”. In the following lightly edited Q&A we pick his brains on the subject.


How are data centre threats evolving?

Data centre operators are facing dual challenges of infrastructure complexity and application velocity as they seek to adhere to global governance regulations such as GDPR. Today’s workloads are increasingly containerised, which means that new management and monitoring paradigms are required to remain compliant. One example of this complexity comes from requirements to patch applications. With bare metal and virtualised servers, we’ve evolved procedures where the operating system and application components within those servers are continuously updated as patches are released. Containerisation flips this paradigm where it’s considered poor practice to patch containerised applications. The preferred solution is to rebuild the container image from patched sources and then redeploy. This one change in procedure requires a reassessment of how applications are built, and importantly where trusted source files are located.

As AI and machine learning is gradually being used to improve data centre operations, is adversarial machine learning also becoming more prevalent?

There’s a lot of potential for bad actors to use AI and machine learning to mount attacks. Machine learning is great at evaluating large data sets and finding patterns. Open source projects are perfect data sets for ML to analyse and assess for potential attack vectors. As we see more AI employed in cybersecurity and data centre operations, it’s reasonable to expect that hackers will also implement this technology, whether that’s to launch phishing attacks or test scenarios that hackers can then use in a malware or DDoS attack.

Machine Learning in security can be a tricky game… Welcome to the world of adversarial machine learning

What are the main ways containers can be compromised?

This is fundamentally a question of trust. For example, most container images are created from some source or base image and then application specific components are added to form the containerised application. With no validation of the security state for a base image, hackers can place modified components in base images which are then widely distributed, putting an entire container ecosystem at risk. The size of the developer community embracing containers continues to grow, offering an expanding target community for hackers.

Looking at this from a non-malicious intent perspective, base images may bundle outdated and insecure components, especially when the underlying operating system is not the most current version.

If one container is compromised, how many other containers within the data centre are at risk?

If we assume an attacker was able to compromise a container, they’re on the other side of the various perimeters, and can attack from the inside. To determine how many other containers are at risk within the data centre, you really need to know how the data centre is set up and how well you’ve prepared for attack. Minimally, if one container is compromised, it likely isn’t the only copy of the container image in use. It also is likely the given attack vector may also be present in other container images. From a defender’s perspective, the key takeaway is that when perimeter defences are removed from the equation, the potential for lateral movement within the data centre increases.

How would you define ‘trust’ in containers?

You need to both “trust” and “verify” container images. It’s essential to establish the bona fides of the publisher of the container image you wish to use, and verify that the contents of that image won’t introduce serious and exploitable security vulnerabilities into your environment. Ideally, you trust the running containers in your cluster to perform the tasks you expect of them, without exposing your organisation to risk. This process starts with establishing trust protocols for all images entering your organisation. Those trust protocols should include an understanding of the image composition, security context required to execute the image, and any services exposed from or consumed by the image.  

What can be done to proactively mitigate risk?

First you need to make sure that all containerised applications were subjected to static code analysis and pen-tested; that you’re able to determine the provenance of the container image through signatures and from trusted repositories; and that appropriate perimeter defences are in place and authorisation controls are gating deployment changes.

It’s also essential to make life harder for attackers by enabling the SELinux or AppArmor security modules for the distro you’re using as a base and have appropriate seccomp profiles in place.

Confused by containers? Check out: Containers: Everything you need to know

Lastly, from a security perspective, you want to minimise the attack surface of the container host. This is readily performed by using one of the available minimal Linux host systems such as Red Hat Enterprise Linux Atomic Host, CoreOS Container Linux or RancherOS

Is there anything else you’d like to share?

Container technologies are becoming the norm for application deployment in modern data centres. As with any technology, there will be paradigm shifts in management protocols which users should embrace. Containers have attributes such as immutable images which facilitate some security tasks such as patching and continuous monitoring. True containerised applications are designed to be tolerant of failure and maintain minimal state. When an application is designed to be stateless, its lifespan can be shortened which in turn condenses the window of attack a given container may experience. Containers also support constrained resource management which when combined with a shortened lifespan can help reduce the potential for resource exhaustion impacting service delivery.