Container Security: Scanning, Signing & Runtime Protection

Containerization has revolutionized software deployment, but it also introduces new security challenges. Ignoring these challenges can expose your applications and infrastructure to serious risks. As the MisuJob engineering team, we’ve learned a lot about securing our containerized infrastructure, which processes 1M+ job listings, and we want to share our experience.

Container Security: Scanning, Signing & Runtime Protection

Securing containers is a multi-layered approach, extending from the initial image build to runtime execution. It’s not enough to just build the image; you need to ensure its integrity, continuously monitor its behavior, and respond to threats in real-time. We’ll cover three critical aspects of container security: image scanning, image signing, and runtime protection. Each layer provides a different set of defenses against various threats. Neglecting any of these layers creates vulnerabilities that can be exploited.

Image Scanning: Identifying Vulnerabilities Early

Image scanning is the process of analyzing container images for known vulnerabilities, misconfigurations, and malware. This is a critical step in the software development lifecycle because it helps you identify and remediate security issues before they reach production. Think of it as a spell check for your container images.

Why Image Scanning is Crucial

Without image scanning, you’re essentially deploying code without knowing if it contains critical vulnerabilities. These vulnerabilities can be exploited by attackers to gain access to your system, steal sensitive data, or disrupt your services. Image scanning helps prevent this by:

• Identifying known vulnerabilities: Scans identify Common Vulnerabilities and Exposures (CVEs) in your container images.
• Enforcing security policies: You can define policies to ensure that only images that meet specific security standards are deployed.
• Reducing the attack surface: By addressing vulnerabilities early, you reduce the potential attack surface of your applications.
• Improving compliance: Image scanning helps you comply with industry regulations and security standards.

Implementing Image Scanning in Your CI/CD Pipeline

Integrating image scanning into your CI/CD pipeline allows you to automatically scan container images as they are built. This ensures that vulnerabilities are detected and addressed early in the development process. Here’s an example using grype, a popular open-source vulnerability scanner:

grype alpine:latest

This command will scan the alpine:latest image for vulnerabilities. You can then use the results to fail the build if any critical vulnerabilities are found.

We use a policy-as-code approach using OPA (Open Policy Agent) to define the criteria for passing/failing builds based on scan results. For example, the following OPA policy prevents images with vulnerabilities rated “Critical” from being deployed:

package misujob.security

deny[msg] {
  input.results[_].vulnerability.severity == "Critical"
  msg := "Critical vulnerability found in image. Deployment blocked."
}

This policy integrates directly into our CI/CD pipeline, preventing vulnerable images from ever reaching production.

Example: Addressing a Critical Vulnerability

Let’s say your image scan reveals a critical vulnerability in a commonly used library, such as openssl. The scanner will provide details about the vulnerability, including the CVE ID and a description of the issue. You can then take steps to address the vulnerability, such as:

1. Updating the library: Upgrade to a version of the library that contains a fix for the vulnerability.
2. Applying a patch: If an upgrade is not immediately available, you may be able to apply a patch to address the vulnerability.
3. Removing the library: If the library is not essential, you may be able to remove it from your image.

Image Signing: Ensuring Image Integrity

Image signing is the process of digitally signing container images to ensure their integrity and authenticity. This helps prevent attackers from tampering with your images or deploying malicious images to your environment.

Why Image Signing is Important

Without image signing, it’s difficult to verify that a container image hasn’t been tampered with. Attackers could potentially inject malicious code into your images, leading to serious security breaches. Image signing helps prevent this by:

• Ensuring image integrity: Verifies that the image hasn’t been modified since it was signed.
• Authenticating the image source: Confirms that the image was created by a trusted source.
• Preventing image tampering: Prevents attackers from injecting malicious code into your images.
• Enhancing supply chain security: Provides assurance that the image hasn’t been compromised during the build or deployment process.

Implementing Image Signing with Notary and Docker Content Trust

Docker Content Trust (DCT) uses Notary to sign and verify container images. To enable DCT, you need to set the DOCKER_CONTENT_TRUST environment variable to 1.

export DOCKER_CONTENT_TRUST=1

Then, you can sign your images using the docker tag and docker push commands. The first time you push a tagged image, Docker will prompt you to create a root key and a repository key. Keep these keys safe, as they are required to manage the trust for your image.

docker tag my-image:latest my-registry/my-image:latest
docker push my-registry/my-image:latest

When pulling an image with DCT enabled, Docker will verify the signature and refuse to pull the image if it’s invalid.

Example: Verifying a Signed Image

To verify that an image is signed, you can use the docker inspect command. The output will include a ContentTrust section that contains information about the image’s signature. If the image is not signed, the ContentTrust section will be empty.

Runtime Protection: Detecting and Responding to Threats

Runtime protection involves monitoring container behavior and detecting and responding to suspicious activities. This is a crucial layer of security because it provides real-time protection against attacks that may bypass other security measures.

Why Runtime Protection is Essential

Even with image scanning and signing, your containers can still be vulnerable to attacks. For example, an attacker could exploit a zero-day vulnerability in a running application. Runtime protection helps prevent this by:

• Detecting anomalous behavior: Identifies unusual activity, such as unexpected network connections or file system modifications.
• Preventing malicious activity: Blocks or mitigates attacks in real-time.
• Providing visibility into container behavior: Gives you insights into how your containers are running and what they are doing.
• Improving incident response: Helps you quickly identify and respond to security incidents.

Implementing Runtime Protection with Falco

Falco is a popular open-source runtime security tool for Kubernetes. It allows you to define rules that detect suspicious behavior in your containers. When a rule is triggered, Falco generates an alert that you can use to investigate the issue.

Here’s an example Falco rule that detects when a shell is spawned inside a container:

- rule: Shell Spawned in Container
  desc: Detects when a shell is spawned inside a container.
  condition: >
    container and
    proc.name = bash or proc.name = sh or proc.name = tcsh or proc.name = zsh
  output: >
    Shell spawned in container (user=%user.name user_loginuid=%user.loginuid
    command=%proc.cmdline container_id=%container.id
    container_name=%container.name image=%container.image.repository)
  priority: WARNING

This rule uses Falco’s powerful filtering capabilities to detect when a shell process is created inside a container. When the rule is triggered, Falco will generate an alert that includes information about the user who spawned the shell, the command that was executed, and the container in which the shell was spawned.

Example: Responding to a Runtime Threat

Let’s say Falco detects that a shell has been spawned inside a container and that the shell is being used to execute malicious commands. You can then take steps to respond to the threat, such as:

1. Isolating the container: Prevent the container from communicating with other containers or services.
2. Killing the container: Terminate the container to prevent further damage.
3. Investigating the incident: Determine how the attacker gained access to the container and take steps to prevent future attacks.

Practical Examples: Securing a MisuJob Microservice

At MisuJob, we apply these principles rigorously. Let’s consider a simplified example of how we secure a microservice responsible for processing job postings.

1. Image Scanning: Before deploying any new version of the microservice, we run a grype scan as part of our CI/CD pipeline. This scan identifies any known vulnerabilities in the base image and the application dependencies. If critical vulnerabilities are found, the build is automatically failed, and the developers are notified to address the issues.
2. Image Signing: Once the image passes the scanning stage, it is signed using Docker Content Trust (DCT) with Notary. This ensures that only authorized images can be deployed to our production environment. Our Kubernetes cluster is configured to only pull signed images, preventing unauthorized images from being deployed.
3. Runtime Protection: We use Falco to monitor the microservice at runtime. We have defined rules that detect suspicious behavior, such as unexpected network connections, file system modifications, or attempts to execute malicious commands. When a rule is triggered, Falco sends an alert to our security team, who investigates the incident and takes appropriate action.

For example, we have a Falco rule that detects when a process attempts to write to a protected directory. This rule helps us prevent attackers from modifying critical system files.

- rule: Write to Protected Directory
  desc: Detects writes to protected directories
  condition: >
    container and
    evt.type = "syscall" and evt.dir </> "SA" and
    evt.category = "file" and
    (evt.name = "open" or evt.name = "openat") and
    args.flags contains "O_WRONLY" and
    args.flags contains "O_CREAT" and
    not proc.name in (allowed_processes) and
    dir in (protected_directories)
  output: >
    File write to protected directory
    (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline
    file=%file.path container_id=%container.id
    container_name=%container.name image=%container.image.repository)
  priority: WARNING

Where allowed_processes and protected_directories are lists defined elsewhere in our Falco configuration.

Salary Implications: Security Engineers in Europe

A strong understanding of container security principles is a valuable asset for security engineers. The demand for skilled security professionals is high across Europe, and this is reflected in the salaries offered. Here’s a comparison of typical salary ranges for Security Engineers with expertise in container security in several European countries:

These figures represent the typical salary ranges for experienced Security Engineers with a strong understanding of container security and related technologies. MisuJob aggregates from multiple sources and processes 1M+ job listings, providing valuable insights into salary trends across Europe.

Key Takeaways

Securing containers requires a comprehensive approach that encompasses image scanning, image signing, and runtime protection. Each layer of security provides a different set of defenses against various threats, and neglecting any of these layers can leave your applications and infrastructure vulnerable.

• Image scanning helps you identify and remediate vulnerabilities before they reach production.
• Image signing ensures the integrity and authenticity of your container images.
• Runtime protection provides real-time protection against attacks that may bypass other security measures.

By implementing these security measures, you can significantly reduce the risk of container-related security incidents and protect your applications and data. Investing in container security is not just a technical necessity, but also a strategic imperative for any organization that relies on containerization. As we continue to expand across Europe, we’re constantly refining our security practices, leveraging AI-powered job matching to connect with the best security talent.