Surprisingly easy: Anchore Engine for container vulnerability scanning in a Jenkins pipeline running on Kubernetes

Anchore Engine is a popular open source tool for container image inspection and vulnerability scanning. It is easily integrated in a Kubernetes environment as an admission controller or in a Jenkins build pipeline using a plugin. A while ago I took a look at Anchore Engine and created a small introductory presentation and Katacoda scenario for it. The Katacoda scenario allows you to try out Anchore Engine without having to setup your own container environment. In this blog I’ll go a step further and illustrate how you can incorporate an Anchore Engine container scan inside the Java build pipeline which I created here. Anchore Engine is deployed to Kubernetes, configured in Jenkins (which also runs on Kubernetes) and incorporated in a Jenkins Pipeline during a build process. Only if the container has been deemed secure by the configured Anchore Engine policy, is it allowed to be deployed to Kubernetes. I will also show how to update policies using the CLI.

Prerequisites

In order for the following steps to work, you need to have a Kubernetes cluster running and kubectl + helm configured to allow deployments to it. The Kubernetes cluster is expected to have internet access in order to download the container image from DockerHub and to update the Anchore Engine vulnerability lists. I’ve used the following Kubernetes environment using the following storage configuration and the following Jenkins installation for this.

Deploying Anchore Engine to Kubernetes

The following can be used to deploy Anchore Engine to Kubernetes. This is based on the following documentation. The Helm chart installs a PostgreSQL database, but you can also use one supplied externally. In production you should definitely use your own PostgreSQL installation which has been deployed in a high-available setup. Of course I would also change Welcome01 to a more secure password if you need it.

 cat << EOF > anchore_values.yaml  
 postgresql:  
  postgresPassword: Welcome01  
  persistence:  
   size: 10Gi  
   
 anchoreGlobal:  
  defaultAdminPassword: Welcome01  
  defaultAdminEmail: maarten.smeets@amis.nl  
 EOF  
   
 helm repo add anchore https://charts.anchore.io  
 helm repo update  
   
 kubectl create ns anchore  
 helm install anchore-release -n anchore -f anchore_values.yaml anchore/anchore-engine

In order to check the deployment, you are provided with several handy commands after the installation is finished such as a command to start a container which has the anchore-cli installed. You also get information on the URL at which you can access Anchore Engine from within the Kubernetes cluster. You need this information for the Jenkins configuration.

 kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=Welcome01 --env ANCHORE_CLI_URL=http://anchore-release-anchore-engine-api.anchore.svc.cluster.local:8228/v1/    
   
 [anchore@anchore-cli anchore-cli]$ anchore-cli system status  
 Service simplequeue (anchore-release-anchore-engine-simplequeue-75b49c55c5-bktbv, http://anchore-release-anchore-engine-simplequeue:8083): up  
 Service policy_engine (anchore-release-anchore-engine-policy-5576ff74b4-8rv5r, http://anchore-release-anchore-engine-policy:8087): up  
 Service apiext (anchore-release-anchore-engine-api-5b5ddb8cd6-k66hm, http://anchore-release-anchore-engine-api:8228): up  
 Service catalog (anchore-release-anchore-engine-catalog-5648d4df64-5rtq9, http://anchore-release-anchore-engine-catalog:8082): up  
 Service analyzer (anchore-release-anchore-engine-analyzer-5588cc6964-489h7, http://anchore-release-anchore-engine-analyzer:8084): up  
   
 Engine DB Version: 0.0.13  
 Engine Code Version: 0.8.1  

Configuring Jenkins

The Jenkins configuration is relatively straightforward. You need to install the Anchore Container Image Scanner plugin:

Next you need to configure the Anchore Engine location and credentials. The location is the URL as previously obtained from the output of the Helm chart installation and the credentials as supplied in the anchore_values.yaml file.

Configure and run your pipeline

I created a minimal Jenkins Java build pipeline a while back which I wanted to expand with Anchore Engine scanning. The pipeline pushes the following image to DockerHub (here) which is based on the base image openjdk:11-jdk-slim. Mind that I use latest here so a next run of the pipeline might create a container using a newer version of the base image.

I added the following to my pipeline:

   stage('Anchore analyse') {  
    steps {  
     writeFile file: 'anchore_images', text: 'docker.io/maartensmeets/spring-boot-demo'  
     anchore name: 'anchore_images'  
    }  
   }  

Next I ran it:

It can take a while before Anchore Engine has scanned the image. After it is completed, you can see the results in Jenkins.

Updating the policy using the CLI

I get a warning about a Dockerfile which I did not supply. This is correct since I created my container using Google Jib which does not require a Dockerfile or Docker daemon installation to build and push images. I don want to see this warning though because it is irrelevant to my build process. Based on the following, you can change the ‘gate action’ when this ‘vulnerability’ is found using the Anchore CLI.

First enter the Anchore CLI container

kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=Welcome01 --env ANCHORE_CLI_URL=http://anchore-release-anchore-engine-api.anchore.svc.cluster.local:8228/v1/    

Next determine and download the used policy

 [anchore@anchore-cli anchore-cli]$ anchore-cli policy list  
 Policy ID                  Active    Created           Updated             
 2c53a13c-1765-11e8-82ef-23527761d060    True     2020-10-04T11:30:19Z    2020-10-04T11:30:19Z      
 [anchore@anchore-cli anchore-cli]$ anchore-cli policy get 2c53a13c-1765-11e8-82ef-23527761d060 --detail > /tmp/policybundle.json  

When you edit the file with vi, you can find the dockerfile warning. You can also see at the bottom of the file a whitelists definition. In this whitelist we can add the following in the items tag to exclude this specific warning.

The gate and trigger_id can be determined from the report given by the Anchore plugin in Jenkins or by looking up the vulnerability in the json file you just downloaded.

Next update the policy and activate it:

 [anchore@anchore-cli anchore-cli]$ anchore-cli policy add /tmp/policybundle.json   
 Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060  
 Active: False  
 Source: local  
 Created: 2020-10-04T11:30:19Z  
 Updated: 2020-10-04T14:03:25Z  
   
 [anchore@anchore-cli anchore-cli]$ anchore-cli policy activate 2c53a13c-1765-11e8-82ef-23527761d060  
 Success: 2c53a13c-1765-11e8-82ef-23527761d060 activated

When you now execute the pipeline again, you will notice the warning is gone (listed as Go) and it is indicated as whitelisted.

Finally

Deploying Anchore Engine to your Kubernetes cluster is relatively easy. Using it in a Jenkins pipeline is also. By adding it to your build process, you can confirm images used do not have vulnerabilities you might want to avoid. Also using the Anchore CLI you can edit the policy which determines if the build succeeds or not. 

Anchore Engine also provides a notification system. If you scan a container and new vulnerabilities are discovered because vulnerability feeds get updated, you can be informed by webhook calls. This allows you to take action when needed or feed dashboards (like an ELK stack).

What this does not provide is in-debt scanning of images, just the libraries which are present in the container. It depends on a publicly available vulnerability database. Anchore has an Enterprise version available which adds additional vulnerability feeds (Snyk). It also provides an option to create a local proxy for the vulnerability feeds so you are not dependent on an internet connection anymore. You get reports on entire Docker registries and a GUI for configuring policies. The Enterprise version has advanced SSO/RBAC options and an easy way to integrate notifications with Slack, Jira, GitHub, etc. If you want to use Anchore Engine at scale, I would take a look at the Enterprise version.

If you want to scan for example Java library dependencies, you can use the OWASP dependency check (of which (amongst others) a Maven plugin and Gradle plugin are available). By using several tools to scan your application/container for vulnerabilities at different levels, you can decrease the likelihood your application or container is compromised. It helps greatly if these are implementing as automated steps in your deployment pipeline. In this blog post I tried to show that implementing Anchore Engine (in a basic setup) is a piece of cake and there is no good reason not to use it.