Build Docker images without Docker - using Kaniko, Jenkins and Kubernetes

Thursday, Aug 8, 2019| Tags: kubernetes, containers, docker, kaniko, jenkins, CICD

Note: I will be using an Amazon EKS cluster on AWS and using Amazon ECR for storing images. There will be minor differences for handling other repositories.

Jenkins is a hugely popular build tool that has been around for ages and used by many people. With huge shift to Kubernetes as a platform you would naturally want to run jenkins on Kubernetes. While running Jenkins in itself on Kubernetes is not a challenge, it is a challenge when you want to build a container image using jenkins that itself runs in a container in the Kubernetes cluster.

The process of running Docker-in-Docker (DIND), and setting it up is not very interesting not to mention the hacking that you need to do to achieve it.

An alternative would be Kaniko which provides a clean approach to building and pushing container images to your repository.

In this post we will build a Jenkins pipeline that will be responsible for pulling code, building image and pushing image to Amazon ECR.

If you don’t already have Jenkins installed then follow the steps in this post

We will follow the below steps:

  1. Create a configmap for docker configuration that will use ECR credential helper
  2. Build a Jenkins pipeline

Step 1: Create a configmap for docker configuration that will use ECR credential helper

Amazon ECR uses AWS IAM authentication to get docker credentials for pushing the images. ECR crdenetial helper makes getting the credentials for pushing images easier. Setting up ECR crdenetial helper for Docker/Kaniko needs a configuration file. Let’s go ahead and create a configuration file.

Create a configmap docker-config.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-config
data:
  config.json: |-
    {
      "credHelpers": {
        "123456789498.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
      }
    }


Replace 123456789498 with your AWS account number.

Run the below command to create the configmap. You must install the configmap in the same namespace where jenkins is installed.

kubectl -n jenkins apply -f docker-config.yaml

Step 2: Create a Jenkins pipeline to build and push the container image

Once you are logged in to Jenkins it’s time to create a new Jenkins pipeline. Follow the steps:

1. Create a New Item

2. Create a new Pipeline

3. Place the pipeline script in the job

Now place the below script in the pipeline script section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
pipeline {
  agent {
    kubernetes {
      //cloud 'kubernetes'
      yaml """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug-539ddefcae3fd6b411a95982a830d987f4214251
    imagePullPolicy: Always
    command:
    - cat
    tty: true
    volumeMounts:
      - name: docker-config
        mountPath: /kaniko/.docker
  volumes:
    - name: docker-config
      configMap:
        name: docker-config
"""
    }
  }
  stages {
    stage('Build with Kaniko') {
      steps {
        git 'https://github.com/prabhatsharma/sample-microservice'
        container(name: 'kaniko') {
            sh '''
            /kaniko/executor --dockerfile `pwd`/Dockerfile --context `pwd` --destination=123456789498.dkr.ecr.us-west-2.amazonaws.com/sample-microservice:latest --destination=123456789498.dkr.ecr.us-west-2.amazonaws.com/sample-microservice:v$BUILD_NUMBER
            '''
        }
      }
    }
  }
}


We are using a sample microservice for which we will build an image and push it to ECR. You must also have the 123456789498.dkr.ecr.us-west-2.amazonaws.com/sample-microservice ECR repository created before running this pipeline.

We are using an older kaniko docker image (tag debug-539ddefcae3fd6b411a95982a830d987f4214251) as the latest kaniko docker image is not compatible with Jenkins Kubernetes plugin.


Now save the pipeline. You are all set up. Next you can click Build Now link to start the build.

Once the build completes your screen should look like below

and you should have a docker image in your repository:

You are all set !!!

Now while we are here, some may argue that kaniko has the drawback that it runs as root. That is indeed true. Also the default docker build is not compatible with kubernetes plugin (It is not really kaniko’s fault in my opinion but jenkins’ kubernetes plgin’s fault) in latest versions, which means you are stuck with using an old version.

In order to mitigate these issues you might want to use img.

PS: You may also be interested in building docker images in kubernetes using img



Comments