Build Docker images without Docker - using Kaniko, Jenkins and Kubernetes

Thursday, Aug 8, 2019 by Prabhat Sharma kubernetes, containers, docker, kaniko, jenkins, CICD

Note: I will be using an Amazon EKS cluster on AWS and using Amazon ECR for storing images. There will be minor differences for handling other repositories. Jenkins is a hugely popular build tool that has been around for ages and used by many people. With huge shift to Kubernetes as a platform you would naturally want to run jenkins on Kubernetes. While running Jenkins in itself on Kubernetes is not a challenge, it is a challenge when you want to build a container image using jenkins that itself runs in a container in the Kubernetes cluster.

Continue Reading

Installing Jenkins on Kubernetes

Thursday, Aug 8, 2019 by Prabhat Sharma kubernetes, containers, docker, jenkins, CICD

In this post I will cover the steps to install Jenkins in a Kubernetes cluster. While Jenkins can be installed on standard VMs, if you are running a Kubernetes cluster then its much easier to install and run Jenkins in it. We will be using Helm to install Jenkins. We will follow the below steps to get up and running: Get the Jenkins values file for configuring our Jenkins installation Install Jenkins Access Jenkins Let’s rock ‘n’ roll:

Continue Reading

Installing Elasticsearch on Kubernetes Using Operator and setting it for Kubernetes logging

Wednesday, Aug 7, 2019 by Prabhat Sharma kubernetes, logging, elasticsearch, fluentd, fluent-bit, kibana, helm

In an earlier blog post I provided the steps to install elastisearch using helm and setting it up for logging using fluent-bit. With the introduction of elasticsearch operator the experience of managing the elasticsearch cluster in kubernetes has improved greatly. We will cover the same goal of setting up elastisearch and configuring it for logging as the earlier blog, with the same ease but much better experience. Using operator allows you benefits in the area of security, upgrades and scalability.

Continue Reading

Using encrypted EBS Volumes with Kubernetes on AWS

Sunday, Aug 4, 2019 by Prabhat Sharma k8s, kubernetes, eks, aws, amazon

Lot of people run Kubernetes on AWS and need to use encrypted EBS volumes for security and compliace. I will lay down the steps below in order to use it. Create a storage class Create a PersistentVolume (or dynamically provisoned PersistentVolumeClaim) using the storage class Create a pod to use the PersistentVolumeClaim 1 2 3 4 5 6 7 8 9 10 11 12 13 14 { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "kms:GenerateDataKeyWithoutPlaintext", "kms:CreateGrant" ], "Resource": "*" } ] } -- 1.

Continue Reading

Installing Airflow on Kubernetes Using Operator

Sunday, Jul 28, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Operator - “A Kubernetes Operator is an abstraction for deploying non-trivial applications on Kubernetes. It wraps the logic for deploying and operating an application using Kubernetes constructs.” –Richard Laub, staff cloud engineer at Nebulaworks Note: I will be using an EKS cluster on AWS. You could use the same steps on other cloud providers too. Well created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes.

Continue Reading

RED Monitoring With Istio and Prometheus

Sunday, Jul 21, 2019 by Prabhat Sharma k8s, kubernetes, monitoring, Prometheus, Istio

In this blog post we will discuss RED method that I really like in terms of simplicity by which it explains the state of an application. We will also discuss how to cpature RED metrics for your application running in kubernetes. Primarily we need to capture 3 things (Request) Rate - the number of requests, per second, you services are serving. (Request) Errors - the number of failed requests per second.

Continue Reading

Kubernetes Slides

Sunday, Jul 14, 2019 by Prabhat Sharma k8s, kubernetes, stateful

Created some slides for running stateful applications on kubernetes. Why containers Basics of Kubernetes- Pod, Replicaset, Deployment, Service Stateful Sets Storage High availability Best Practices

Continue Reading

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading

eksuser - Amazon EKS user management tool

Tuesday, Dec 4, 2018 by Prabhat Sharma eks, kubernetes, aws, authentication, iam, eksuser

Amazon EKS user management You have setup the EKS cluster and are able to use it. Now you want your teammates to access to the cluster too, so that they can build and run the applications. User management for EKS is done via the aws-auth configmap in kube-system namespace. You can learn how to add users manually by modifying the aws-auth configmap using the official documentation which shows how to add users to EKS by editing the aws-auth configmap.

Continue Reading

Logging in Kubernetes using Elasticsearch: The easy way

Monday, Nov 26, 2018 by Prabhat Sharma k8s, kubernetes, logging, elasticsearch, fluentd, fluent-bit

Note: This blog is deprecated. Please follow the new blog that uses operators In my conversations with various development teams I regularly come across this common question of how to do logging on kubernetes. While there are existing solutions like EFK (elasticsearch, fluentd, kibana) stack, it takes good amount of effort for setting these up and making them work. I was wondering if I can provide some easy steps to people who just want to get started with logging in not so cumbersome and tedious way.

Continue Reading

Amazon EKS IAM authentication: How to add an IAM user

Friday, Oct 26, 2018 by Prabhat Sharma eks, kubernetes, aws, authentication, iam

Note: This blog provides a deep dive on EKS authentication. If your goal is to be able to just add/delete eks users then follow this eksuser-amazon-eks-user-management-tool When you create an EKS cluster it uses credentials of the user creating the cluster to set things up and assigns the user cluster-admin rights on the cluster through kubernetes RBAC. EKS uses IAM for authentication. It uses aws-iam-authenticator for authentication using webhook token authentication of kubernetes.

Continue Reading

Reserving Compute Resources for System Daemons in Kubernetes using node-allocatable

Friday, Oct 19, 2018 by Prabhat Sharma k8s, kubernetes, cgroup, node-allocatable, qos

I was discussing resource management on k8s worker nodes with someone and they mentioned that they had faced a situation earlier where scheduler will schedule a lot of pods on the worker node which would consume all of the node’s CPU and memory leaving other system processes starved of resources, that ultimately leads the node to crash. Kubernetes provides a feature to solve this problem called Node Allocatable that allows you to reserve resources for system daemons so that they are not starved of resources and continue to function well.

Continue Reading

How to connect to Kubernetes cluster resources(services, pods) using OpenVPN for local development

Friday, Oct 19, 2018 by Prabhat Sharma kubernetes, helm, openvpn, local development, development workflow, developer productivity

I wanted to do development locally on my laptop and I had some services that were running in my kubernetes cluster. I was trying to figure out the easiest way to access those resources without exposing them via a loadbalancer or nodeport for the specific service. This is when I thought if I could vpn into the cluster then it would allow me to access cluster resources from my laptop.

Continue Reading

Helm tutorial - The package manager for kubernetes - Part 1

Saturday, Oct 13, 2018 by Prabhat Sharma kubernetes, helm, package manager

I have been asked by various people on what is the best way to install and manage applications on kubernetes. In fact a lot of people want to install various readily available solutions like mysql, mariadb, mongodb, nginx etc. For stateless applications its generally straightforward and nobody is bothered much, however when it comes to stateful applications one starts to think how they will manage to install it. Does one have to go through the rigmarole of installing the application just like installing it on instances?

Continue Reading

How I passed Certified Kubernetes Administrator exam on first attempt

Wednesday, Oct 3, 2018 by Prabhat Sharma kubernetes, cka, certification, exam

I have been working for past couple of months on kubernetes and advising people on how to implement best practices using kubernetes for building applications. I wanted to test my knowledge and figured I should take the CKA exam to validate my knowledge of things. I began preparing about one and a half months ago for the certification and found its preparation to be as hard as AWS SA Pro exam which is considered to be one of the difficult certifications.

Continue Reading