Using encrypted EBS Volumes with Kubernetes on AWS

Sunday, Aug 4, 2019 by Prabhat Sharma k8s, kubernetes, eks, aws, amazon

Lot of people run Kubernetes on AWS and need to use encrypted EBS volumes for security and compliace. I will lay down the steps below in order to use it. Create a storage class Create a PersistentVolume (or dynamically provisoned PersistentVolumeClaim) using the storage class Create a pod to use the PersistentVolumeClaim 1 2 3 4 5 6 7 8 9 10 11 12 13 14 { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "kms:GenerateDataKeyWithoutPlaintext", "kms:CreateGrant" ], "Resource": "*" } ] } -- 1.

Continue Reading

Installing Airflow on Kubernetes Using Operator

Sunday, Jul 28, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Operator - “A Kubernetes Operator is an abstraction for deploying non-trivial applications on Kubernetes. It wraps the logic for deploying and operating an application using Kubernetes constructs.” –Richard Laub, staff cloud engineer at Nebulaworks Note: I will be using an EKS cluster on AWS. You could use the same steps on other cloud providers too. Well created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes.

Continue Reading

RED Monitoring With Istio and Prometheus

Sunday, Jul 21, 2019 by Prabhat Sharma k8s, kubernetes, monitoring, Prometheus, Istio

In this blog post we will discuss RED method that I really like in terms of simplicity by which it explains the state of an application. We will also discuss how to cpature RED metrics for your application running in kubernetes. Primarily we need to capture 3 things (Request) Rate - the number of requests, per second, you services are serving. (Request) Errors - the number of failed requests per second.

Continue Reading

Kubernetes Slides

Sunday, Jul 14, 2019 by Prabhat Sharma k8s, kubernetes, stateful

Created some slides for running stateful applications on kubernetes. Why containers Basics of Kubernetes- Pod, Replicaset, Deployment, Service Stateful Sets Storage High availability Best Practices

Continue Reading

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading

Logging in Kubernetes using Elasticsearch: The easy way

Monday, Nov 26, 2018 by Prabhat Sharma k8s, kubernetes, logging, elasticsearch, fluentd, fluent-bit

Note: This blog is deprecated. Please follow the new blog that uses operators In my conversations with various development teams I regularly come across this common question of how to do logging on kubernetes. While there are existing solutions like EFK (elasticsearch, fluentd, kibana) stack, it takes good amount of effort for setting these up and making them work. I was wondering if I can provide some easy steps to people who just want to get started with logging in not so cumbersome and tedious way.

Continue Reading

Reserving Compute Resources for System Daemons in Kubernetes using node-allocatable

Friday, Oct 19, 2018 by Prabhat Sharma k8s, kubernetes, cgroup, node-allocatable, qos

I was discussing resource management on k8s worker nodes with someone and they mentioned that they had faced a situation earlier where scheduler will schedule a lot of pods on the worker node which would consume all of the node’s CPU and memory leaving other system processes starved of resources, that ultimately leads the node to crash. Kubernetes provides a feature to solve this problem called Node Allocatable that allows you to reserve resources for system daemons so that they are not starved of resources and continue to function well.

Continue Reading