Blog

post-image

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading
post-image

DBeaver - SQL GUI Client for Amazon Redshift

Sunday, Dec 30, 2018 by Prabhat Sharma redshift, data warehousing, aws, amazon, gui

I use redshift occasionally for playing with some data. Its an amazing service(a datawarehousing solution) from AWS that you can use for analyzing large amounts of data. Every time I need to use redshift (every couple of months) I begin my search for a decent client that I can use. A google search does not result in something concrete right off with full of discussions on quora and other places.

Continue Reading
post-image

eksuser - Amazon EKS user management tool

Tuesday, Dec 4, 2018 by Prabhat Sharma eks, kubernetes, aws, authentication, iam, eksuser

Amazon EKS user management You have setup the EKS cluster and are able to use it. Now you want your teammates to access to the cluster too, so that they can build and run the applications. User management for EKS is done via the aws-auth configmap in kube-system namespace. You can learn how to add users manually by modifying the aws-auth configmap using the official documentation which shows how to add users to EKS by editing the aws-auth configmap.

Continue Reading
post-image

Logging in Kubernetes using Elasticsearch: The easy way

Monday, Nov 26, 2018 by Prabhat Sharma k8s, kubernetes, logging, elasticsearch, fluentd, fluent-bit

In my conversations with various development teams I regularly come across this common question of how to do logging on kubernetes. While there are existing solutions like EFK (elasticsearch, fluentd, kibana) stack, it takes good amount of effort for setting these up and making them work. I was wondering if I can provide some easy steps to people who just want to get started with logging in not so cumbersome and tedious way.

Continue Reading
post-image

Amazon EKS IAM authentication: How to add an IAM user

Friday, Oct 26, 2018 by Prabhat Sharma blog, eks, kubernetes, aws, authentication, iam

Note: This blog provides a deep dive on EKS authentication. If your goal is to be able to just add/delete eks users then follow this eksuser-amazon-eks-user-management-tool When you create an EKS cluster it uses credentials of the user creating the cluster to set things up and assigns the user cluster-admin rights on the cluster through kubernetes RBAC. EKS uses IAM for authentication. It uses aws-iam-authenticator for authentication using webhook token authentication of kubernetes.

Continue Reading
post-image

Reserving Compute Resources for System Daemons in Kubernetes using node-allocatable

Friday, Oct 19, 2018 by Prabhat Sharma k8s, kubernetes, cgroup, node-allocatable, qos

I was discussing resource management on k8s worker nodes with someone and they mentioned that they had faced a situation earlier where scheduler will schedule a lot of pods on the worker node which would consume all of the node’s CPU and memory leaving other system processes starved of resources, that ultimately leads the node to crash. Kubernetes provides a feature to solve this problem called Node Allocatable that allows you to reserve resources for system daemons so that they are not starved of resources and continue to function well.

Continue Reading
post-image

How to connect to Kubernetes cluster resources(services, pods) using OpenVPN for local development

Friday, Oct 19, 2018 by Prabhat Sharma kubernetes, helm, openvpn, local development, development workflow, developer productivity

I wanted to do development locally on my laptop and I had some services that were running in my kubernetes cluster. I was trying to figure out the easiest way to access those resources without exposing them via a loadbalancer or nodeport for the specific service. This is when I thought if I could vpn into the cluster then it would allow me to access cluster resources from my laptop.

Continue Reading
post-image

Helm tutorial - The package manager for kubernetes - Part 1

Saturday, Oct 13, 2018 by Prabhat Sharma kubernetes, helm, package manager

I have been asked by various people on what is the best way to install and manage applications on kubernetes. In fact a lot of people want to install various readily available solutions like mysql, mariadb, mongodb, nginx etc. For stateless applications its generally straightforward and nobody is bothered much, however when it comes to stateful applications one starts to think how they will manage to install it. Does one have to go through the rigmarole of installing the application just like installing it on instances?

Continue Reading
post-image

Best way to build website and blog in 2018

Tuesday, Oct 9, 2018 by Prabhat Sharma website, blog, hosting, wordpress

Beware!!! opinions ahead!!! I am asked by multiple people on what is the best way to build a small website and maybe build a blog as well. For long time wordpress has been the choice of application with other CMSs like drupal, joomla, etc as contenders. On the blogging front the contenders have been like ghost. Also there have been online applications like medium and tumblr that are doing the job pretty well.

Continue Reading
post-image

How I passed Certified Kubernetes Administrator exam on first attempt

Wednesday, Oct 3, 2018 by Prabhat Sharma kubernetes, cka, certification, exam

I have been working for past couple of months on kubernetes and advising people on how to implement best practices using kubernetes for building applications. I wanted to test my knowledge and figured I should take the CKA exam to validate my knowledge of things. I began preparing about one and a half months ago for the certification and found its preparation to be as hard as AWS SA Pro exam which is considered to be one of the difficult certifications.

Continue Reading