Installing Elasticsearch on Kubernetes Using Operator and setting it for Kubernetes logging

Wednesday, Aug 7, 2019 by Prabhat Sharma kubernetes, logging, elasticsearch, fluentd, fluent-bit, kibana, helm

In an earlier blog post I provided the steps to install elastisearch using helm and setting it up for logging using fluent-bit. With the introduction of elasticsearch operator the experience of managing the elasticsearch cluster in kubernetes has improved greatly. We will cover the same goal of setting up elastisearch and configuring it for logging as the earlier blog, with the same ease but much better experience. Using operator allows you benefits in the area of security, upgrades and scalability.

Continue Reading

Installing Airflow on Kubernetes Using Operator

Sunday, Jul 28, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Operator - “A Kubernetes Operator is an abstraction for deploying non-trivial applications on Kubernetes. It wraps the logic for deploying and operating an application using Kubernetes constructs.” –Richard Laub, staff cloud engineer at Nebulaworks Note: I will be using an EKS cluster on AWS. You could use the same steps on other cloud providers too. Well created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes.

Continue Reading

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading

How to connect to Kubernetes cluster resources(services, pods) using OpenVPN for local development

Friday, Oct 19, 2018 by Prabhat Sharma kubernetes, helm, openvpn, local development, development workflow, developer productivity

I wanted to do development locally on my laptop and I had some services that were running in my kubernetes cluster. I was trying to figure out the easiest way to access those resources without exposing them via a loadbalancer or nodeport for the specific service. This is when I thought if I could vpn into the cluster then it would allow me to access cluster resources from my laptop.

Continue Reading

Helm tutorial - The package manager for kubernetes - Part 1

Saturday, Oct 13, 2018 by Prabhat Sharma kubernetes, helm, package manager

I have been asked by various people on what is the best way to install and manage applications on kubernetes. In fact a lot of people want to install various readily available solutions like mysql, mariadb, mongodb, nginx etc. For stateless applications its generally straightforward and nobody is bothered much, however when it comes to stateful applications one starts to think how they will manage to install it. Does one have to go through the rigmarole of installing the application just like installing it on instances?

Continue Reading