airflow

Installing Airflow on Kubernetes Using Operator

Sunday, Jul 28, 2019 by Prabhat Sharma kubernetes, containers, docker, airflow, helm, data engineering

Operator - “A Kubernetes Operator is an abstraction for deploying non-trivial applications on Kubernetes. It wraps the logic for deploying and operating an application using Kubernetes constructs.” –Richard Laub, staff cloud engineer at Nebulaworks Note: I will be using an EKS cluster on AWS. You could use the same steps on other cloud providers too. Well created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes.

Continue Reading

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading