Lately I have been using VS code dev containers quite a lot. I also use AWS pretty extensively. So when I started using dev containers and found that calls to AWS services are failing from my dev container, I looked for a solution. Here is how I solved it. On you local machine when you try to call aws services e.g. aws s3 ls - It fetches your credentials from ~/.
Continue ReadingNote: I will be using an Amazon EKS cluster on AWS and using Amazon ECR for storing images. There will be minor differences for handling other repositories. Jenkins is a hugely popular build tool that has been around for ages and used by many people. With huge shift to Kubernetes as a platform you would naturally want to run jenkins on Kubernetes. While running Jenkins in itself on Kubernetes is not a challenge, it is a challenge when you want to build a container image using jenkins that itself runs in a container in the Kubernetes cluster.
Continue ReadingIn this post I will cover the steps to install Jenkins in a Kubernetes cluster. While Jenkins can be installed on standard VMs, if you are running a Kubernetes cluster then its much easier to install and run Jenkins in it. We will be using Helm to install Jenkins. We will follow the below steps to get up and running: Get the Jenkins values file for configuring our Jenkins installation Install Jenkins Access Jenkins Let’s rock ’n’ roll:
Continue ReadingWell created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes. We had earlier seen how to install airflow on kubernetes using helm charts. While helm charts help you get started fast, they may not be suitable for day 2 operatios like: Upgrades Backup & restore Auto recovery Automatic/On-demand scalability Configuration management Deep insights Let’s find how to install airflow on kubernetes using airflow operator.
Continue ReadingData engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.
Continue Reading