Blog

In search of a Search Engine, beyond Elasticsearch: Introducing Zinc

Saturday, Dec 11, 2021 by Prabhat Sharma Elasticsearch, OpenSearch, fulltext, full text, search engine, go, golang, Zinc, Zincsearch, Zinc search

Introduction A search engine is a software that can search for any text in the data that has been indexed for full text. While there are public search engines like Google and Bing, we are not talking about them in this blog, but of softwares that would allow you to search for text in your own data. Think of a this kind of search engine software as google or bing but just for your own data.

Continue Reading

Video Encoding using ffmpeg to reduce file size before uploading to youtube

Monday, Sep 6, 2021 by Prabhat Sharma video, encoding, youtube, ffmpeg

When uploading videos to youtube thay can get huge and uploads might take a while. FFMPEG can reduce size of the files pretty heavily. I had a screencast of 190 MB that ffmpeg converted to 17 MB, which saved me a ton of time during upload considering upload bandwidth that my ISP gives is far lower than download bandwidth. Below is the simple no-nosense command that you can use. ffmpeg -i RawInput.

Continue Reading

Argo Workflow Authentication using Cognito

Wednesday, Jul 21, 2021 by Prabhat Sharma argo, workflow, kubernetes

Argo workflow supports SSO using 3rd part identity providers. Amazon cognito is a very popular authentication provider that is almost free for most use cases that you can use for authenticating argo workflow. In order to setup conto authentication you will have to do the following: Create a cognito user pool Setup a kubernetes secret containg cognito details Configure argo workflow controller configmap. Here are the sample values for congito secret:

Continue Reading

VScode Dev Container AWS Credentials

Saturday, Jan 30, 2021 by Prabhat Sharma aws, vscode, development, containers, docker

Lately I have been using VS code dev containers quite a lot. I also use AWS pretty extensively. So when I started using dev containers and found that calls to AWS services are failing from my dev container, I looked for a solution. Here is how I solved it. On you local machine when you try to call aws services e.g. aws s3 ls - It fetches your credentials from ~/.

Continue Reading

Elasticsearch License Change and what it means for the community and you

Wednesday, Jan 20, 2021 by Prabhat Sharma search, elasticsearch, license, open source

Elastic changed the licensing terms for elasticsearch yesterday. It means that any new versions of elasticsearch will not be Apache 2.0 licensed, but SSPL licensed. What does it mean? From https://www.elastic.co/blog/license-change-clarification Our on-prem or Elastic Cloud customers will not be impacted. The vast majority of our users will not be impacted. The folks who take our products and sell them directly as a service will be impacted, such as the Amazon Elasticsearch Service.

Continue Reading

Max Pods in EKS per EC2 Instance

Tuesday, Dec 15, 2020 by Prabhat Sharma kubernetes, eks

Maximum pods in an EC2 instance in EKS is dependent on the ENIs the ec2 instance has and the number of secondary IP addresses each ENI can have. You can find the details of maximum ENIs per instance and IP addresses per ENI at - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI As for the details of max pods per ec2 instance, it can be found at: https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt

Continue Reading

Embedding Quicksight Dashboards Using Go

Thursday, Sep 24, 2020 by Prabhat Sharma quicksight, analytics, business intelligence, amazon, reporting, aws

Amazon Quicksight is an amazingly simple Business Intelligence tool that is fully managed and allows one to get started with dashboarding and reporting in a matter of minutes. Its dashboard embedding feature is allows you to embed the dashboards natively in your web applications. Essentially you have 3 steps to perform Assume an IAM role (Let’s call it quicksight-embedded) that has 2 required permissions: quicksight:RegisterUser quicksight:GetDashboardEmbedUrl For example you have your application running on an EC2 instance/Lambda/ECS Task that has the role myAppIAMRole .

Continue Reading

CKA Practice Test 2

Tuesday, Aug 18, 2020 by Prabhat Sharma kubernetes, cka, certification, exam, practice test

2nd practice test for CKA exam preparation. First is at CKA exam practice test 1 Display all the pods sorted by start time Create a pod that uses secrets Create a secret Pull secrets from environment variables Pull secrets from a volume Dump the secrets out via kubectl to show it worked Create a job that runs every 3 minutes and prints out the current time. Create a job that runs 20 times, 5 containers at a time, and prints “Hello parallel world” Create a horizontal autoscaling group that starts with 2 pods and scales when CPU usage is over 50%.

Continue Reading

CKA Practice Test 1

Tuesday, Aug 18, 2020 by Prabhat Sharma kubernetes, cka, certification, exam, practice test

More than 2 years ago when I passed CKA, there were fewer resources available for me to study. I had to create a lot of resources myself during my cert preparation. During many of the certification exams that I have taken over a period I have realized that having a set of of test questions is incredibly helpful. I am posting the below practice exam questions that will be helpful to those aspiring to pass CKA.

Continue Reading

Stream Video From Raspberry Pi using Kinesis Video Streams

Sunday, May 10, 2020 by Prabhat Sharma raspberry pi, diy, video, streaming, aws

Kinesis video streams is one of the easiest way to stream your videos to internet. We will try it today using a raspberry pi. I am using the below hardware for streaming video. Hardware that I will use: Raspberry pi - https://www.raspberrypi.org/products/raspberry-pi-4-model-b/ Camera module - https://www.amazon.com/gp/product/B07L82XBNM/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 Steps: 1. On raspberry pi Enable camera by running below and selecting appropriate options raspi-config Install dependencies sudo apt-get install byacc flex sudo apt-get install cmake git clone https://github.

Continue Reading

AWS Workshops

Friday, May 1, 2020 by Prabhat Sharma workshop, training, aws

AWS teams have built multiple workshops to help customers learn its services through self paced workshops. You can follow them at your own pacve and learn the cool stuff. These are hands on guides. Here is a list of the ones I have come through: Sagemaker workshop - https://sagemaker-workshop.com/ Yet another Sagemaker workshop - https://www.sagemakerworkshop.com/ Yet another Sagemaker workshop - https://www.getstartedonsagemaker.com/ Quicksight - https://embed-workshop.learnquicksight.online EKS - https://eksworkshop.com/ ECS - https://ecsworkshop.

Continue Reading

The majestic Taj Mahal

Tuesday, Apr 21, 2020 by Prabhat Sharma architecture, beautiful, old, nostalgia

Found some old photographs from May 2009 when I visited majestic Taj Mahal. Here are the pics.

Continue Reading

My First PC Hardware Project

Tuesday, Apr 21, 2020 by Prabhat Sharma iot, car, hardware, college, old, nostalgia

Just found some old photographs of my first computer hardware project that I did as my final year bachelor’s graduation project (Bachelor of Computer Applications) in 2003 from OU PG College Osmania University in Secunderabad. We took a remote controlled car and connected its controls to computer using parallel port (Yeah that was a thing then). We finished it the morning before we were supposed to demo it to our examiner’s at college.

Continue Reading

Live Debugging of microservices in Kubernetes

Wednesday, Apr 8, 2020 by Prabhat Sharma reinvent, AWS, k8s, kubernetes, microservices

I did this session at AWS reinvent 2019. Keeping it here for reference. I also did a screen recording of the practice session before reinvent, details of which can be found at - https://github.com/prabhatsharma/ecommerce-microservices/blob/master/live-debugging.md

Continue Reading

Understanding Zookeeper by Doing

Wednesday, Jan 29, 2020 by Prabhat Sharma k8s, kubernetes, zookeeper, kafka, distributed systems

Software ecosystem of distrubuted systems is vast and each of the pieces do a specific task. Many a times you will find many systems that provide similar functionality. Let’s take a look at the problem of distributed coordination today, and one of the most popular systems to help solve it is zookeeper. Part of the hadoop ecosystem, zookeeper is very popular. Let’s understand what it is and how it works.

Continue Reading

How to stay updated With AWS developments

Tuesday, Jan 7, 2020 by Prabhat Sharma aws

With AWS releasing myriad of services and features every week how do you even get to know what got released and what is relevant to you. On top of the new services/features getting released there are a lot of blogs that get released by AWS solution architects and engineers that provide useful instructions to achieve specific use cases. Imagine being able to get all the relevant details in your inbox automatically and is separate from your email that allows you to focus.

Continue Reading

How to make reads from mongodb efficient using replicas

Monday, Sep 16, 2019 by Prabhat Sharma mongodb, database

Generally you run mongodb in a cluster that has replicas. Together the replicas are part of a replicaset. Replicasets allow you to have fault tolerance and high availability for your mongodb. You generally have one primary and 2 or more secondary replicas in the cluster. Data is written to primary and it gets replicated to replicas. While you would always be writing to primary, you have the option to read from primary or replicas.

Continue Reading

Build Docker images without Docker using Img and Jenkins on Kubernetes

Monday, Aug 12, 2019 by Prabhat Sharma kubernetes, CICD, Jenkins

Note: I will be using an Amazon EKS cluster on AWS and using Amazon ECR for storing images. There will be minor differences for handling other repositories.In the earlier post we learnt on how to to build a docker image using kubernete plugin in Jenkins on Kuberenetes. In this post we will look at how to build a docker image using Img. We will follow the same steps as the earlier post but will have a different pipeline script.

Continue Reading

Build Docker images without Docker - using Kaniko, Jenkins and Kubernetes

Thursday, Aug 8, 2019 by Prabhat Sharma kubernetes, containers, docker, kaniko, jenkins, CICD

Note: I will be using an Amazon EKS cluster on AWS and using Amazon ECR for storing images. There will be minor differences for handling other repositories.Jenkins is a hugely popular build tool that has been around for ages and used by many people. With huge shift to Kubernetes as a platform you would naturally want to run jenkins on Kubernetes. While running Jenkins in itself on Kubernetes is not a challenge, it is a challenge when you want to build a container image using jenkins that itself runs in a container in the Kubernetes cluster.

Continue Reading

Installing Jenkins on Kubernetes

Thursday, Aug 8, 2019 by Prabhat Sharma kubernetes, containers, docker, jenkins, CICD

In this post I will cover the steps to install Jenkins in a Kubernetes cluster. While Jenkins can be installed on standard VMs, if you are running a Kubernetes cluster then its much easier to install and run Jenkins in it. We will be using Helm to install Jenkins. We will follow the below steps to get up and running: Get the Jenkins values file for configuring our Jenkins installation Install Jenkins Access Jenkins Let’s rock ‘n’ roll:

Continue Reading

Installing Elasticsearch on Kubernetes Using Operator and setting it for Kubernetes logging

Wednesday, Aug 7, 2019 by Prabhat Sharma kubernetes, logging, elasticsearch, fluentd, fluent-bit, kibana, helm

In an earlier blog post I provided the steps to install elastisearch using helm and setting it up for logging using fluent-bit. With the introduction of elasticsearch operator the experience of managing the elasticsearch cluster in kubernetes has improved greatly. We will cover the same goal of setting up elastisearch and configuring it for logging as the earlier blog, with the same ease but much better experience. Using operator allows you benefits in the area of security, upgrades and scalability.

Continue Reading

Using encrypted EBS Volumes with Kubernetes on AWS

Sunday, Aug 4, 2019 by Prabhat Sharma k8s, kubernetes, eks, aws, amazon

Lot of people run Kubernetes on AWS and need to use encrypted EBS volumes for security and compliace. I will lay down the steps below in order to use it. Create a storage class Create a PersistentVolume (or dynamically provisoned PersistentVolumeClaim) using the storage class Create a pod to use the PersistentVolumeClaim 1. Create a storage class You must create a storage class that can be used for creating a PV/PVC.

Continue Reading

Installing Airflow on Kubernetes Using Operator

Sunday, Jul 28, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Well created Kubernetes Operators pack a lot of power and help run and manage stateful applications on kubernetes. We had earlier seen how to install airflow on kubernetes using helm charts. While helm charts help you get started fast, they may not be suitable for day 2 operatios like: Upgrades Backup & restore Auto recovery Automatic/On-demand scalability Configuration management Deep insights Let’s find how to install airflow on kubernetes using airflow operator.

Continue Reading

RED Monitoring With Istio and Prometheus

Sunday, Jul 21, 2019 by Prabhat Sharma k8s, kubernetes, monitoring, Prometheus, Istio

In this blog post we will discuss RED method that I really like in terms of simplicity by which it explains the state of an application. We will also discuss how to cpature RED metrics for your application running in kubernetes. Primarily we need to capture 3 things (Request) Rate - the number of requests, per second, you services are serving. (Request) Errors - the number of failed requests per second.

Continue Reading

Kubernetes Slides

Sunday, Jul 14, 2019 by Prabhat Sharma k8s, kubernetes, stateful

Created some slides for running stateful applications on kubernetes. Why containers Basics of Kubernetes- Pod, Replicaset, Deployment, Service Stateful Sets Storage High availability Best Practices

Continue Reading

Increase i3 instance store restore performance by 2.5x

Monday, Jun 3, 2019 by Prabhat Sharma ebs, ec2, instance store, aws, performance, IO, st1

EC2 i3 instances are great for scenarios where you require very high read/write performance as it provides you with locally attached instance store volumes upto 2 million read IOPS and 1.6 million write IOPS (for i3en.24xlarge). You can find details on available performance on these volumes at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html#i2-instances-diskperf . Many AWS customers use these instances for high IOPS sensitive applictions like databases. However instance store volumes are ephemeral. In order to ensure durability you need to be able to push the data on local SSD volumes to EBS on a regular basis.

Continue Reading

Installing Airflow on Kubernetes

Friday, Feb 1, 2019 by Prabhat Sharma k8s, kubernetes, containers, docker, airflow, helm, data engineering

Data engineering is a difficult job and tools like airflow make that streamlined. Let’s take a look at how to get up and running with airflow on kubernetes. Prerequisites A kubernetes cluster - You can spin up on AWS, GCP, Azure or digitalocean or you can start one on your local machine using minikube Helm - If you do not already have helm installed then follow this tutorial to get it installed Installing airflow using helm 1.

Continue Reading

DBeaver - SQL GUI Client for Amazon Redshift

Sunday, Dec 30, 2018 by Prabhat Sharma redshift, data warehousing, aws, amazon, gui

I use redshift occasionally for playing with some data. Its an amazing service(a datawarehousing solution) from AWS that you can use for analyzing large amounts of data. Every time I need to use redshift (every couple of months) I begin my search for a decent client that I can use. A google search does not result in something concrete right off with full of discussions on quora and other places.

Continue Reading

eksuser - Amazon EKS user management tool

Tuesday, Dec 4, 2018 by Prabhat Sharma eks, kubernetes, aws, authentication, iam, eksuser

Amazon EKS user management You have setup the EKS cluster and are able to use it. Now you want your teammates to access to the cluster too, so that they can build and run the applications. User management for EKS is done via the aws-auth configmap in kube-system namespace. You can learn how to add users manually by modifying the aws-auth configmap using the official documentation which shows how to add users to EKS by editing the aws-auth configmap.

Continue Reading

Logging in Kubernetes using Elasticsearch: The easy way

Monday, Nov 26, 2018 by Prabhat Sharma k8s, kubernetes, logging, elasticsearch, fluentd, fluent-bit

In my conversations with various development teams I regularly come across this common question of how to do logging on kubernetes. While there are existing solutions like EFK (elasticsearch, fluentd, kibana) stack, it takes good amount of effort for setting these up and making them work. I was wondering if I can provide some easy steps to people who just want to get started with logging in not so cumbersome and tedious way.

Continue Reading

Amazon EKS IAM authentication: How to add an IAM user

Friday, Oct 26, 2018 by Prabhat Sharma eks, kubernetes, aws, authentication, iam

Note: This blog provides a deep dive on EKS authentication. If your goal is to be able to just add/delete eks users then follow this eksuser-amazon-eks-user-management-toolWhen you create an EKS cluster it uses credentials of the user creating the cluster to set things up and assigns the user cluster-admin rights on the cluster through kubernetes RBAC. EKS uses IAM for authentication. It uses aws-iam-authenticator for authentication using webhook token authentication of kubernetes.

Continue Reading

Reserving Compute Resources for System Daemons in Kubernetes using node-allocatable

Friday, Oct 19, 2018 by Prabhat Sharma k8s, kubernetes, cgroup, node-allocatable, qos

I was discussing resource management on k8s worker nodes with someone and they mentioned that they had faced a situation earlier where scheduler will schedule a lot of pods on the worker node which would consume all of the node’s CPU and memory leaving other system processes starved of resources, that ultimately leads the node to crash. Kubernetes provides a feature to solve this problem called Node Allocatablethat allows you to reserve resources for system daemons so that they are not starved of resources and continue to function well.

Continue Reading

How to connect to Kubernetes cluster resources(services, pods) using OpenVPN for local development

Friday, Oct 19, 2018 by Prabhat Sharma kubernetes, helm, openvpn, local development, development workflow, developer productivity

I wanted to do development locally on my laptop and I had some services that were running in my kubernetes cluster. I was trying to figure out the easiest way to access those resources without exposing them via a loadbalancer or nodeport for the specific service. This is when I thought if I could vpn into the cluster then it would allow me to access cluster resources from my laptop.

Continue Reading

Helm tutorial - The package manager for kubernetes - Part 1

Saturday, Oct 13, 2018 by Prabhat Sharma kubernetes, helm, package manager

I have been asked by various people on what is the best way to install and manage applications on kubernetes. In fact a lot of people want to install various readily available solutions like mysql, mariadb, mongodb, nginx etc. For stateless applications its generally straightforward and nobody is bothered much, however when it comes to stateful applications one starts to think how they will manage to install it. Does one have to go through the rigmarole of installing the application just like installing it on instances?

Continue Reading

Best way to build website and blog in 2018

Tuesday, Oct 9, 2018 by Prabhat Sharma website, blog, hosting, wordpress

Beware!!! opinions ahead!!! I am asked by multiple people on what is the best way to build a small website and maybe build a blog as well. For long time wordpress has been the choice of application with other CMSs like drupal, joomla, etc as contenders. On the blogging front the contenders have been like ghost. Also there have been online applications like medium and tumblr that are doing the job pretty well.

Continue Reading

How I passed Certified Kubernetes Administrator exam on first attempt

Wednesday, Oct 3, 2018 by Prabhat Sharma kubernetes, cka, certification, exam

Update on Aug 18th 2020 - Added practice tests link at the end I have been working for past couple of months on kubernetes and advising people on how to implement best practices using kubernetes for building applications. I wanted to test my knowledge and figured I should take the CKA exam to validate my knowledge of things. I began preparing about one and a half months ago for the certification and found its preparation to be as hard as AWS SA Pro exam which is considered to be one of the difficult certifications.

Continue Reading