What is Karpenter?
Designed as an open-source Kubernetes cluster autoscaler, Karpenter seamlessly integrates with AWS (Amazon Web Services) to enhance application availability and cluster efficiency. By swiftly provisioning compute resources of the right size, it effectively responds to fluctuations in application load, ensuring optimal performance.
How Karpenter works?
Once installed in your cluster, Karpenter actively monitors the combined resource requests of unscheduled pods and takes actions to minimize scheduling delays and reduce infrastructure expenses by launching or terminating nodes. This is accomplished by observing Kubernetes cluster events and issuing commands to the underlying compute service of the cloud provider, such as Amazon EC2.
Let’s Deploy…
Prerequisites
- AWS IAM access with admin privileges
- AWS CLI
- Kubectl
- Eksctl
- Helm
Step 1: Setup ENV variables
1.1 Declare variables for the stack
## Set karpenter version
export KARPENTER_VERSION=v0.29.0
## if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export AWS_PARTITION="aws"
## Set your desired cluster name
export CLUSTER_NAME="karpenter-demo"
## Set AWS region
export AWS_DEFAULT_REGION="us-east-1"
## Set AWS Account ID
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
1.2 Verify the variables
echo $KARPENTER_VERSION $CLUSTER_NAME $AWS_DEFAULT_REGION $AWS_ACCOUNT_ID
Step 2: Setup Infra
2.1 Setup infrastructure needed by the EKS cluster
- Here we will use the getting started CloudFormation templates provided by the Karpenter to set up the infrastructure needed by the EKS cluster.
- This template will set up required roles, policies, etc...
- Fetch the CloudFormation yaml to your local
curl -fsSL https://raw.githubusercontent.com/aws/karpenter/"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > cloudformation.yaml
- Now we can deploy the CloudFormation stack
aws cloudformation deploy \
--stack-name "${CLUSTER_NAME}" \
--template-file cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "ClusterName=${CLUSTER_NAME}"
Screenshot of the CloudFormation stack:
- Successfully deployed, Now Let’s create the EKS cluster
2.2 Provision of EKS cluster
- Here, We will create the EKS cluster using the Eksctl template
- It will establish a Kubernetes service account and an AWS IAM Role, then utilize IRSA to associate them, allowing Karpenter to initiate instance launches.
- VPC, Subnets, and Security Groups will be created
- Mapping of Karpenter node role to the aws-auth configmap to allow nodes to connect.
- Uses AWS EKS-managed node groups for the kube-system and karpenter namespaces
eksctl create cluster -f - <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ${AWS_DEFAULT_REGION}
version: "1.27"
tags:
karpenter.sh/discovery: ${CLUSTER_NAME}
iam:
withOIDC: true
serviceAccounts:
- metadata:
name: karpenter
namespace: karpenter
roleName: ${CLUSTER_NAME}-karpenter
attachPolicyARNs:
- arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}
roleOnly: true
iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
managedNodeGroups:
- instanceType: t3a.large
amiFamily: AmazonLinux2
name: ${CLUSTER_NAME}-ng
desiredCapacity: 2
minSize: 1
maxSize: 10
EOF
CloudFormation created by Eksctl:
EKS Cluster:
- Successfully created the EKS cluster
- Set KARPENTER_IAM_ROLE_ARN variables. Which will be used during the Karpenter installation
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
echo $CLUSTER_ENDPOINT $KARPENTER_IAM_ROLE_ARN
- Create a role to allow spot instances.
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || true
# If the role has already been successfully created, you will see:
# An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForEC2Spot has been taken in this account, please try a different suffix.
Step 3: Install Karpenter
3.1 Installation of Karpenter using HELM
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
--set settings.aws.clusterName=${CLUSTER_NAME} \
--set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
--set settings.aws.interruptionQueueName=${CLUSTER_NAME} \
--set controller.resources.requests.cpu=1 \
--set controller.resources.requests.memory=1Gi \
--set controller.resources.limits.cpu=1 \
--set controller.resources.limits.memory=1Gi \
--wait
Helm Output:
3.2 Configuration of Provisioner
- Karpenter provisioners can efficiently manage numerous pod shapes by utilizing pod attributes like labels and affinity, thus streamlining scheduling and provisioning decisions and eliminating the necessity to handle multiple node groups.
- More about Provisioners, Node Templates
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: "node.kubernetes.io/instance-type"
operator: In
values: ["t3a.large"]
limits:
resources:
cpu: 1000
providerRef:
name: default
consolidation:
enabled: true
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: default
spec:
subnetSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
securityGroupSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
EOF
Provisioner output:
- Karpenter has been activated and is fully prepared to initiate node provisioning.
Step 4: Deploy the Application
4.1 Upscale deployment
- Now, Let’s create a deployment and watch Karpenter provision nodes in response
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 0
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 0
containers:
- name: nginx
image: nginx
resources:
requests:
cpu: 1
EOF
Before Scaling:
- Deployed nginx with zero replicas, Now Let’s scale the deployment and watch the Karpenter logs
## Scale deployment
kubectl scale deployment nginx --replicas 5
## To monitor the karpenter logs
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
After Scaling:
- Based on your Requirements and Configuration, You can modify the provisioner and node templates
4.1 Downscale deployment
- Next, proceed to downscale the deployment. In a brief duration, Karpenter will automatically terminate the empty nodes as part of the consolidation process.
## DownScale deployment
kubectl scale deployment nginx --replicas 0
## To monitor the karpenter logs
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
After Downscaling:
- We have completed the demo of Karpenter in EKS
Conclusion
- Karpenter provides enhanced scaling capabilities, enabling it to detect precise workload requirements and allocate the resources accordingly intelligently. This results in dynamic scaling that adapts to actual usage, allowing it to scale up as needed efficiently.