Kubernetes has become the de facto standard for container orchestration, enabling developers to manage and deploy containerized applications at scale. Amazon’s Elastic Kubernetes Service (EKS) is a fully managed Kubernetes service that makes it easy to run Kubernetes on AWS. However, as applications grow, managing resources effectively can become a challenge. That’s where Karpenter comes in.
Karpenter is a just-in-time capacity provisioner for any Kubernetes cluster, not just EKS. It is designed to simplify and optimize resource management in Kubernetes clusters by intelligently provisioning the right resources based on workload requirements. Although Karpenter can be used with any Kubernetes cluster, in this article, we will focus on its benefits and features within the context of AWS and EKS.
EKS best practices recommend using Karpenter over Autoscaling Groups, Managed Node Groups, and Kubernetes Cluster Autoscaler in most cases. Here’s why Karpenter has the edge:
Key Features of Karpenter Karpenter offers several powerful features that set it apart from traditional autoscaling solutions:
Karpenter is a powerful tool that optimizes resource usage in Kubernetes clusters, resulting in significant cost savings for users. By assessing the resource requirements of pending pods, Karpenter intelligently selects the most suitable instance type for running them, ensuring efficient resource utilization. Furthermore, this smart solution can automatically scale-in or terminate instances no longer in use, reducing waste and lowering costs.
In addition to these capabilities, Karpenter offers a unique consolidation feature that intelligently reorganizes pods and either deletes or replaces nodes with more cost-effective alternatives to further optimize cluster costs. This proactive approach to cost management ensures users get the most value from their Kubernetes clusters while minimizing waste and maximizing efficiency. With Karpenter, users can trust that their clusters are optimized for performance, scalability, and cost-effectiveness.
With so many reasons to obvious reasons to use karpenter, let’s give it a try while you figure out your favourite use-case.
Karpenter deploys as any other component in a kubernetes cluster. Helm charts are published by AWS to simplify the installation process. Here is a step-by-step guide to install karpeneter on any running EKS Cluster and and test it on a sample application workload.
As a Pre-requisite, you need access to cluster using kubectl and helm, alongwith access to AWS Account using AWS CLI ( to configure IAM roles and related settings )
First create an IAM role that will be associated to the worker nodes , provisioned by Karpenter. This Allows worker nodes to attach to EKS cluster and add to Available capacity. All the required IAM policies for this role can be added with these steps
aws iam create-role --role-name eks-karpenter-node-role \ | |
--assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}' | |
aws iam attach-role-policy --role-name eks-karpenter-node-role \ | |
--policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy | |
aws iam attach-role-policy --role-name eks-karpenter-node-role \ | |
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore | |
aws iam attach-role-policy --role-name eks-karpenter-node-role \ | |
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly | |
aws iam attach-role-policy --role-name eks-karpenter-node-role \ | |
--policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy | |
aws iam get-instance-profile-for-role --role-name eks-karpenter-node-role |
Take note of the IAM instance profile ARN , this will be passed to karpenter helm config for installation. Create a custom values.yaml file, replace cluster_name, cluster_endpoint and node_iam_instance_profile_arn (from above step)
nodeSelector: | |
kubernetes.io/os: linux | |
clusterName: cluster_name | |
clusterEndpoint: cluster_endpoint | |
aws: | |
defaultInstanceProfile: node_iam_instance_profile_arn |
Next, install karpenter to your EKS cluster, while passing above values.yaml file as an argument to helm install command
helm repo add karpenter https://awslabs.github.io/karpenter/charts | |
helm install karpenter karpenter/karpenter --values values.yaml |
Verify that karpenter is deployed successfully
kubectl get all -n karpenter
Get the name of Kubernetes service account created by Karpenteraws eks describe-cluster --name dev-skaf --query "cluster.identity.oidc.issuer" --output text
Replacekubectl get serviceaccount -n karpenter
OIDC_PROVIDER_ARN
and KARPENTER_SA_NAME
with the values fetched in above commands. Note the role ARN from command output. aws iam create-role \ | |
--role-name karpenter-irsa \ | |
--assume-role-policy-document '{ | |
"Version": "2012-10-17", | |
"Statement": [ | |
{ | |
"Effect": "Allow", | |
"Principal": { | |
"Federated": "arn:aws:iam::accountid:oidc-provider/<OIDC_PROVIDER_ARN>" | |
}, | |
"Action": "sts:AssumeRoleWithWebIdentity", | |
"Condition": { | |
"StringEquals": { | |
"OIDC provider ARN:sub": "system:serviceaccount:karpenter:<KARPENTER_SA_NAME>" | |
} | |
} | |
} | |
] | |
}' |
Create an IAM policy with required permissions and note policy ARN from output.
# Create IAM Policy | |
aws iam create-policy \ | |
--policy-name karpenter-irsa-policy \ | |
--policy-document '{ | |
"Statement": [ | |
{ | |
"Action": [ | |
"ssm:GetParameter", | |
"pricing:GetProducts", | |
"iam:PassRole", | |
"ec2:RunInstances", | |
"ec2:DescribeSubnets", | |
"ec2:DescribeSpotPriceHistory", | |
"ec2:DescribeSecurityGroups", | |
"ec2:DescribeLaunchTemplates", | |
"ec2:DescribeInstances", | |
"ec2:DescribeInstanceTypes", | |
"ec2:DescribeInstanceTypeOfferings", | |
"ec2:DescribeImages", | |
"ec2:DescribeAvailabilityZones", | |
"ec2:DeleteLaunchTemplate", | |
"ec2:CreateTags", | |
"ec2:CreateLaunchTemplate", | |
"ec2:CreateFleet" | |
], | |
"Effect": "Allow", | |
"Resource": "*", | |
"Sid": "Karpenter" | |
}, | |
{ | |
"Action": "ec2:TerminateInstances", | |
"Condition": { | |
"StringLike": { | |
"ec2:ResourceTag/Name": "*karpenter*" | |
} | |
}, | |
"Effect": "Allow", | |
"Resource": "*", | |
"Sid": "ConditionalEC2Termination" | |
} | |
], | |
"Version": "2012-10-17" | |
}' |
Attach the IAM policy to the role, and annotate karpenter service account with ARN of the role. ( Replace arn_of_karpenter_irsa_policy and arn_of_karpenter_irsa_role with noted values )
# Attach Policy to the role | |
aws iam attach-role-policy --role-name karpenter-irsa \ | |
--policy-arn <arn_of_karpenter_irsa_policy> | |
# Annotate service account with arn of role | |
kubectl annotate serviceaccount -n karpenter \ | |
karpenter-irsa eks.amazonaws.com/role-arn=<arn_of_karpenter_irsa_role> |
Now Karpenter is ready with the required IAM permissions to creaet worker nodes, but it still needs the node provisioning loginc an dhence let’s define the Karpenter Provisioner configuration.
Here you again need to specify EKS Cluster name and Subnet Selector for the Nodes . Replace cluster_name and private_subnet_name with correct values. Create a file and use kubectl apply -f <filename>command to apply this provisioner configuration
apiVersion: karpenter.sh/v1alpha5 | |
kind: Provisioner | |
metadata: | |
name: karpenter-provisioner | |
spec: | |
labels: | |
Services: "app" | |
requirements: | |
- key: karpenter.sh/capacity-type | |
operator: In | |
values: ["on-demand"] | |
- key: karpenter.k8s.aws/instance-size | |
operator: NotIn | |
values: ["nano", "micro", "small"] | |
- key: "karpenter.k8s.aws/instance-category" | |
operator: In | |
values: ["c", "m", "r"] | |
- key: "karpenter.k8s.aws/instance-cpu" | |
operator: In | |
values: ["4", "8", "16", "32"] | |
- key: karpenter.k8s.aws/instance-hypervisor | |
operator: In | |
values: ["nitro"] | |
- key: "topology.kubernetes.io/zone" | |
operator: In | |
values: ["us-west-2a", "us-west-2b"] | |
- key: "kubernetes.io/arch" | |
operator: In | |
values: ["arm64", "amd64"] | |
- key: "karpenter.sh/capacity-type" # If not included, the webhook for the AWS cloud provider will default to on-demand | |
operator: In | |
values: ["spot", "on-demand"] | |
providerRef: | |
name: karpenter-node-template | |
ttlSecondsAfterEmpty: 300 | |
--- | |
apiVersion: karpenter.k8s.aws/v1alpha1 | |
kind: AWSNodeTemplate | |
metadata: | |
name: karpenter-node-template | |
spec: | |
subnetSelector: | |
Name: <private_subnet_name> # required | |
securityGroupSelector: | |
aws:eks:cluster-name: <cluster_name> # required |
Now when Karpenter is all set for action, let’s see how it can be harnessed to add capacity to EKS cluster , on demand and based on workload requirement. For this example, we create a simple nginx deployment , define the requirements and see how karpenter chooses the right ec2 instance for worker node
kubectl apply -f - <<EOF | |
apiVersion: apps/v1 | |
kind: Deployment | |
metadata: | |
name: nginx-deployment | |
labels: | |
app: nginx | |
spec: | |
replicas: 1 | |
selector: | |
matchLabels: | |
app: nginx | |
template: | |
metadata: | |
labels: | |
app: nginx | |
spec: | |
nodeSelector: | |
"ECK-Services": "true" | |
containers: | |
- name: nginx | |
image: nginx:latest | |
ports: | |
- containerPort: 80 | |
EOF |
Verify the deployment , see that pod is running and a new node is provisioned.
Get the node IP and verify it on AWS Console
Karpenter understands many available scheduling constraint definitions in kubernetes by default, including
Karpenter can detect availability zone of Kubernetes Persistent volumes backed by Amazon EBS and launch the worker nodes accordingly to avoid cross-AZ mounting issues with EBS storage class.
We just witnessed how karpenter can create and optimized EC2 capacity on demand for your EKS clusters. Installing and configuring karpeneter may seem like a tak but our terraform module for EKS Bootstrap adds such useful drivers to your eks cluster seamlessly.
Next : best practices for using karpenter