This content originally appeared on HackerNoon and was authored by Alexander Sharov
Running GPU workloads on Amazon EKS requires configuring GPU-enabled nodes, installing necessary drivers, and ensuring proper scheduling. Follow these steps to set up GPU nodes in your EKS cluster.
1. Create an Amazon EKS Cluster
First, create an EKS cluster without worker nodes using eksctl (for simplicity, we don’t use Terraform/OpenTofu ):
eksctl create cluster --name kvendingoldo–eks-gpu-demo --without-nodegroup
2. Create a Default CPU Node Group
A separate CPU node group ensures that
Kubernetes system components (
kube-systempods) have a place to run.The GPU Operator and its dependencies will be deployed successfully.
Non-GPU workloads don’t end up on GPU nodes.
\
Create at least one CPU node to maintain cluster stability:
eksctl create nodegroup --cluster kvendingoldo–eks-gpu-demo \
--name cpu-nodes \
--node-type t3.medium \
--nodes 1 \
--nodes-min 1 \
--nodes-max 3 \
--managed
\
3. Create a GPU Node Group
GPU nodes should have appropriate taints to prevent non-GPU workloads from running on them. Use an NVIDIA-compatible instance type (you can check all options at instances.vantage.sh, but typically, it’s g4dn.xlarge or p3.2xlarge) for such nodes:
\
eksctl create nodegroup --cluster kvendingoldo–eks-gpu-demo \
--name gpu-nodes \
--node-type g4dn.xlarge \
--nodes 1 \
--node-taints only-gpu-workloads=true:NoSchedule \
--managed
\
A custom taint only-gpu-workloads=true:NoSchedule guarantees that only pods with the same toleration configuration are scheduled on these nodes.
4. Install the NVIDIA GPU Operator
The NVIDIA GPU Operator installs drivers, CUDA, toolkit, and monitoring tools. To install it, use the following steps:
\ Create gpu-operator-values.yaml:
tolerations:
- key: "only-gpu-workloads"
value: "true"
effect: "NoSchedule"
\ Deploy the gpu-operator via Helm:
helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update
helm install gpu-operator nvidia/gpu-operator -f gpu-operator-values.yaml
\ Pay attention to two things:
- YAML deployment of k8s-device-plugin shouldn’t be used for production.
- By using
gpu-operator-values.yamlvalues, we set up tolerations for the gpu-operator daemonset; without that, nodes will not work, and you won’t be able to schedule GPU workloads there.
5. Verify GPU Availability
After deploying the GPU Operator, check if NVIDIA devices are correctly detected on the GPU by the following command:
kubectl get nodes -o json | jq '.items[].status.allocatable' | grep nvidia
Check the GPU Status on the Node Using AWS SSM (In Case of Issues)
If you need to manually debug a GPU node, connect using AWS SSM (Systems Manager Session Manager) instead of SSH.
Step 1: Attach the SSM IAM Policy
Ensure your EKS worker nodes have the AmazonSSMManagedInstanceCore policy:
aws iam attach-role-policy --role-name <NodeInstanceRole> \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Step 2: Start an SSM Session
Find the Instance ID of your GPU node:
aws ec2 describe-instances --filters "Name=tag:eks:nodegroup-name,Values=gpu-nodes" \
--query "Reservations[].Instances[].InstanceId" --output text
\ Start the AWS SSM session:
aws ssm start-session --target <Instance-ID>
\ Inside the node, check the GPU state:
lspci | grep -i nvidiato check if the GPU hardware is detectednvidia-smito verify the NVIDIA driver and GPU status
\ If Nvidia-smi fails or the GPU is missing, it may indicate that:
- GPU Operator is not installed correctly.
- K8S node does not have an NVIDIA GPU.
- NVIDIA driver failed to load.
\ Check the official Nvidia documentation to solve these issues.
6. Schedule a GPU Pod
Deploy a test pod to verify GPU scheduling. This pod:
- Requests a GPU.
- Uses tolerations to run on GPU nodes.
- Runs nvidia-smi to confirm GPU access.
---
apiVersion: v1
kind: Pod
metadata:
name: kvendingoldo-gpu-test
spec:
tolerations:
- key: "only-gpu-workloads"
value: "true"
effect: "NoSchedule"
nodeSelector:
nvidia.com/gpu: "true"
containers:
- name: cuda-container
image: nvidia/cuda:12.0.8-base
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1
7. Handling “Insufficient nvidia.com/gpu” Errors
Typically, users may face failing pods and errors like:
0/2 nodes are available: 1 Insufficient nvidia.com/gpu
\ It means that all GPUs are already allocated or Kubernetes is not recognizing available GPUs. Here, the following fixes may help.
\
Check GPU Allocations kubectl describe node <gpu-node-name> | grep "nvidia.com/gpu"
\
If you don’t see any nvidia.com labels on your GPU node, it means the operator isn’t working, and you should debug it. It is typically caused by taints or tolerations. Pay attention that the nvidia-device-plugin pod should exist on each GPU node.
\ Verify the GPU Operator
Check the status of operator pods: \n kubectl get pods -n gpu-operator
\
If some pods are stuck in Pending or CrashLoopBackOff, restart the operator: \n kubectl delete pod -n gpu-operator — all
\ Restart kubectl
Sometimes, the kubelet gets stuck. In such cases, login into a node and restarting Kubelet may be helpful.
\
**Scale Up GPU Nodes \n **Increase GPU node count: \n eksctl scale nodegroup --cluster=kvendingoldo–eks-gpu-demo --name=gpu-nodes --nodes=3
Conclusion
Congrats! Your EKS cluster is all set to tackle GPU workloads. Whether you’re running AI models, processing videos, or crunching data, you’re ready to go. Happy deploying!
\
This content originally appeared on HackerNoon and was authored by Alexander Sharov
Alexander Sharov | Sciencx (2025-03-11T23:09:05+00:00) A Guide on How to Use GPU Nodes in Amazon EKS. Retrieved from https://www.scien.cx/2025/03/11/a-guide-on-how-to-use-gpu-nodes-in-amazon-eks/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.