any AWS EKS skills around

[georgelza]

if I kubectl get pvc -n <namespace> then it jsut shows the pv, and pending

When the pods are in pending state, have a look at the generated PVC using kubectl, and see why the volume is not being provisioned.

Strangely think this is the problem... permissions, I found a doc thats about pv's and pvc's but it is trying to enable a different driver for EBS.

relevant region in your AWS account to attach, check that the provisioner has the correct rights to allow this, if I remember correctly, this is not something that gets attached by default.
</quote>

ye, aware of the EBS and az limitation, thanks.

Also keep in mind if you are trying to build a multi-az setup with fault tolerance, EBS volumes are single AZ only and set to ReadWriteOnce inside the cluster, they will only be able to attach to a single node/pod at one time, so trying to provision multiple pods across multiple nodes with the same EBS PVC will not work and only one pod will start correctly and attach to the volume (https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)

 

[Chiller89]

if I kubectl get pvc -n <namespace> then it jsut shows the pv, and pending



Strangely think this is the problem... permissions, I found a doc thats about pv's and pvc's but it is trying to enable a different driver for EBS.


ye, aware of the EBS and az limitation, thanks.

Rather do kubectl describe pvc -n $namespace - it should give you events for the provisioning process and give you somewhere else to look.

 

[georgelza]

other than trying to figure out why... this is failing,

from creating a new EKS cluster using eksctl... is there anything I need to do before I can do a pvc claim, is there a "do this to get permissions ready", or a create a EBS first like this...
what am I missing.
G

 

[Chiller89]

other than trying to figure out why... this is failing,

from creating a new EKS cluster using eksctl... is there anything I need to do before I can do a pvc claim, is there a "do this to get permissions ready", or a create a EBS first like this...
what am I missing.
G

Apologies been a little bit since I set up our clusters.
You need the EBS/EFS CSI Driver installed in the cluster to enable the cluster to provision EBS volumes
Driver specific info:

Use Kubernetes volume storage with Amazon EBS - Amazon EKS

The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver manages the lifecycle of Amazon EBS volumes as storage for Kubernetes Volumes.

docs.aws.amazon.com

This covers persistent storage in EKS in general and runs through EBS CSI driver and deploying permissions etc for it: https://aws.amazon.com/premiumsupport/knowledge-center/eks-persistent-storage/

 

[georgelza]

sweet, let me recreate cluster... so that I'm working from a clean slate, do the above and then retry my deployment.
I'm aware of that doc... but was thinking it can't be relevant as it installs a CSI driver and I should already have a working gp2 driver working...
G

 

[georgelza]

... so new cluster, with the above EBS CSI driver deployed.

then executed the elastic search quickstart deployment, to the TTTTTT

as can be seen, still pending

Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic > kubectl describe pvc elasticsearch-data-quickstart-es-default-0
Name: elasticsearch-data-quickstart-es-default-0
Namespace: default
StorageClass: gp2
Status: Pending
Volume:
Labels: common.k8s.elastic.co/type=elasticsearch
elasticsearch.k8s.elastic.co/cluster-name=quickstart
elasticsearch.k8s.elastic.co/statefulset-name=quickstart-es-default
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: quickstart-es-default-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 2m11s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 12s (x9 over 2m11s) persistentvolume-controller waiting for pod quickstart-es-default-0 to be scheduled
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic > kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-data-quickstart-es-default-0 Pending gp2 2m16s
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic > kubectl get pv
No resources found
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic >

 

[Chiller89]

... so new cluster, with the above EBS CSI driver deployed.

then executed the elastic search quickstart deployment, to the TTTTTT

as can be seen, still pending

What happens if you describe the storage class?
kubectl describe storageclass gp2
Should show you provisioning events for the storage class, might point to something

 

[georgelza]

Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic > kubectl describe sc
Name: gp2
IsDefaultClass: Yes
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/aws-ebs
Parameters: fsType=ext4,type=gp2
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events: <none>
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic >
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic >
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic > kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 132m
Georges-MacBook-Pro.local:/Users/george/Downloads/dev/devlab_aws/elastic >

 

[georgelza]

@Chiller89, see PM.