georgelza
Expert Member
Looking for someone that would be willing to field AWS EKS questions from a Noob.
It seems the AWS Forum is not that busy...
G
It seems the AWS Forum is not that busy...
G
-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: What is your preferred Chinese car brand?
Herr der Verboten
Honorary Master
georgelza
Expert Member
hehehehe
what did YOU doooooo
guys, how can we chat... can always / potentially do it here, will become a long mixed up thread...
G
georgelza
Expert Member
georgelza
Expert Member
we have multiple projects in development scope/phase at the moment thats being done using cloud native based architecture,a conversation here would be great.
i'll ask a question
do you really need EKS/k8s, or do you just think you need it?
For some of them sure we can potentially do it using ECS, but for now I have a Sandbox, where I'm playing, and have various questions popping up, or need assistance as I go...
G
yeah, really depends on your use case and team
We went through a similar process.
At the end of the day, we/I decided that ECS with Fargate was a better for us as a pure development company with many clients.
AWS SSO (remove developer IAM & multi account complexities) + Terraform obviously simplified this a lot.
This also obviously aligned with one of our higher level strategic goals - Eliminate all self-managed services in production
Possibly if I was evaluating for a product based team, then my needs might be different.
We went through a similar process.
At the end of the day, we/I decided that ECS with Fargate was a better for us as a pure development company with many clients.
AWS SSO (remove developer IAM & multi account complexities) + Terraform obviously simplified this a lot.
This also obviously aligned with one of our higher level strategic goals - Eliminate all self-managed services in production
Possibly if I was evaluating for a product based team, then my needs might be different.
RedViking
Nord of the South
@Johnatan56 maybe
georgelza
Expert Member
so lets fire the first question going, and start first with the lab environment...
VPC:
EBS PV will be used with associated pvc.
first question, can multiple pvc's be hosted on a single pv ? and how do I tell my helm scripts to use the pre created pv. - happy to share my scripts and point to the gitrepo where the above helm chart comes from, where the configurable values.yaml is located.
I understand a EBS that houses the pv is "restricted" to a az and EFS is not allowed, otherwise the prometheus server could have run on any node in my management subnet, but due to the EBS restriction I got to use a nodeSelector to pin it to a node in management subnet in az1.
I would like to pre create my pv and then have the above helm chart do the pic claim from my pre created pv.
more questions to come...
G
VPC:
- 2 App Subnets over 2 AZ's (az1,2) 10.7.23.0/24 and 10.7.24.0/24
- 2 DB Subnets over 2 AZ's (az1,2) 10.7.33.0/24 and 10.7.34.0/24
- 2 Management Subnets over 2 AZ's (az1,2) 10.7.43.0/24 and 10.7.44.0/24
- the app nodes have been "wrapped" together into a node_group : app_ng
- the db nodes have been "wrapped" together into a node_group : db_ng
- the management nodes have been "wrapped" together into a node_group : management_ng
EBS PV will be used with associated pvc.
first question, can multiple pvc's be hosted on a single pv ? and how do I tell my helm scripts to use the pre created pv. - happy to share my scripts and point to the gitrepo where the above helm chart comes from, where the configurable values.yaml is located.
I understand a EBS that houses the pv is "restricted" to a az and EFS is not allowed, otherwise the prometheus server could have run on any node in my management subnet, but due to the EBS restriction I got to use a nodeSelector to pin it to a node in management subnet in az1.
I would like to pre create my pv and then have the above helm chart do the pic claim from my pre created pv.
more questions to come...
G
georgelza
Expert Member
sure i might also end there, but I prefer to learn first in a environment where things are hard defined, add the learning bits, and then progress to a more dynamic setup, where I then take my knew knowledge and then stop it up into the new functionality.
G
georgelza
Expert Member
I'm using: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack as a helm option to figure some concepts out, once I got this one working going to drop it and try another... just to see how they differ and how they are deployed and how my understanding translates...
What I do might not make sense, might not be the best way at times, it's me unravelling how things are done, and what has impact on what...
Command being executed
contents of values.yaml
EKS create script
What I do might not make sense, might not be the best way at times, it's me unravelling how things are done, and what has impact on what...
Command being executed
helm install prometheus kube-prometheus-stack/values.yaml prometheus-community/kube-prometheus-stack \
--namespace monitoring
--create-namespace
contents of values.yaml
namespaceOverride: "monitoring"
alertmanager:
nodeSelector:
ng-tier: "management-ng"
failure-domain.beta.kubernetes.io/zone: af-south-1a
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: gp2
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
prometheusOperator:
nodeSelector:
ng-tier: "management-ng"
failure-domain.beta.kubernetes.io/zone: af-south-1a
prometheus:
nodeSelector:
ng-tier: "management-ng"
failure-domain.beta.kubernetes.io/zone: af-south-1a
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp2
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
EKS create script
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: devlab-a
region: af-south-1
vpc:
subnets:
# must provide 'private' and/or 'public' subnets by availibility zone as shown
private:
# App
private-one: # az1 10.7.23.0
id: "subnet-06e792826b572f6f7"
private-two: # az2 10.7.24.0
id: "subnet-0ce3bc55b7de14e25"
# DB
private-three: # az1 10.7.33.0
id: "subnet-04867640599bb00db"
private-four: # az2 10.7.34.0
id: "subnet-079c46d587812b38d"
# Management
private-five: # az1 10.7.43.0
id: "subnet-029cbef1f8e78ce9f"
private-six: # az2 10.7.44.0
id: "subnet-030f96716406bac51"
managedNodeGroups:
- name: app-ng-az1
privateNetworking: true
subnets:
- private-one
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 4
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: app-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "app-ng"
ng-location: "az1"
- name: app-ng-az2
privateNetworking: true
subnets:
- private-two
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 4
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: app-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "app-ng"
ng-location: "az2"
- name: db-ng-az1
privateNetworking: true
subnets:
- private-three
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 4
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: db-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "db-ng"
ng-location: "az1"
- name: db-ng-az2
privateNetworking: true
subnets:
- private-four
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 4
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: db-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "db-ng"
ng-location: "az2"
- name: management-ng-az1
privateNetworking: true
subnets:
- private-five
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 3
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: management-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "management-ng"
ng-location: "az1"
- name: management-ng-az2
privateNetworking: true
subnets:
- private-six
instanceType: t3.micro
desiredCapacity: 1
minSize: 1
maxSize: 3
volumeSize: 20
volumeType: gp2
tags:
team: devlab
eks: management-eks-nodes
tier: app
ekscluster: devlab-a
ssh: # use existing EC2 key
publicKeyName: devlab_jumpserver_az1
allow: true
labels:
ng-tier: "management-ng"
ng-location: "az2"
Johnatan56
Honorary Master
Kubernetes, sure, not AWS EKS.
georgelza
Expert Member
ye... they might both be based on K8S, but with the introduction of multiple AZ's it does bring in some special considerations when designing a deployment architecture.
G
Concentric
Expert Member
- Joined
- Feb 16, 2017
- Messages
- 1,028
- Reaction score
- 197
Multiple PVCs can belong to a single pv.
We use this for our logging.
All our apps log to the same location being an EFS share
georgelza
Expert Member
hmm, wonder how this changes/applies to a EBS... EFS being a file share, NFS, it's sort of the base architecture of the share.
would like to create one (or two) PV per AZ, onto which the various pvc's are bounded.
G
would like to create one (or two) PV per AZ, onto which the various pvc's are bounded.
G
georgelza
Expert Member
Want to qualify something here, do you do this on AWS ? asking as I've just watched a YT video from AWS where they say a single PV can only be associated with a single PVC...
My thinking g was to create a nice 100TB PV per AZ and then carve out of there for various pvc's but this fact now stops that...
G
My thinking g was to create a nice 100TB PV per AZ and then carve out of there for various pvc's but this fact now stops that...
G
georgelza
Expert Member
hi hi, I have some pv/pvc specific on EKS (sure ECS) will also apply questions, any chance you willing to take a teams call...
or... in writing... then maybe everyone can see answer and be knowledge for all...
I've created a fresh EKS cluster. with multiple nodes.
I'm trying to deploy (prometheus and/or Elastic stack).
Which of course have pvc requirements...
My pods stay in pending state, pretty sure it's related to fact that there is no supporting pv, me default StorageClass is gp2... which should have the cluster auto create supporting pi's... but seems not.
Is there something I have to do first...
I've tried create multiple EBS volumes of the right size in each az, but did not help.
G
or... in writing... then maybe everyone can see answer and be knowledge for all...
I've created a fresh EKS cluster. with multiple nodes.
I'm trying to deploy (prometheus and/or Elastic stack).
Which of course have pvc requirements...
My pods stay in pending state, pretty sure it's related to fact that there is no supporting pv, me default StorageClass is gp2... which should have the cluster auto create supporting pi's... but seems not.
Is there something I have to do first...
I've tried create multiple EBS volumes of the right size in each az, but did not help.
G
Chiller89
Well-Known Member
When the pods are in pending state, have a look at the generated PVC using kubectl, and see why the volume is not being provisioned. Since the default volume type is gp2, the cluster is going to try and create a gp2 EBS volume for you in the relevant region in your AWS account to attach, check that the provisioner has the correct rights to allow this, if I remember correctly, this is not something that gets attached by default.
Also keep in mind if you are trying to build a multi-az setup with fault tolerance, EBS volumes are single AZ only and set to ReadWriteOnce inside the cluster, they will only be able to attach to a single node/pod at one time, so trying to provision multiple pods across multiple nodes with the same EBS PVC will not work and only one pod will start correctly and attach to the volume (https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)