<3> eks-node-viewer 설치 1 # IP 주소 확인 : 172.30.0.0/16 VPC 대역에서 172.30.1.0/24 대역을 사용 중 ip -br -c addr 2 # EKS Node Viewer 설치 : 현재 ec2 spec에서는 설치에 다소 시간이 소요됨 (3분 걸림) go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest 3 # [터미널1] bin 확인 및 사용 tree ~/go/bin cd ~/go/bin ./eks-node-viewer -h ./eks-node-viewer # EKS 배포 완료 후 실행 하자 <4> 환경 변수 확인 1 aws configure ap-northeast-2 # 환경변수 정보 확인 export CLUSTER_NAME=myeks2 export | egrep 'ACCOUNT|AWS_|CLUSTER' | egrep -v 'SECRET|KEY' # 환경변수 설정 export KARPENTER_VERSION=v0.27.5 export TEMPOUT=$(mktemp) echo $KARPENTER_VERSION $CLUSTER_NAME $AWS_DEFAULT_REGION $AWS_ACCOUNT_ID $TEMPOUT v0.27.5 myeks2 ap-northeast-2 476286675138 /tmp/tmp.QOU73aDEKI // 5개가 다 나와야 함. 2 # CloudFormation 스택으로 IAM Policy, Role, EC2 Instance Profile 생성 : 5분 정도 소요 curl -fsSL https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-karpenter/cloudformation.yaml > $TEMPOUT && aws cloudformation deploy --stack-name "Karpenter-${CLUSTER_NAME}" --template-file "${TEMPOUT}" --capabilities CAPABILITY_NAMED_IAM --parameter-overrides "ClusterName=${CLUSTER_NAME}" Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - Karpenter-myeks2 3 # 클러스터 생성 : myeks2 EKS 클러스터 생성 20분 정도 소요 eksctl create cluster -f - < 95s v1.24.13-eks-0a3 ip-192-168-91-59.ap-northeast-2.compute.internal Ready 98s v1.24.13-eks-0a2 eksctl get cluster NAME REGION EKSCTL CREATED myeks2 ap-northeast-2 True eksctl get nodegroup --cluster $CLUSTER_NAME eksctl get iamidentitymapping --cluster $CLUSTER_NAME eksctl get iamserviceaccount --cluster $CLUSTER_NAME eksctl get addon --cluster $CLUSTER_NAME 5 # [터미널1] eks-node-viewer cd ~/go/bin && ./eks-node-viewer 6 # k8s 확인 kubectl cluster-info kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone kubectl get pod -n kube-system -owide kubectl describe cm -n kube-system aws-auth 7 # 카펜터 설치를 위한 환경 변수 설정 및 확인 export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)" export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter" echo $CLUSTER_ENDPOINT $KARPENTER_IAM_ROLE_ARN 8 # EC2 Spot Fleet 사용을 위한 service-linked-role 생성 확인 : 만들어있는것을 확인하는 거라 아래 에러 출력이 정상! # If the role has already been successfully created, you will see: # An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForEC2Spot has been taken in this account, please try a different suffix. aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || true # docker logout : Logout of docker to perform an unauthenticated pull against the public ECR docker logout public.ecr.aws 9 # karpenter 설치 helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} --set settings.aws.clusterName=${CLUSTER_NAME} --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} --set settings.aws.interruptionQueueName=${CLUSTER_NAME} --set controller.resources.requests.cpu=1 --set controller.resources.requests.memory=1Gi --set controller.resources.limits.cpu=1 --set controller.resources.limits.memory=1Gi --wait 10 # 확인 kubectl get-all -n karpenter kubectl get all -n karpenter kubectl get cm -n karpenter karpenter-global-settings -o jsonpath={.data} | jq kubectl get crd | grep karpenter <5> 옵션 : ExternalDNS, kube-ops-view 유틸 watch로 모니터링하자. 1 # ExternalDNS MyDomain=<자신의 도메인> echo "export MyDomain=<자신의 도메인>" >> /etc/profile MyDomain=masterseo1.link echo "export MyDomain=masterseo1.link" >> /etc/profile MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text) echo $MyDomain, $MyDnzHostedZoneId curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f - 2 # kube-ops-view helm repo add geek-cookbook https://geek-cookbook.github.io/charts/ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set env.TZ="Asia/Seoul" --namespace kube-system kubectl patch svc -n kube-system kube-ops-view -p '{"spec":{"type":"LoadBalancer"}}' kubectl annotate service kube-ops-view -n kube-system "external-dns.alpha.kubernetes.io/hostname=kubeopsview.$MyDomain" echo -e "Kube Ops View URL = http://kubeopsview.$MyDomain:8080/#scale=1.5" 8080 포트로 접속 하세요 <6> Create Provisioner 제공자 (정책) Provisioner라는 K8S 커스텀 리소스가 ASG 를 대체한다. 오토스케일링 그룹을 대체한다. kind Provisioner 시작 템플릿이 없다. 보안 그룹과 서브넷은 필수이다. 태그를 기반으로 적용하는 방식이 있다. 1 # 프로비저너를 하나 만들어보자 cat < Add optional monitoring with Grafana (옵션) 1 https://grafana.com/grafana/dashboards/?search=karpenter 2 # helm repo add grafana-charts https://grafana.github.io/helm-charts helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring 3 # 프로메테우스 설치 curl -fsSL https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-karpenter/prometheus-values.yaml | tee prometheus-values.yaml helm install --namespace monitoring prometheus prometheus-community/prometheus --values prometheus-values.yaml --set alertmanager.enabled=false 4 # 그라파나 설치 curl -fsSL https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-karpenter/grafana-values.yaml | tee grafana-values.yaml helm install --namespace monitoring grafana grafana-charts/grafana --values grafana-values.yaml --set service.type=LoadBalancer 5 # admin 암호 kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo # 그라파나 접속 kubectl annotate service grafana -n monitoring "external-dns.alpha.kubernetes.io/hostname=grafana.$MyDomain" echo -e "grafana URL = http://grafana.$MyDomain" ----------- grafana URL = http://grafana.masterseo1.link admin 추출한 암호 6 # pause 파드 1개에 CPU 1개 최소 보장 할당. pod가 빠르게 올라갔나 내려간다. cat < 2m37s v1.24.13-eks-0a21954 spot c5a.2xlarge ip-192-168-57-91.ap-northeast-2.compute.internal Ready 13m v1.24.13-eks-0a21954 ON_DEMAND m5.large ip-192-168-75-253.ap-northeast-2.compute.internal Ready 13m v1.24.13-eks-0a21954 ON_DEMAND m5.large 8 Scale down deployment : ttlSecondsAfterEmpty 30초 # Now, delete the deployment. // After 30 seconds (ttlSecondsAfterEmpty), Karpenter should terminate the now empty nodes. kubectl delete deployment inflate kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller 30초 기다리다, 아무일도 안하네.. 삭제 함. <8> Consolidation 강화하다.통합하다. 1 https://www.eksworkshop.com/docs/autoscaling/compute/karpenter/consolidation/ https://aws.amazon.com/ko/blogs/containers/optimizing-your-kubernetes-compute-costs-with-karpenter-consolidation/ https://ec2spotworkshops.com/karpenter/050_karpenter/consolidation.html 2 # 기존의 정책은 삭제한다. kubectl delete provisioners default // cat < 97s v1.24.13-eks- ip-192-168-191-216.ap-northeast-2.compute.internal Ready 97s v1.24.13-eks- ip-192-168-79-228.ap-northeast-2.compute.internal Ready 97s v1.24.13-eks- ip-192-168-81-155.ap-northeast-2.compute.internal Ready 97s v1.24.13-eks- kubectl get node --label-columns=eks.amazonaws.com/capacityType,karpenter.sh/capacity-type kubectl get node --label-columns=node.kubernetes.io/instance-type,topology.kubernetes.io/zone node가 빠르게 증설된다 !! 5 줄여보자 12개인데 5개로 줄이면? # Next, scale the number of replicas back down to 5: kubectl scale deployment inflate --replicas 5 # The output will show Karpenter identifying specific nodes to cordon, drain and then terminate: kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller 6 1개로 줄이면??? 불필요한거 줄이고, 합친다. 스펙도 자동 조정한다. 2개 뜨는거보다 1개로 성능 좋은 pod로 뜨게 한다. 자동 조정한다. 비용 효율적으로 운영하는데 도움을 준다. # Next, scale the number of replicas back down to 1 kubectl scale deployment inflate --replicas 1 kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller 7 # 인스턴스 확인 kubectl get node -l type=karpenter kubectl get node --label-columns=eks.amazonaws.com/capacityType,karpenter.sh/capacity-type kubectl get node --label-columns=node.kubernetes.io/instance-type,topology.kubernetes.io/zone 8 # 삭제 kubectl delete deployment inflate kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller 빠르게 다운된다. <9> 실습 리소스 삭제 1 # kubectl delete svc -n monitoring grafana helm uninstall -n kube-system kube-ops-view helm uninstall karpenter --namespace karpenter 2 # 위 삭제 완료 후 아래 삭제 aws ec2 describe-launch-templates --filters Name=tag:karpenter.k8s.aws/cluster,Values=${CLUSTER_NAME} | jq -r ".LaunchTemplates[].LaunchTemplateName" | xargs -I{} aws ec2 delete-launch-template --launch-template-name {} 3 # 클러스터 삭제 export CLUSTER_NAME=myeks2 eksctl delete cluster --name "${CLUSTER_NAME}" 4 # aws cloudformation delete-stack --stack-name "Karpenter-${CLUSTER_NAME}" # 위 삭제 완료 후 아래 삭제 aws cloudformation delete-stack --stack-name ${CLUSTER_NAME}