Karpenter V1 업그레이드 및 마이그레이션

Upgrade Procedure

Configure environment variables for the cluster you’re upgrading:

export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export KARPENTER_NAMESPACE=kube-system
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
Shell
복사

Determine your current Karpenter version:

kubectl get deployment -A -l app.kubernetes.io/name=karpenter -ojsonpath="{.items[0].metadata.labels['app\.kubernetes\.io/version']}{'\n'}"
Shell
복사

To upgrade to v1, you must be running a Karpenter version between v0.33 and v0.37. If you are on an older version, you must upgrade before continuing with this guide.

Before upgrading to v1, we’re going to upgrade to a patch release that supports rollback. Set the KARPENTER_VERSION environment variable to the latest patch release for your current minor version. The following releases are the current latest:

•

0.37.7

•

0.36.9

•

0.35.12

•

v0.34.13

•

v0.33.12

# Note: v0.33.x and v0.34.x include the v prefix, omit it for versions v0.35+
export KARPENTER_VERSION="0.37.7" # Replace with your minor version
Shell
복사

Upgrade Karpenter to the latest patch release for your current minor version. Note that webhooks must be enabled.

# Service account annotation can be dropped when using pod identity
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
  --set settings.clusterName=${CLUSTER_NAME} \
  --set settings.interruptionQueue=${CLUSTER_NAME} \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --set webhook.enabled=true \
  --set webhook.port=8443 \
  --wait
Shell
복사

Apply the latest patch version of your current minor version’s Custom Resource Definitions (CRDs). Applying this version of the CRDs will enable the use of both the v1 and v1beta1 APIs on this version via the conversion webhooks. Note that this is only for rollback purposes, and new features available with the v1 APIs will not work on your minor version.

helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
    --set webhook.enabled=true \
    --set webhook.serviceName="karpenter" \
    --set webhook.port=8443
Shell
복사

Note
To properly template the conversion field in the CRD, the karpenter-crd chart must be used. If you’re using a GitOps solution to manage your Karpenter installation, you should use this chart to manage your CRDs. You should set skipCrds to true for the main karpenter chart (e.g. Argo CD).
Alternatively, you can install the CRDs with the main chart and apply the following patches. However, we strongly recommend using the dedicated CRD chart.

SERVICE_NAME="karpenter"
SERVICE_NAMESPACE="kube-system"
SERVICE_PORT="8443"
CRDS=("nodepools.karpenter.sh" "nodeclaims.karpenter.sh" "ec2nodeclasses.karpenter.k8s.aws")
for crd in ${CRDS[@]}; do
    kubectl patch customresourcedefinitions ${crd} --patch-file=/dev/stdin <<-EOF
spec:
  conversion:
    webhook:
      clientConfig:
        service:
          name: "${SERVICE_NAME}"
          namespace: "${SERVICE_NAMESPACE}"
          port: ${SERVICE_PORT}
EOF
done
Shell
복사

Note
Helm uses annotations on resources it provisions to track ownership. Switching to the new chart may result in Helm failing to install the chart due to invalid ownership metadata. If you encounter errors at this step, consult this troubleshooting entry to resolve.

Validate that Karpenter is operating as expected on this patch release. If you need to rollback after upgrading to v1, this is the version you will need to rollback to.

Note
The conversion webhooks must be able to communicate with the API server to operate correctly. If you see errors related to the conversion webhooks, ensure that your security groups and network policies allow traffic between the webhooks and the API server.

We’re now ready to begin the upgrade to v1. Set the KARPENTER_VERSION environment variable to the latest v1.0.x release.

export KARPENTER_VERSION="1.0.9"
Shell
복사

Attach the v1 policy to your existing NodeRole. Notable Changes to the IAM Policy include additional tag-scoping for the eks:eks-cluster-name tag for instances and instance profiles. We will remove this additional policy later once the controller has been migrated to v1 and we’ve updated the Karpenter cloudformation stack.

POLICY_DOCUMENT=$(mktemp)curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/13d6fc014ea59019b1c3b1953184efc41809df11/website/content/en/v1.0/upgrading/get-controller-policy.sh | sh | envsubst > ${POLICY_DOCUMENT}
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1"
ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_ARN="$(aws iam create-policy --policy-name "${POLICY_NAME}" --policy-document "file://${POLICY_DOCUMENT}" | jq -r .Policy.Arn)"
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
Shell
복사

Apply the v1 Custom Resource Definitions (CRDs):

helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
    --set webhook.enabled=true \
    --set webhook.serviceName="karpenter" \
    --set webhook.port=8443
Shell
복사

10.

Upgrade Karpenter to the latest v1.0.x release.

# Service account annotion can be dropped when using pod identity
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
  --set settings.clusterName=${CLUSTER_NAME} \
  --set settings.interruptionQueue=${CLUSTER_NAME} \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait
Shell
복사

Note
Karpenter has deprecated and moved a number of Helm values as part of the v1 release. Ensure that you upgrade to the newer version of these helm values during your migration to v1. You can find detail for all the settings that were moved in the v1 Upgrade Reference.

Note
Karpenter versions 0.32.x through 0.37.x have a number of mutating webhooks and validating webhooks that are not present in 1.0.x; if you are only using helm to generate manifests and are not using it to deploy them, you must clean up these webhook configurations by hand: specifically, you will need to remove the following resources:
    ◦ defaulting.webhook.karpenter.k8s.aws
    ◦ validation.webhook.karpenter.sh
    ◦ validation.webhook.config.karpenter.sh
    ◦ validation.webhook.karpenter.k8s.aws

11.

Upgrade your cloudformation stack and remove the temporary v1 controller policy.

TEMPOUT=$(mktemp)curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}"
aws cloudformation deploy \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file "${TEMPOUT}" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"

ROLE_NAME="${CLUSTER_NAME}-karpenter"
POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1"
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='${POLICY_NAME}'].Arn" --output text)
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"
Shell
복사

번역

업그레이드하는 클러스터에 대한 환경 변수 구성:  

현재 Karpenter 버전 확인:  

v1로 업그레이드하려면 v0.33~v0.37 사이의 Karpenter 버전을 실행해야 합니다. 이전 버전을 사용하는 경우 이 가이드를 계속 진행하기 전에 업그레이드해야 합니다.

v1로 업그레이드하기 전에 롤백을 지원하는 패치 릴리스로 업그레이드합니다. KARPENTER_VERSION 환경 변수를 현재 마이너 버전의 최신 패치 릴리스로 설정합니다. 다음 릴리스가 현재 최신 릴리스입니다.  

Karpenter를 현재 마이너 버전의 최신 패치 릴리스로 업그레이드합니다. 웹훅을 활성화해야 합니다.  

현재 마이너 버전의 사용자 지정 리소스 정의(CRD)의 최신 패치 버전을 적용합니다. 이 버전의 CRD를 적용하면 변환 웹훅을 통해 이 버전에서 v1 및 v1beta1 API를 모두 사용할 수 있습니다. 이것은 롤백 목적으로만 사용되며, v1 API에서 사용할 수 있는 새로운 기능은 마이너 버전에서는 작동하지 않습니다.

Karpenter가 이 패치 릴리스에서 예상대로 작동하는지 확인합니다. v1로 업그레이드한 후 롤백해야 하는 경우 이 버전으로 롤백해야 합니다.  

이제 v1로 업그레이드를 시작할 준비가 되었습니다. KARPENTER_VERSION 환경 변수를 최신 v1.0.x 릴리스로 설정합니다.  

기존 NodeRole에 v1 정책을 연결합니다. IAM 정책의 주요 변경 사항에는 인스턴스 및 인스턴스 프로필에 대한 eks:eks-cluster-name 태그에 대한 추가 태그 범위가 포함됩니다. 

컨트롤러가 v1로 마이그레이션되고 Karpenter cloudformation 스택이 업데이트되면 이 추가 정책을 나중에 제거합니다.

v1 사용자 지정 리소스 정의(CRD)를 적용합니다.  

10.

Karpenter를 최신 v1.0.x 릴리스로 업그레이드합니다.  

11.

cloudformation 스택을 업그레이드하고 임시 v1 컨트롤러 정책을 제거합니다.