Debugging “webidentityerr” Error in AWS LB

When you use the AWS Load Balancer Controller in Amazon EKS, you might see the following error:

"failed to find existing LoadBalancer due to WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403"

or it could be this:

"error":"WebIdentityErr: failed to retrieve credentials\ncaused by: ValidationError: Request ARN is invalid\n\tstatus code: 400, request id: 05f00b10-0e7f-4db8-9d9e-797ebf5eadfd"}

The error occurs for the following reasons:

• Incorrect service account configurations
• Incorrect trust relationship of the AWS Identity and Access Management (IAM) role that you use in the service account

Resolution:

a. Incorrect service account configurations:

1. Verify the service account name that’s defined in your deployment:

kubectl describe deploy <Name of load balancer controller> -n kube-system | grep -i "Service Account"

• • In the above namespace used is kube-system which is the default namespace, incase you’ve changed the namespace of the LB controller the same needs to be changed in the above command too.

2. Describe the service account:

kubectl describe sa <Name of load balancer controller> -n kube-system

3. Verify the service account annotation for the IAM role:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/<AMAZON_EKS_LOAD_BALANCER_CONTROLLER_ROLE>

4. If this annotation is missing or incorrect, then update the annotation. Make sure that you properly associated the IAM role to a service account:

kubectl annotate serviceaccount -n SERVICE_ACCOUNT_NAMESPACE  SERVICE_ACCOUNT_NAME \ eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME

b. Incorrect trust relationship between the IAM role used and service account:

1. The IAM role or trust relationship isn’t properly defined for the “sts:AssumeRoleWithWebIdentity” action

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC-PROVIDER_ID"
},
"Action": "sts:AssumeRoleWithWebIdentity", #<<<< This action is important
"Condition": {
"StringEquals": {
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"
}
}
}
]
}

2. In case the same IAM needs to be used for multiple clusters separate “Action”: “sts:AssumeRoleWithWebIdentity” needs to be present for the clusters:

{
"Version":  "2012-10-17",
"Statement":  [
{
"Effect":  "Allow",
"Principal":  {
"Federated":  "arn:aws:iam::AWS-ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC-PROVIDER_ID"
},
"Action":  "sts:AssumeRoleWithWebIdentity",
"Condition":  {
"StringEquals":  {
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub":  "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud":  "sts.amazonaws.com"
}
}
},
{
"Effect":  "Allow",
"Principal":  {
"Federated":  "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID"
},
"Action":  "sts:AssumeRoleWithWebIdentity",
"Condition":  {
"StringEquals":  {
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:sub":  "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
"oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:aud":  "sts.amazonaws.com"
}
}
}
]
}

The other common mistakes are:

a. Incorrect OIDC provider ID when creating an Amazon EKS cluster: Properly create and verify an OpenID Connect (OIDC) provider for your Amazon EKS cluster. Verify that the OIDC provider ID and the associated AWS Region are correctly listed

b. Service account name or its namespace not correctly entered

References:

1. Installing AWS controller add-on
2. Creating an IAM OIDC provider for your cluster.
3. Kubernetes service and ingress resources.
4. AWS troubleshooting guide