EKS Anywhere: Benefits, Challenges, and Real-World Insights

Hello there! In this blog, I’ll share my hands-on experience with AWS’s on-premise Kubernetes solution, EKS Anywhere. I’ll cover key comparisons, practical insights, and useful tricks that can help you decide whether EKS Anywhere is the right choice for your environment. My goal is to turn this into a blog series as time allows, and if I publish more articles related to EKS Anywhere, I’ll list them at the end of this post.

To start, let me share a personal challenge: transitioning from RKE2 to EKS Anywhere was a true “change my mind” moment for me. While EKS has been my go-to choice for managed Kubernetes in the public cloud, my on-premise preference had long been RKE2. However, after extensively working with EKS Anywhere in a live environment, I’ve gained valuable insights that shifted my perspective.

A few years ago, I even wrote a comparison between EKS Anywhere and RKE2: 👉 EKS Anywhere vs. RKE2: Kubernetes Distribution Comparison for On-Premises

That said, a lot has changed in the past three years. Running EKS Anywhere in production, preparing for deployment, and troubleshooting real-world challenges have all given me a much deeper understanding of its capabilities and limitations.

So let’s make a very general introduction to the subject;

Again: What is EKS Anywhere?

EKS Anywhere is Amazon Web Services’ on-premises Kubernetes solution, designed to bring the benefits of Amazon EKS (Elastic Kubernetes Service) to your own data center or private cloud. It allows organizations to deploy, manage, and operate Kubernetes clusters outside of AWS, while still leveraging AWS tooling and best practices.

Unlike fully managed EKS in the cloud, EKS Anywhere gives you complete control over your cluster infrastructure, networking, and security policies, making it a strong choice for organizations prioritizing data control, low-latency workloads, or hybrid cloud strategies. It is particularly well-suited for VMware vSphere environments, but also supports bare-metal deployments with additional configuration.

By adopting EKS Anywhere, teams can achieve a consistent Kubernetes experience across on-premise and AWS environments, benefiting from EKS-compatible tooling, Cluster API-based lifecycle management, and built-in automation. However, as with any self-managed Kubernetes solution, it also comes with operational responsibilities and infrastructure considerations, which I’ll explore in the upcoming sections.

A ‘Semi-Managed’ Kubernetes Experience On-Premises

You’ve containerized your workloads and microservices, set up Kubernetes, and successfully scaled your pods. You’re already benefiting from Kubernetes’ elasticity, but now you want to take it a step further. However, managing upgrades, node scaling, right-sizing, and security hardening in an on-premises environment is increasing your operational overhead. If this sounds familiar, you’re in the right place.

The good news? You can transform your on-prem Kubernetes environment into a ‘semi-managed’ platform. In this section, we’ll dive into immutable nodes, declarative cluster management with manifests, Cluster API (CAPI), automated provisioning, and more — all of which help streamline operations and reduce management complexity.

Is EKS Anywhere the Only Option?

Of course not! Red Hat OpenShift and VMware Tanzu are two strong competitors in the on-prem Kubernetes space. In fact, you could think of EKS Anywhere as their quiet yet powerful rival.

Having used OpenShift extensively in production environments and experimented with Tanzu mostly in sandbox setups, I know that directly comparing them wouldn’t be entirely fair — each has its own strengths and weaknesses. That said, today’s focus is on EKS Anywhere and what it brings to the table.

Why Should I Switch to EKS Anywhere? Or Should I?

For years, we successfully met our Kubernetes needs with RKE2, EKS-Distro images, and custom configurations. However, as operational requirements evolved, so did the need for a different approach.

If we talk about RKE2, it offers a simple installation with a single service, straightforward configuration, and “meh” documentation (most solutions require digging through GitHub issues and correlating multiple discussions to solve a problem). Despite these challenges, RKE2 is an effective solution that can sustain many environments throughout their entire lifecycle.

A quick side note: If you’re considering setting up Kubernetes using kubeadm by following the official Kubernetes documentation — stop right there! RKE2 is a much simpler and more practical alternative.

So, what made us reconsider RKE2 and move to EKS Anywhere? To answer that, let’s first look at our infrastructure setup before migrating to EKS Anywhere:

Before EKS Anywhere: Our Kubernetes Landscape

• We had hard-tenant, independent Kubernetes clusters across different environments and projects.
• Our monitoring setup was configured to federate data from cluster-level sources to a central monitoring system.
• Each cluster required individual ArgoCD management.
• Scaling a cluster meant manually adding nodes via Ansible.
• Cluster upgrades became a recurring operational challenge.
• Istio service mesh was in use across all clusters.
• Logging and application traces were centralized but stored outside the clusters.

What Did EKS Anywhere Improve?

• Easier Cluster Lifecycle Management: As our cluster count grew, provisioning and deleting clusters became seamless. We set up workload clusters under a central management cluster, leveraging Cluster API (CAPI). Managing active clusters became as simple as running: “*kubectl get cluster” *With CAPV, we provisioned workloads in our VMware environment without needing a Terraform + Ansible stack. The only essential artifact to keep was our cluster’s YAML configuration file.
• Centralized Monitoring, GitOps & Logging: We merged monitoring, GitOps, and logging into a centralized management cluster and dedicated a separate Kubernetes cluster for standalone applications previously running on VMs. (I won’t get into the numerous optimizations we made on the monitoring side — that’s for another post! 😃)
• Simplified Service Mesh with Istio: We defined service entries for inter-cluster communication.
• ArgoCD’s “App-of-Apps” Model Became Fully Functional:
• New project?
• Apply the cluster configuration
• Add it as a destination in ArgoCD
• Deploy the standartized stack
• Within minutes, your platform, logging, monitoring, CD pipelines, and applications are up *and running.
• Scaling Became More Flexible:
• Scaling a cluster? Easier than ever.
• Need to resize VMs? Just update the VsphereMachineConfig.
• Most importantly, no dependency on specific VMs — everything is immutable.
• If someone manually tweaks a VM without documentation, it won’t cause future problems — that VM will simply be replaced in the next update.
• Replaced HAProxy with MetalLB. Instead of an external load balancer, we now leverage MetalLB within the cluster.

What Became More Challenging?

• Managing vSphere Was a Hassle:
• Immutable VMs prevent manual interventions — every change must be made at the template (OVA) level.
• Properly managing template tags is crucial.
• Standalone ETCD Nodes Were Mandatory:
• Even for a sandbox cluster, ETCD nodes are required, sometimes leading to wasted resources (mostly should be)
• Limited CNI Options:
• Fewer choices compared to other Kubernetes distributions.
• Documentation Issues (The Biggest Drawback!):
• While the documentation looks comprehensive, it’s not intuitive.
• Identifying problems is easy, but solving them often isn’t.
• Fragile Management Cluster:
• If you’re using separate management and workload clusters, keep the management cluster isolated and well-backed-up.
• No Customization on Worker Nodes:
• Almost all forms of worker node customization are restricted. (I will talk about the GPU based EKS Anywhere cluster we created for Kubeflow in a different article :) )

So, Does It Make Our Lives Easier?

I could give a political answer to this: Out of the box, no. With customizations, absolutely yes.

EKS Anywhere’s default OVA images and configurations might cause issues in the long run — unless that’s exactly what you’re looking for. For example, you can use Bottlerocket images as ready-to-use OVA templates. However, keep in mind that Bottlerocket is a hardened, minimal image with no root user access. If you’re not a fan of RPM-based distributions (like me), Bottlerocket may not be the best option. But if security is your top priority and you don’t want to spend time hardening an OS yourself, then Bottlerocket is perfectly designed for your needs.

On the other hand, creating a custom Ubuntu 22.04 or 24.04 image is fairly straightforward. With AWS’s image-builder tool, you can generate a custom Ubuntu image with just a little effort.

Challenges in Air-Gapped Environments

If you’re working in an air-gapped environment with no internet access, things get trickier. AWS provides a list of public resources that EKS Anywhere needs to access — your bootstrap cluster VM must be able to reach these addresses. The installation process also differs for air-gapped setups.

Some pain points I’ve encountered:

• Defining a custom image registry can be frustrating. For example, when configuring Harbor, you may run into hardcoded ports that take too long to adjust.
• Static ETCD download scripts can break things in an air-gapped setup. The ETCD nodes rely on hardcoded scripts to fetch binaries, which can be a real headache in offline environments.

CAPI & CAPV: Handle with Care

Throughout your cluster’s lifecycle, you need to manage CAPI and CAPV carefully. A small mistake — especially one that affects the management cluster — can bring down your entire Kubernetes setup.

Let’s say you break your EKS Anywhere management cluster in production.

• You’ve tried everything.
• You’ve searched GitHub issues, but found no solution.
• You need to roll back ASAP.

At this point, backups are your best friend — but they’re often overlooked during installation. Although EKS Anywhere documentation mentions backups, they are easy to miss.

By default, every time you modify a kind:cluster object, the configuration is backed up on the administrative machine running the bootstrap cluster. But is this enough? Absolutely not.

Backup Strategy for Disaster Recovery

• You must automate ETCD backups — this is mentioned in the official docs, but many users skip it.
• If you need to replace the management cluster, you must restore from an ETCD backup.
• Good news: After restoring, you can re-import existing workload clusters into the new management cluster.

Conclusion

EKS Anywhere is a solid Kubernetes distribution for on-premise environments. However, customizing it might be frustrating at times. If you have prior experience with Tanzu or OpenShift, the setup process should be fairly smooth. Depending on your use case, I would recommend it in certain scenarios.

However, if you have a small-scale setup — for example, if you are migrating from standalone Docker-based applications to Kubernetes — EKS Anywhere is not the best choice.

What’s Next?

In my upcoming posts, I will share:

• Step-by-step deployment insights based on real-world actions we took.
• How we integrated Kubeflow into our environment.
• How we configured NVIDIA GPU-powered VMs for workloads.

Thank you for taking the time to read! If you have any questions, feel free to reach out on LinkedIn.