AWS EKS BottleRocket Nodes

First published at the PerfectScale blog. PerfectScale helps you ensure peak performance and cut costs by up to 50% with data-driven, autonomous actions that continuously optimize each layer of your K8s stack. Start your 30-day free trial now!

AWS BottleRocket is an open-source Linux distribution. Think of it like the Alpine of AMI’s (Amazon Machine Image). It is intended to provide a minimal operating system on which to run containerized workloads. The benefits of AWS EKS BottleRocket address some of the most fundamental issues present in the default Amazon Linux 2 distribution, which you are provided by default when you create an EKS (Elastic Kubernetes Service) Cluster on AWS.

Having your EKS workloads on BottleRocket essentially solves many problems just by its OS structure alone, such as improved security and resource usage. The scenario referenced in this article is the creation of an EKS cluster with a BottleRocket node group from scratch, or the migration of an existing one.

What are the benefits of AWS BottleRocket?

AWS BottleRocket is a purpose-built, Linux-based operating system built for running containers. It includes all the necessary software required to run containers while ensuring the underlying software is always secure. In short, it provides the following advantages:

• Increased uptime for container applications by changing the system update mechanism
• Ability to manage the host by an actual HTTP API, interchangeable with TOML configuration file used to define a BottleRocket instance.
• Open-source development model enables custom builds. As you know, errors cannot slip under the radar as easily as enterprise software, and it allows for full control of customization of the source code..
• Lower management overhead and operational costs. This is done by supporting EKS automatic updates.
• Improved security and resource utilization. Fewer packages bloating the OS means fewer resources needed and less attack surface.
• Optimized performance through AWS integrations. Built-in features inserted by AWS allow it to easily integrate with other AWS services.
• 3 years of support, including community support in the open-source Github project. Who doesn’t love support?

There are, however, some hidden inconveniences explained later in this article that only a true implementer of AWS EKS BottleRocket would encounter. These include the OS difference between BottleRocket as a Linux distribution as opposed to other distributions, and its impact on Kubernetes as a resource allocator. After showing the method of installation, I’ll address them.

AWS BottleRocket Installation Guide

Getting Started

First, you’re going to have to define an EC2 Launch Template, which is an AWS resource that will supply the template upon which your EKS cluster will create BottleRocket OS EC2 Nodes. For this example, we will use Terraform, as it is a popular IaC tool for deploying EKS clusters. As for the AWS EKS BottleRocket configuration file, it will also be addressed in this section.

For starters, provide your Terraform code with a reference to the latest (or specific) AWS BottleRocket AMI version, which is present in the AWS AMI catalog available for creating EC2 instances. We can retrieve the Amazon Machine Image (AMI) ID for Amazon EKS optimized AMIs by querying the AWS Systems Manager Parameter Store API. Using this parameter, you don’t need to manually look up Amazon EKS optimized AMI IDs:

locals {
  eks_version = "1.28"
}

data "aws_ssm_parameter" "bottlerocket_image_id" {
  name = "/aws/service/bottlerocket/aws-k8s-${local.eks_version}/x86_64/latest/image_id"
}

data "aws_ami" "bottlerocket_image" {
  owners = ["amazon"]
  filter {
    name   = "image-id"
    values = [data.aws_ssm_parameter.bottlerocket_image_id.value]
  }
}

Now that we got the value of the AMI’s Image ID, we can use it to create a launch template. I’ve also added some default settings, to save you time from ruffling through the docs:

resource "aws_eks_cluster" "eks-cluster" {
  ...
}

resource "aws_launch_template" "eks-general-launch-template" {

  image_id               = data.aws_ami.bottlerocket_image.id

  instance_type          = ["c6a.xlarge"]
  name                   = "my-bottlerocket-launch-template"
  update_default_version = true

  # A Basic EBS Volume
  block_device_mappings {
    device_name = "/dev/xvdb"
    ebs {
      volume_type = "gp3"
      volume_size = 20
    }
  }

  # Included Tags for cluster-autoscaler
  tag_specifications {
    resource_type = "instance"
    tags = {
      "Name"                                                          = "my-bottlerocket-launch-template"
      "kubernetes.io/cluster/${aws_eks_cluster.eks-cluster.name}"     = "owned"
      "k8s.io/cluster-autoscaler/${aws_eks_cluster.eks-cluster.name}" = "owned"
      "k8s.io/cluster-autoscaler/enabled"                             = "true"
    }
  }

  # Private Instance
  network_interfaces {
    associate_public_ip_address = false
  }

  # Enables IMDSv2, EC2 Instance Metadata Security Update
  metadata_options {
    http_tokens                 = "required"
    http_put_response_hop_limit = "2"
  }

  # BottleRocket Configuration File. See Below This Code Block.
  user_data = base64encode(templatefile("${path.module}/config.toml",
    {
      "cluster_name"             = aws_eks_cluster.eks-cluster.name
      "endpoint"                 = aws_eks_cluster.eks-cluster.endpoint
      "cluster_auth_base64"      = aws_eks_cluster.eks-cluster.certificate_authority[0].data
      "aws_region"               = "us-east-1"
      "enable_admin_container"   = false
      "enable_control_container" = true
    }
  ))
}

Great! We’ve got an EKS Cluster, a Launch Template, and now we are missing an EKS node group to actually use this launch template:

resource "aws_iam_role" "my-node-group-role" {
 ...
}

resource "aws_eks_node_group" "my_node_group" {

  cluster_name    = aws_eks_cluster.eks-cluster.name
  
  node_group_name = "my_node_group"

  node_role_arn   = aws_iam_role.my-node-group-role.arn

  subnet_ids      = local.aws_subnets

  # Basic 1 to 2 Autoscaling
  scaling_config {
    desired_size = 1
    min_size     = 1
    max_size     = 2
  }

  # Reference to Our Launch Template
  launch_template {
    id      = aws_launch_template.my-bottlerocket-launch-template.id
    version = "$Latest"
  }
}

The Configuration File

The BottleRocket configuration file is written in TOML format. It contains some basic configuration options and allows for tweaking endless knobs regarding the operating system. The variables in curly braces are provided via Terraform templating.

# https://github.com/bottlerocket-os/bottlerocket/blob/develop/README.md#description-of-settings
[settings.kubernetes]
api-server = "${endpoint}"
cluster-certificate = "${cluster_auth_base64}"
cluster-name = "${cluster_name}"

# Hardening based on https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md

# Enable kernel lockdown in "integrity" mode.
# This prevents modifications to the running kernel, even by privileged users.
[settings.kernel]
lockdown = "integrity"

# The admin host container provides SSH access and runs with "superpowers".
# It is disabled by default, but can be disabled explicitly.
[settings.host-containers.admin]
enabled = ${enable_admin_container}

# The control host container provides out-of-band access via SSM.
# It is enabled by default, and can be disabled if you do not expect to use SSM.
# This could leave you with no way to access the API and change settings on an existing node!
[settings.host-containers.control]
enabled = ${enable_control_container}

The most important settings in the file are whether to enable a control container and an admin container. AWS BottleRocket does not have SSH access, so there are two mechanisms to connect it via terminal:

1. A Control Container. Allows for SSM (AWS Systems Manager) connection from the AWS console, which does this through the container without interacting with the infamously problematic SSH port.
2. An Admin Container. If you do require a direct SSH connection, it’s done through an Admin Container that runs on the machine and provides access through it. This is a risky option and should be evaluated a per-need basis.

It is also important to note that BottleRocket OS allows for multiple kernel security settings, which provide elevated access. The integrity is recommended, however, you could go even deeper with the confidentiality mode, which will block even eBPF software. Only use this if you’re very serious about your security and know what you’re doing.

A major difference between Amazon Linux and BottleRocket

There is an extremely specific Linux operating system flag, which has an insanely dramatic effect on containerization. It is the version of the component named cgroups (control groups) — a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, etc) of a collection of processes. This is the kernel feature that allows Linux to isolate container workloads, and it is being used by container engines such as containerd, which are the engine behind Kubernetes versions 1.24 and up.

Currently, there are two versions of cgroups. cgroup v1 and cgroup v2.

As of writing this article, the default cgroups version of Amazon Linux 2 AMI is v1, while AWS BottleRocket has v2 as the default version.

This might be critical to the stability of your workloads. Without getting into too much technical detail, the algorithm with which Linux (and Kubernetes) calculates the usage of resources, especially Memory, is different between the versions.

The practical implication is that Kubernetes itself might report different CPU/memory usage as to what you’ve seen your application use before the migration. Most of us will be upgrading from Amazon Linux 2 to AWS BottleRocket, so we will automatically receive cgroup v2. If we do not adjust our Kubernetes requests/limits accordingly, our pods might get OOM, get over/under-provisioned, or just lose stability. For some, readjusting your workload Requests and Limits would be passable, but sometimes that means provisioning more nodes for the same amount of previous workload, which might be unacceptable.

(I’ve personally noticed this behavior on NodeJS workloads)

If you want to prevent this from happening, or return to cgruop v1, there is a solution. You’ll have to tweak the kernel settings of the AWS BottleRocket AMI to force the usage of v1. For this, you’ll have to add the following settings to the aforementioned config.toml (0 = 1, 1 = 2):

[settings.boot]
reboot-to-reconcile = true

[settings.boot.init-parameters]
"systemd.unified_cgroup_hierarchy" = ["0"]

Now you’re all set to experience a clean setup of the BottleRocket-based EKS cluster. AWS BottleRocket is being actively developed by the open-source community but is perfectly safe for Production use, as long as necessary testing is done beforehand. Cheers!‍

Amitai G is a freelance DevOps consultant with an extensive background in DevOps practices and Kubernetes’ ongoing maintenance and operations. As the author of the @elementtech.dev Medium channel, he specializes in writing articles that provide engineers with a deeper understanding and best practice guidance across the various aspects of the Cloud Native landscape.