Have your Lambda Functions Connect to RDS through RDS Proxy

Overview

I recently went through the task of setting up an RDS Proxy for a new serverless architecture I was building. From an IaC (Infrastructure as Code) perspective, I found the setup to be a little tricky, with quite a few odds and ends. Since many of the resources available to help with this task are not complete solutions, I thought it would be useful for others to see the entire step-by-step process. I use Terraform in the example code, but I’m sure it will still be useful to you if you are using CloudFormation or any other IaC solution.

What is RDS Proxy?

RDS Proxy is a database connection pool designed to handle and scale many concurrent connections. Essentially, it is AWS’s serverless response to Lambda not being able to utilize a standard connection pool, like the one available in the mysql2 library.

Problems Without a Connection Pool

Consider a case without RDS Proxy: you have Lambda integrations with API Gateway and an RDS instance. We know that each Lambda invocation creates a new instance if none are available, and each instance creates a new database connection. Instead of reusing available connections from a pool, we are inefficiently eating away at a finite resource. In general, the formula for determining your max connections for a MySQL /MariaDB instance is DBInstanceClassMemory/12582880. A t2.micro has 512MB of memory available or 512000000B, meaning that this database instance can support a maximum of approximately 40 connections.

The Benefits

• The connection pool and reuse of connections reduce CPU and memory overhead and improve scalability.
• Automatically throttles incoming connections to avoid failure.
• The RDS Proxy infrastructure is highly available (multiple AZ’s) with compute, memory, and storage independent from your database instance or cluster.
• Handles database failover without impacting current connections.
• Better security by connecting to the proxy through TLS /IAM and storing database credentials in Secrets Manager.

Overall, RDS Proxy adds efficiency to your serverless architecture without having to scale up. As a result, using services like API Gateway, Lambda, Aurora, and RDS Proxy will give you a fully serverless solution.

Sample Code

If I am working with a service for the first time and need to write it as code, I usually like to set it up through the console first to get a better idea of how it works. This gives me a better idea of which arguments will be available before I approach the Terraform docs. This AWS blog/tutorial is useful for working through that:

Using Amazon RDS Proxy with AWS Lambda | Amazon Web Services

Step 1: Setup a secret in Secrets Manager

The secret holds our database credentials and connection details. Note that this assumes we have already created anaws_db_instance resource or similar (the rds-aurora module is popular for Aurora) named database.

rds-proxy-secrets-manager.tf:

resource "aws_secretsmanager_secret" "rds_secret" {
  name_prefix = "rds-proxy-secret"
  recovery_window_in_days = 7
  description = "Secret for RDS Proxy"
}

resource "aws_secretsmanager_secret_version" "rds_secret_version" {
  secret_id     = aws_secretsmanager_secret.rds_secret.id
  secret_string = jsonencode({
    "username"             = "my_username"
    "password"             = "my_password"
    "engine"               = "mysql"
    "host"                 = aws_db_instance.database.address
    "port"                 = 3306
    "dbInstanceIdentifier" = aws_db_instance.database.id
  })
}

This uses the default AWS managed KMS key named aws/secretsmanager. I also assume we are working with MySQL rather than Postgres (but RDS proxy supports both).

Step 2: Setup the Necessary Security Rules

The easiest way to set this up is to have a single security group for each of our resources (Lambda, RDS Proxy, and RDS).

• sg_lambda sends requests to sg_rds_proxy. I will leave this outbound (egress) rule open.
• sg_rds_proxy only accepts inbound (ingress) traffic from sg_lambda on port 3306/TCP.
• sg_rds only accepts inbound traffic from sg_rds_proxy on port 3306/TCP.

In general, you can leave outbound rules open. Note that this assumes our Lambda functions will sit inside a vpc and subnet.

rds-proxy-sg-rules.tf:

resource "aws_security_group" "sg_lambda" {
  vpc_id      = module.some_vpc.vpc_id

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
  }
}

resource "aws_security_group" "sg_rds_proxy" {
  vpc_id      = module.some_vpc.vpc_id

  ingress {
    description      = "MySQL TLS from sg_lambda"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    security_groups  = [aws_security_group.sg_lambda.id]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
  }
}

 resource "aws_security_group" "sg_rds" {
  vpc_id      = module.some_vpc.vpc_id

  ingress {
    description      = "MySQL TLS from sg_rds_proxy"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    security_groups  = [aws_security_group.sg_rds_proxy.id]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
  }
}

A common pitfall here is giving the proxy and the database the same security group. This will result in the proxy target being unavailable because that security group does not give access to itself.

Step 3: Setup the Necessary IAM Policies and IAM Role

Next, we have to make sure that RDS Proxy has permission to get and decrypt the database credentials from the Secrets Manager. In this example, it is accomplished by assuming the RDS service role and attaching the necessary policies to it. The actionsts.AssumeRole gives RDS Proxy the same role as RDS so it can perform the same functions.

rds-proxy-policies.tf:

data "aws_iam_policy_document" "assume_role" {

  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["rds.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "rds_proxy_policy_document" {

  statement {
    sid = "AllowProxyToGetDbCredsFromSecretsManager"

    actions = [
      "secretsmanager:GetSecretValue"
    ]

    resources = [
      aws_secretsmanager_secret.rds_secret.arn
    ]
  }

  statement {
    sid = "AllowProxyToDecryptDbCredsFromSecretsManager"

    actions = [
      "kms:Decrypt"
    ]

    resources = [
      "*"
    ]

    condition {
      test     = "StringEquals"
      values   = ["secretsmanager.${var.my_aws_region}.amazonaws.com"]
      variable = "kms:ViaService"
    }
  }
}

resource "aws_iam_policy" "rds_proxy_iam_policy" {
  name   = "rds-proxy-policy"
  policy = data.aws_iam_policy_document.rds_proxy_policy_document.json
}

resource "aws_iam_role_policy_attachment" "rds_proxy_iam_attach" {
  policy_arn = aws_iam_policy.rds_proxy_iam_policy.arn
  role       = aws_iam_role.rds_proxy_iam_role.name
}

resource "aws_iam_role" "rds_proxy_iam_role" {
  name               = "rds-proxy-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

Notice that I left the resources set to all for the action kms:Decrypt. This will work, but of course, we want to give the least amount of privilege necessary. If you used a CMK (Customer Managed Key) for rds_secret, use the output arn from that resource. If you left at as default (AWS managed), then you can use something like data aws_kms_key to get the arn.

Step 4: Setup the actual Proxy along with its Group and Target

The target group associated with the proxy controls the settings related to the database connection.

rds-proxy.tf:

resource "aws_db_proxy_default_target_group" "rds_proxy_target_group" {
  db_proxy_name = aws_db_proxy.db_proxy.name

  connection_pool_config {
    connection_borrow_timeout = 120
    max_connections_percent = 100
  }
}

resource "aws_db_proxy_target" "rds_proxy_target" {
  db_instance_identifier = aws_db_instance.database.id
  db_proxy_name          = aws_db_proxy.db_proxy.name
  target_group_name      = aws_db_proxy_default_target_group.rds_proxy_target_group.name
}

resource "aws_db_proxy" "db_proxy" {
  debug_logging          = false
  engine_family          = "MYSQL"
  idle_client_timeout    = 1800
  require_tls            = true
  role_arn               = aws_iam_role.rds_proxy_iam_role.arn
  vpc_security_group_ids = [aws_security_group.sg_rds_proxy.id]
  vpc_subnet_ids         = module.some_vpc.my_database_subnets

  auth {
    auth_scheme = "SECRETS"
    iam_auth    = "REQUIRED"
    secret_arn  = aws_secretsmanager_secret.rds_secret.arn
  }
}

The best practice is to keep the RDS Proxy in the same subnet as the database. Make sure that all the services (Lambda, Proxy, and RDS) are set up with the security groups we created. That’s pretty much it, you can test the connection by running the javascript sample code provided in the AWS blog linked above.

Step 5: Give Lambda Permission to Access RDS Proxy

One last thing to note is that in order for Lambda to have permission to access RDS Proxy, its role needs to be modified with the correct policy. If you set up Lambda manually (i.e. through Add Database Proxy) and check out the role afterward, you will see that the action is rds-db:connect and the resource arn is set up in the format:

`arn:aws:rds-db:${region}:${awsAccountId}:dbuser:${proxyId}/*`

So for example, if your proxy arn is arn:aws:rds:us-east-1:424824108755:db-proxy:prx-1126716789abcdef07, then the resource arn for the policy should bearn:aws:rds-db:us-east-1:424824108755:dbuser:prx-1126716789abcdef07.

Further Reading

How to Integrate Proxy with Python Requests