Message brokers play a crucial role in modern software architecture by enabling **asynchronous communication, decoupling services, and improving system scalability**. They help applications exchange data efficiently and ensure reliability in distributed systems.

If you’re a developer looking to integrate message brokers into your applications, here are the **top 5 message brokers** you should know.

### 1️⃣ RabbitMQ — The Feature-Rich Message Broker 🐰

**RabbitMQ** is a widely used message broker that supports **multiple messaging protocols** and is known for its **reliability, scalability, and ease of use**.

### ✅ Why Use RabbitMQ?

* **Supports AMQP (Advanced Message Queuing Protocol)**, making it highly compatible.
* **Ensures message delivery** with features like acknowledgments and persistence.
* **Supports complex routing** with exchanges and queues.
* **Scales easily** with clustering and high availability configurations.

### 🚀 How to Get Started?

Install RabbitMQ on Linux:

```
# Install RabbitMQsudo apt updatesudo apt install rabbitmq-server# Enable and start the servicesudo systemctl enable rabbitmq-serversudo systemctl start rabbitmq-server
```

[**Spring Boot RabbitMQ Producer and Consumer Example**](https://www.javaguides.net/2022/07/spring-boot-rabbitmq-producer-and-consumer-example.html)

[**Spring Boot RabbitMQ Send and Receive JSON Messages**](https://www.javaguides.net/2022/07/spring-boot-rabbitmq-send-and-receive-json-messages.html)

[**Spring Boot RabbitMQ Multiple Queues Example**](https://www.javaguides.net/2022/07/spring-boot-rabbitmq-multiple-queues.html)

[**How RabbitMQ Works and RabbitMQ Core Concepts**](http://www.javaguides.net/2018/12/how-rabbitmq-works-and-rabbitmq-core-concepts.html)

Check out my Udemy course to learn Spring Boot RabbitMQ — [**Spring Boot + RabbitMQ (Includes Event-Driven Microservices)**](https://www.udemy.com/course/spring-boot-rabbitmq-course/?referralCode=597B5A8827E03AB85417)

### 2️⃣ Apache Kafka — The High-Throughput Streaming Platform 📊

**Apache Kafka** is an event streaming platform designed for **high-performance data pipelines, real-time analytics, and event-driven applications**.

### ✅ Why Use Kafka?

* **Handles high-throughput data streams efficiently**.
* **Distributed, fault-tolerant, and scalable architecture**.
* **Supports real-time processing with stream processing frameworks**.
* **Ideal for event-driven microservices architectures**.

### 🚀 How to Get Started?

Install Kafka and start a broker:

```
# Install Kafka (Linux)wget https://downloads.apache.org/kafka/latest/kafka_2.13-3.0.0.tgztar -xvzf kafka_2.13-3.0.0.tgzcd kafka_2.13-3.0.0# Start Zookeeper (Kafka dependency)bin/zookeeper-server-start.sh config/zookeeper.properties# Start Kafka brokerbin/kafka-server-start.sh config/server.properties
```

**Tutorials on Kafka:**

[**Spring Boot Apache Kafka Tutorial**](https://www.javaguides.net/2022/06/spring-boot-apache-kafka-tutorial.html)

[**Spring Boot Kafka Producer and Consumer Example Tutorial**](https://www.javaguides.net/2022/05/spring-boot-kafka-producer-consumer-example-tutorial.html)

[**Spring Boot Kafka JsonSerializer and JsonDeserializer Example**](https://www.javaguides.net/2022/05/spring-boot-kafka-jsonserializer-and-Jsondeserializer-example.html)

YouTube Complete playlist on Apache Kafka: [**https://youtube.com/playlist?list=PLGRDMO4rOGcNLwoack4ZiTyewUcF6y6BU**](https://youtube.com/playlist?list=PLGRDMO4rOGcNLwoack4ZiTyewUcF6y6BU)

### 3️⃣ NATS — The Lightweight and Fast Message Broker 🚀

**NATS** is a high-performance messaging system designed for **cloud-native applications, IoT, and microservices communication**.

### ✅ Why Use NATS?

* **Super lightweight and easy to deploy**.
* **Supports at-most-once, at-least-once, and exactly-once messaging**.
* **Designed for high-speed and low-latency communications**.
* **Auto-scaling and self-healing capabilities**.

### 🚀 How to Get Started?

Install and run NATS:

```
# Install NATS serverwget https://github.com/nats-io/nats-server/releases/latest/download/nats-server-linux-amd64.zipunzip nats-server-linux-amd64.zipchmod +x nats-server# Start NATS server./nats-server
```

### 4️⃣ IBM MQ — The Enterprise Messaging Standard 🏢

**IBM MQ** is a powerful enterprise message broker used by large organizations for **high-volume, secure, and reliable messaging**.

### ✅ Why Use IBM MQ?

* **Ensures secure and reliable message delivery**.
* **Works across multiple platforms including cloud, mainframes, and hybrid environments**.
* **Supports transaction-based messaging for business-critical applications**.
* **Integrates with various enterprise applications like SAP and Salesforce**.

### 🚀 How to Get Started?

Install and configure IBM MQ:

```
# Install IBM MQ (Trial Version)wget https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqadv/mqadv_dev902_linux_x86-64.tar.gztar -xvzf mqadv_dev902_linux_x86-64.tar.gzcd MQServer# Run installation scriptsudo ./mqlicense.sh -acceptsudo ./install-mq-server.sh
```

### 5️⃣ ActiveMQ — The Enterprise-Grade Message Broker 🏢

**Apache ActiveMQ** is a powerful, **Java-based message broker** widely used in **enterprise applications** for its rich features and flexibility.

### ✅ Why Use ActiveMQ?

* **Supports multiple protocols** (AMQP, MQTT, STOMP, JMS, and WebSockets).
* **Provides robust message durability and failover mechanisms**.
* **Ideal for large-scale applications with enterprise-level messaging needs**.
* **Supports message prioritization, scheduling, and clustering**.

### 🚀 How to Get Started?

Install and start ActiveMQ:

```
# Download and extract ActiveMQwget https://archive.apache.org/dist/activemq/5.17.0/apache-activemq-5.17.0-bin.tar.gztar -xvzf apache-activemq-5.17.0-bin.tar.gzcd apache-activemq-5.17.0# Start ActiveMQ./bin/activemq start
```

### 🎯 Conclusion

Message brokers are essential for **handling asynchronous communication, processing large-scale data, and enabling event-driven applications**. Choosing the right message broker depends on your **use case, performance needs, and scalability requirements**.

### Quick Recap:

✔ **RabbitMQ** — Great for traditional messaging with AMQP support.

✔ **Kafka** — Best for high-throughput real-time data streaming.

✔ **NATS** — Lightweight and scalable for cloud-native applications.

✔ **IBM MQ** — Enterprise messaging standard with high reliability.

✔ **ActiveMQ** — Enterprise-grade broker with multiple protocol support.

Start exploring these message brokers and integrate them into your applications to build **robust, scalable, and efficient systems!** 🚀

Original article: [Top 5 Message Brokers Every Developer Should Know](https://www.javaguides.net/2024/12/top-5-message-brokers-every-developer-should-know.html)

Message brokers play a crucial role in modern software architecture by enabling asynchronous communication, decoupling services, and improving system scalability. They help applications exchange data efficiently and ensure reliability in distributed systems.
If you’re a developer looking to integrate message brokers into your applications, here are the top 5 message brokers you should know.
<h3 id="1️⃣-rabbitmqthe-feature-rich-message-broker">1️⃣ RabbitMQ — The Feature-Rich Message Broker 🐰</h3>
RabbitMQ is a widely used message broker that supports multiple messaging protocols and is known for its reliability, scalability, and ease of use.
<h3 id="-why-use-rabbitmq">✅ Why Use RabbitMQ?</h3>
<ul>
<li>Supports AMQP (Advanced Message Queuing Protocol), making it highly compatible.</li>
<li>Ensures message delivery with features like acknowledgments and persistence.</li>
<li>Supports complex routing with exchanges and queues.</li>
<li>Scales easily with clustering and high availability configurations.</li>
</ul>
<h3 id="-how-to-getstarted">🚀 How to Get Started?</h3>
Install RabbitMQ on Linux:
<pre><code class="code-highlight"># Install RabbitMQsudo apt updatesudo apt install rabbitmq-server# Enable and start the servicesudo systemctl enable rabbitmq-serversudo systemctl start rabbitmq-server
</code></pre>
<a href="https://www.javaguides.net/2022/07/spring-boot-rabbitmq-producer-and-consumer-example.html">Spring Boot RabbitMQ Producer and Consumer Example</a>
<a href="https://www.javaguides.net/2022/07/spring-boot-rabbitmq-send-and-receive-json-messages.html">Spring Boot RabbitMQ Send and Receive JSON Messages</a>
<a href="https://www.javaguides.net/2022/07/spring-boot-rabbitmq-multiple-queues.html">Spring Boot RabbitMQ Multiple Queues Example</a>
<a href="http://www.javaguides.net/2018/12/how-rabbitmq-works-and-rabbitmq-core-concepts.html">How RabbitMQ Works and RabbitMQ Core Concepts</a>
Check out my Udemy course to learn Spring Boot RabbitMQ — <a href="https://www.udemy.com/course/spring-boot-rabbitmq-course/?referralCode=597B5A8827E03AB85417">Spring Boot + RabbitMQ (Includes Event-Driven Microservices)</a>
<h3 id="2️⃣-apache-kafkathe-high-throughput-streaming-platform">2️⃣ Apache Kafka — The High-Throughput Streaming Platform 📊</h3>
Apache Kafka is an event streaming platform designed for high-performance data pipelines, real-time analytics, and event-driven applications.
<h3 id="-why-usekafka">✅ Why Use Kafka?</h3>
<ul>
<li>Handles high-throughput data streams efficiently.</li>
<li>Distributed, fault-tolerant, and scalable architecture.</li>
<li>Supports real-time processing with stream processing frameworks.</li>
<li>Ideal for event-driven microservices architectures.</li>
</ul>
<h3 id="-how-to-getstarted-1">🚀 How to Get Started?</h3>
Install Kafka and start a broker:
<pre><code class="code-highlight"># Install Kafka (Linux)wget https://downloads.apache.org/kafka/latest/kafka_2.13-3.0.0.tgztar -xvzf kafka_2.13-3.0.0.tgzcd kafka_2.13-3.0.0# Start Zookeeper (Kafka dependency)bin/zookeeper-server-start.sh config/zookeeper.properties# Start Kafka brokerbin/kafka-server-start.sh config/server.properties
</code></pre>
Tutorials on Kafka:
<a href="https://www.javaguides.net/2022/06/spring-boot-apache-kafka-tutorial.html">Spring Boot Apache Kafka Tutorial</a>
<a href="https://www.javaguides.net/2022/05/spring-boot-kafka-producer-consumer-example-tutorial.html">Spring Boot Kafka Producer and Consumer Example Tutorial</a>
<a href="https://www.javaguides.net/2022/05/spring-boot-kafka-jsonserializer-and-Jsondeserializer-example.html">Spring Boot Kafka JsonSerializer and JsonDeserializer Example</a>
YouTube Complete playlist on Apache Kafka: <a href="https://youtube.com/playlist?list=PLGRDMO4rOGcNLwoack4ZiTyewUcF6y6BU">https://youtube.com/playlist?list=PLGRDMO4rOGcNLwoack4ZiTyewUcF6y6BU</a>
<h3 id="3️⃣-natsthe-lightweight-and-fast-message-broker">3️⃣ NATS — The Lightweight and Fast Message Broker 🚀</h3>
NATS is a high-performance messaging system designed for cloud-native applications, IoT, and microservices communication.
<h3 id="-why-usenats">✅ Why Use NATS?</h3>
<ul>
<li>Super lightweight and easy to deploy.</li>
<li>Supports at-most-once, at-least-once, and exactly-once messaging.</li>
<li>Designed for high-speed and low-latency communications.</li>
<li>Auto-scaling and self-healing capabilities.</li>
</ul>
<h3 id="-how-to-getstarted-2">🚀 How to Get Started?</h3>
Install and run NATS:
<pre><code class="code-highlight"># Install NATS serverwget https://github.com/nats-io/nats-server/releases/latest/download/nats-server-linux-amd64.zipunzip nats-server-linux-amd64.zipchmod +x nats-server# Start NATS server./nats-server
</code></pre>
<h3 id="4️⃣-ibm-mqthe-enterprise-messaging-standard">4️⃣ IBM MQ — The Enterprise Messaging Standard 🏢</h3>
IBM MQ is a powerful enterprise message broker used by large organizations for high-volume, secure, and reliable messaging.
<h3 id="-why-use-ibmmq">✅ Why Use IBM MQ?</h3>
<ul>
<li>Ensures secure and reliable message delivery.</li>
<li>Works across multiple platforms including cloud, mainframes, and hybrid environments.</li>
<li>Supports transaction-based messaging for business-critical applications.</li>
<li>Integrates with various enterprise applications like SAP and Salesforce.</li>
</ul>
<h3 id="-how-to-getstarted-3">🚀 How to Get Started?</h3>
Install and configure IBM MQ:
<pre><code class="code-highlight"># Install IBM MQ (Trial Version)wget https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqadv/mqadv_dev902_linux_x86-64.tar.gztar -xvzf mqadv_dev902_linux_x86-64.tar.gzcd MQServer# Run installation scriptsudo ./mqlicense.sh -acceptsudo ./install-mq-server.sh
</code></pre>
<h3 id="5️⃣-activemqthe-enterprise-grade-message-broker">5️⃣ ActiveMQ — The Enterprise-Grade Message Broker 🏢</h3>
Apache ActiveMQ is a powerful, Java-based message broker widely used in enterprise applications for its rich features and flexibility.
<h3 id="-why-use-activemq">✅ Why Use ActiveMQ?</h3>
<ul>
<li>Supports multiple protocols (AMQP, MQTT, STOMP, JMS, and WebSockets).</li>
<li>Provides robust message durability and failover mechanisms.</li>
<li>Ideal for large-scale applications with enterprise-level messaging needs.</li>
<li>Supports message prioritization, scheduling, and clustering.</li>
</ul>
<h3 id="-how-to-getstarted-4">🚀 How to Get Started?</h3>
Install and start ActiveMQ:
<pre><code class="code-highlight"># Download and extract ActiveMQwget https://archive.apache.org/dist/activemq/5.17.0/apache-activemq-5.17.0-bin.tar.gztar -xvzf apache-activemq-5.17.0-bin.tar.gzcd apache-activemq-5.17.0# Start ActiveMQ./bin/activemq start
</code></pre>
<h3 id="-conclusion">🎯 Conclusion</h3>
Message brokers are essential for handling asynchronous communication, processing large-scale data, and enabling event-driven applications. Choosing the right message broker depends on your use case, performance needs, and scalability requirements.
<h3 id="quick-recap">Quick Recap:</h3>
✔ RabbitMQ — Great for traditional messaging with AMQP support.
✔ Kafka — Best for high-throughput real-time data streaming.
✔ NATS — Lightweight and scalable for cloud-native applications.
✔ IBM MQ — Enterprise messaging standard with high reliability.
✔ ActiveMQ — Enterprise-grade broker with multiple protocol support.
Start exploring these message brokers and integrate them into your applications to build robust, scalable, and efficient systems! 🚀
Original article: <a href="https://www.javaguides.net/2024/12/top-5-message-brokers-every-developer-should-know.html">Top 5 Message Brokers Every Developer Should Know</a>

Top 5 Message Brokers Every Developer Should Know

![](https://cdn-images-1.medium.com/max/1600/1*XMf_cR1ST_NpeJrjNq0R-w.png)

### The Problem

You’ve built a nice little API using AWS API Gateway - serving as a web-hook endpoint for an external Order Management System.
It takes orders from your web app, saves them to a database, and calls a few downstream services.
It runs through this stack:
 **API Gateway → Lambda → Database**.

It works perfectly on a normal day.

Then one day…

* Your marketing team runs a big sale.
* Or a partner’s integration script gets stuck in a loop.
* Or a new feature upstream pushes thousands of requests by mistake.

Suddenly you’re facing an **event storm -** a huge, unexpected spike in requests hitting your API all at once.

### What’s an Event Storm?

An **event storm** is when the number of incoming requests in a short time is way higher than your system normally handles.
 It’s not a slow increase. It’s a flood.

**Example triggers:**

* Black Friday or Diwali sale on your e-commerce site.
* A cron job gone rogue sending thousands of API calls per second.
* A botnet hammering your public endpoint.
* A sudden surge in user signups after media coverage.

**Quick note on Lambda scaling:** Lambda doesn’t go from 0 → 100K concurrent executions instantly. Each region has a burst concurrency limit (for example \~3000 in `us-east-1`) and then scales up by about 500–1000 per minute.
So when I say “100K requests hit,” I mean the **inbound spike hitting API Gateway**, not the actual concurrency Lambda reached. The scaling ramp-up actually makes the need for a buffer more obvious -SQS can absorb the full burst immediately, while Lambda drains at the pace it can realistically scale to.

### Why It’s a Problem in AWS Lambda

AWS Lambda scales fast, which sounds great until you realise:

* Every Lambda execution counts toward your account concurrency limit.
* If you hit that limit, all other Lambdas in your account may start failing too.
* Downstream systems (like a database) may crash from too many parallel requests.

With API Gateway directly invoking Lambda, an event storm can burn through concurrency in seconds.

### The Safer Approach

We’ll redesign the flow so that:

* API Gateway sends requests to **SQS** instead of Lambda directly.
* If SQS accepts it, API Gateway returns **202 Accepted** with an **ack id** (SQS `MessageId`).

```
HTTP/1.1 202 Accepted{"status":"accepted","messageId":"c1c7...","receivedAt":1735296623123}
```

* Processing happens later via the consumer Lambda.
* Lambda reads from SQS in **controlled batches** and upserts into database
* Concurrency is **capped** so we never overload the database.
* Failed messages go to a **DLQ** for later reprocessing.

This way, the **storm is buffered** in SQS, and you can process at a steady pace without losing data.

### Architecture Overview

1. **API Gateway** -> receives incoming HTTP requests.
2. **SQS (Standard Queue)** -> buffers all requests.
3. **Lambda Consumer ->** pulls messages from SQS in small batches.
4. **DLQ ->** stores failed messages for later replay.

**Don’t forget about caching at the API layer.**
If your API is serving repetitive data (like catalog lookups, product info, or configuration files), you can enable **API Gateway caching**. This stores responses in an in-memory cache so repeated requests don’t even reach Lambda or SQS.
In my case the requests were mostly unique and user-specific, so caching didn’t help much. But for many APIs, caching at API GW can drastically reduce Lambda concurrency and costs.

### Step 0: Recommended Starting Settings

You can tweak these later, but start here:

* **Lambda timeout:** 30s
* **Lambda reserved concurrency (***the max number of concurrent executions this function can ever use*)**:** 50
* **SQS batch size:** 10
* **Max batching window** (*how long Lambda waits to gather more records before invoking the function*)**:** 2s
* **SQS visibility timeout** (*the time a message stays hidden after being picked up, before it can be retried*)**:** 90s
* **DLQ maxReceiveCount (***the number of times a message can fail before being sent to the DLQ*)**:** 5

Why these help:

* Max parallel work = 50 (*Lambda reserved concurrency***)** × 10 (*SQS batch size***)** = **500 messages at once**.
* SQS buffers the rest during a spike.
* Visibility timeout prevents duplicate processing.
* DLQ ensures nothing is lost.

### Step 1: Create SQS and DLQ

**Create DLQ**

```
aws sqs create-queue --queue-name orders-dlq
```

**Create main queue with DLQ attached**

```
DLQ_URL=$(aws sqs get-queue-url --queue-name orders-dlq --query 'QueueUrl' --output text)DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $DLQ_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)aws sqs create-queue \ --queue-name orders \ --attributes '{ "RedrivePolicy":"{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"5\"}", "VisibilityTimeout":"90" }'
```

### Step 2: Connect API Gateway to SQS

#### Option A: Direct integration

* No Lambda in between.
* API Gateway sends request body directly to SQS.

**IAM role for API Gateway**

```
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["sqs:SendMessage"], "Resource": "arn:aws:sqs:<REGION>:<ACCOUNT_ID>:orders" }]}
```

#### Option B: Producer Lambda

If you need validation/auth before pushing to SQS:

```
// producer/index.mjsimport { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";const sqs = new SQSClient({});const QUEUE_URL = process.env.QUEUE_URL;export const handler = async (event) => { const body = JSON.parse(event.body || "{}"); if (!body.orderId) { return { statusCode: 400, body: JSON.stringify({ error: "orderId required" }) }; } await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify(body) })); return { statusCode: 202, body: JSON.stringify({ status: "accepted" }) };};
```

**API Gateway caching reminder:** If traffic includes repeated requests, enable API Gateway caching. It cuts load before it even reaches SQS or Lambda.

**Step 3: Create the Consumer Lambda**

```
// consumer/index.mjsexport const handler = async (event) => { for (const record of event.Records) { try { const msg = JSON.parse(record.body); await processOrder(msg); } catch (err) { console.error("Failed for messageId", record.messageId, err); throw err; // forces SQS to retry or send to DLQ } }};async function processOrder(msg) { await new Promise(r => setTimeout(r, 150)); // simulate work}
```

**Event Source Mapping**

* Batch size: 10
* Max batching window: 2s

**Reserved concurrency:** 50 (prevents overload)

### Step 3.1: Don’t kill your database connections

One hidden problem when Lambda talks to RDS: each Lambda instance may open a new DB connection. During a spike, you can exhaust DB connections fast.

The fix is to use **RDS Proxy**. It sits between Lambda and the database, pools connections, and reuses them across Lambda invocations. This keeps your database safe from connection storms.

Alternatives:

* For PostgreSQL/MySQL, some teams also run PgBouncer on ECS/Fargate.
* Or, for workloads with unpredictable spikes, consider DynamoDB which doesn’t have connection limits.

### Step 4: Tune for Storms

* **Visibility timeout** ≥ 3 × Lambda timeout.
* Do throughput math:

```
parallelism = reserved_concurrency × batch_sizerecords/sec ≈ parallelism / avg_processing_seconds
```

### Step 5: Alerts

**Queue depth alarm**

```
aws cloudwatch put-metric-alarm \ --alarm-name "orders-queue-depth-high" \ --metric-name ApproximateNumberOfMessagesVisible \ --namespace AWS/SQS \ --dimensions Name=QueueName,Value=orders \ --statistic Average \ --period 60 \ --threshold 50000 \ --comparison-operator GreaterThanThreshold \ --evaluation-periods 5 \ --alarm-actions <SNS_TOPIC_ARN>
```

**Oldest message age alarm ->** Warns you when messages have been waiting too long in the queue (in **seconds**).

```
aws cloudwatch put-metric-alarm \ --alarm-name "orders-oldest-message-too-old" \ --namespace "AWS/SQS" \ --metric-name "ApproximateAgeOfOldestMessage" \ --dimensions "Name=QueueName,Value=orders" \ --statistic "Maximum" \ --period 60 \ --threshold 600 \ # warn if oldest message > 10 minutes old --comparison-operator "GreaterThanThreshold" \ --evaluation-periods 5 \ # for 5 minutes if period=60 --treat-missing-data "notBreaching" \ --unit "Seconds" \ --alarm-actions "<SNS_TOPIC_ARN>"
```

### Step 6: DLQ Replay

Fix the bug → replay messages from DLQ to main queue with a small Lambda.

```
import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand, SendMessageCommand } from "@aws-sdk/client-sqs";const sqs = new SQSClient({});const DLQ_URL = process.env.DLQ_URL;const MAIN_URL = process.env.MAIN_URL;export const handler = async () => { const res = await sqs.send(new ReceiveMessageCommand({ QueueUrl: DLQ_URL, MaxNumberOfMessages: 10, WaitTimeSeconds: 1 })); if (!res.Messages) return; for (const m of res.Messages) { await sqs.send(new SendMessageCommand({ QueueUrl: MAIN_URL, MessageBody: m.Body })); await sqs.send(new DeleteMessageCommand({ QueueUrl: DLQ_URL, ReceiptHandle: m.ReceiptHandle })); }};
```

* Pick `THRESHOLD_SECONDS` well below your SQS **message retention** (default 4 days) but high enough to avoid noise. Start with 600-1800 seconds for most APIs, or align it with your SLA.
* Use `Maximum` (not `Average`) - you care about the single **oldest** message.
* Pair this with the **queue depth** alarm you already have.

### Important Pointers..

* **Don’t** let Lambda’s scaling run wild in a spike.
* **Do** buffer in SQS and process at a pace your system can handle.
* **Always** set a DLQ - losing data is worse than slow processing.
* **Monitor** queue depth and message age - that’s your early warning.
* **Use RDS Proxy** when Lambdas talk to relational databases.
* **Add API Gateway caching** if traffic is repetitive.

### **One last note,**

This whole design works best for **asynchronous APIs-**&#x70;laces where you can acknowledge a request and process it later.

But what if your API is **synchronous** and the client needs a response *right away*? In that case, queues won’t help. You need other tools like rate limiting, provisioned concurrency, and sometimes containers.

I wrote a follow-up on handling **synchronous API traffic spikes** [here](https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d):
[https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d](https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d)

Together, these two posts cover both sides of the coin:

* Async APIs → buffer with SQS.
* Sync APIs → throttle, pre-warm, or containerize.

### ☕️ Found This Helpful?

If this guide helped you debug a tricky front-end issue or saved you from a wild goose chase :
you can [buy me a coffee](http://buymeacoffee.com/subodhshet0)

### A message from our Founder

**Hey,** [**Sunil**](https://linkedin.com/in/sunilsandhu) **here.** I wanted to take a moment to thank you for reading until the end and for being a part of this community.

Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? **We don’t receive any funding, we do this to support the community. ❤️**

If you want to show some love, please take a moment to **follow me on** [**LinkedIn**](https://linkedin.com/in/sunilsandhu)**,** [**TikTok**](https://tiktok.com/@messyfounder), [**Instagram**](https://instagram.com/sunilsandhu). You can also subscribe to our [**weekly newsletter**](https://newsletter.plainenglish.io/).

And before you go, don’t forget to **clap** and **follow** the writer️!

<img src="https://cdn-images-1.medium.com/max/1600/1*XMf_cR1ST_NpeJrjNq0R-w.png" alt="">
<h3 id="the-problem">The Problem</h3>
You’ve built a nice little API using AWS API Gateway - serving as a web-hook endpoint for an external Order Management System.
It takes orders from your web app, saves them to a database, and calls a few downstream services.
It runs through this stack:
 API Gateway → Lambda → Database.
It works perfectly on a normal day.
Then one day…
<ul>
<li>Your marketing team runs a big sale.</li>
<li>Or a partner’s integration script gets stuck in a loop.</li>
<li>Or a new feature upstream pushes thousands of requests by mistake.</li>
</ul>
Suddenly you’re facing an event storm - a huge, unexpected spike in requests hitting your API all at once.
<h3 id="whats-an-eventstorm">What’s an Event Storm?</h3>
An event storm is when the number of incoming requests in a short time is way higher than your system normally handles.
 It’s not a slow increase. It’s a flood.
Example triggers:
<ul>
<li>Black Friday or Diwali sale on your e-commerce site.</li>
<li>A cron job gone rogue sending thousands of API calls per second.</li>
<li>A botnet hammering your public endpoint.</li>
<li>A sudden surge in user signups after media coverage.</li>
</ul>
Quick note on Lambda scaling: Lambda doesn’t go from 0 → 100K concurrent executions instantly. Each region has a burst concurrency limit (for example ~3000 in <code>us-east-1</code>) and then scales up by about 500–1000 per minute.
So when I say “100K requests hit,” I mean the inbound spike hitting API Gateway, not the actual concurrency Lambda reached. The scaling ramp-up actually makes the need for a buffer more obvious -SQS can absorb the full burst immediately, while Lambda drains at the pace it can realistically scale to.
<h3 id="why-its-a-problem-in-awslambda">Why It’s a Problem in AWS Lambda</h3>
AWS Lambda scales fast, which sounds great until you realise:
<ul>
<li>Every Lambda execution counts toward your account concurrency limit.</li>
<li>If you hit that limit, all other Lambdas in your account may start failing too.</li>
<li>Downstream systems (like a database) may crash from too many parallel requests.</li>
</ul>
With API Gateway directly invoking Lambda, an event storm can burn through concurrency in seconds.
<h3 id="the-saferapproach">The Safer Approach</h3>
We’ll redesign the flow so that:
<ul>
<li>API Gateway sends requests to SQS instead of Lambda directly.</li>
<li>If SQS accepts it, API Gateway returns 202 Accepted with an ack id (SQS <code>MessageId</code>).</li>
</ul>
<pre><code class="code-highlight">HTTP/1.1 202 Accepted{"status":"accepted","messageId":"c1c7...","receivedAt":1735296623123}
</code></pre>
<ul>
<li>Processing happens later via the consumer Lambda.</li>
<li>Lambda reads from SQS in controlled batches and upserts into database</li>
<li>Concurrency is capped so we never overload the database.</li>
<li>Failed messages go to a DLQ for later reprocessing.</li>
</ul>
This way, the storm is buffered in SQS, and you can process at a steady pace without losing data.
<h3 id="architecture-overview">Architecture Overview</h3>
<ol>
<li>API Gateway -> receives incoming HTTP requests.</li>
<li>SQS (Standard Queue) -> buffers all requests.</li>
<li>Lambda Consumer -> pulls messages from SQS in small batches.</li>
<li>DLQ -> stores failed messages for later replay.</li>
</ol>
Don’t forget about caching at the API layer.
If your API is serving repetitive data (like catalog lookups, product info, or configuration files), you can enable API Gateway caching. This stores responses in an in-memory cache so repeated requests don’t even reach Lambda or SQS.
In my case the requests were mostly unique and user-specific, so caching didn’t help much. But for many APIs, caching at API GW can drastically reduce Lambda concurrency and costs.
<h3 id="step-0-recommended-startingsettings">Step 0: Recommended Starting Settings</h3>
You can tweak these later, but start here:
<ul>
<li>Lambda timeout: 30s</li>
<li>Lambda reserved concurrency (the max number of concurrent executions this function can ever use): 50</li>
<li>SQS batch size: 10</li>
<li>Max batching window (how long Lambda waits to gather more records before invoking the function): 2s</li>
<li>SQS visibility timeout (the time a message stays hidden after being picked up, before it can be retried): 90s</li>
<li>DLQ maxReceiveCount (the number of times a message can fail before being sent to the DLQ): 5</li>
</ul>
Why these help:
<ul>
<li>Max parallel work = 50 (Lambda reserved concurrency**)** × 10 (SQS batch size**)** = 500 messages at once.</li>
<li>SQS buffers the rest during a spike.</li>
<li>Visibility timeout prevents duplicate processing.</li>
<li>DLQ ensures nothing is lost.</li>
</ul>
<h3 id="step-1-create-sqs-anddlq">Step 1: Create SQS and DLQ</h3>
Create DLQ
<pre><code class="code-highlight">aws sqs create-queue --queue-name orders-dlq
</code></pre>
Create main queue with DLQ attached
<pre><code class="code-highlight">DLQ_URL=$(aws sqs get-queue-url --queue-name orders-dlq --query 'QueueUrl' --output text)DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $DLQ_URL --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)aws sqs create-queue \ --queue-name orders \ --attributes '{ "RedrivePolicy":"{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"5\"}", "VisibilityTimeout":"90" }'
</code></pre>
<h3 id="step-2-connect-api-gateway-tosqs">Step 2: Connect API Gateway to SQS</h3>
<h4 id="option-a-direct-integration">Option A: Direct integration</h4>
<ul>
<li>No Lambda in between.</li>
<li>API Gateway sends request body directly to SQS.</li>
</ul>
IAM role for API Gateway
<pre><code class="code-highlight">{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["sqs:SendMessage"], "Resource": "arn:aws:sqs:&#x3C;REGION>:&#x3C;ACCOUNT_ID>:orders" }]}
</code></pre>
<h4 id="option-b-producerlambda">Option B: Producer Lambda</h4>
If you need validation/auth before pushing to SQS:
<pre><code class="code-highlight">// producer/index.mjsimport { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";const sqs = new SQSClient({});const QUEUE_URL = process.env.QUEUE_URL;export const handler = async (event) => { const body = JSON.parse(event.body || "{}"); if (!body.orderId) { return { statusCode: 400, body: JSON.stringify({ error: "orderId required" }) }; } await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify(body) })); return { statusCode: 202, body: JSON.stringify({ status: "accepted" }) };};
</code></pre>
API Gateway caching reminder: If traffic includes repeated requests, enable API Gateway caching. It cuts load before it even reaches SQS or Lambda.
Step 3: Create the Consumer Lambda
<pre><code class="code-highlight">// consumer/index.mjsexport const handler = async (event) => { for (const record of event.Records) { try { const msg = JSON.parse(record.body); await processOrder(msg); } catch (err) { console.error("Failed for messageId", record.messageId, err); throw err; // forces SQS to retry or send to DLQ } }};async function processOrder(msg) { await new Promise(r => setTimeout(r, 150)); // simulate work}
</code></pre>
Event Source Mapping
<ul>
<li>Batch size: 10</li>
<li>Max batching window: 2s</li>
</ul>
Reserved concurrency: 50 (prevents overload)
<h3 id="step-31-dont-kill-your-database-connections">Step 3.1: Don’t kill your database connections</h3>
One hidden problem when Lambda talks to RDS: each Lambda instance may open a new DB connection. During a spike, you can exhaust DB connections fast.
The fix is to use RDS Proxy. It sits between Lambda and the database, pools connections, and reuses them across Lambda invocations. This keeps your database safe from connection storms.
Alternatives:
<ul>
<li>For PostgreSQL/MySQL, some teams also run PgBouncer on ECS/Fargate.</li>
<li>Or, for workloads with unpredictable spikes, consider DynamoDB which doesn’t have connection limits.</li>
</ul>
<h3 id="step-4-tune-forstorms">Step 4: Tune for Storms</h3>
<ul>
<li>Visibility timeout ≥ 3 × Lambda timeout.</li>
<li>Do throughput math:</li>
</ul>
<pre><code class="code-highlight">parallelism = reserved_concurrency × batch_sizerecords/sec ≈ parallelism / avg_processing_seconds
</code></pre>
<h3 id="step-5alerts">Step 5: Alerts</h3>
Queue depth alarm
<pre><code class="code-highlight">aws cloudwatch put-metric-alarm \ --alarm-name "orders-queue-depth-high" \ --metric-name ApproximateNumberOfMessagesVisible \ --namespace AWS/SQS \ --dimensions Name=QueueName,Value=orders \ --statistic Average \ --period 60 \ --threshold 50000 \ --comparison-operator GreaterThanThreshold \ --evaluation-periods 5 \ --alarm-actions &#x3C;SNS_TOPIC_ARN>
</code></pre>
Oldest message age alarm -> Warns you when messages have been waiting too long in the queue (in seconds).
<pre><code class="code-highlight">aws cloudwatch put-metric-alarm \ --alarm-name "orders-oldest-message-too-old" \ --namespace "AWS/SQS" \ --metric-name "ApproximateAgeOfOldestMessage" \ --dimensions "Name=QueueName,Value=orders" \ --statistic "Maximum" \ --period 60 \ --threshold 600 \ # warn if oldest message > 10 minutes old --comparison-operator "GreaterThanThreshold" \ --evaluation-periods 5 \ # for 5 minutes if period=60 --treat-missing-data "notBreaching" \ --unit "Seconds" \ --alarm-actions "&#x3C;SNS_TOPIC_ARN>"
</code></pre>
<h3 id="step-6-dlqreplay">Step 6: DLQ Replay</h3>
Fix the bug → replay messages from DLQ to main queue with a small Lambda.
<pre><code class="code-highlight">import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand, SendMessageCommand } from "@aws-sdk/client-sqs";const sqs = new SQSClient({});const DLQ_URL = process.env.DLQ_URL;const MAIN_URL = process.env.MAIN_URL;export const handler = async () => { const res = await sqs.send(new ReceiveMessageCommand({ QueueUrl: DLQ_URL, MaxNumberOfMessages: 10, WaitTimeSeconds: 1 })); if (!res.Messages) return; for (const m of res.Messages) { await sqs.send(new SendMessageCommand({ QueueUrl: MAIN_URL, MessageBody: m.Body })); await sqs.send(new DeleteMessageCommand({ QueueUrl: DLQ_URL, ReceiptHandle: m.ReceiptHandle })); }};
</code></pre>
<ul>
<li>Pick <code>THRESHOLD_SECONDS</code> well below your SQS message retention (default 4 days) but high enough to avoid noise. Start with 600-1800 seconds for most APIs, or align it with your SLA.</li>
<li>Use <code>Maximum</code> (not <code>Average</code>) - you care about the single oldest message.</li>
<li>Pair this with the queue depth alarm you already have.</li>
</ul>
<h3 id="important-pointers">Important Pointers..</h3>
<ul>
<li>Don’t let Lambda’s scaling run wild in a spike.</li>
<li>Do buffer in SQS and process at a pace your system can handle.</li>
<li>Always set a DLQ - losing data is worse than slow processing.</li>
<li>Monitor queue depth and message age - that’s your early warning.</li>
<li>Use RDS Proxy when Lambdas talk to relational databases.</li>
<li>Add API Gateway caching if traffic is repetitive.</li>
</ul>
<h3 id="one-lastnote">One last note,</h3>
This whole design works best for asynchronous APIs-places where you can acknowledge a request and process it later.
But what if your API is synchronous and the client needs a response right away? In that case, queues won’t help. You need other tools like rate limiting, provisioned concurrency, and sometimes containers.
I wrote a follow-up on handling synchronous API traffic spikes <a href="https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d">here</a>:
<a href="https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d">https://medium.com/aws-in-plain-english/surviving-traffic-surges-in-sync-apis-rate-limits-warm-lambdas-and-smart-scaling-d04488ad94db?sk=6a2f4645f254fd28119b2f5ab263269d</a>
Together, these two posts cover both sides of the coin:
<ul>
<li>Async APIs → buffer with SQS.</li>
<li>Sync APIs → throttle, pre-warm, or containerize.</li>
</ul>
<h3 id="️-found-thishelpful">☕️ Found This Helpful?</h3>
If this guide helped you debug a tricky front-end issue or saved you from a wild goose chase :
you can <a href="http://buymeacoffee.com/subodhshet0">buy me a coffee</a>
<h3 id="a-message-from-ourfounder">A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community.
Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community. ❤️
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>.
And before you go, don’t forget to clap and follow the writer️!

How to Stop AWS Lambda from Melting When 100K Requests Hit at Once

![](https://cdn-images-1.medium.com/max/1600/1*d2N6_gZM39tZ6FYTVM0_5Q.jpeg)

[https://unsplash.com/photos/DZe1jK6pk5c](https://unsplash.com/photos/DZe1jK6pk5c)

#### Introduction

[S3 Deployment](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html) is a [CDK module](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html) from AWS (currently “experimental” during June 2021 at the time of writing)that allows populating an S3 bucket with the contents of .zip files from other S3 buckets or from a local disk.

***

#### TLDR

* You should use S3-Deployment only if you need to upload under 100 files
* **Never use S3-Deployment if you need to upload more 100 files/pages,** this will be most of the time the usual case for people.

***

#### Quick Example to see the main issue

This is what happens under the hood:

1. When this stack is deployed (either via `cdk deploy` or via CI/CD), the contents of the local `website-dist` directory will be archived and uploaded to an **intermediary assets bucket**. This will be the main issue if you have websites with hundreds of pages.
2. The `BucketDeployment` construct synthesizes a custom CloudFormation resource of type `Custom::CDKBucketDeployment` into the template. The source bucket/key is set to point to the assets bucket.
3. The custom resource downloads the .zip archive, extracts it and issues `aws s3 sync --delete` against the destination bucket (in this case `websiteBucket`).

***

#### The problem

The problem we get is that the intermediary bucket is not really needed and is mostly limited to copying the files 2 times!

If you have 5 files you won’t see the problem but with websites of hundreds or thousands of pages you will be stuck in a long deployment that will also be closed automatically after 15 minutes and that will do upload basically twice!

This is due to the fact that this CDK module under the hood uses a Lambda to move the files from the intermediary bucket to the destination bucket.

***

#### Alternatives or solutions?

1. **The Best solution ever:** just use the most simple command to upload files to S3 called “aws s3 cp” like this:

```
aws s3 cp DIRECTORY_OR_FILE s3://BUCKET_NAME — recursive — quiet
```

2\. **Second best choice:** use a Github action to upload again the files without remembering the previous S3 command: [https://github.com/shallwefootball/upload-s3-action](https://github.com/shallwefootball/upload-s3-action)

```
name: Upload to S3on: [pull_request]jobs:  upload:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@master      - uses: shallwefootball/s3-upload-action@master        with:          aws_key_id: ${{ secrets.AWS_KEY_ID }}          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY}}          aws_bucket: BUCKET_NAME          source_dir: 'DIRECTORY_OR_FILE'
```

3\. not suggested solution: increasing the memory of the Lambda with the memoryLimit parameter, from the default 128 to 1024, this might help but won’t make a difference with more than 200 files.

***

> ***💡*****Also learn how to efficiently store one million small objects to S3 with AWS Athena integration:**

[**How to write one million small objects to S3**
*Learn how to efficiently store one million small objects to S3 for your stock analytics app with AWS Athena…*&#x64;iffer.blog](https://differ.blog/p/how-to-write-one-million-small-objects-to-s3-c69e8c "https://differ.blog/p/how-to-write-one-million-small-objects-to-s3-c69e8c")

> *👉* To read more such articles, [**sign up for free on Differ**](https://differ.blog/).

***

#### Resources

* [docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html)
* [gist.github.com/riccardogiorato/428d015dee054d3cb9b06941851ab018](https://gist.github.com/riccardogiorato/428d015dee054d3cb9b06941851ab018)
* [docs.aws.amazon.com/cli/latest/reference/s3/cp.html](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html)
* [github.com/shallwefootball/upload-s3-action](https://github.com/shallwefootball/upload-s3-action)

#### If you want to help me out you can:

* **Join Medium Membership and get my future Articles:** [https://riccardogiorato.medium.com/membership](https://riccardogiorato.medium.com/membership)
* **Follow me on Twitter suggesting other Topics to cover:** [https://twitter.com/riccardogiorato](https://twitter.com/riccardogiorato)

<img src="https://cdn-images-1.medium.com/max/1600/1*d2N6_gZM39tZ6FYTVM0_5Q.jpeg" alt="">
<a href="https://unsplash.com/photos/DZe1jK6pk5c">https://unsplash.com/photos/DZe1jK6pk5c</a>
<h4 id="introduction">Introduction</h4>
<a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html">S3 Deployment</a> is a <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html">CDK module</a> from AWS (currently “experimental” during June 2021 at the time of writing)that allows populating an S3 bucket with the contents of .zip files from other S3 buckets or from a local disk.
<hr>
<h4 id="tldr">TLDR</h4>
<ul>
<li>You should use S3-Deployment only if you need to upload under 100 files</li>
<li>Never use S3-Deployment if you need to upload more 100 files/pages, this will be most of the time the usual case for people.</li>
</ul>
<hr>
<h4 id="quick-example-to-see-the-mainissue">Quick Example to see the main issue</h4>
This is what happens under the hood:
<ol>
<li>When this stack is deployed (either via <code>cdk deploy</code> or via CI/CD), the contents of the local <code>website-dist</code> directory will be archived and uploaded to an intermediary assets bucket. This will be the main issue if you have websites with hundreds of pages.</li>
<li>The <code>BucketDeployment</code> construct synthesizes a custom CloudFormation resource of type <code>Custom::CDKBucketDeployment</code> into the template. The source bucket/key is set to point to the assets bucket.</li>
<li>The custom resource downloads the .zip archive, extracts it and issues <code>aws s3 sync --delete</code> against the destination bucket (in this case <code>websiteBucket</code>).</li>
</ol>
<hr>
<h4 id="the-problem">The problem</h4>
The problem we get is that the intermediary bucket is not really needed and is mostly limited to copying the files 2 times!
If you have 5 files you won’t see the problem but with websites of hundreds or thousands of pages you will be stuck in a long deployment that will also be closed automatically after 15 minutes and that will do upload basically twice!
This is due to the fact that this CDK module under the hood uses a Lambda to move the files from the intermediary bucket to the destination bucket.
<hr>
<h4 id="alternatives-or-solutions">Alternatives or solutions?</h4>
<ol>
<li>The Best solution ever: just use the most simple command to upload files to S3 called “aws s3 cp” like this:</li>
</ol>
<pre><code class="code-highlight">aws s3 cp DIRECTORY_OR_FILE s3://BUCKET_NAME — recursive — quiet
</code></pre>
2. Second best choice: use a Github action to upload again the files without remembering the previous S3 command: <a href="https://github.com/shallwefootball/upload-s3-action">https://github.com/shallwefootball/upload-s3-action</a>
<pre><code class="code-highlight">name: Upload to S3on: [pull_request]jobs: upload: runs-on: ubuntu-latest steps: - uses: actions/checkout@master - uses: shallwefootball/s3-upload-action@master with: aws_key_id: ${{ secrets.AWS_KEY_ID }} aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY}} aws_bucket: BUCKET_NAME source_dir: 'DIRECTORY_OR_FILE'
</code></pre>
3. not suggested solution: increasing the memory of the Lambda with the memoryLimit parameter, from the default 128 to 1024, this might help but won’t make a difference with more than 200 files.
<hr>
<blockquote>
💡Also learn how to efficiently store one million small objects to S3 with AWS Athena integration:
</blockquote>
<a href="https://differ.blog/p/how-to-write-one-million-small-objects-to-s3-c69e8c" title="https://differ.blog/p/how-to-write-one-million-small-objects-to-s3-c69e8c">How to write one million small objects to S3
Learn how to efficiently store one million small objects to S3 for your stock analytics app with AWS Athena…differ.blog</a>
<blockquote>
👉 To read more such articles, <a href="https://differ.blog/">sign up for free on Differ</a>.
</blockquote>
<hr>
<h4 id="resources">Resources</h4>
<ul>
<li><a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html">docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html</a></li>
<li><a href="https://gist.github.com/riccardogiorato/428d015dee054d3cb9b06941851ab018">gist.github.com/riccardogiorato/428d015dee054d3cb9b06941851ab018</a></li>
<li><a href="https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html">docs.aws.amazon.com/cli/latest/reference/s3/cp.html</a></li>
<li><a href="https://github.com/shallwefootball/upload-s3-action">github.com/shallwefootball/upload-s3-action</a></li>
</ul>
<h4 id="if-you-want-to-help-me-out-youcan">If you want to help me out you can:</h4>
<ul>
<li>Join Medium Membership and get my future Articles: <a href="https://riccardogiorato.medium.com/membership">https://riccardogiorato.medium.com/membership</a></li>
<li>Follow me on Twitter suggesting other Topics to cover: <a href="https://twitter.com/riccardogiorato">https://twitter.com/riccardogiorato</a></li>
</ul>

Do you want to wait hours to upload a React, Next.js, Gatsby, or Nuxt.js project? We hope you don’t and are looking for fast and easy solutions!

Stop Using the CDK S3 Deployment Module

#### Implementation Plan:

* Use AWS Systems Manager to take action on AWS resources, EC2 instances in this case.
* Create maintenance windows to schedule and automate the start and stop of EC2 instances. This will automatically perform tasks in a defined time window.
* Register the EC2 instance(s) as targets. The schedule/window applies to these targets.
* Registering tasks, these tasks will be performed on the targets.
* IAM role which SSM assumes and is allowed to make changes to the EC2 instances.

#### IAM role:

* Trust relationship

![](https://cdn-images-1.medium.com/max/1600/1*Xc6mftzJBIUEeEmRFyKD3g.png)

* Permissions
  \- Use built-in permissions policy [AmazonSSMAutomationRole](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMAutomationRole.html):

```
{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "lambda:InvokeFunction"            ],            "Resource": [                "arn:aws:lambda:*:*:function:Automation*"            ]        },        {            "Effect": "Allow",            "Action": [                "ec2:CreateImage",                "ec2:CopyImage",                "ec2:DeregisterImage",                "ec2:DescribeImages",                "ec2:DeleteSnapshot",                "ec2:StartInstances",  --> Ec2 start instance allow action                "ec2:RunInstances",                "ec2:StopInstances",   --> Ec2 stop instance allow action                "ec2:TerminateInstances",                "ec2:DescribeInstanceStatus",                "ec2:CreateTags",                "ec2:DeleteTags",                "ec2:DescribeTags",                "cloudformation:CreateStack",                "cloudformation:DescribeStackEvents",                "cloudformation:DescribeStacks",                "cloudformation:UpdateStack",                "cloudformation:DeleteStack"            ],            "Resource": [                "*"            ]        },        {            "Effect": "Allow",            "Action": [                "ssm:*"            ],            "Resource": [                "*"            ]        },        {            "Effect": "Allow",            "Action": [                "sns:Publish"            ],            "Resource": [                "arn:aws:sns:*:*:Automation*"            ]        }    ]}
```

#### Systems Manager Maintenance Windows

* I used Maintenance Windows sub-feature of Change Management under Systems Manager to start and stop the EC2 instances.
* **Schedule:**

![](https://cdn-images-1.medium.com/max/1600/1*beFnpWIfS8MRcO0vTBrCBw.png)

* **Targets:**
  \- Here we have a few options. I chose to select instance manually since I only had once instance to work with. It auto populates all the running instances.

![](https://cdn-images-1.medium.com/max/1600/1*-0brAU_HfVSKS9osWGtcYw.png)

![](https://cdn-images-1.medium.com/max/1600/1*fKMOoYJZmSnA9gRe3CTWaQ.png)

\- A **resource group** can also be selected in case resources are grouped together based on a criteria.
 - The selection criteria in these set of screenshots are:
**Resource Types → AWS::EC2::Instance**
**Tags → Key:env , value:dev**

![](https://cdn-images-1.medium.com/max/1600/1*UxaehQBtSWXYj2hcta6aBQ.png)

The resources fulfilling the criteria can then be selected after selecting the Preview group resources option:

![](https://cdn-images-1.medium.com/max/1600/1*LFT6jUJ5YIHCLA3uNelRKg.png)

#### Tasks:

* Navigate to tasks:

![](https://cdn-images-1.medium.com/max/1600/1*H5b2eeHorQnagHqPMyA5XQ.png)

* Select the Automation Document:
  \- I selected EC2 Start Instance document

![](https://cdn-images-1.medium.com/max/1600/1*XunRjyIL1gO9_u_ldxifhg.png)

-Since our target type is `INSTANCE`, `{{RESOURCE_ID}}` parameter yields the instance ID only. When the maintenance window task runs, it passes the correct values instead of the pseudo parameter placeholders. More information [here](https://docs.aws.amazon.com/systems-manager/latest/userguide/maintenance-window-tasks-pseudo-parameters.html#pseudo-parameters).

![](https://cdn-images-1.medium.com/max/1600/1*BHnSnjGlkEvSZF6eDatoww.png)

#### Additional Pointers:

* Execution history of tasks can be checked under the history tab:

![](https://cdn-images-1.medium.com/max/1600/1*t1gtfUqZsVwKf9HSiPP_Lg.png)

* If for some reason, the schedule needs to be changed, click on edit in the window description tab:

![](https://cdn-images-1.medium.com/max/1600/1*Ox0phC8yrHnr_oaQJcYSQg.png)

The current schedule is set to run everyday at 9:15 AM :

**cron(0 15 9 ? \* \* \*)**

This can be changed, for example, to run at 11:30 PM everyday:

**cron(0 30 23 ? \* \* \*)**

#### Testing:

* Stop Instance maintenance window:
  \- At a set schedule, the task execution starts and the instance stops as shown in the screenshots below:

![](https://cdn-images-1.medium.com/max/1600/1*h066wHZznkk0_1N-dBj-6A.png)

![](https://cdn-images-1.medium.com/max/1600/1*1v_lytqYAVuPFOyqzKjozg.png)

* Start Instance maintenance window:
  \- At a set schedule, the task execution starts and the instance goes into a running state again as shown in the screenshots below:

![](https://cdn-images-1.medium.com/max/1600/1*NpkspcCPwP4N35K4Ki5C8g.png)

![](https://cdn-images-1.medium.com/max/1600/1*TdlZH7AYg1vDqHWtLoBNag.png)

#### For further reading:

* I will implement this feature in my upcoming blog [here](https://medium.com/p/e58612d35f83/edit).
* I will also be making sure that:
  \- Docker service on the instance keeps running after the instance is stopped and started following the schedule.
  \- Docker container keeps running and serving the web application as usual even after the instance is re-started.

Thanks for reading!

<h4 id="implementation-plan">Implementation Plan:</h4>
<ul>
<li>Use AWS Systems Manager to take action on AWS resources, EC2 instances in this case.</li>
<li>Create maintenance windows to schedule and automate the start and stop of EC2 instances. This will automatically perform tasks in a defined time window.</li>
<li>Register the EC2 instance(s) as targets. The schedule/window applies to these targets.</li>
<li>Registering tasks, these tasks will be performed on the targets.</li>
<li>IAM role which SSM assumes and is allowed to make changes to the EC2 instances.</li>
</ul>
<h4 id="iam-role">IAM role:</h4>
<ul>
<li>Trust relationship</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*Xc6mftzJBIUEeEmRFyKD3g.png" alt="">
<ul>
<li>Permissions
- Use built-in permissions policy <a href="https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMAutomationRole.html">AmazonSSMAutomationRole</a>:</li>
</ul>
<pre><code class="code-highlight">{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": [ "arn:aws:lambda:*:*:function:Automation*" ] }, { "Effect": "Allow", "Action": [ "ec2:CreateImage", "ec2:CopyImage", "ec2:DeregisterImage", "ec2:DescribeImages", "ec2:DeleteSnapshot", "ec2:StartInstances", --> Ec2 start instance allow action "ec2:RunInstances", "ec2:StopInstances", --> Ec2 stop instance allow action "ec2:TerminateInstances", "ec2:DescribeInstanceStatus", "ec2:CreateTags", "ec2:DeleteTags", "ec2:DescribeTags", "cloudformation:CreateStack", "cloudformation:DescribeStackEvents", "cloudformation:DescribeStacks", "cloudformation:UpdateStack", "cloudformation:DeleteStack" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "ssm:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "sns:Publish" ], "Resource": [ "arn:aws:sns:*:*:Automation*" ] } ]}
</code></pre>
<h4 id="systems-manager-maintenance-windows">Systems Manager Maintenance Windows</h4>
<ul>
<li>I used Maintenance Windows sub-feature of Change Management under Systems Manager to start and stop the EC2 instances.</li>
<li>Schedule:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*beFnpWIfS8MRcO0vTBrCBw.png" alt="">
<ul>
<li>Targets:
- Here we have a few options. I chose to select instance manually since I only had once instance to work with. It auto populates all the running instances.</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*-0brAU_HfVSKS9osWGtcYw.png" alt="">
<img src="https://cdn-images-1.medium.com/max/1600/1*fKMOoYJZmSnA9gRe3CTWaQ.png" alt="">
- A resource group can also be selected in case resources are grouped together based on a criteria.
 - The selection criteria in these set of screenshots are:
Resource Types → AWS::EC2::Instance
Tags → Key:env , value:dev
<img src="https://cdn-images-1.medium.com/max/1600/1*UxaehQBtSWXYj2hcta6aBQ.png" alt="">
The resources fulfilling the criteria can then be selected after selecting the Preview group resources option:
<img src="https://cdn-images-1.medium.com/max/1600/1*LFT6jUJ5YIHCLA3uNelRKg.png" alt="">
<h4 id="tasks">Tasks:</h4>
<ul>
<li>Navigate to tasks:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*H5b2eeHorQnagHqPMyA5XQ.png" alt="">
<ul>
<li>Select the Automation Document:
- I selected EC2 Start Instance document</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*XunRjyIL1gO9_u_ldxifhg.png" alt="">
-Since our target type is <code>INSTANCE</code>, <code>{{RESOURCE_ID}}</code> parameter yields the instance ID only. When the maintenance window task runs, it passes the correct values instead of the pseudo parameter placeholders. More information <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/maintenance-window-tasks-pseudo-parameters.html#pseudo-parameters">here</a>.
<img src="https://cdn-images-1.medium.com/max/1600/1*BHnSnjGlkEvSZF6eDatoww.png" alt="">
<h4 id="additional-pointers">Additional Pointers:</h4>
<ul>
<li>Execution history of tasks can be checked under the history tab:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*t1gtfUqZsVwKf9HSiPP_Lg.png" alt="">
<ul>
<li>If for some reason, the schedule needs to be changed, click on edit in the window description tab:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*Ox0phC8yrHnr_oaQJcYSQg.png" alt="">
The current schedule is set to run everyday at 9:15 AM :
cron(0 15 9 ? * * *)
This can be changed, for example, to run at 11:30 PM everyday:
cron(0 30 23 ? * * *)
<h4 id="testing">Testing:</h4>
<ul>
<li>Stop Instance maintenance window:
- At a set schedule, the task execution starts and the instance stops as shown in the screenshots below:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*h066wHZznkk0_1N-dBj-6A.png" alt="">
<img src="https://cdn-images-1.medium.com/max/1600/1*1v_lytqYAVuPFOyqzKjozg.png" alt="">
<ul>
<li>Start Instance maintenance window:
- At a set schedule, the task execution starts and the instance goes into a running state again as shown in the screenshots below:</li>
</ul>
<img src="https://cdn-images-1.medium.com/max/1600/1*NpkspcCPwP4N35K4Ki5C8g.png" alt="">
<img src="https://cdn-images-1.medium.com/max/1600/1*TdlZH7AYg1vDqHWtLoBNag.png" alt="">
<h4 id="for-furtherreading">For further reading:</h4>
<ul>
<li>I will implement this feature in my upcoming blog <a href="https://medium.com/p/e58612d35f83/edit">here</a>.</li>
<li>I will also be making sure that:
- Docker service on the instance keeps running after the instance is stopped and started following the schedule.
- Docker container keeps running and serving the web application as usual even after the instance is re-started.</li>
</ul>
Thanks for reading!

Automating the Start and Stop state of EC2 instance

Deploying COBOL Projects on AWS: Practical Guide

Mastering AWS CDK: Setting Up a Custom Domain for Your HTTP Gateway

Exploring The New Claude Model On Amazon Bedrock

Image To Text With Claude 3 Sonnet

How integrating Terraform within AWS Control Tower can streamline networking setups and strengthen connectivity between on-premises and cloud environments in a natural, intuitive way.

Exploring AWS Control Tower Account Factory for Terraform (AFT) Customizations

Learn how to navigate AWS costs effectively to trim unnecessary expenses and boost your bottom line. Discover essential tips and best practices for optimizing your AWS spending and achieving cost-efficiency.

Mastering AWS Costs: Key Strategies to Optimize Spending and Maximize Savings

Being aware of the most common & hidden service that can potentially drive up your AWS usage costs.

AWS Costs: An Important Service To Watch Out For To Reduce Costs

Everything you need to know to start with the AWS Cloud Up for Her Solutions Architect Associate Program and get certified!

Main Menu

Follow Us

AWS