AWS SNS Events: SNS to Lambda vs SNS to SQS to Lambda

When working with AWS SNS Events that require processing in AWS Lambda, there are two common event-driven design patterns because SNS events can target multiple destinations such as Lambda, SNS, SQS :

1. Invoke the Lambda directly through the SNS event.

2. Send the SNS event to the SQS queue which in turn will feed the Lambda.

SNS Events Directly Trigger Lambda

Amazon SNS invokes Lambda function asynchronously with an event that contains a message and metadata.

• For asynchronous invocation, Lambda queues the message and handles retries.

• If SNS can’t reach Lambda or the message is rejected, SNS retries at increasing intervals over several hours and then the message is lost after a few retries i.e., there is no guarantee of delivery.

• If Lambda fails an event, that particular event is lost. In the case of DLQ (Dead Letter Queue) configured, provisions need to be made for reading that separately and processing the message.

• One thing to consider in SNS is only one message is received at once, so the advantage of batching is not available and it needs to be ensured to have enough concurrent Lambda functions. You can apply for an increase in Lambda concurrency in your AWS account.

• There is no persistence in SNS. Whichever consumer is present at the time of message arrival gets the message and the message is deleted. If no consumers are available then the message is lost after a few retries.

SNS Events to SQS to Lambda

Lambda polls the SQS queue and invokes Lambda function synchronously with an event that contains queue messages.

• All error handling and retry of synchronous invocation apply here.

• Lambda can read messages in batches and invoke function once for each batch. When the function successfully processes a batch, it deletes the messages from the queue. This helps in cost saving on Lambda invocations.

• Lambda automatically scales up and down based on the number of inflight messages in the queue.

• The polling, reading and removing of messages from the queue will be automatically handled by the built-in functionality, without needing to explicitly configure these steps inside Lambda function code.

• Polling inherently introduces some latency in message delivery in SQS, unlike SNS where messages are immediately pushed to subscribers.

• Having SQS in between SNS and Lambda allows reprocessing older unprocessed events in case Lambda fails to process.

• SQS allows to put a delay, so that message gets processed after some time, it may be useful in the scenario where data takes time to be available.

• Messages can be persisted for some configurable duration if no consumer is available (maximum two weeks), so the consumer does not have to be up when messages are added to the queue.

Conclusion

To summarize, you can use SNS to SQS to Lambda if:

• Your events are critical, you don't want to miss out on them

• You want to reprocess events in case of failure and configure how many times a message should be retried

• You want a longer retention period for events for example when there is a long-running job and the Lambda polls one by one from the job queue

• You want to optimize cost and run batch processing in Lambda

You can choose to use SNS to Lambda if:

• You do not care about lost messages

• You want to ingest/process messages faster, keeping in mind there is enough Lambda concurrency to process all messages

In both cases, there can be duplicate messages (in the cases of retry) and there is no guarantee of order.

Thank you for reading! If you found this helpful, here are some next steps you can take:

1. Send some claps my way & follow me on Medium!

2. Subscribe below to get a notification whenever I publish!

3. Connect with me on LinkedIn & Twitter for more tech blogs!