When working with AWS S3 Events that require processing in AWS Lambda, there are two common event-driven design patterns because S3 Notifications can target Lambda, SNS and SQS:
-
Invoke the Lambda directly through S3 event
-
Send the S3 event to an SNS/SQS queue which in turn triggers the Lambda
1: S3 Events directly trigger Lambda
Amazon S3 invokes Lambda function asynchronously with an event that contains details about the object in S3 bucket.
-
Amazon S3 event notifications are designed to be delivered at least once however, it doesn’t guarantee delivery.
-
Retry and error handling mechanisms of asynchronous invocation are applicable. A dead-letter queue can be configured on the function to capture events that weren’t successfully processed/discarded.
-
There is no pricing associated with S3 Notifications and Lambda targets are subject to normal AWS Lambda billing.
-
For push model invocations, Lambda scales with the number of incoming requests until concurrency or account limit is reached.
-
S3 events are limited to a single destination and don’t support fan-out. However, multiple S3 events can be configured to same Lambda function.
-
If message ordering is not of critical importance, this approach can be used.
-
S3 events can directly trigger lambda in case TPS is low or batch processing of events is not required.
-
To directly trigger Lambda from S3 notifications, Lambda function must be in the same AWS region as the S3 bucket.
2: S3 Events to SNS to Lambda
Amazon SNS invokes Lambda function asynchronously with an event that contains a message and metadata.
-
Both S3 and SNS events are asynchronous event sources and behave the same way. There is no added advantage.
-
Unnecessary costs & infrastructure management of SNS.
-
SNS to Lambda delivery is not reliable as if SNS can’t reach Lambda or the message is rejected, after retries, it discards the message and event is lost.
-
If there is an use-case to fan-out events, SNS can provide flexibility to send (fan-out) the notifications to multiple destinations — another Lambda, SQS, an HTTPS endpoint, or even email which can be useful in developing new capabilities that need the same event.
3: S3 Events to SQS to Lambda
Lambda polls the SQS queue and invokes function synchronously with an event that contains queue messages.
-
If it is critical that all published messages be successfully processed, SQS is a better alternative as it provides guaranteed delivery and event re-driving capabilities.
-
SQS should be used in case message ordering is of critical importance. Here, we can use SNS with FIFO SQS queue.
-
SQS should be used in case of high TPS systems so that system would be more resilient to traffic spikes and traffic to Lambda can be moderated.
-
SQS should be used in case events need to be processed in batches. With SQS, Lambda can read messages in batches and invoke function once for each batch. This also results in cost saving on Lambda invocations.
-
SQS provides in-built capability for message reprocessing. If a message fails to be processed multiple times, Amazon SQS can send it to a dead-letter queue (DLQ). After the visibility timeout, Lambda receives the message again.
-
SQS allows to put message delay so that message gets processed after some time. It may be useful in the scenario where downstream data takes time to be available.
-
SQS can also serve as a contingency store when downstream services are unavailable as message can be retained in SQS for 14 days.
-
SQS can be used in case if want to keep Lambda & S3 bucket in different AWS regions.
Conclusion
The most suitable approach depends on your system requirements of scale (TPS), error-handling (DLQ & event redriving), fan-out (multiple destinations), message ordering, batch processing, cost, AWS Region etc.
Thank you for reading!
This blog is part of my System Design Choices series. Check out the other blogs of the series.