Beyond the 10MB Limit: The Smart Way to Handle File Uploads in AWS 🚀

Handling file uploads in AWS presents challenges, especially when working with Amazon API Gateway. One common roadblock engineers face is API Gateway’s 10MB payload limit, which can prevent applications from handling large file transfers.

Many developers make the mistake of pushing files directly through API Gateway, which leads to failed uploads, unnecessary costs, and increased latency. But there’s a better, more scalable approach: leveraging Amazon S3 Pre-Signed URLs with AWS Lambda.

This blog will discuss the problem, the better architecture, and how to implement a more efficient solution for handling file uploads in AWS.

The Problem: API Gateway’s 10MB Limit & Why It Fails

API Gateway is an excellent service for managing APIs, but it’s not built to handle large file uploads. Here’s why forcing file uploads through API Gateway is a bad idea:

❌ 1. API Gateway’s Hard 10MB Payload Limit

AWS API Gateway rejects requests that are more significant than 10MB, making it unsuitable for handling large file uploads.

❌ 2. Increased Costs & Inefficiency

API Gateway charges per request and payload size. Uploading large files through API Gateway increases both costs and latency.

❌ 3. Unnecessary Backend Load

Processing large file uploads through API Gateway adds stress to backend services, causing performance degradation and potential throttling.

What’s the alternative? Offloading file uploads to Amazon S3 directly using Pre-Signed URLs.

The Better Architecture: Pre-Signed URLs with Lambda

Instead of pushing files through API Gateway, a better approach is to:

• Request a Pre-Signed URL via API Gateway & Lambda.
• Upload the file directly to S3 using the Pre-Signed URL.

How It Works (Step-by-Step Breakdown)

🔹 Step 1: Client Requests an Upload URL

• The mobile/web client sends a request to API Gateway requesting a Pre-Signed URL for file upload.

🔹 Step 2: API Gateway Invokes Lambda

• API Gateway triggers an AWS Lambda function, which generates a temporary, time-limited upload URL.

🔹 Step 3: Lambda Generates a Pre-Signed URL

• The Lambda function uses AWS SDK to generate a Pre-Signed URL that allows direct uploads to S3.

🔹 Step 4: API Gateway Returns the URL

• Lambda sends the Pre-Signed URL back to the client via API Gateway.

🔹 Step 5: Client Uploads File Directly to S3

• The client uses the Pre-Signed URL to upload the file directly to S3, bypassing API Gateway entirely.

🔹 Step 6: Success!

• The file is securely stored in S3 without hitting API Gateway’s payload limit or causing unnecessary backend overhead.

Why This Approach Works Better

✅ No API Gateway Payload Limit → Files go directly to S3, so the 10MB limit no longer applies. ✅ Faster Uploads & Lower Latency → Uploads go straight to S3, reducing API Gateway processing time. ✅ Lower Costs → Avoids API Gateway request & payload charges, making it more cost-efficient. ✅ Improved Scalability → S3 handles massive file uploads without overloading API Gateway or Lambda. ✅ Enhanced Security → Pre-Signed URLs are temporary & controlled, preventing unauthorized access.

Implementation: Generate a Pre-Signed URL Using AWS Lambda

Here’s a simple AWS Lambda function (Python) to generate a Pre-Signed URL for uploading files to S3:

import boto3
import oss3 = boto3.client('s3')

def lambda_handler(event, context):
    bucket_name = os.getenv('S3_BUCKET_NAME')
    file_name = event['queryStringParameters']['file_name']

    url = s3.generate_presigned_url(
        'put_object',
        Params={'Bucket': bucket_name, 'Key': file_name},
        ExpiresIn=3600  # URL expires in 1 hour
    )
    return {
        "statusCode": 200,
        "body": url
    }

How This Code Works

✔️ Retrieves file name from the client request. ✔️ Generates a Pre-Signed URL valid for 1 hour. ✔️ Returns the URL to the client, allowing a direct S3 upload.

Wrapping Up

If you’re still sending large files through API Gateway, it’s time to switch to a better, scalable solution.

Using AWS Lambda and S3 Pre-Signed URLs eliminates payload limits, improves upload speed, and reduces costs, all while keeping your application secure and efficient.