Access files from AWS S3 using pre-signed URLs in Python

Introduction:

Let's assume that you need to share files from your AWS S3 bucket(private) without providing AWS access to a user. How would you do that? Well, we have pre-signed URLs that are shortly lived, which can be shared and used to access the content shared.

Pre-signed URLs:

What is a pre-signed URL?

A pre-signed URL gives you **temporary **access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object.

That is, if you receive a pre-signed URL to upload an object, you can upload the object only if the creator of the pre-signed URL has the necessary permissions to upload that object. Same applies for download as well.

We will see how to generate pre-signed URLs for S3 bucket programmatically using python and boto3.

When we say, the creator of the presigned URL should have access what does it mean?

It means, the URL generator should have a aws access with right credentials(may be in a lambda)and to achieve this, we could expose a REST API to the customer to request for a URL based on the upload/download operation. This ensures the user need not be provided with the AWS credentials.

The pre-signed URL will expire based on the expiry value configured while generating it. We shall look at it shortly.

A high-level design:

In the above design, a user requests the URL from the UI(could be a web portal) via a REST API based on the operation required. This hits the API gateway which triggers a lambda. The lambda executes the code to generate the pre-signed URL for the requested S3 bucket and key location.

The most prevalent operations are but not limited to upload/download objects to and from S3 buckets which are performed using

1. put_object

2. get_ object.

Let’s look at the code which goes in the lambda

1. Generating pre-signed URL for download

import boto3
from botocore.exceptions import ClientError
from botocore.config import Config
import requests

def generate_presigned_url(bucket_name, object_key, expiry=3600):

    client = boto3.client("s3",region_name=REGION_NAME,
                          aws_access_key_id=ACCESS_KEY,
                          aws_secret_access_key="SECRET_KEY",
                          aws_session_token="SESSION_TOKEN")
    try:
        response = client.generate_presigned_url('get_object',
                                                  Params={'Bucket': bucket_name,'Key': object_key},
                                                  ExpiresIn=expiry)
        print(response)
    except ClientError as e:
        print(e)

Please note that the awssession token is an optional parameter. This may be required if your organization is providing credentials that expire. If you are using your personal account and do not have any configuration for session expiry they may not be required.

The ‘get_object’ specifies the URL is being generated for a download operation. The bucket name and object should be passed as part of the params dictionary.

2. Generating pre-signed URL for upload

We use the same create presigned url with put_object method to create a presigned URL for uploading a file.

def create_presigned_upload(expiration=3600):


    # Generate a presigned S3 POST URL
    s3_client = boto3.client("s3", config=Config(signature_version='s3v4'),
                             region_name="us-west-2",
                             aws_access_key_id="ACCESS_KEY",
                             aws_secret_access_key="SECRET_KEY",
                             aws_session_token="SESSION_TOKEN")
    try:
        response = s3_client.generate_presigned_url('put_object',
                   Params= {'Bucket': "BUCKET_NAME",
                            "Key":"OBJECT_KEY"},
                            ExpiresIn=3600)
    except ClientError as e:
        return None

    # The response contains the presigned URL and required fields
    return response

Output:
[https://experiment.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-2818d7/0185-43b6-4e82-90-f80ddeb/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64](https://gehc-us-west-2-551934631674-dev-eai-experiment-foipmbja.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-12cf632818d7/018979a5-43b6-4e82-9ba0-f80dd9584deb/annotation/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64)

This is a sample pre-signed URL output.

Try accessing the presigned URL either through browser or programmatically.

Signature Invalid

Oops:

We have hit a roadblock. The URL throws a signature does not match error.

Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Photo by Jonathan Farber on Unsplash

Why?

The reason for this is, it’s not recommended to use generate_presigned_url with put_object parameter to generate pre-signed URLs for uploading files though it wouldn’t throw any error while generating.

I had deliberately used it here because I had run into this issue and wanted to share this learning.

Please refer to this github link for more information about this.

Let’s move to the recommended solution. While generating URLs for upload, it’s always better to use generate_presigned_post method as this includes the proper header information and other parameters required for the URL.

3. Pre-signed URL post

def create_presigned_post(bucket_name=None, object_name=None,
                          fields=None, conditions=None, expiration=3600):


    # Generate a presigned S3 POST URL
    s3_client = boto3.client("s3", config=Config(signature_version='s3v4'),
                             region_name="us-west-2",
                             aws_access_key_id="ACCESS_KEY",
                             aws_secret_access_key="SECRET_KEY",
                             aws_session_token="SESSION_TOKEN")
    try:
        response = s3_client.generate_presigned_post(Bucket="BUCKET_NAME",
                                                     Key="OBJECT_PATH",
                                                     ExpiresIn=3600)
    except ClientError as e:
        return None

    # The response contains the presigned URL and required fields
    return response


resp = create_presigned_post()

# Extract the URL and other fields from the response
post_url = resp['url']
data = resp['fields']
key = data['key']

# Upload the file using requests module
response = requests.post(url=post_url, data=data,
                         files={'file': open(r'C:\Users\212757215\Desktop\Dockerfile.txt', 'rb')})

print(response)

Output: A sample presigned URL
[https://experiment.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-2818d7/0185-43b6-4e82-90-f80ddeb/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64](https://gehc-us-west-2-551934631674-dev-eai-experiment-foipmbja.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-12cf632818d7/018979a5-43b6-4e82-9ba0-f80dd9584deb/annotation/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64)

Caveats:

• If the server-side encryption of S3 is set to KMS, you may need to set the signature version to v4 while creating the boto3 object.

• Boto3 by default supports signature v4. However for S3, the objects should explicitly set the signature version to v4 in case of KMS.

• Not setting the signature to v4 may result in 403 error while trying to access the URL though you have the right permissions.

Setting signature version explicitly:

s3_client=boto3.client("s3",config=Config(signature_version='s3v4'))

Summary:

• Pre-signed URLs could be used to provide temporary access to users without providing aws access to users

• URLs could be generated to upload and download files

References:

• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_post

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html