Uploading Files to an AWS S3 Bucket Using C++ in Linux

With Amazon's AWS becoming the standard in cloud infrastructure, one of the basic tasks for a software developer working with AWS is to programatically upload files to the cloud. However, when I was first faced with this task (without knowing much about how AWS works), I found it very hard to find a simple step-by-step guide on a simple file upload using C++ in a Linux environment. So here with a brief outline of how to write the "Hello World, Upload Me!" of AWS uploading.

Set Up the AWS SDK for C++

First of all, you must set up your local environment. I won't go over setting up C++, just the AWS C++ SDK. I'm using Ubuntu, but it's similar for most Linux versions. You can find the official instructions here.

You start by downloading the source code:

git clone --- recurse-submodules https://github.com/aws/aws-sdk-cpp

I prefer generating the build files in a different folder from the AWS source directory:

mkdir build
cd build

Then finally you generate the build files, build the SDK binaries and install the SDK. Building all the AWS services can take hours, so this is the simpler version of building just the S3 services:

cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Debug -DCMAKE_PREFIX_PATH=/usr/local/ -DCMAKE_INSTALL_PREFIX=/usr/local/ -DBUILD_ONLY="s3"
make
sudo make install
sudo ldconfig

Also, this is very important, your cmake version must be between 3.13 and 3.21. To find what version you have, run:

cmake --version

Changing the cmake version is not very straight-forward. You can find multiple ways of doing it, here's what worked for me. First, remove your current version of cmake:

sudo apt update
sudo apt remove --purge --auto-remove cmake

Then go to the official CMake downloads page, https://cmake.org/files and choose a version. I went with 3.21.6 since it's the highest supported by the AWS SDK.

wget https://cmake.org/files/v3.21/cmake-3.21.6.tar.gz
tar -xzvf cmake-3.21.6.tar.gz
cd cmake-3.21.6/

Finally, install it and check that the correct version is working:

./bootstrap
make -j$(nproc)
sudo make install
cmake --version

Configure the AWS Connection

First, you go into AWS Console, into the S3 section and create a bucket where you'll store the files. I assume you already have the AWS account.

Then, you have to create your access keys. In the console, go to the Security Credentials page (under your account drop-down at the top right), find the Access Keys section and click Create Access Key. You'll get an Access Key ID and a Secret Access Key. Write them down immediately, as you won't be able to see the secret key again! To set up the connection, you use the AWS CLI:

aws configure

It will request the 2 keys generated before and then you're all set to start writing code!

The C++ Uploading Code

Here is a basic example of a started code to upload a file to the S3 bucket you created, using the AWS C++ SDK in Linux (Ubuntu):

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/core/auth/AWSCredentialsProviderChain.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <iostream>
#include <fstream>
using namespace Aws;
using namespace Aws::Auth;

bool PutObject(const Aws::String &bucketName, const Aws::String &fileName, const Aws::Client::ClientConfiguration &clientConfig) {
Aws::S3::S3Client s3_client(clientConfig);

    Aws::S3::Model::PutObjectRequest request;
    request.SetBucket(bucketName);
    request.SetKey(fileName);
    std::shared_ptr<Aws::IOStream> inputData = Aws::MakeShared<Aws::FStream>("SampleAllocationTag", fileName.c_str(), std::ios_base::in | std::ios_base::binary);

    if (!*inputData) {
        std::cerr << "Error unable to read file " << fileName << std::endl;
        return false;
    }

    request.SetBody(inputData);
    Aws::S3::Model::PutObjectOutcome outcome = s3_client.PutObject(request);

    if (!outcome.IsSuccess()) {
        std::cerr << "Error: PutObject: " << outcome.GetError().GetMessage() << std::endl;
    }
    else {
        std::cout << "Added object '" << fileName << "' to bucket '" << bucketName << "'.n";
    }

    return outcome.IsSuccess();

}

int main(int argc, char \*\*argv) {
Aws::SDKOptions options;
Aws::InitAPI(options);

    Aws::Client::ClientConfiguration clientConfig;
    const Aws::String bucket_name = "bucketname/subfolder";
    const Aws::String object_name = "s3simpleupload.cpp";
    PutObject(bucket_name, object_name, clientConfig);

    Aws::ShutdownAPI(options);
    return 0;

}

To compile it, use:

g++ s3simpleupload.cpp -laws-cpp-sdk-core -laws-cpp-sdk-s3 -laws-c-event-stream -o s3simpleupload

And that's it! You created your first AWS uploading program in C++! For further reading, you can see the official AWS examples or the AWS sample code on Github. And if you don't need the flexibility of using your custom code, you can use the AWS CLI to sync and entire local folder to the S3 bucket:

aws s3 sync /home/user/C/files s3://bucketname/subfolder

Chris Fotache is an AI researcher with CYNET.ai based in Florida. He covers topics related to artificial intelligence in our life, Python and C++ programming, machine learning, computer vision, natural language processing, robotics and more.