If you’ve ever built an application that handles image uploads from users then you are probably aware that at least to start with you can try storing those images in your application’s database. The problem you will eventually run into is that this method will not scale efficiently as your application grows.
The more efficient and cost-effective option is to use AWS’s S3 service for storing the image files. Using S3 is a very low-cost option. Effectively, all you are paying for is transferring files into an S3 bucket and serving those images to your users. It is important to also restrict access to those files so only users should have access to them. You definitely don’t want to make them publicly available because you’d be vulnerable to someone requesting the same files over and over and running up your AWS bill for data transfer.
In this article, I will be demonstrating how to build a simple NodeJS/Express application that will allow users to upload image files, store them in S3, and restrict access to where only the users that uploaded those files will be able to view them within the app.
Prerequisites
Before we jump in, to follow along with the example in this article you will need a few things:
- Basic understanding of JavaScript
- NodeJS installed
- An AWS account
Let’s Code
First, we will need to do to get our project started is to create a new folder and then generate a new Node project within that newly created folder. You can name the folder whatever you want, but in the example commands below I used s3-images-example-app.
$ cd s3-images-example-app
$ npm init -y
This will generate an empty package.json so we can add our dependencies through npm next. Our dependencies include:
Back in the terminal run the following command to install these:
$ npm install --save express express-formidable aws-sdk
Just for reference on version numbers used in this example, here is what the package.json file looks like after we run the previous command:
{
"name": "s3-images-example-app",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"aws-sdk": "^2.920.0",
"express": "^4.17.1",
"express-formidable": "^1.2.0"
}
}
To get things started with our development, create a new file called server.js and add the following code. If you’re familiar with Express this should look really straightforward. Just creating a simple web server that is listening for requests on port 3000.
const express = require("express");
const port = process.env.PORT || 3000;
const app = express();
app.use(express.urlencoded({ limit: "100mb" }));
app.get("/health", (req, res) => {
res.sendStatus(200);
});
// Start the app
app.listen(port, () => console.log(`API listening on ${port}`));
The only route we declare initially is a health route to test if the API is up and running. Run one of the following commands to start the server.
$ node server.jsOR$ npx nodemon server.js
If you use the second option (with nodemon) you will not need to restart the server for code changes, it will automatically restart whenever you make changes.
After starting the server make a GET HTTP request to the following URL to make sure the server is up and accepting requests:
GET http://localhost:3000/health
If all is working as expected you should receive this response:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
Content-Length: 2
ETag: W/"2-nOO9QiTIwXgNtWtBJezz8kv3SLc"
Date: Thu, 03 Jun 2021 02:50:49 GMT
Connection: close OK
PostgreSQL Database
For our application, we will need a PostgreSQL instance running to have a data store for our example app. You can run your database instance from anywhere, but for my demonstration, I’ll be spinning up a Postgres instance with Docker. For a thorough walkthrough on how to do this check out this article by Lorenz Vanthillo.
To start the instance all we need to do is run the following command:
docker run -d -p 5432:5432 --name my-postgres -e POSTGRES_PASSWORD=secretpassword postgres
Once it’s started run the following command to open up a bash shell within the Postgres instance:
docker exec -it my-postgres bash
Next, we need to log in with psql and then create the database for our application:
root@4b88f7a00f5f:/### psql -U postgres
psql (13.3 (Debian 13.3-1.pgdg100+1))
Type "help" for help.postgres=### CREATE DATABASE developmentdb;
CREATE DATABASEpostgres=#
Sequelize ORM
For my NodeJS projects, I like to use Sequelize, an ORM that can help us create data models and manage database migrations. To add Sequelize to our project and bootstrap the migrations we will need to run the following commands.
$ npm install --save sequelize pg
$ npx sequelize-cli init
You will notice that this will create a few new folders in our project. If you want to check out the full documentation on Sequelize migrations that can be found here.
Now that we’ve bootstrapped our project with Sequelize migrations we need to update config/config.json with the following code. Note that you will just need to update the configuration details to connect with your instance if you are not running your database locally.
{
"development": {
"username": "postgres",
"password": "secretpassword",
"database": "developmentdb",
"host": "127.0.0.1",
"dialect": "postgres"
}
}
That takes care of setting up the connection between our application and our database. Next up we are ready to create migrations for our database tables. We will be creating two tables, one for our users and the other for the image uploads. Run the following commands to create the models and migrations for those tables.
$ npx sequelize-cli model:generate --name user --attributes email:string,password:string
$
$ npx sequelize-cli model:generate --name upload --attributes id:uuid,file_name:string,user_id:bigint
That will create a couple of new files in both the models and migrations folders. We need to update our user model and migration with the following code.
'use strict';
const {
Model
} = require('sequelize');
module.exports = (sequelize, DataTypes) => {
class user extends Model {
/**
* Helper method for defining associations.
* This method is not a part of Sequelize lifecycle.
* The `models/index` file will call this method automatically.
*/
static associate(models) {
// define association here
}
};
user.init({
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: DataTypes.INTEGER
},
email: DataTypes.STRING,
password: DataTypes.STRING
}, {
sequelize,
modelName: 'user',
});
return user;
};
'use strict';
module.exports = {
up: async (queryInterface, Sequelize) => {
await queryInterface.createTable('users', {
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: Sequelize.INTEGER
},
email: {
type: Sequelize.STRING
},
password: {
type: Sequelize.STRING
},
createdAt: {
allowNull: false,
type: Sequelize.DATE
},
updatedAt: {
allowNull: false,
type: Sequelize.DATE
}
});
},
down: async (queryInterface, Sequelize) => {
await queryInterface.dropTable('users');
}
};
We’ll also need to update the upload model and migration with the following code.
"use strict";
const { Model } = require("sequelize");
module.exports = (sequelize, DataTypes) => {
class upload extends Model {
/**
* Helper method for defining associations.
* This method is not a part of Sequelize lifecycle.
* The `models/index` file will call this method automatically.
*/
static associate(models) {
// define association here
}
}
upload.init(
{
id: {
type: DataTypes.UUID,
primaryKey: true,
},
file_name: DataTypes.STRING,
user_id: {
type: DataTypes.BIGINT,
references: {
model: "users",
key: "id",
},
},
},
{
sequelize,
modelName: "upload",
}
);
return upload;
};
"use strict";
module.exports = {
up: async (queryInterface, Sequelize) => {
await queryInterface.createTable("uploads", {
id: {
type: Sequelize.UUID,
allowNull: false,
primaryKey: true,
},
file_name: {
type: Sequelize.STRING,
},
user_id: {
type: Sequelize.BIGINT,
references: {
model: "users",
key: "id",
},
},
createdAt: {
allowNull: false,
type: Sequelize.DATE,
},
updatedAt: {
allowNull: false,
type: Sequelize.DATE,
},
});
},
down: async (queryInterface, Sequelize) => {
await queryInterface.dropTable("uploads");
},
};
That should be all the models we need for our application. All that is left to do for this section is to run the following command to execute the migrations on our database.
npx sequelize-cli db:migrate
If all goes well this will create the tables in the database and also add the migrations to the SequelizeMeta table so that next time the migrations are run it knows what has already been executed and won’t run them again. If for whatever reason you need to undo the last executed migration run the following command.
npx sequelize-cli db:migrate:undo
Passport and User Login/Registration
The next step for our application is to set up user registration and logins. To accomplish this we will utilize Passport which is popular authentication middleware for NodeJS. To set Passport up we will be following closely the example created by Jared Hanson here.
We will need to run the following command next to install Passport and several other dependencies.
npm install --save passport passport-local express-session ejs connect-ensure-login body-parser
For the application, we need to create two different forms. One for registering a new user and the other for logging in users. We will be using EJS templates for the views so we will create a folder called views. Next, we create three files within the views folder:
login.ejsregister.ejshome.ejs
<div>
<form action="/login" method="post">
<div>
<label>Username:</label>
<input type="text" name="username" /><br />
</div>
<div>
<label>Password:</label>
<input type="password" name="password" />
</div>
<div>
<input type="submit" value="Submit" />
</div>
</form>
<div>
<p>
<a href="/register">Register</a>
</p>
</div>
</div>
<form action="/register" method="post">
<div>
<label>Username:</label>
<input type="text" name="username" /><br />
</div>
<div>
<label>Password:</label>
<input type="password" name="password" />
</div>
<div>
<input type="submit" value="Register" />
</div>
</form>
<% if (!user) { %>
<p>
Welcome! Please <a href="/login">log in</a> or
<a href="/register">register</a>.
</p>
<% } else { %>
<a href="/logout">Log out</a>
<% } %>
We will be adding another template later on for the upload form, but for now, these are all the templates we need to get started.
Now we need to add some code to server.js to hook up the templates and set up Passport. Add the following code towards the top of the file.
const passport = require("passport");
const Strategy = require("passport-local").Strategy;
const models = require("./models");
passport.use(
new Strategy(async (username, password, cb) => {
const user = await models.user.findOne({
where: {
email: username,
},
});
if (!user) {
return cb(null, false);
}
if (user.password !== password) {
return cb(null, false);
}
return cb(null, user);
})
);
passport.serializeUser((user, cb) => {
cb(null, user.id);
});
passport.deserializeUser(async (id, cb) => {
const user = await models.user.findByPk(id);
if (!user) {
return cb({});
}
cb(null, user);
});
In this code, we define for Passport where to find users which are located in our local Postgres database we set up in the previous step. Please note that we are storing passwords unencrypted in the example code which you definitely wouldn’t want to do in a real production application. This is just for demonstration.
Add the following code towards the bottom of the same file:
app.set("views", __dirname + "/views");
app.set("view engine", "ejs");
app.use(
require("express-session")({
secret: "changeme",
resave: false,
saveUninitialized: false,
})
);
app.use(passport.initialize());
app.use(passport.session());
app.use(express.urlencoded({ limit: "100mb" }));
app.use(require("body-parser").urlencoded({ extended: true }));
app.get("/health", (req, res) => {
res.sendStatus(200);
});
app.get("/", (req, res) => {
res.render("home", { user: req.user });
});
app.get("/login", (req, res) => {
res.render("login");
});
app.post(
"/login",
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/register", (req, res) => {
res.render("register");
});
app.post(
"/register",
async (req, res, next) => {
await models.user.create({
email: req.body.username,
password: req.body.password,
});
next();
},
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/logout", (req, res) => {
req.logout();
res.redirect("/");
});
Here we complete the setup for Passport and define the routes for the application. For reference here is the current state of the server.js file looks like.
const express = require("express");
const passport = require("passport");
const Strategy = require("passport-local").Strategy;
const models = require("./models");
passport.use(
new Strategy(async (username, password, cb) => {
const user = await models.user.findOne({
where: {
email: username,
},
});
if (!user) {
return cb(null, false);
}
if (user.password !== password) {
return cb(null, false);
}
return cb(null, user);
})
);
passport.serializeUser((user, cb) => {
cb(null, user.id);
});
passport.deserializeUser(async (id, cb) => {
const user = await models.user.findByPk(id);
if (!user) {
return cb({});
}
cb(null, user);
});
const port = process.env.PORT || 3000;
const app = express();
app.set("views", __dirname + "/views");
app.set("view engine", "ejs");
app.use(
require("express-session")({
secret: "changeme",
resave: false,
saveUninitialized: false,
})
);
app.use(passport.initialize());
app.use(passport.session());
app.use(express.urlencoded({ limit: "100mb" }));
app.use(require("body-parser").urlencoded({ extended: true }));
app.get("/health", (req, res) => {
res.sendStatus(200);
});
app.get("/", (req, res) => {
res.render("home", { user: req.user });
});
app.get("/login", (req, res) => {
res.render("login");
});
app.post(
"/login",
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/register", (req, res) => {
res.render("register");
});
app.post(
"/register",
async (req, res, next) => {
await models.user.create({
email: req.body.username,
password: req.body.password,
});
next();
},
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/logout", (req, res) => {
req.logout();
res.redirect("/");
});
// Start the app
app.listen(port, () => console.log(`API listening on ${port}`));
This is a good point to test your code by running the server and navigating to [http://localhost:3000](http://http//localhost:3000) in a browser. We can now register a new user, log in, and log out with our just created views.
Upload Form
Now that we have set up users for access, we need to create a new template and route so they can upload an image and we can store it in our S3 image store.
We will need a couple more dependencies installed. Formidable is a middleware we can use to parse files from the upload form and UUID gives us a way to create a unique id that we can use for a key both in S3 and in the database.
npm install --save formidable uuid
Next, create a new file in the views folder named upload.ejs and add the following code:
<form action="/upload" enctype="multipart/form-data" method="post">
<div>
<label>File:</label>
<input type="file" name="file" /><br />
</div>
<div>
<input type="submit" value="Upload" />
</div>
</form>
Also, we need to add a link to the upload form for authenticated users in home.ejs
<% if (!user) { %>
<p>
Welcome! Please <a href="/login">log in</a> or
<a href="/register">register</a>.
</p>
<% } else { %>
<div>
<p><a href="/upload">Upload</a></p>
<p><a href="/logout">Log out</a></p>
</div>
<% } %>
One more step left to finish up the upload form. We need to add the code to our server to accept the uploaded files, upload them to S3 and create a new row in the uploads table. If you remember when we declared our upload model up above we made the id column for the uploads table a UUID. This is where we generate that UUID when a new file is uploaded. This generated UUID is used both for the file key in S3 and the id in the database table.
...
const formidable = require("formidable");
const { v4: uuidv4 } = require("uuid");
const fs = require("fs");
const AWS = require("aws-sdk");
const S3 = new AWS.S3({
signatureVersion: "v4",
apiVersion: "2006-03-01",
accessKeyId: "YOUR_ACCESS_KEY_HERE",
secretAccessKey: "SECRET_ACCESS_KEY_HERE",
region: "us-west-2",
});
...
app.get("/upload", (req, res) => {
res.render("upload");
});
app.post("/upload", (req, res, next) => {
const form = formidable({ multiples: true });
form.parse(req, async (err, fields, files) => {
if (err) {
next(err);
return;
}
const id = uuidv4();
S3.putObject(
{
Bucket: "YOUR_BUCKET_NAME_HERE",
Key: id,
ContentType: files.file.type,
ContentLength: files.file.size,
Body: fs.createReadStream(files.file.path),
},
async (data) => {
await models.upload.create({
id,
file_name: files.file.name,
user_id: req.user.id,
});
res.redirect("/");
}
);
});
});
This is the state of the full server.js file after the latest updates.
const express = require("express");
const passport = require("passport");
const Strategy = require("passport-local").Strategy;
const models = require("./models");
const formidable = require("formidable");
const { v4: uuidv4 } = require("uuid");
const fs = require("fs");
const AWS = require("aws-sdk");
const S3 = new AWS.S3({
signatureVersion: "v4",
apiVersion: "2006-03-01",
accessKeyId: "YOUR_ACCESS_KEY_HERE",
secretAccessKey: "YOUR_SECRET_ACCESS_KEY_HERE",
region: "us-west-2",
});
passport.use(
new Strategy(async (username, password, cb) => {
const user = await models.user.findOne({
where: {
email: username,
},
});
if (!user) {
return cb(null, false);
}
if (user.password !== password) {
return cb(null, false);
}
return cb(null, user);
})
);
passport.serializeUser((user, cb) => {
cb(null, user.id);
});
passport.deserializeUser(async (id, cb) => {
const user = await models.user.findByPk(id);
if (!user) {
return cb({});
}
cb(null, user);
});
const port = process.env.PORT || 3000;
const app = express();
app.set("views", __dirname + "/views");
app.set("view engine", "ejs");
app.use(
require("express-session")({
secret: "changeme",
resave: false,
saveUninitialized: false,
})
);
app.use(passport.initialize());
app.use(passport.session());
app.use(express.urlencoded({ limit: "100mb" }));
app.use(require("body-parser").urlencoded({ extended: true }));
app.get("/health", (req, res) => {
res.sendStatus(200);
});
app.get("/", (req, res) => {
res.render("home", { user: req.user });
});
app.get("/login", (req, res) => {
res.render("login");
});
app.post(
"/login",
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/register", (req, res) => {
res.render("register");
});
app.post(
"/register",
async (req, res, next) => {
await models.user.create({
email: req.body.username,
password: req.body.password,
});
next();
},
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/upload", (req, res) => {
res.render("upload");
});
app.post("/upload", (req, res, next) => {
const form = formidable({ multiples: true });
form.parse(req, async (err, fields, files) => {
if (err) {
next(err);
return;
}
const id = uuidv4();
S3.putObject(
{
Bucket: "YOUR_BUCKET_NAME_HERE",
Key: id,
ContentType: files.file.type,
ContentLength: files.file.size,
Body: fs.createReadStream(files.file.path),
},
async (data) => {
await models.upload.create({
id,
file_name: files.file.name,
user_id: req.user.id,
});
res.redirect("/");
}
);
});
});
app.get("/logout", (req, res) => {
req.logout();
res.redirect("/");
});
// Start the app
app.listen(port, () => console.log(`API listening on ${port}`));
Retrieving User Files
The final step we need to complete to fully round our demonstration is to add a page for users to view their previous uploads and download their files. First, we will create a new template in the views folder and create a new file called files.ejs
<ul>
<% uploads.forEach(upload => { %>
<li>
<a download href="<%= upload.url %>"><%= upload.file_name %></a>
</li>
<% }); %>
</ul>
The template is pretty simple. We supply the list of uploads in the data and then loop through the list and show a download link for each.
We also need to add a link to the file list from the home page.
<% if (!user) { %>
<p>
Welcome! Please <a href="/login">log in</a> or
<a href="/register">register</a>.
</p>
<% } else { %>
<div>
<p><a href="/files">View Files</a></p>
<p><a href="/upload">Upload</a></p>
<p><a href="/logout">Log out</a></p>
</div>
<% } %>
The last thing we need to do is add a new endpoint to retrieve the upload data for the current user and serve the template we just created.
function getSignedUrl(key) {
return new Promise((resolve, reject) => {
S3.getSignedUrl(
"getObject",
{
Bucket: "image-upload-example-app",
Key: key,
},
function (err, url) {
if (err) reject(err);
resolve(url);
}
);
});
}
app.get("/files", async (req, res) => {
let uploads = await models.upload.findAll({
where: {
user_id: req.user.id,
},
});
uploads = await Promise.all(
uploads.map(async (upload) => {
const url = await getSignedUrl(upload.id);
return {
...upload.toJSON(),
url,
};
})
);
res.render("files", { uploads });
});
Note that we have to call getObject for each file in the list to generate an authenticated URL. This is because our files in S3 are not publicly available. This allows us to restrict access so that each user is only viewing their own files and not files from another user. Note by default these authenticated URLs expire after 10 minutes, but that length of time can be changed by passing a value with the Expires key when calling getObject.
Below is the final contents of server.js.
const express = require("express");
const passport = require("passport");
const Strategy = require("passport-local").Strategy;
const models = require("./models");
const formidable = require("formidable");
const { v4: uuidv4 } = require("uuid");
const fs = require("fs");
const AWS = require("aws-sdk");
const S3 = new AWS.S3({
signatureVersion: "v4",
apiVersion: "2006-03-01",
accessKeyId: "YOUR_ACCESS_KEY_HERE",
secretAccessKey: "YOUR_SECRET_ACCESS_KEY_HERE",
region: "us-east-1",
});
passport.use(
new Strategy(async (username, password, cb) => {
const user = await models.user.findOne({
where: {
email: username,
},
});
if (!user) {
return cb(null, false);
}
if (user.password !== password) {
return cb(null, false);
}
return cb(null, user);
})
);
passport.serializeUser((user, cb) => {
cb(null, user.id);
});
passport.deserializeUser(async (id, cb) => {
const user = await models.user.findByPk(id);
if (!user) {
return cb({});
}
cb(null, user);
});
const port = process.env.PORT || 3000;
const app = express();
app.set("views", __dirname + "/views");
app.set("view engine", "ejs");
app.use(
require("express-session")({
secret: "changeme",
resave: false,
saveUninitialized: false,
})
);
app.use(passport.initialize());
app.use(passport.session());
app.use(express.urlencoded({ limit: "100mb" }));
app.use(require("body-parser").urlencoded({ extended: true }));
app.get("/health", (req, res) => {
res.sendStatus(200);
});
app.get("/", (req, res) => {
res.render("home", { user: req.user });
});
app.get("/login", (req, res) => {
res.render("login");
});
app.post(
"/login",
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/register", (req, res) => {
res.render("register");
});
app.post(
"/register",
async (req, res, next) => {
await models.user.create({
email: req.body.username,
password: req.body.password,
});
next();
},
passport.authenticate("local", { failureRedirect: "/login" }),
(req, res) => {
res.redirect("/");
}
);
app.get("/upload", (req, res) => {
res.render("upload");
});
app.post("/upload", (req, res, next) => {
const form = formidable({ multiples: true });
form.parse(req, async (err, fields, files) => {
if (err) {
next(err);
return;
}
const id = uuidv4();
S3.putObject(
{
Bucket: "YOUR_BUCKET_HERE",
Key: id,
ContentType: files.file.type,
ContentLength: files.file.size,
Body: fs.createReadStream(files.file.path),
},
async (data) => {
await models.upload.create({
id,
file_name: files.file.name,
user_id: req.user.id,
});
res.redirect("/");
}
);
});
});
function getSignedUrl(key) {
return new Promise((resolve, reject) => {
S3.getSignedUrl(
"getObject",
{
Bucket: "YOUR_BUCKET_HERE",
Key: key,
},
function (err, url) {
if (err) reject(err);
resolve(url);
}
);
});
}
app.get("/files", async (req, res) => {
let uploads = await models.upload.findAll({
where: {
user_id: req.user.id,
},
});
uploads = await Promise.all(
uploads.map(async (upload) => {
const url = await getSignedUrl(upload.id);
return {
...upload.toJSON(),
url,
};
})
);
res.render("files", { uploads });
});
app.get("/logout", (req, res) => {
req.logout();
res.redirect("/");
});
// Start the app
app.listen(port, () => console.log(`API listening on ${port}`));
Conclusion
That wraps things up for our code. We have built everything we need to let users register, log in, upload files, and view and download their uploaded files. For this example, we’ve utilized AWS’s S3 to store the image files which is the scalable way to handle file storage for your applications.
No to put it kindly our basic example app does the job but is quite ugly. A possible next step for this project could be to convert the UI from EJS templates to a SPA framework like React or Vue. Other ideas for improvements to be made on this example are:
- show a loading progress indicator for file uploads
- kick users back to the login page if they end up on one of the other routes without an active session
- allow multiple file uploads at one time
- create a thumbnail of each image on upload and also store those in S3
I hope you have found this example useful. You can find the complete code example on Github. Thank you for reading!