Introduction
Queues can solve a lot of different problems in an interesting way, to reduce the load on processes on one server by breaking up separating heavy processes into other servers. In Nodejs there are some good libraries to use for queuing processes like bee-queue, bull, and others. In this practice, we will try it by using a bull. Simple case study user wants to download data, here I will give an example with a little dummy data.
Here we will create two applications that have separate roles as follows:
-
Producer: Node program to add jobs to the queue
-
Consumer: Node program that defines the function of the process, and the work to be done
Creating the Application
First, create a folder and initialize our application using the following command in the terminal by pointing to the folder that was created earlier:
$ npm init
After initialize is done, then we will install the bull and dotenv packages, with this command in the terminal:
$ npm install bull dotenv
Then create a file with the name config.js and enter the code as shown below, for the REDIS_HOST and REDIS_PORT variables, fill in the Redis host and port they have.
config.js
After that create a file with the name handler.js, consumer.js and produer.js like this:
Now let's test the application, first we test the application using a delay of 1 minute, open two terminals navigate to our folder and run it as in the video below:
If you don't want any delay in the queue then you can comment the code in the delay section in producer.js in this section.
Conclusion
A Queue has the concept of FIRST-IN-FIRST-OUT which can map orders that are carried out simultaneously (asynchronously) to be ordered and take turns one by one (synchronous).
In a real case, I use queue with bull to export more than 1 million data into the xlsx format. It doesn't slow down the server and is smooth when exporting.
The link for the full source code is here:
GitHub - firmanJS/producer-consumer-example
Thank you for reading!