How to Retrieve All Data via Paginated API Calls in Vanilla JavaScript

A few months ago, as I was building a real-time web app to monitor 🚌 bus arrival times, one of the primary requirements was to render all bus stops onto the basemap on the first load:

Screenshot by Author | An image of rendered Bus Stops on the web application | Data was retrieved via a paginated API from transport provider

There were 5,098 bus stops in total and the transportation data API returned a maximum of 500 items per page.

In order to render all bus stops on the first load, it was necessary to first aggregate all paginated data chunks on the Node.js backend server.

Illustration by Author | A diagram which represents the above API pagination format and offset

While there are several code examples online for reference, all the code snippets I have encountered *either *centered around (1) Recursion or (2) Iteration algorithm approaches. For convenience, both approaches + their respective code snippets have been included in the following part of this article for reference. Feel free to tweak and use them for your use-cases.

(1) Recursion

Here is the recursion code snippet I have implemented to aggregate all data chunks:

recursion.js

// ----------------------------------------------------------------------
// ➜ Recursion
// ----------------------------------------------------------------------
const API_ENDPOINT = "http://datamall2.mytransport.sg/ltaodataservice";
const PAGE_SIZE = 500; // How many records the API returns in a page.
const LTA_API_KEY = process.env.LTA_API_KEY;

router.get("/ltaodataservice/:transportation", (req, res) => {
  let params = req.params;
  let transportation = params["transportation"];

  var arr_result = [];
  var offset = 0;

  function callAPIService(transportation, offset) {
    request(
      {
        url: `${API_ENDPOINT}/${transportation}?$skip=${offset}`,
        method: "GET",
        json: true,
        headers: {
          AccountKey: LTA_API_KEY,
          accept: "application/json",
        },
      },
      (err, response, body) => {
        let result = {};
        if (err || response.statusCode !== 200) {
          return res.status(500).json({
            type: "error",
            message:
              err !== null && typeof err.message !== "undefined"
                ? err.message
                : "Error. Unable to retrieve data from datamall.lta.gov.sg Bus Routing API.",
          });
        } else {
          result = body.value;
          arr_result = arr_result.concat(result);
          offset += PAGE_SIZE;
          if (result.length < PAGE_SIZE) {
            return res.status(200).json(arr_result);
          } else {
            callAPIService(transportation, offset);
          }
        }
      }
    );
  } // end recursive method
  callAPIService(transportation, offset);
});

Main points to note:

• Stop Condition: if(result.length<PAGE_SIZE)

Explanation: When the no. of items returned is less than 500 (the max no. of items returned per page), the function callAPIService(transportation, offset) returns the array arr_result where all data chunks containing 500 items/per chunk from previous API calls have been aggregated.

• When Stop Condition is not met: Recursion i.e. the method itself is executed instead to propagate the recursion process.

(2) Iteration — While-Loop + Async-Await & Promises

Alternatively, it is possible to implement iteration via a loop with async-await syntax to aggregate all data chunks instead.

iteration.js

// ----------------------------------------------------------------------
// ➜ Looping with Async, Await, Promise
// ----------------------------------------------------------------------
const API_ENDPOINT = "http://datamall2.mytransport.sg/ltaodataservice";
const PAGE_SIZE = 500; // How many records the API returns in a page.
const LTA_API_KEY = process.env.LTA_API_KEY;

router.get("/ltaodataservice/all/:transportation", async (req, res) => {
	let params=req.params;
	let transportation=params["transportation"];

	function resolveAsyncCall(reqOptions) {
		return new Promise(resolve => {
			request(reqOptions, ((err, res, body) => {
				let result=body.value;
				resolve(result);
			});
		});
	}
	async function asyncCall(transportation) {
		var arr_result=[];
		var offset = 0;

		var options={
			url: `${API_ENDPOINT}/${transportation}?$skip=${offset}`,
			method: "GET",
			json: true,
			headers: {
				"AccountKey" : LTA_API_KEY,
				"accept" : "application/json"
			}
		};

		var result = [];
		var toContinue=true;

		while(toContinue) {
			if(offset==0 || result.length==PAGE_SIZE) {
				result = await resolveAsyncCall(options);
				offset += PAGE_SIZE;
				options.url=`${API_ENDPOINT}/${transportation}?$skip=${offset}`;
			} else if(result.length < PAGE_SIZE) {
				toContinue=false;
			}
			arr_result=arr_result.concat(result);
		}
		return new Promise(resolve => {
			resolve(arr_result);
		});
	};

	try {
		let entireListing=await asyncCall(transportation);
		return res.status(200).json(entireListing);
	} catch(err) {
		return res.status(500).json({
			type: "error",
			message: (err !== null && typeof err.message !== "undefined") ? err.message : `Error. Unable to retrieve data from datamall.lta.gov.sg ${transportation} Routing API.`
		});
	}
});

Using Async-Await & Promise: To implement the async-await syntax, async must always be present at the function where await is implemented in:

**async** function func() {

  let dataObj = **await **getData();
}

Explanation of Code Logic: Similar to recursion, there is a termination condition for the while-loop which is if(result.length<PAGE_SIZE)

• Stop condition is met: The while-loop stops running and response is returned. Boolean is set to false to stop iterationtoContinue=false;

• Else: while-loop continues running and more data chunks are concatenated to the array arr_result

Overall Thoughts on Recursion vs Iteration:

In the above use-case where the no. of bus stops ~5K, I would personally prefer using recursion due to the shorter length of code as compared to iteration.

However, in situations where the total no. of items returned in the aggregated data array is unknown and the final no. of items could be enormously huge, it would be wiser to use an iterative approach such as a while-loop for each API call instead. This is because, in recursion algorithms, a large volume of API calls can lead to **stack overflows and cause your program to crash during runtime **🙃

Another caution to note is that recursion algorithm complexity O(n) increases exponentially with more API calls. Hence when no. of items retrieved is huge and the expected number of API calls is correspondingly high, it would be wiser to stick to the iteration approach rather than recursion to avoid compromising your application’s performance.

Many thanks for persisting to the end of this article! ❤ Hope you found the above 2 code snippets useful and feel free to follow me on Medium if you would like more Data Analytics & Web application-related content. Would really appreciate it 😀

FYI: Feel free to check out the following articles if you are interested in the aforementioned Bus Route web application.

Building a real-time web app in NodeJS Express with Socket.io library

Tackling Heroku H12 timeout errors of Node.js Web APIs — Handling Long Response Times