Building Asynchronous Queues in Javascript

I found this design pattern as part of my previous series and thought I’d expand it to its own mini-post.

First some background: sometimes we have a series of asynchronous operations that need to occur in a particular order. For example, in a human resources app:

  1. A job applicant uploads their CV, then

  2. the server saves that raw data and processes it for particulars (e.g.: university GPA, if it’s for a graduate role, or perhaps keywords instead), then

  3. the server process then serialises that data and saves it as a database entry, then

  4. the server responds with a “We’ve successfully saved your application and look forward to speaking with you soon”, or some other such canned message.

We can take these fundamentally asynchronous operations – file I/O, parsing, and database calls – and process them as Promises:'/apply', (req, res, next) => {

    .then(filePointer => processApplicationDocument(filePointer))
    .then(serialisedData => saveToDatabase(serialisedData))
    .then(dbResponse => res.send('Thank you.'))
    .catch(error => {


Or we can use async / await syntax:'/apply', async (req, res, next) => {

  try {
    const filePointer = await saveUploadedFile(res.body);
    const serialisedData = await processApplicationDocument(filePointer);
    const dbResponse = await saveToDatabase(serialisedData);
    res.send('Thank you.');
  } catch (error) {


And both of these work. Unfortunately, they’re brittle – the structure of the code depends on the business logic in which it’s operating. What if we’re performing the same process, of performing asynchronous operations in a queue, over and over again?

Well, in the series I just finished writing I stumbled on a useful pattern to handle this. I’m sure there are others, but it boils down to this:

Each step is its own function, which returns a Promise:

// Gross oversimplification of what really happens:
function saveUploadedFile(fileData) {

  return new Promise((resolve, reject) => {

    const filePointer = './abcde.pdf';

    fs.writeFile(filePointer, fileData, (err) => {
      if (err) {
        return reject(err);
      } else {
        return resolve(filePointer);



These functions are then collected in an Array:

const queue = [

That Array is then reduced with an asynchronous processing function:

const result = queue.reduce(async (prev, next) => {

  try {
    const result = await prev;
  } catch (err) {

    // and optionally...
    throw err;

  return next(result);

}, true);

What’s great is that we can then refactor this abstraction as its own function, and pass arbitrary numbers of asynchronous operations to it.

I’ve got a hunch that this already exists in a library somewhere, but haven’t taken the time to research that. If you do know, let me know by email or hit me up on Reddit – I’d love to get in touch.

There’s a companion repo on GitHub here if you want to immediately clone and get going.

458 Words

2019-04-06 00:00 +0000