Building Asynchronous Queues in Javascript

I found this design pattern as part of my previous series and thought I’d expand it to its own mini-post.

First some background: sometimes we have a series of asynchronous operations that need to occur in a particular order. For example, in a human resources app:

  1. A job applicant uploads their CV, then

  2. the server saves that raw data and processes it for particulars (e.g.: university GPA, if it’s for a graduate role, or perhaps keywords instead), then

  3. the server process then serialises that data and saves it as a database entry, then

  4. the server responds with a “We’ve successfully saved your application and look forward to speaking with you soon”, or some other such canned message.

We can take these fundamentally asynchronous operations – file I/O, parsing, and database calls – and process them as Promises:

app.post('/apply', (req, res, next) => {

  saveUploadedFile(res.body)
    .then(filePointer => processApplicationDocument(filePointer))
    .then(serialisedData => saveToDatabase(serialisedData))
    .then(dbResponse => res.send('Thank you.'))
    .catch(error => {
      handleError(error);
      res.send('Whoops!');
    });

});

Or we can use async / await syntax:

app.post('/apply', async (req, res, next) => {

  try {
    const filePointer = await saveUploadedFile(res.body);
    const serialisedData = await processApplicationDocument(filePointer);
    const dbResponse = await saveToDatabase(serialisedData);
    res.send('Thank you.');
  } catch (error) {
    handleError(error);
    res.send('Whoops!');
  }

});

And both of these work. Unfortunately, they’re brittle – the structure of the code depends on the business logic in which it’s operating. What if we’re performing the same process, of performing asynchronous operations in a queue, over and over again?

Well, in the series I just finished writing I stumbled on a useful pattern to handle this. I’m sure there are others, but it boils down to this:

Each step is its own function, which returns a Promise:

// Gross oversimplification of what really happens:
function saveUploadedFile(fileData) {

  return new Promise((resolve, reject) => {

    const filePointer = './abcde.pdf';

    fs.writeFile(filePointer, fileData, (err) => {
      if (err) {
        return reject(err);
      } else {
        return resolve(filePointer);
      }
    })

  });

}

These functions are then collected in an Array:

const queue = [
  saveUploadedFile,
  processApplicationDocument,
  saveToDatabase,
];

That Array is then reduced with an asynchronous processing function:

const result = queue.reduce(async (prev, next) => {

  try {
    const result = await prev;
  } catch (err) {
    handleError(err);

    // and optionally...
    throw err;
  }

  return next(result);

}, true);

What’s great is that we can then refactor this abstraction as its own function, and pass arbitrary numbers of asynchronous operations to it.

I’ve got a hunch that this already exists in a library somewhere, but haven’t taken the time to research that. If you do know, let me know by email or hit me up on Reddit – I’d love to get in touch.

There’s a companion repo on GitHub here if you want to immediately clone and get going.


458 Words

2019-04-06 00:00 +0000