Asynchronous Operations

All of our code up until now has involved purely synchronous operations. Every operation we run, or function we call, gives us back a result (or produces some output) by the time it is finished running. Operations are normally synchronized with each other: each operation waits to start only after the previous one is finished.

console.log('first');

const fruits = ['apple', 'banana', 'cherry'];
console.log(fruits);

console.log('last');

If we run the above code, we will see the output in the order we expect: “first”, followed by the fruits, and then “last”.

On the other hand, asynchronous operations do not wait for other operations to finish; they run whenever they are ready. Asynchronous operations come about in two circumstances:

when specific timing is involved, or
when dealing with external data (any data from outside of our JavaScript code).

There are three ways of handling asynchronous operations: callbacks, promises, and async/await (which is simplified syntax for using promises).

Using Callbacks

If a function performs an asynchronous operation, it is up to us to decide what to do after the operation is complete. In order to give us flexibility, the asynchronous function may take in a callback function as an argument. We get to define the callback function to do whatever it is we want to do after the asynchronous operation, and provide this callback to the asynchronous function. It is the asynchronous function’s job to call our callback function (that’s what makes it a “callback”).

const imaginaryAsyncFunc = (callback) => {
  // ... do some asynchronous work here ...

  callback();
};

const afterOperation = () => {
  console.log('The async operation is complete!');
};

imaginaryAsyncFunc(afterOperation);

Timing example

setTimeout() is a built-in asynchronous function that allows us to do something after a specified time. It takes two arguments: a callback function and a number of milliseconds. The callback will run after the specified number of milliseconds. The reason setTimeout() takes a callback as an argument is to allow for some action to be performed after some time. To be flexible (i.e., to allow for any action), a function makes the most sense here. Callbacks are often used for asynchronous operations for exactly this reason. However, the use of callbacks is not always evidence of asynchronous code. As we have seen before, callbacks are frequently used in synchronous code as well.

Let’s try mixing some asynchronous code into the first example.

console.log('first');

const fruits = ['apple', 'banana', 'cherry'];

setTimeout(() => {
  console.log(fruits);
}, 1000);

console.log('last');

Now, even though the code to print the fruits comes before the code to print “last”, we will see “last” show up before the fruits in the output. In fact, it will take one second (1000 milliseconds) before the fruits show up. Even if we set the timeout delay to 0, the fruits will still be the last to print. This is because asynchronous code only runs after all of the synchronous code is finished. In running the above example, when it comes to the setTimeout() code, the callback is put aside, into an imaginary asynchronous box, and then the program carries on with the rest of the synchronous code. After all of the synchronous code is finished, everything inside the asynchronous box gets to run.

console.log('first');

const fruits = ['apple', 'banana', 'cherry'];

fruits.forEach(fruit => {
  console.log(fruit);
});

console.log('last');

The above code uses a callback to print the fruit, but there are no asynchronous operations going on. The output order is the same as in the first example: “first”, the fruits, then “last”.

External data example

Let’s decide on a task: read in a file of text and copy all of the text before ‘---’ to a second file. For example, we can have a file ‘first.txt’ that contains the lines:

here
are some lines
to copy
---
but none of this
should get copied

Our program should copy only the first three lines to another file ‘second.txt’.

Node comes with a built-in module fs for dealing with any file system operations. Because these operations deal with data from outside of our code, they are asynchronous. The readFile() function will allow us to read in a file and store its contents. According to the documentation, readFile() takes in three arguments: a path to the file (string), an encoding of the file contents (string), and a callback.¹ The callback will run after the operation is complete. In order to provide the callback, we also need to know what arguments it takes in. Remember, it is not our job to call the callback; it is readFile() that will call it, passing it specific arguments. We only decide what to do with the information given to the callback’s arguments. The documentation says the callback takes in two arguments: a possible error (object) and the file’s contents (string). When dealing with external data, things can go wrong in many different ways (e.g., trying to read a file that doesn’t exist), so handling errors is important.

We now have enough information to start. Let’s print the file’s contents.

const fs = require('fs'); // Import the module as one big object

const afterRead = (err, data) => {
  if (err) throw err; // Print the error and quit
  console.log('File contents:');
  console.log(data);
};

// Read a file called 'first.txt' in the same directory as this js file
// utf8 is the most common text encoding
fs.readFile('first.txt', 'utf8', afterRead);

Try it out and compare what happens when the file ‘first.txt’ doesn’t exist and after you create it.

The next step is to extract the part we want to copy from the contents. At this point, we are dealing with a plain string, so we can use any string manipulation techniques. The split() method works well here.

const fs = require('fs'); // Import the module as one big object

const afterRead = (err, data) => {
  if (err) throw err; // Print the error and quit
  const parts = data.split('---');
  console.log('Contents to copy:');
  console.log(parts[0]);
};

fs.readFile('first.txt', 'utf8', afterRead);

For dealing with the second file, the writeFile() function allows us to write to a file. It will create the file if it doesn’t exist. According to the documentation, writeFile() takes in four arguments: a path to the file (string), the data to write (string), an encoding of the file contents (string), and a callback. This time, the callback only takes a single argument: a possible error (object). The file writing operation either succeeds or it doesn’t; there is no extra data to work with. The callback will be called after the operation is complete. We can now use writeFile() to finish the task.

const fs = require('fs'); // Import the module as one big object

const afterWrite = err => {
  if (err) throw err; // Print the error and quit
  console.log('Successfully copied the contents!');
};

const afterRead = (err, data) => {
  if (err) throw err; // Print the error and quit
  const parts = data.split('---');
  fs.writeFile('second.txt', parts[0], 'utf8', afterWrite);
};

fs.readFile('first.txt', 'utf8', afterRead);

console.log('Starting task...'); // This prints first!

There are two important things to notice with our finished code. First, the line that has been added to the end will be first to print because it is synchronous while the rest of the code is triggered by the asynchronous readFile(). Second, the code does not read very linearly. It takes significant mental effort to trace the order of operations in this code. We can make a change, replacing the callbacks with their equivalent anonymous functions.

const fs = require('fs'); // Import the module as one big object

fs.readFile('first.txt', 'utf8', (err, data) => {
  if (err) throw err; // Print the error and quit
  const parts = data.split('---');
  fs.writeFile('second.txt', parts[0], 'utf8', err => {
    if (err) throw err; // Print the error and quit
    console.log('Successfully copied the contents!');
  });
});

console.log('Starting task...'); // This prints first!

In the above code, the order of operations now matches the written order (top to bottom), with the exception of the last line. However, there is a cascading effect. As more asynchronous operations are added (and we follow standard code styling), their callbacks are indented more and more. If we added ten more successive asynchronous operations, the last one’s code would be indented quite a lot! It seems we have to make a decision with unfortunate tradeoffs. This is where promises are helpful.

Using Promises

As a sidenote, this material will not be about creating promises. It is much more important to learn how to use them. Just like we don’t need to know how to create console.log() or other built-in functions from scratch, even if the knowledge may be interesting. In practice, you can go a long way without ever needing to create your own promises. Once you are comfortable with using promises, you should be able to easily learn how to create them as well.

Promises are special objects for dealing with asynchronous operations. A promise has three potential states: pending, fulfilled, and rejected. This suits the pattern of what happens when dealing with external data. Taking the previous example of reading in the contents of a text file on our computer, the operation ‘read in the contents of first.txt’ will first be pending while node looks for the file (this is what makes it asynchronous). Once the file is found and its contents are read in, the operation is fulfilled and we can then do what we want with the file’s contents. If instead the file is not found because it doesn’t exist, the operation is rejected and we can print a message saying so.

The pending state is entered any time we call a function that returns a promise. To handle the fulfilled state, promises have a then() method. To handle the rejected state, there is a catch() method. Both then() and catch() take in a single argument: a callback. In the case of then(), the callback may have an argument to store some data that is provided by the completed asynchronous operation. For catch(), the callback typically only has the asynchronous operation’s error as its argument.

Both then() and catch() do something interesting: they return a promise containing the result of the callback. This allows these promise methods to be used on each other, chaining them in sequence.

External data example

We use the same task as before: read in a file of text and copy all of the text before ‘---’ to a second file.

Node also includes a version of the fs module that uses promises instead of callbacks for asynchronous functions. Depending on your version of node, it can be imported either as:

const fs = require('fs/promises'); // Node.js v14.x or newer

Or:

const fs = require('fs').promises; // Node.js v13.x or older

According to the documentation, the promise version of readFile() takes in the same arguments as its counterpart, minus the callback. This leaves us with two arguments: a path to the file (string), and an encoding of the file contents (string). The function returns a promise which provides the contents of the file upon being fulfilled, or an error (object) upon being rejected.

Let’s print the file’s contents, this time using promises.

const fs = require('fs').promises; // Import the module as one big object

const readPromise = fs.readFile('first.txt', 'utf8');

readPromise.then(data => {
  console.log('File contents:');
  console.log(data);
});

readPromise.catch(err => { // Catch any errors with readFile
  console.log('Something went wrong with readFile:');
  console.log(err);
});

As different as it may look, this code works the same as the corresponding step in the callback example. Typically, promises are not used quite this way. It is more common to take advantage of the chaining aspect of promises as follows.

const fs = require('fs').promises; // Import the module as one big object

fs.readFile('first.txt', 'utf8')
  .then(data => {
    console.log('File contents:');
    console.log(data);
  })
  .catch(err => { // Catch any errors with readFile
    console.log('Something went wrong with readFile:');
    console.log(err);
  });

If anything goes wrong in the file reading operation, the then() will be skipped and the catch() will run instead. This is advantageous over the callback example, since the error handling is not in the same block of code as dealing with the file’s contents.

The rest of the task can be completed using writeFile() similarly as before.

const fs = require('fs').promises; // Import the module as one big object

fs.readFile('first.txt', 'utf8')
  .then(data => {
    const parts = data.split('---');
    fs.writeFile('second.txt', parts[0], 'utf8')
      .then(() => {
        console.log('Successfully copied the contents!');
      })
      .catch(err => { // Catch any errors with writeFile
        console.log('Something went wrong with writeFile:');
        console.log(err);
      });
  })
  .catch(err => { // Catch any errors with readFile
    console.log('Something went wrong with readFile:');
    console.log(err);
  });

console.log('Starting task...');

The above code can be cleaned up by again taking advantage of chaining. If we instead return the writeFile() result, which is a promise, we can chain another then() after the first one. A side effect of this is that the catch() will catch any errors from both readFile() and writeFile(), for better or for worse.

const fs = require('fs').promises; // Import the module as one big object

fs.readFile('first.txt', 'utf8')
  .then(data => {
    const parts = data.split('---');
    return fs.writeFile('second.txt', parts[0], 'utf8');
  })
  .then(() => {
    console.log('Successfully copied the contents!');
  })
  .catch(err => { // Catch any errors
    console.log('Something went wrong:');
    console.log(err);
  });

console.log('Starting task...');

From here, we could continue the chaining pattern to add successive asynchronous operations and our code would remain linear without excessive nesting. While this is an improvement on the pure callback approach, we are still dealing with callbacks inside of then() and catch(). The following section shows how we can avoid callbacks altogether and write linear-reading code that involves asynchronous operations.

Using Async/Await

The async and await keywords are syntactic sugar over promises, meaning they use promises exactly the same way as described above but simply make the code look different. Instead of having a function explicitly return a promise, it can be defined with the word ‘async’ before it.

const someFunc = async () => {
  // ... do something asynchronous stuff ...
};

And instead of using then() on a function returning a promise, it can be called with the word ‘await’ before it.

const data = await someFunc();

There are two rules to using async/await:

If a function is defined with ‘async’, then it can be called with ‘await’.
‘await’ can only be used inside a function defined with ‘async’.

So to transform the first part of our previous exercise from using explicit promises to async/await, we can ‘await’ the call to readFile().

const fs = require('fs').promises;

const data = await fs.readFile('first.txt', 'utf8');
console.log('File contents:');
console.log(data);

However, there is a catch. The above code won’t run in node because we violated rule 2.² We have used ‘await’ outside of an async function. There is a simple trick to fix this: we can put the code inside an async function, then call the function.

const fs = require('fs').promises;

const go = async () => {
  const data = await fs.readFile('first.txt', 'utf8');
  console.log('File contents:');
  console.log(data);
};

go();

With the exception of the ‘go’ function, we already have a cleaner result than its promise or callback counterpart. The steps appear linear even though we are dealing with an asynchronous operation.

Applying the async/await syntax to the rest of the task gives us the following result.

const fs = require('fs').promises;

const go = async () => {
  console.log('Starting task...');
  const data = await fs.readFile('first.txt', 'utf8');
  const parts = data.split('---');
  await fs.writeFile('second.txt', parts[0], 'utf8');
  console.log('Successfully copied the contents!');
};

go();

But there’s still one thing missing. The above code will work so long as nothing goes wrong with either asynchronous operation. This is not a fair comparison to the final code using callbacks or promises unless we include the same error handling. With async/await, the way to handle errors is by using the try...catch statement. It works a lot like an if...else statement. The try block runs if there are no errors, otherwise the catch block runs and catches the error as well.

The final code is as follows.

const fs = require('fs').promises;

const go = async () => {
  try {
    console.log('Starting task...');
    const data = await fs.readFile('first.txt', 'utf8');
    const parts = data.split('---');
    await fs.writeFile('second.txt', parts[0], 'utf8');
    console.log('Successfully copied the contents!');
  } catch (err) {
    console.log('Something went wrong:');
    console.log(err);
  }
};

go();

Comparison

Task

We start with a file ‘notes.txt’ that has public and private text, separated by a line ‘---’. Copy the public notes into a file ‘public.txt’, then copy the private notes into a file ‘private.txt’. Finally, remove the original file ‘notes.txt’.

With callbacks

const fs = require('fs');

const afterCopy = err => {
  if (err) {
    console.log('Something went wrong:');
    throw err;
  }

  console.log('Removing original file...');
  fs.unlink('notes.txt', err => {
    if (err) {
      console.log('Something went wrong:');
      throw err;
    }

    console.log('Finished!');
  });
};

const afterRead = (err, notes) => {
  if (err) {
    console.log('Something went wrong:');
    throw err;
  }

  const [publicNotes, privateNotes] = notes.split('---');

  console.log('Copying public notes...');
  fs.writeFile('public.txt', publicNotes, 'utf8', err => {
    if (err) {
      console.log('Something went wrong:');
      throw err;
    }

    console.log('Copying private notes...');
    fs.writeFile('private.txt', privateNotes, 'utf8', afterCopy);
  });
};

fs.readFile('notes.txt', 'utf8', afterRead);

With promises

const fs = require('fs').promises;

fs.readFile('notes.txt', 'utf8')
  .then(notes => {
    const [publicNotes, privateNotes] = notes.split('---');

    console.log('Copying public notes...');
    return fs.writeFile('public.txt', publicNotes, 'utf8')
      .then(() => {
        console.log('Copying private notes...');
        return fs.writeFile('private.txt', privateNotes, 'utf8');
      })
  })
  .then(() => {
    console.log('Removing original file...');
    return fs.unlink('notes.txt');
  })
  .then(() => {
    console.log('Finished!');
  })
  .catch(err => {
    console.log('Something went wrong:');
    console.log(err);
  });

With async/await

const fs = require('fs').promises;

const go = async () => {
  try {
    console.log('Gathering notes...');
    const notes = await fs.readFile('notes.txt', 'utf8');
    const [publicNotes, privateNotes] = notes.split('---');

    console.log('Copying public notes...');
    await fs.writeFile('public.txt', publicNotes, 'utf8');

    console.log('Copying private notes...');
    await fs.writeFile('private.txt', privateNotes, 'utf8');

    console.log('Removing original file...');
    await fs.unlink('notes.txt');

    console.log('Finished!');
  } catch (err) {
    console.log('Something went wrong:');
    console.log(err);
  }
};

go();

Technically, readFile()’s arguments are more flexible than described here but we can ignore the optional arguments for our purposes.↩︎
Since Node.js v14.8, top-level ‘await’ is allowed.↩︎

Prev lesson

Project - batteries.js

Top