Node.js - File System

Overview

This article is Part 5 of Working With NodeJS, a series of articles to help simplify learning Node.js. In this topic I will cover working with the file system.

Introduction

To be able to work with the file system using Node.js you need to understand the fs and path modules. These modules provide the means to interact with a file system. The path module is important for path manipulation and normalization across platforms, and the fs module provides APIs for reading, writing, file system meta-data, and file system watching.

Paths

When navigating a file system it is helpful to have a reference point. The __dirname and __filename variables can provide such a point. These two variables are defined whenever Node.js executes a modules code. The __filename variable holds the absolute path to the currently executing file, and __dirname holds the absolute path to the directory where the currently executing file resides.

console.log(__dirname); // Path to current working directoryconsole.log(__filename); // Path to current executing file

With these reference points, we can build paths for files and directories programmatically. The most commonly used method to build a path is the join method. The join method joins all given path segments together using the platform-specific separator as a delimiter, then normalizes the resulting path.

const { join } = require('path');console.log(join(__dirname, 'out.txt'));

The above code snippet will output the absolute path for out.txt to the console. You can add as many segments to the path as you need. Lets say you wanted to construct the path for out.txt so the path leads to a subdirectory called text.

const { join } = require('path');console.log(join(__dirname, 'text', 'out.txt'));

In the above code snippet, the segment text is now apart of the absolute path for out.txt. Its important to know the join method does not create the directory text, but only a path which leads to it. There are other methods within the path module which can be used to construct a path, however, the join method is the most commonly used. In addition to path builders, there are path deconstructors. A path deconstructor will break down a path into its properties.

const { parse } = require('path');console.log(parse('/home/user/dir/file.txt'));

output

{ root: '/',  dir: '/home/user/dir',  base: 'file.txt',  ext: '.txt',  name: 'file' }

In the above code snippet, the parse method is used to deconstruct the path into the segments which compose it. The important thing to understand about the path module is that its platform specific. This means if you are working on a different operating system all the methods within the module are still applicable.

Files - Reading and Writing

The fs module provides methods for interacting with file system. The module contains both synchronous and asynchronous methods. All the names of synchronous methods in the fs module end with Sync. This important to remember because synchronous methods will block the main process until the operation has completed.

'use strict';const { readFileSync } = require('fs');const contents = readFileSync(__filename);console.log(contents);

output

<Buffer 27 75 73 65 20 73 74 72 69 63 74 27 0a 63 6f 6e 7374 20 7b 20 72 65 61 64 46 69 6c 65 53 79 6e 63 20 7d 20 3d20 72 65 71 75 69 72 65 28 27 66 73 27 ... 66 more bytes>

In the above code snippet, the readFileSync is used to read its own file contents. If no encoding is specified then a Buffer is returned. The encoding can be set by configuring an options object on readFileSync.

'use strict';const { readFileSync } = require('fs');const contents = readFileSync(__filename, {encoding: 'utf8'});console.log(contents);

In the above code snippet, the encoding was set to utf8 which will allow a string to be returned instead of a Buffer. Some encoding options you can set for read methods are: ascii, base64, hex, utf8, and binary.

Writing synchronously can be accomplished using the writeFileSync method. At a minimum the method takes a filename and the data to be written to the file.

'use strict';const { join } = require('path');const { writeFileSync } = require('fs');const out = join(__dirname, 'out.txt');writeFileSync(out, 'Hello');

If the file already exists then the operation will replace the file, and if not the file will be created. Like the read method, writeFileSync can be configured with options as well. If instead of replacing the entire file you wish to add to it, you can configure writeFileSync with the flag option and set it to append.

'use strict';const { join } = require('path');const { writeFileSync } = require('fs');const out = join(__dirname, 'out.txt');writeFileSync(out, 'Hello', {    flag: 'a'});

For a full list of supported flags, see File System Flags section of the Node.js Documentation. It is important to remember that synchronous read and write operations should be surrounded with try/catch blocks for error handling.

'use strict';const { readFileSync } = require('fs');try {    const contents = readFileSync('noFile.txt');    console.log(contents);} catch (err) {    console.error(err.message);    return;}

output

ENOENT: no such file or directory, open 'noFile.txt'

Up to this point, the reading and writing methods have been synchronous operations. However, Node.js is single threaded and therefore works best with asynchronous operations. The fs module provides both callback and promise based methods to perform asynchronous operations.

'use strict';const { readFile } = require('fs');readFile(__filename, {encoding: 'utf8'}, (err, contents) => {  if (err) {    console.error(err);    return;  }  console.log(contents);});

In the above code snippet, the readFile method is used instead of the readFileSync method. Both methods perform the same operation, however, readFile is an asynchronous operation. Let see an asynchronous writing operation.

'use strict'const { join } = require('path');const { writeFile } = require('fs');const out = join(__dirname, 'out.txt');writeFile(out, 'Hello', { flag: 'a' }, (err) => {    if (err) { console.error(err); }    else console.log('Write Successful');});

In the above code snippet, the writeFile method performs the same operation as the writeFileSync, but now its an asynchronous operation. The important thing to remember is that Node.js is single threaded so you want to avoid using operations that block further execution of code. All methods within the fs module that will block further execution of code end in Sync.

The fs module also supports promise based asynchronous operations. To use the promise based operations append the .promises on the require('fs') function.

const { readFile, writeFile } = require('fs').promises;

In the above code snippet, the both readFile and writeFile are asynchronous operations, but now they now return promises. This is useful because now async/await can be used. This gives the code a cleaner and traditional look. Lets see an example. I'll use the readFile method to read its own content and then use the writeFile method to write the content to the file out.txt.

'use strict';const { join } = require('path');const { readFile, writeFile } = require('fs').promises;async function run () {  const contents = await readFile(__filename, {encoding: 'utf8'});  const out = join(__dirname, 'out.txt');  await writeFile(out, contents);}run().catch((err) => {    console.error(err);});

out.txt

'use strict';const { join } = require('path');const { readFile, writeFile } = require('fs').promises;async function run () {  const contents = await readFile(__filename, {encoding: 'utf8'});  const out = join(__dirname, 'out.txt');  await writeFile(out, contents);}run().catch((err) => {    console.error(err);});

File Streams

Remember the fs module has four API types: Synchronous, Callback-based, Promise-based, and Stream-based. The fs module has createReadStream and createWriteStream methods which allow us to read and write files in chunks. Streams are ideal when handling very large files that can be processed incrementally.

'use strict';const { pipeline } = require('stream');const { join } = require('path');const { createReadStream, createWriteStream } = require('fs');pipeline(    createReadStream(__filename),    createWriteStream(join(__dirname, 'out.txt')),    (err) => {        if (err) {            console.error(err);        }        console.log('Finished writing');    });

In the above code snippet, its the same as before, we read the content of the file and write it to out.txt, but now we are using streams. Quick note, notice that we didn't pass any content to createWriteStream like we did for writeFile. This is because we're using a pipeline and the data is automatically passed from one phase of the pipe to the next.

Before moving on to directories lets take a look at some other useful file methods within the fs module.

change the permissions of a file

'use strict';const { chmodSync, readFileSync } = require('fs');chmodSync('out.txt', 0o000);try {    const content = readFileSync('out.txt');    console.log(content);} catch (error) {    console.error(error.message);}

The above code snippet uses the chmodSync to change the permissions of the file out.txt. When the readFileSync method is used an error is thrown EACCES: permission denied, open 'out.txt'. To learn more about chmod or chmodSync see Node.js Documentation

change the owner of a file

'use strict';const { chown } = require('fs');chown('out.txt', 6101, 120, (err) => {    if (err) {        console.error(err);    }    console.log('Owner changed');});

copying a file

const { copyFile } = require('fs');copyFile('out.txt', 'out.backup.txt', (err) => {    if (err) {        console.error(err.message);    }    console.log('File copied successful');});

deleting a file

const { rm } = require('fs');rm('out.backup.txt', (err) => {    if (err) {        console.error(err.message);    }    console.log('File deleted');});

Directories

The fs module provides the means for working with directories. Similar to files, the fs module provides three main operations to read the content of a directory.

1. Synchronous
2. Callback
3. Promise

To read the content of a directory you can use the readdirSync, readdir which is callback based, or its asynchronous counterpart readdir which is promise-based obtained through require('fs').promises. Lets see some examples.

Synchronous

'use strict'const { readdirSync } = require('fs');try {    const content = readdirSync(__dirname);    console.log(content);} catch (error) {    console.error(error);    }

Callback

'use strict'const { readdir } = require('fs');readdir(__dirname, (err, files) => {    if (err) {        console.error(err);        return;    }    console.log('callback', files);});

Promise

'use strict';const { readdir } = require('fs').promises;async function run () {    const files = await readdir(__dirname);    console.log('promise', files);}run().catch((err) => {    console.error(err);})

In the above code snippets, all three snippets do the same thing, that is read the content of the directory of the currently executing file.

File Meta Data

Metadata about files can be obtained with the following methods:

• fs.stat, fs.statSync, fs.promises.stat
• fs.lstat, fs.lstatSync, fs.promises.lstat

The only difference between the stat and lstat methods is that stat follows symbolic links, and lstat will get meta data for symbolic links instead of following them. Lets see an example of how we could read the content of a directory and determine wether if the content returned was a file or directory.

const { readdirSync, statSync } = require('fs');const files = readdirSync(__dirname);for (name of files) {    const stat = statSync(name);    const typeLabel = stat.isDirectory() ? 'dir: ' : 'file: ';    console.log(typeLabel, name);}

Watching Files and Directories

The fs module has the capabilities for watching changes to a file or directory using the watch method. Whenever changes are made the watch method is triggered and its callback function is executed. Let see a simple example.

'use strict';const { watch } = require('fs');watch(__dirname, (event, filename) => {    console.log(event, filename);});

The callback function of the watch method takes an event and filename as parameters. Both parameters correspond to the file or directory in which the change happened. For the final example we'll use a combination of what we learned in conjunction with the watch method. We'll create a small program(index.js) that will monitor the content of our working directory.

index.js

'use strict';const { join, resolve } = require('path');const { watch, readdirSync, statSync } = require('fs');const cwd = resolve(__dirname);const files = new Set(readdirSync(__dirname));watch(__dirname, (evt, filename) => {    try {        const { ctimeMs, mtimeMs } = statSync(join(cwd, filename));        if (files.has(filename) === false) {            evt = 'created';            files.add(filename);        } else {            if (ctimeMs === mtimeMs) evt = 'content-updated';            else evt = 'status-updated';        }    } catch (err) {        if (err.code === 'ENOENT') {            files.delete(filename);        } else {            console.error(err);        }    } finally {        console.log(evt, filename);    }});

In the above code snippet, we use Set, (a unique list), initializing it with the array of files already present in the current workings directory. To get the current working directory the resolve(__dirname) method is used. Alternatively, one could have used resolve('.') to achieve the same effect. Next, we begin to watch our current working directory. If a change is emitted within our working directory, we'll extract the ctimeMs and mtimeMs properties of the file using the statSync method. If the file is not apart of our list then we set the evt variable to created and add the filename using the add method. However, if the file is apart of our list then we'll check to see if the ctimeMs and mtimeMs are equal. If both properties are equal evt is set to content-updated and if not evt is set to status-updated. Finally, we log the event and filename to the console with within the finally block.

There is a lot more to discover within the path and fs modules of Node.js and you should take some time to explore the full power of its APIs. As always thank you for taking time to read this article and if you found it helpful subscribe to the series because more is on the way. Take care.