Command-line utilities with Node.js

One often overlooked feature of node.js is the ability to create command-line utilities. Here I want to demonstrate how simple it can be to create useful command-line utilities with the help of node.js. I’m going to create a small utility which allows for quick searching of GitHub for reposities based on keywords, owners and languages. If you’d like to jump straight into the source code, it’s available here.

Understanding the command-line

Before writing anything for the Unix command-line, no matter what scripting language you are using, it’s important to understand common patterns used for input. The most basic pattern consists of three main components: the command, options and arguments.

Command

Commands are categorized into three types:

1. Internal - a command recognized and processed by the command-line which is not dependent upon any external executable file.

2. Included - a command that requires a separate executable file that is always included with the OS and generally considered part of the OS.

3. External - a command that requires an external executable file which is not part of the OS and instead added by a third party.

Options

Command-line options can be used to modify the operation of a command. Options on Unix like systems are often indicated by a hyphen and separated by a space.

Arguments

A command-line argument is an item of information passed to the command when run. Arguments are often used to identify sources of information, or alter the operation of the command.

Creating a command-line utility

Before I dive too deep into any implementation it’s worth pointing out that this tutorial has been written and tested on a unix-like operating system (OSX). Everything should work on other unix-like operating systems but this may require some extra research.

The only dependency to getting started is Node.js. You can run which node from the command-line to find out if it’s already installed. If you already have node installed you should receive a similar response to below, else if the response is blank then node is not installed and you can download the installer from nodejs.org.

$ which node
/usr/local/bin/node

To start, create a new JavaScript file named gitsearch.js and add a shebang to the first line of it. This informs the system which interpreter to use to run our file. In our case, we want to run the file with the node interpreter (see this discussion on StackOverflow as to why we use /usr/bin/env node instead of something like /usr/local/bin/node).

#!/usr/bin/env node

Your script needs to be “executable” (so it can be run by the program loader). In order to make it executable, run chmod +x gitsearch.js, which modifies your script’s access permissions so that the program loader can execute it.

Creating the command

The simplest way to create the command would be to call it’s file path and file name which would execute your script.

./gitsearch.js

It is good practice with command-line utilities to make sure there are no other commands already in use on your system with the same name. This can be checked by using the which command and the name of our command which commandName. In this example we will be using the command gitsearch, If which gitsearch returns blank then the command is not in use.

Because this is a NodeJS script, we’re going to make this script installable using Node’s package manager, because that’s a normal thing to do if writing NodeJS scripts. Doing this will mean you can just type your script’s name, without having to worry about where you are (and where it is). Don’t worry if you don’t really know what npm is. Discussing package managers is way beyond the scope of this post - if you want to learn more about npm, read this excellent article: Package Managers: An Introductory Guide For The Uninitiated Front-End Developer

In order to make a NodeJS script installable via npm we need to create an accompanying package.json file in the same directory as your gitsearch.js script.

{
    "name": "gitsearch",
    "version": "0.0.1",

    "description": "A simple command-line tool for searching git repositories",
    "author": "Glynn Phillips",
    "engines": {
      "node": ">=0.10"
    },
    "dependencies": {
    },
    "bin": {
      "gitsearch": "gitsearch.js"
    }
}

The important part here is "bin": {"gitsearch": "gitsearch.js"} as this maps the gitsearch command to your gitsearch.js script. On the command-line, navigate to the directory which contains your files and install your script globally via npm (this might require using sudo).

cd ./path/to/directory/
sudo npm install -g

The only draw back to this is after every change to gitsearch.js you will need to re-run the npm install -g command to see your changes reflected globally.

Now running the gitsearch command will execute your script. To test this add a console.log("Hello World") to your script, re-run npm install -g and then run the command.

Options and Arguments

Command-line utilities are particularly useful when it comes to input and output operations. Options and arguments passed into a command can be accessed via process.argv. Adding console.log(process.argv); to your script and running your command with an option should return something like this:

gitsearch -g
[ 'node', '/path/to/script/gitsearch.js', '-g' ]

One of Node’s most valuable features is its community of developers and the packages they have contributed. Often packages are lightweight and have been developed to perform a specific task. A great example of this is Commander, which is designed to help build command-line interfaces and provides great functionality for handling options and arguments.

On the command-line, install commander via npm npm install commander --save (by adding the --save option the npm command will automatically update the dependencies in your package.json).

Now replace your script with the following:

#!/usr/bin/env node

var program = require('commander');

program
    .version('0.0.1')
    .usage('<keywords>')
    .parse(process.argv);

if(!program.args.length) {
    program.help();
} else {
    console.log('Keywords: ' + program.args);   
}

Here we have used node’s require function to load the commander module into the script, and then started a basic structure using commander.

Commanders .args contains just the arguments passed to a command similar to process.argv so here we have used it to check arguments exist as this utility will require at least one argument to use as a keyword for the search.

Now running gitsearch with an argument like gitsearch jquery will output Keywords: jquery (if you fail to pass an argument it will output the command’s help). Another benefit to using commander is auto generated help based on the information you provide for your options, which can also be run manually via gitsearch -h.

Using the commands required argument we can build our GitHub search api endpoint.

program
    .version('0.0.1')
    .usage('<keywords>')
    .parse(process.argv);

if(!program.args.length) {
    program.help();
} else {
    var keywords = program.args;
    var url = 'https://api.github.com/search/repositories?sort=stars&order=desc&q='+keywords;
}

As the GitHub api uses HTTP endpoints we will need to make a HTTP request. To help simplify this request we will use the Request package.

npm install request --save

#!/usr/bin/env node

var program = require('commander');
var request = require('request');

Now we can use request to make a GET request to our base url created from the required arguments.

request({
    method: 'GET',
    headers: {
        'User-Agent': 'yourGithubUsername'
    },
    url: url
}, function(error, response, body) {

    if (!error && response.statusCode == 200) {
        var body = JSON.parse(body);
        console.log(body);
    } else if (error) {
        console.log('Error: ' + error);
    }
});

Note that Github’s api requires all requests to have a valid User-Agent header which must be your username or your application name Github user-agent.

Now when you pass jquery to your gitsearch command it will return a json output of up to 100 repositories mentioning jquery and ordered by star count. This output contains a lot of data so to make it easier to scan we will use chalk to style our output.

npm install chalk --save

#!/usr/bin/env node

var program = require('commander');
var request = require('request');
var chalk = require('chalk');

For this example I have decided to loop over the response and pick out the repository name, owner, description and clone url, whilst using chalk to add the styling.

if (!error && response.statusCode == 200) {
    var body = JSON.parse(body);

    for(var i = 0; i < body.items.length; i++) {
        console.log(chalk.cyan.bold.underline('Name: ' + body.items[i].name));
        console.log(chalk.magenta.bold('Owner: ' + body.items[i].owner.login));
        console.log(chalk.grey('Desc: ' + body.items[i].description + '\n'));
        console.log(chalk.grey('Clone url: ' + body.items[i].clone_url + '\n'));
    }
} else if (error) {
    console.log(chalk.red('Error: ' + error));
}

To help refine results we can also pass in more options and arguments. There are lots of options available via the Github api and I have chosen to limit by owner and language for now.

program
    .version('0.0.1')
    .usage('[options] <keywords>')
    .option('-o, --owner [name]', 'Filter by the repositories owner')
    .option('-l, --language [language]', 'Filter by the repositories language')
    .parse(process.argv);

if(!program.args.length) {
    program.help();
} else {
    var keywords = program.args;

    var url = 'https://api.github.com/search/repositories?sort=stars&order=desc&q='+keywords;

    if(program.owner) {
        url = url + '+user:' + program.owner;
    }

    if(program.language) {
        url = url + '+language:' + program.language;
    }

    […]
}

Now running the command gitsearch jquery -o jquery -l JavaScript will return repositories that mention jquery, are owned by jquery, and use JavaScript.

Exit codes

It is important to make sure you exit your scripts correctly, once again this uses the process object. In the case of an error, the value of process.exit must be greater than 0, whereas a successful exit should equal 0. Here I have added the exit code for the successful HTTP requests and errors. When using commanders .help() an exit code is not needed as commanders handles the exiting for us.

if (!error && response.statusCode == 200) {
    var body = JSON.parse(body);
    for(var i = 0; i < body.items.length; i++) {
        console.log(chalk.cyan.bold('Name: ' + body.items[i].name));
        console.log(chalk.magenta.bold('Owner: ' + body.items[i].owner.login));
        console.log(chalk.grey('Desc: ' + body.items[i].description + '\n'));
        console.log(chalk.grey('Clone url: ' + body.items[i].clone_url + '\n'));
    }
    process.exit(0);
} else if (error) {
    console.log(chalk.red('Error: ' + error));
    process.exit(1);
}

Combining utilities

Finally I have added the --full option which outputs the full response without any manipulation or styling.

program
    .version('0.0.1')
    .usage('[options] <keywords>')
    .option('-o, --owner [name]', 'Filter by the repositories owner')
    .option('-l, --language [language]', 'Filter by the repositories language')
    .option('-f, --full', 'Full output without any styling')
    .parse(process.argv);

if (!error && response.statusCode == 200) {
    var body = JSON.parse(body);
    if(program.full) {
        console.log(body);
    } else {
        for(var i = 0; i < body.items.length; i++) {
            console.log(chalk.cyan.bold('Name: ' + body.items[i].name));
            console.log(chalk.magenta.bold('Owner: ' + body.items[i].owner.login));
            console.log(chalk.grey('Desc: ' + body.items[i].description + '\n'));
            console.log(chalk.grey('Clone url: ' + body.items[i].clone_url + '\n'));
        }
        process.exit(0);
    }
} else if (error) {
    console.log(chalk.red('Error: ' + error));
    process.exit(1);
}

This can be useful to take advantage of the other utilities available on the command-line such as grep, less and pbcopy. One easy way to combine utilties is to use the pipeline to chain operations where the output for one command becomes the input for the next.

pbcopy

pbcopy is an easy way to copy the output from a command. Piping the output to pbcopy will then allow you to paste the output into other programs.

gitsearch jquery -f | pbcopy

less

less is a pager command which breaks content down into more manageable chunks, showing only one screens worth of content at a time.

gitsearch jquery -f | less

grep

grep is a utility for searching plain text data sets using regular expressions

gitsearch jquery -f | grep watchers

Conclusion

Command-line utilities are great for simplifying tasks or automating repetitive operations and NodeJS can be a great stepping stone for developers looking to build commands without learning a shell script.

This was a basic example of how to get started with NodeJS on the command-line, if you want to view the full source could I have created a gist here. There are plenty of other resources available including lots of very useful tools already published on npm and GitHub.

Further reading

• Command Line Utilities with Node.js
• More Than Web: Six Node.js CLI Apps
• Unix and Node: Command-line Arguments
• Writing a Command Line Node Tool
• Eight Terminal Utilities Every OS X Command Line User Should Know