Nullivex

Linux and Node.js in Production using Daemontools and NDT

Posted: Sep 26, 2015 by Bryan Tong

Any developer who has written a Node.js application or any company looking to use a Node.js application comes to a point where its time to finally put the application to the test.

This is where, for me, a lot of questions started happening. In fact, before writing this blog, I believed there wasn't a good way to run Node.js reliably on Linux.

Why I built my own?

First of all, lets go over the existing solutions I know of.

  • Skip it and use Heroku or our upcoming service NodeHD :)
  • Install screen and use something like screen -a -S myApp and then run the app on the following terminal
  • Use a process manager such as pm2

Now I want to address why I dislike all these solutions.

Not Heroku

Currently, I have projects that span well beyond the scope of Heroku. Also, I can imagine most companies want to run their production applications in their own environment. That is what leaves Heroku out for me.

  • Not private
  • Feature limitations
  • Scale limitations
  • Cost at scale

Not Screen

Next, when I first started running Node applications in production, I went the screen route. Its not bad because it captures all the logging and you can always check the interactive shell when you need to look at your app. However, there are a few bad things with this approach.

  • It doesn't start at boot time
  • Apps that generate a lot of output will be lost by the scrollback buffer
  • If someone kills your screen, your app dies too
  • No watchdog or crash handling

Not PM2

Okay, so PM2 is good. I am not going to say it wont work for people especially in smaller environments. However, at scale I ran into enough problems with PM2 that I couldn't continue using it and had to move on to creating my own solutions.

What happened?

  • The monolithic design of PM2 means that all of your applications share a single process. This means that if something crashes the pm2 process, all of your apps die.
  • PM2 does not release bound ports without a full process restart (which kills all of your apps). Sometimes, it is useful to run an application in the foreground for debugging purposes.
  • With PM2 it is difficult to control the environment in which the application runs.
  • Inefficient CLI API.

Daemontools and NDT to the rescue!

Finally, its time to deliver the good news. It is time to start running your applications in the cleanest most resilient way possible!

I built this workflow from the ground up to be conscious of the developer and the system administrator. That way it is easy to debug in production, doesn't get in the way of development, and provides multiple layers of crash handling. Also, built in scalability!

How it Works

Daemontools is a great run system on linux. It will restart your process if it dies, and starts your process on system start up. Daemontools also logs your programs output. I find it to be far easier to use with applications then using the traditional init.d system with Linux. Where I find the problem with Daemontools is the somewhat cryptic and non user-friendly interface.

  • Daemontools will run your application.
  • NDT will control Daemontools (friendlier API)
  • Infant will control your application and process clustering

Implementation

Okay, so finally its time to go over how to set this up. I am going to assume you are running Debian or Ubuntu to set this up otherwise please adapt to your favorite flavor.

Install Daemontools

On Debian this can be done easily.

$ apt-get install daemontools daemontools-run

In order to make the Debian version of Daemontools act like the GNU distribution a symlink must be made.

$ ln -s /etc/service /service

Install NDT

Check out NDT here https://www.npmjs.com/package/ndt

NDT is a wrapper around daemontools that will create its cryptic file structure and manage your Node.js application by creating a dt.json file which will describe how the application is to be ran in production.

I generally refrain from adding a dt.json to my repositories but rather create the file in the folder where code is ran from. This is where environmental changes can be made.

Installation is simple.

$ npm -g install ndt

Get Application, Generate dt.json

In this example I am going to use bowercdn (http://www.bowercdn.net) as an example.

$ cd /opt
$ git clone [email protected]:eSited/bowercdn.git
$ cd /opt/bowercdn
$ npm install
$ env | ndt generate --command "node app" --stdin

Okay at this point the code is checked out, and we have created a dt.json using the current folder and environment to generate it. At this point I recommend checking over the dt.json file and verifying it before continuing.

Here is an example dt.json

{
  "name": "bowercdn",
  "cwd": "/opt/bowercdn",
  "user": "node",
  "command": "node app",
  "env": {
    "NODE_ENV": "production",
    "HOME": "/home/node",
    "LANG": "en_US.UTF-8",
    "LOGNAME": "node",
    "MAIL": "/var/spool/mail/node",
    "PATH": "/usr/local/bin:/bin:/usr/bin:/home/node/bin",
    "PYTHON": "/usr/bin/python2.6",
    "SHELL": "/bin/bash",
    "SHLVL": "1",
    "TERM": "dumb",
    "USER": "node"
  },
  "log": {
    "user": "node",
    "command": "multilog s16777215 t /var/log/node/bowercdn"
  }
}

Create a 'node' user

Notice in the dt.json output we are using a 'node' user. I prefer to have all my node.js applications run under this user.

Before continuing it is good to make sure this user exists and owns your code files.

$ useradd node
$ cd /opt/bowercdn
$ chown -R node. .

This will create the user and then own your code folder and all children to the node user.

Installing the Application

Now it is time to install the application so continuing from the previous example.

$ cd /opt/bowercdn
$ ndt install
$ ndt save

Once the ndt install command is ran, the ndt will create the folder structure that daemontools requires and will link the newly created dt folder to /service/bowercdn . Once daemontools sees this folder it will scan it and automatically start your application.

Checking the log

It is imperative to be able to check the logs from your application for errors or other useful output. This can be done easily using the logging that comes with our instance of daemontools.

$ cd /opt/bowercdn
$ tail -f log/current

Controlling the Application

Now comes the part where ndt makes life a lot easier.

Stop

$ ndt bowercdn stop

Start

$ ndt bowercdn start

Restart

$ ndt bowercdn restart

Restart all running Applications

$ ndt all restart

Thats it! Now you are running applications like a pro on Linux and daemontools will keep your application online if it crashes since it has automatic watchdog abilities built in.

Check out my next blog on how to use https://www.npmjs.com/package/infant to make your Node.js applications scale!