My desire: find a way to run a Node application from a Git(Hub) repository using a generic Docker container and be able to refresh the running container on the fly whenever the sources in the repo are updated. The process of producing containers for each application and upon each change of the application is too cumbersome and time consuming for certain situations – including rapid development/test cycles and live demonstrations. I am looking for a convenient way to run a Node application anywhere I can run a Docker container – without having to build and push a container image – and to continuously update the running application in mere seconds rather than minutes. This article describes what I created to address that requirement.
Key ingredient in the story: nodemon – a tool that monitors a file system for any changes in a node.js application and automatically restarts the server when there are such changes. What I had to put together:
a generic Docker container based on the official Node image – with npm and a git client inside
- adding nodemon (to monitor the application sources)
- adding a background Node application that can refresh from the Git repository – upon an explicit request, based on a job schedule and triggered by a Git webhook
- defining an environment variable GITHUB_URL for the url of the source Git repository for the Node application
- adding a startup script that runs when the container is ran first (clone from Git repo specified through GITHUB_URL and run application with nodemon) or restarted (just run application with nodemon)
I have been struggling a little bit with the Docker syntax and operations (CMD vs RUN vs ENTRYPOINT) and the Linux bash shell scripts – and I am sure my result can be improved upon.
The Dockerfile that builds the Docker container with all generic elements looks like this:
FROM node:8 #copy the Node Reload server - exposed at port 4500 COPY package.json /tmp COPY server.js /tmp RUN cd tmp && npm install EXPOSE 4500 RUN npm install -g nodemon COPY startUpScript.sh /tmp COPY gitRefresh.sh /tmp CMD ["chmod", "+x", "/tmp/startUpScript.sh"] CMD ["chmod", "+x", "/tmp/gitRefresh.sh"] ENTRYPOINT ["sh", "/tmp/startUpScript.sh"]
Feel free to pick any other node base image – from https://hub.docker.com/_/node/. For example: node:10.
The startUpScript that is executed whenever the container is started up – that takes care of the initial cloning of the Node application from the Git(Hub) URL to directory /tmp/app and the running of that application using nodemon is shown below. Note the trick (inspired by StackOverflow) to run a script only when the container is ran for the very first time.
#!/bin/sh
CONTAINER_ALREADY_STARTED="CONTAINER_ALREADY_STARTED_PLACEHOLDER"
if [ ! -e $CONTAINER_ALREADY_STARTED ]; then
touch $CONTAINER_ALREADY_STARTED
echo "-- First container startup --"
# YOUR_JUST_ONCE_LOGIC_HERE
cd /tmp
# prepare the actual Node app from GitHub
mkdir app
git clone $GITHUB_URL app
cd app
#install dependencies for the Node app
npm install
#start both the reload app and (using nodemon) the actual Node app
cd ..
(echo "starting reload app") & (echo "start reload";npm start; echo "reload app finished") &
cd app;
echo "starting nodemon for app cloned from $GITHUB_URL";
nodemon
else
echo "-- Not first container startup --"
cd /tmp
(echo "starting reload app and nodemon") & (echo "start reload";npm start; echo "reload app finished") & (cd app; echo "start nodemon") &
cd app;
echo "starting nodemon for app cloned from $GITHUB_URL";
nodemon
fi
The startup script runs the live reloader application in the background – using (echo “start reload”;npm start)&. That final ampersand (&) takes care of running the command in the background. This npm start command runs the server.js file in /tmp. This server listens at port 4500 for requests. When a request is received at /reload, the application will execute the gitRefresh.sh shell script that performs a git pull in the /tmp/app directory where the git clone of the repository was targeted.
/* this program listens for /reload request at port 4500.
or a GitHub WebHook trigger (see: https://technology.amis.nl/2018/03/20/handle-a-github-push-event-from-a-web-hook-trigger-in-a-node-application/)
When it receives such a request, it will perform a Git pull in the app sub directory (from where this application runs)
TODO
- add the option to schedule an automatic periodic git pull
- use https://www.npmjs.com/package/simple-git instead of shelljs plus local Git client (this could allow usage of a lighter base image - e.g. node-slim)
*/
const RELOAD_PATH = '/reload'
const GITHUB_WEBHOOK_PATH = '/github/push'
var http = require('http');
var server = http.createServer(function (request, response) {
console.log(`method ${request.method} and url ${request.url}`)
if (request.method === 'GET' && request.url === RELOAD_PATH) {
console.log(`reload request starting at ${new Date().toISOString()}...`);
refreshAppFromGit();
response.write(`RELOADED!!${new Date().toISOString()}`);
response.end();
console.log('reload request handled...');
}
else if (request.method === 'POST' && request.url === GITHUB_WEBHOOK_PATH) {
let body = [];
request.on('data', (chunk) => {
body.push(chunk);
}).on('end', () => {
body = Buffer.concat(body).toString();
// at this point, `body` has the entire request body stored in it as a string
console.log(`GitHub WebHook event handling starting ${new Date().toISOString()}...`);
var githubEvent = JSON.parse(body)
console.debug(`github event: ${JSON.stringify(githubEvent)}`)
// - githubEvent.head_commit is the last (and frequently the only) commit
// - githubEvent.pusher is the user of the pusher pusher.name and pusher.email
// - timestamp of final commit: githubEvent.head_commit.timestamp
// - branch: githubEvent.ref (refs/heads/master)
try {
var commits = {}
if (githubEvent.commits)
commits = githubEvent.commits.reduce(
function (agg, commit) {
agg.messages = agg.messages + commit.message + ";"
agg.filesTouched = agg.filesTouched.concat(commit.added).concat(commit.modified).concat(commit.removed)
// .filter(file => file.indexOf("src/js/jet-composites/input-country") > -1)
return agg
}
, { "messages": "", "filesTouched": [] })
var push = {
"finalCommitIdentifier": githubEvent.after,
"pusher": githubEvent.pusher,
"timestamp": githubEvent.head_commit.timestamp,
"branch": githubEvent.ref,
"finalComment": githubEvent.head_commit.message,
"commits": commits
}
console.log("WebHook Push Event: " + JSON.stringify(push))
if (push.commits.filesTouched.length > 0) {
console.log("This commit involves changes to the Node application, so let's perform a git pull ")
refreshAppFromGit();
}
} catch (e) {
console.error("GitHub WebHook handling failed with error "+e)
}
response.write('handled');
response.end();
console.log(`GitHub WebHook event handling complete at ${new Date().toISOString()}`);
});
}
else {
// respond
response.write('Reload is live at path ' + RELOAD_PATH);
response.end();
}
}); //http.createServer
server.listen(4500);
console.log('Server running and listening at Port 4500');
//https://stackoverflow.com/questions/44647778/how-to-run-shell-script-file-using-nodejs
// https://www.npmjs.com/package/shelljs
var shell = require('shelljs');
var pwd = shell.pwd()
console.info(`current dir ${pwd}`)
function refreshAppFromGit() {
try {
if (shell.exec('./gitRefresh.sh').code !== 0) {
shell.echo('Error: Git Pull failed');
// shell.exit(1);
} else {
// shell.exec('npm install')
// shell.exit(0);
}
} catch (e) {
console.error("Error while trying to execute ./gitRefresh " + e)
}
}
Using the node-run-live-reload image
Now that you know a little about the inner workings of the image, let me show you how to use it (also see instructions here: https://github.com/lucasjellema/docker-node-run-live-reload).
To build the image yourself, clone the GitHub repo and run
docker build -t "node-run-live-reload:0.1" .
using of course your own image tag if you like. I have pushed the image to Docker Hub as lucasjellema/node-run-live-reload:0.1. You can use this image like this:
docker run --name express -p 3011:3000 -p 4505:4500 -e GITHUB_URL=https://github.com/shapeshed/express_example -d lucasjellema/node-run-live-reload:0.1
In the terminal window – we can get the logging from within the container using
docker logs express --follow
After the application has been cloned from GitHub, npm has installed the dependencies and nodemon has started the application, we can access it at <host>:3011 (because of the port mapping in the docker run command):
When the application sources are updated in the GitHub repository, we can use a GET request (from CURL or the browser) to <host>:4505 to refresh the container with the latest application definition:
The logging from the container indicates that a git pull was performed – and returned no new sources:
Because there are no changed files, nodemon will not restart the application in this case.
A GitHub WebHook can be configured on the GitHub Reppository. It should be configured with the endpoint host:4500/github/push. When this is in place – and the host is exposed on the public internet – then any commit to the application’s GitHub repository will send a signal to the reload utility in the container (similar to the manual call to the /reload endpoint) and the refresh of the application takes place (git pull, npm install, restart by nodemon).
One requirement at this moment for this generic container to work is that the Node application has a package.json with a scripts.start entry in its root directory; nodemon expects that entry as instruction on how to run the application. This same package.json is used with npm install to install the required libraries for the Node application.
Summary
The next figure gives an overview of what this article has introduced. If you want to run a Node application whose sources are available in a GitHub repository, then all you need is a Docker host and these are your steps:
- Pull the Docker image: docker pull lucasjellema/node-run-live-reload:0.1
(this image currently contains the Node 8 runtime, npm, nodemon, a git client and the reloader application)
Alternatively: build and tag the container yourself. - Run the container image, passing the GitHub URL of the repo containing the Node application; specify required port mappings for the Node application and the reloader (port 4500): docker run –name express -p 3011:3000 -p 4500:4500 -e GITHUB_URL=<GIT HUB REPO URL> -d lucasjellema/node-run-live-reload:0.1
- When the container is started, it will clone the Node application from GitHub
- Using npm install, the dependencies for the application are installed
- Using nodemon the application is started (and the sources are monitored so to restart the application upon changes)
- Now the application can be accessed at the host running the Docker container on the port as mapped per the docker run command
- With an HTTP request to the /reload endpoint, the reloader application in the container is instructed to
- git pull the sources from the GitHub repository and run npm install to fetch any changed or added dependencies
- if any sources were changed, nodemon will now automatically restart the Node application
- the upgraded Node application can be accessed
Next Steps
Some next steps I am contemplating with this generic container image – and I welcome your pull requests – include:
- allow an automated periodic application refresh to be configured through an environment variable on the container (and/or through a call to an endpoint on the reload application) instructing the reloader to do a git pull every X seconds.
- use https://www.npmjs.com/package/simple-git instead of shelljs plus local Git client (this could allow usage of a lighter base image – e.g. node-slim instead of node)
- force a restart of the Node application – even it is not changed at all
- allow for alternative application startup scenarios besides running the scripts.start entry in the package.json in the root of the application
Resources
GitHub Repository with the resources for this article – including the Dockerfile to build the container: https://github.com/lucasjellema/docker-node-run-live-reload
My article on my previous attempt at creating a generic Docker container for running a Node application from GitHub: https://technology.amis.nl/2017/05/21/running-node-js-applications-from-github-in-generic-docker-container/
Article and Documentation on nodemon: https://medium.com/lucjuggery/docker-in-development-with-nodemon-d500366e74df and https://github.com/remy/nodemon#nodemon
NPM module shelljs that allows shell commands to be executed from Node applications: https://www.npmjs.com/package/shelljs
