Writing a NodeJS CI server

A homegrown toy CI server (~100 LOC) that runs this blog.

A Continuous Integration (CI) server can do lots of useful things for a project, for example:

• automatically run unit tests on every commit
• automatically deploy an application on every commit

There are lots of good third party options out there but just for fun I created a little homegrown one for my own personal use on my projects. This post briefly describes how it works.

High level

I use Github as a safe location for my Git repositories. Each project gets its own repository.

The idea is to create a publicly available webhook that Github can notify whenever a new commit it pushed to the repository we'd like to implement CI for.

Github has a nice webhook feature that's really easy to configure with our HTTP end point, so all we need to do is build and deploy something that can respond to the Github webhook HTTP calls.

The webhook server

Some kind person has already created githubhook, a Javascript library that abstracts away most of the details of the Github webhook HTTP traffic. We can leverage that to handle the HTTP side and call us on the event of our choosing.

In my case I am only interested in the push event, and can write a callback function that githubhook will call for me:

// --- server.js ---
const githubhook = require('githubhook');
const onPush     = require('./onPush');

const github = githubhook({
  path: '/github',
  port: 3420, // the port to listen on, defaults to 3420
  secret: 'SOME_SECRET', // you configure Github to sign requests with this
});
github.on('push', onPush);
github.listen();

// --- onPush.js ---
function onPush(repo, ref, data) {
    // Check `repo` to see if it's a project we are interested in
    // Check `ref` to see if it's a branch we are interested in
    // Do some CI stuff for this push!
}

So my CI server is just a thin layer over the top of githubhook that coordinates cloning/pulling the updated repo before running any configured scripts in the repo directory.

I chose to store each project's CI configuration and cloned repo in a directory outside the webhook:

• data/
  • {project}/ - data for this project
    • config.js - CI configuration file, looks like { ref, repoPath, scripts }
    • repo - the directory to clone/pull the repo to

// --- onPush.js ---
function onPush(repo, ref, data) {
  const projectConfig = lookupProjectConfig(repo);

  if (!projectConfig) {
    console.log('[onPush] Config for project not found', repo);
    return;
  }

  if (projectConfig.ref !== ref) {
    console.log('[onPush] Ignoring push to ref', ref);
    return;
  }

  console.log('[onPush] Handling push to ref', repo, ref);

  // Clone repo from Github to `projectConfig.repoPath` if it doesn't exist
  // Pull latest version from Github to `projectConfig.repoPath`
  // Run any scripts from `projectConfig.scripts`

  console.log('[onPush] Finished!');
}

Configuration

The CI server authenticates against the Github repository using an SSH key, which is configured in Github as a "deploy key" for the repository.

In order to configure the CI server to use the correct SSH key for multiple Github repositories, we do something a touch sneaky with our SSH config:

# ~/.ssh/config
Host fake1.github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/fake1_rsa

That tells SSH to use the given host, user and SSH key when connecting to fake1.github.com. A project called foobar for user someuser would actually have the Github SSH url of git@github.com:someuser/foobar.git but if we instead force our CI server to use git@fake1.github.com:someuser/foobar.git then the above config will ensure it sends the SSH key of our choice.

CI actions

All of the projects I'm using CI for are NodeJS projects, so in my case I add NPM scripts in the project's package.json for any CI actions I want (e.g. "tests": "mocha"...), then configure the CI server to run them, npm run tests.

For this blog, the CI is configured with two scripts: one to generate the static site; then one to deploy it to my web hosting over SSH (using a different SSH keypair as a security precaution).

There you have it, a basic homegrown CI server in less than 100 lines of code!