Fabian Enos

Writing test cases are considered good practices because they allow you to ensure that your code works and that changes over time do not break the underlying functionality. But what if you want to use those test cases as a base to ensure that any person’s code built to specification passes?

Why do this? So that user submitted code samples can be tested for correctness. Then the advantage of using test case syntax is alluring because of ease of reading assertions and the ability to have anyone easily contribute to a pool of testable problems.

For example, when testing a generic search function:

it('should return an array', function () {
    const haystack = [1,2,3,4,5,6];
    const needle = 6;

    try {
        let result = search(haystack, needle);

        expect(result)
            .to.be.an('array', "Invalid value returned by search, it must be an array of all needles")
            .and.to.have.lengthOf(1);
    } catch(err) {
        expect.fail("Error thrown by search function");
    }
});

This works well but test suites, in practice, run by invocation from the shell. And while we can use plugins or reporters to make this work, the easier approach would be to run the test suite programmatically.

When working on this, since we were using the mocha and chai for testing at the time, we looked at the mocha API first and eventually came across the official using mocha programmatically guide. This was simple enough to then get running.

const mocha = new Mocha({});

mocha.addFile('test-file.js')

mocha.run()
on('test', function(test) {
    console.log('Test started: ' + test.title);
})
.on('test end', function(test) {
    console.log('Test done: ' + test.title);
})
.on('pass', function(test) {
    console.log('Test passed');
    console.log(test);
})
.on('fail', function(test, err) {
    console.log('Test fail');
    console.log(test);
    console.log(err);
})
.on('end', function() {
    console.log('All done');
});

We added some of our own code to make things work with our user submissions and then ran into our next problem:

Running multiple tests would cause inconsistencies in the results that got returned. Looking closer, it was very clearly a race condition.

The reason was, for one problem specification, if we had two solutions, A and B, since the same set of tests were running on both, B would get marked as passed if A entered first and passed even if B failed. The same happened in reverse. It depended on the queue.

Here is where the rabbit hole started.

Because we are trying to test user submitted code, the path to the submitted code was always guaranteed to be unique but the test files would always have the same path. In order to make the test run for every user uniquely, we either had to have the submitted code path be the same. This was not an option as submissions would come in at any time, so instead we had to find a way to set the path to the submitted file manually. We did this by playing around with the mocha suite information and it worked to pass the path to the test file. We then rewrote out test files to use this.

Initial testing seemed to work but then we ran into an issue where even though the file paths were being set correctly, the bug persisted.

Turned out that the initial guide had offered a hint to this issue:

Note that run (via loadFiles, which it calls) relies on Node’s require to execute the test interface functions. Thus, files loaded by Mocha will be stored in Node’s require cache and therefore tests in these files will not be re-run if mocha.run() is called again

So clearing the require cache was a necessity to fix this run.

mocha.suite.on('require', function (global, file) {
    delete require.cache[file];
});

That fixed it. Or so it seemed.

When the code was getting submissions, at times, the tests were failing to run.

It seemed as though this was because of too many requests. Setting up mocha to run in parallel mode seemed a good first option but because some tests needed to run in a certain order, this option was ruled out.

After some debugging, it turned out that mocha needs you to set cleanReferencesAfterRun manually to ensure that the mocha instance clears all the references it has set up.

mocha.cleanReferencesAfterRun = false;

This sorted it out and things started to work correctly.

The final “problem” that arose was constantly getting the following warning:
(node:18921) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit

This is caused because mocha attaches listeners directly to the process. After some research, running mocha.dispose() when the test suite was done took care of this.

The focus then became adding more functionality which necessitated moving away from mocha since it had become a bottleneck and we needed more control over how the code ran including having time out limits and code analysis.