tackling 2 lines of code to improve code quality : the mystery of new promise(setimmediate)

On the project I am currently working on, we put a specific focus on testing, specifically integration testing. It is our main quality tool and allows us to have confidence in the code that we are writing every day. However, this confidence was affected in some of our tests, in which we used a specific function we were not sure about :

To reassure the complete quality of our codebase, we decided to tackle this subject with one objective : 0 unjustified use of these 2 lines. We spent a day, following the 6 steps Kaizen methodology, to improve this very specific subject of our test suites. Here is what we learned.

? What is Kaizen? A methodology to improve iteratively on a specific measurable subject. Performance, time of compilation, and code quality are very compatible with Kaizen.

Understanding setImmediate and the Node event loop

To understand how ++code>const flushPromises = () => new Promise(setImmediate);++/code> works we took one of the test using it and we debugged it.

The tested component is a form page that uses Formik, a React library to handle forms. One feature of Formik is to validate a form, checking if all fields are filled for instance. This validation is asynchronous, which is why we needed to use ++code>new Promise(setImmediate)++/code> to make the test pass.

How does new Promise(setImmediate) work

We first needed to understand the actual syntax of ++code>new Promise(setImmediate)++/code> .

- ++code>new Promise(function)++/code> is a shorthand of ++code>new Promise((resolve,reject)? function(resolve,reject))++/code>

- so, ++code>new Promise(setImmediate)++/code> is equivalent to ++code>new Promise((resolve) ? setImmediate(resolve)) ++/code>

After clarifying this confusing syntax, we needed to understand clearly what ++code>setImmediate++/code> does. One key sentence can be found in node.js docs.

Any function passed as the setImmediate() argument is a callback that's executed in the next iteration of the event loop.

That's where we get to the fun part. The argument of setImmediate is the resolve function of our promise. So ++code>new Promise(setImmediate++/code>) puts ++code>()?resolve()++/code> in the next iteration of the event loop. The question we had : when does that happen exactly?

Diving into the event loop : microtask and task queues

To answer this, we had to dig deeper and understand theoretically how the event loop of Node.js works. The event loop is what allows the use of the single thread of JavaScript to manage asynnchronous elements, API calls for example.

Three elements of the event loop are essential to make this line work : the callstack, the task queue and the microtask queue.

Callstack : what is currently executing.

Microtask queue : next microtask to execute ? in our case, mainly promises.

(Macro) Task queue : next task to execute. Each execution of one task represents an iteration of the event loop.

Source : https://medium.com/@saravanaeswari22/microtasks-and-macro-tasks-in-event-loop-7b408b2949e0

There is one big difference between the microtask queue and the task queue, allowing ++code>new Promise(setImmediate)++/code> to work its magic.

The microtasks are executed until the microtask queue is empty,  during the same iteration of the event loop, meaning that before moving on to the next task, all microstasks have to be executed. The next iteration of the event loop is thus after the callstack and the microtask queue is empty.

On the other hand, the task queue iterates on only one element at a time. One iteration of the event loop is equivalent to one task being executed.

There is another consequence of the microtask queue being emptied completely before iterating on the event loop. If a microtask is created while iterating on the microtask queue, it will be executed during the same iteration of the event loop, before moving on to the next one.

When we were able to grasp these differences, we had a better understanding of the use of ++code>setImmediate++/code> in the code. It was now time to take from our learnings and draw what was going on exactly in our test.

Drawing the event loop

That's the face you make when you understand asynchronous JavaScript

Drawing the state of the event loop components on each step really was the game changer for us. It allowed us to understand completely what was going on and why ++code>() => new Promise(setImmediate);++/code> made the tests pass.

In our use, it meant that ++code>setImmediate(resolve)++/code> put the Promise resolve function in the task queue, therefore acting as a "sweep vehicle" function. Because the resolve is placed in the task queue, it allows all promises created by Formik validation to be resolved/rejected before making the assertions of the test.

Here is a gif reproducing the states of the callstack, the microtask queue and the task queue on every step.

Checking the theory of the event loop and setimmediate

We now needed to check if the theory was right. The first step was to verify that Formik validation did in fact create chained promise (a promise creating another promise). We checked the code of Formik searching for at least two chained Promises, then we checked in the test how many Promises were created :

After resolving 9 promises, the test failed, but with 10 promises it passed. We had the validation that Formik is indeed creating several chained promises.

Success !

The question we have to ask ourselves now is : how does learning this specific behavior help us work better? Well this day of technical investigation in the node.js event loop had several impacts :

- We understood what ++code>new Promise(setImmediate) ++/code>does in our tests : it awaits for all existing and chained promises to be resolved before making the assertions. However, this behavior is very implicit, which makes it a liability in our code base

- We found an alternative solution for our tests : as we are used React-Native-Testing-Library, we could use the waitFor wrapper, which is more comprehensible and explicit..

- We deleted this asynchronous part from synchronous tests.

- We regained the confidence of our test suite, which was our main goal ?

++code>waitFor++/code> is actually using the same properties of the Event loop, but we have more confidence in React-Native-Testing-Library and using ++code>waitFor++/code> makes the test more readable.

And the biggest achievement of this day : we improved as developers and as a team. We investigated deeply on a highly technical subject, learning many rules and behaviors of asynchronous JS, and can now transfer this knowledge inside the team and the company.

Docs on setImmediate and the event loop

NodeJS docs :

https://nodejs.dev/learn/understanding-setimmediate

https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/

In The Loop, a talk by Jake Archibald : https://www.youtube.com/watch?v=cCOL7MC4Pl0

About the event loop : https://medium.com/dkatalis/eventloop-in-nodejs-macrotasks-and-microtasks-164417e619b9