The Network is Unreliable

Hello, I hope you are doing well.

Last week, I had an application that is written in TypeScript and running on Node.js, which sent requests to another system through HTTP, but they failed intermittently. I checked the CPU, RAM, disk usage, and everything seemed fine. However, when I checked the network usage, it exceeded the server’s bandwidth. This is a well-known problem; bandwidth is finite, and it is one of the falacies of distributed computing. Unfortunately, many developers are not aware of this.

With a limited time to solve the problem, I devised the quickest solution I could think of. I decided to add a retry mechanism for the HTTP request. After some research, I found Cockatiel a library to make the function calls fault-tolerant by retrying them, adding a circuit breaker and implementing a timeout. Cockatiel also provides a backoff retry mechanism to prevent flooding the target system with requests. I used this to avoid adding additional load to the application.

To install the library run the following command:

npm install --save-exact cockatiel

Here is how to use it in your code:

import { ExponentialBackoff, retry, handleAll } from "cockatiel";

async function main() {
  const request = async () => {
    // http request
  };

  await retry(handleAll, {
    backoff: new ExponentialBackoff(),
  }).execute(() => request());
}

After implementing it, if there’s a failed HTTP request, Cockatiel will automatically retry it. You can customize options like maxDelay, exponent, and initialDelay; you can find more information in the Cockatiel GitHub repository. For now, I’m using the default options until I see a need for customization.

Of course, this is just a workaround. I still need to address how to scale the network bottleneck, but that’s a topic for another article I plan to write in the future.

That’s all from me. I hope you find something useful in this article.

Thank you.