Two Days I Lost to the Network, Not the Code

TL;DR

• Local: the server stopped receiving client requests because my macOS IP address changed.
• A brief network drop triggered the dynamic IP to reset, splitting client and server across addresses.
• Metro kept working, which masked it — its cache was holding the old address for some services.
• Production: aggregate errors in Node turned out to be unresolved hostnames.
• AWS was healthy, which pointed at Bluehost, our DNS provider.

Two separate incidents where I went looking for a bug in my code and the problem was the network. Different environments, same lesson.

Local: the IP moved

The server wasn’t receiving client requests. I went through the application layers first and eventually rebuilt the Metro cache, which produced connection timeouts — the first honest error I’d seen.

The cause was that my macOS system’s IP address had changed. Client and server ended up on different addresses, so the client was dialling a machine that no longer answered to that name. A brief loss of network connectivity had triggered the dynamic IP system to reset.

What made this hard is that Metro kept working. Its caching held the older IP for some services, so parts of the setup functioned normally while others failed. A completely broken connection is easy to diagnose; a partially working one sends you hunting for what’s different about the failing paths.

Checking dynamic network configuration early would have saved the session.

Production: aggregate errors

Different day, sharper stakes. Our server couldn’t resolve hostnames to IP addresses, surfacing as an aggregate error in Node.

Aggregate errors are worth recognising on sight: they come from repeated failed attempts, typically timeouts, which usually means a network endpoint problem with some kind of backoff behind it. The error is a summary of many failures rather than a description of one, so reading it as a single fault sends you the wrong way.

I checked AWS’s system status first, since if their routing were degraded that would explain everything and there’d be nothing for me to do. AWS was fine.

That left our DNS provider, Bluehost, as the likely source of the unresolved hostnames.

The common thread

Both times I assumed the fault was in something I’d written, because that’s usually true and it’s where I have leverage. Both times the code was correct and its assumptions about the network weren’t.

Cheap check worth doing first: can the thing actually reach the thing?