Every IoT platform we have been asked to rescue had the same root cause, and it was not the one the team expected. It was not the database, the broker or the cloud provider. It was that the system had been sized against an average that never actually occurs.
Why averages lie in telemetry
Consider eighteen thousand electricity meters, each reporting once every fifteen minutes. Averaged over the interval that is twenty messages a second, which any laptop can handle. But meters do not report at random. They report on the quarter hour, because that is what the tariff period demands. The real profile is fourteen minutes of near silence followed by eighteen thousand messages arriving inside a few seconds.
The peak is not twenty messages a second. It is closer to nine thousand. A system sized for the average will drop the difference, and — this is the part that hurts — it will usually drop them silently.
The synchronisation trap
Devices synchronise for reasons that are entirely reasonable in isolation. Tariff periods, shift changes, scheduled sampling, and the most common one of all: a power cut across a site, followed by every device on it booting and reconnecting at the same moment. That last case is the one to design for, because it combines a connection storm with a backlog replay.
If you have not decided what happens when every device on a site reconnects at once, the system has decided for you, and it has chosen to fall over.
What to do about it
- Ask for the peak, then ask what causes it. The cause tells you whether it will get worse as the fleet grows.
- Add jitter on the device. A random offset of a few seconds around the reporting boundary costs nothing and flattens a synchronised spike dramatically.
- Put a durable buffer between accept and process, so the ingest tier can accept at line rate while consumers drain at their own pace.
- Rate-limit reconnection on the device with exponential backoff, so a site-wide power cut does not become a connection storm.
- Load test at the peak multiplied by your expected growth, not at the average.
The cheapest fix is on the device
Cloud-side capacity to absorb a synchronised spike is expensive and permanent. A few lines of jitter in the firmware are free, and they are also the hardest to add later, once units are deployed. This is a good example of the general rule: decisions that live on the device are the ones to get right first, because they are the ones you cannot revise from a laptop.