Connections fail, so plan for it
Any device that depends on a network will sometimes lose it. Wi-Fi routers restart, cellular coverage fades in a basement, a gateway loses power, a remote site has a storm, or a provider has an outage. Connectivity problems are not unusual events but a normal feature of the environment, and well-designed devices treat them that way. The goal is not to prevent every drop, which is impossible, but to make sure a drop does not cause lost data, unsafe behavior or a mess when the link returns.
Decide what the device must do alone
Start by deciding what the device should keep doing when it cannot reach the network. This is a design choice and varies by purpose.
- A temperature logger should keep measuring and storing readings.
- A smart thermostat should keep following its stored schedule and respond to the local temperature, even if it cannot reach its cloud service.
- A leak or smoke detector should keep sounding a local alarm, because its main safety function must not depend on the internet.
- A door lock should keep working with a physical or local method, so people are not locked out by an outage.
The more a device’s essential job depends on remote decisions, the more fragile it is. This is one reason designers weigh the trade-offs described in the comparison of cloud processing and local processing, and why some home systems favor local control through smart hubs.
Buffering: keep the readings
When the link drops, a device that still collects data needs somewhere to put it. This is buffering: holding readings in local memory or storage until they can be sent. Key design questions:
- How much can be stored? Small devices have limited memory, so the buffer fills up. A device that takes a reading each minute produces 1,440 per day, and the buffer size determines how long an outage it can survive.
- What happens when it is full? Options include dropping the oldest readings, dropping the newest, or storing readings less often (for example, summarizing). Which is best depends on whether recent or complete history matters more.
- Does it survive a restart? A buffer held only in volatile memory is lost if the device resets or loses power. Writing to non-volatile storage protects data but adds wear and complexity, especially on flash memory with limited write cycles.
- Is each reading timestamped at the time of measurement? If the device records the time when it measures, not when it finally sends, the history stays accurate after delayed delivery. This depends on the device having a reasonable clock, a point revisited in the article on time series data.
Gateways can also hold data on behalf of simple devices. A local IoT gateway with more memory than the sensors can store readings during an outage.
Retrying without making things worse
After a failure, a device tries again. How it retries matters more than it seems. If thousands of devices lose their connection at the same moment, such as after a power cut or server outage, and all retry immediately and repeatedly, they can overwhelm the service when it recovers and prolong the problem.
The usual remedy has two parts:
- Exponential backoff. The device waits a short time before the first retry, then a longer time before the next, and so on, up to a maximum. This eases the load and saves battery.
- Jitter. A small random variation is added to each wait so that devices do not all retry in lockstep.
Retrying also costs energy. A battery device that keeps trying to connect with its radio on can drain its cells quickly, so it makes sense to limit attempts and sleep between them, as discussed under duty cycling and battery life.
Catching up when the link returns
Once the connection is back, the device must deliver its backlog without causing new problems.
- Send in order, or mark the order. Sending the oldest data first keeps history tidy, but if the backlog is large, you may want the newest reading first so the current state is quickly known. Whichever is chosen, the timestamps should make the true order clear.
- Pace the upload. Sending a large backlog all at once can saturate a weak link or a server.
- Expect duplicates. If a device did not receive a confirmation, it may resend something that actually arrived. Receiving systems should be able to recognize and discard repeated messages, for example by using message identifiers. This is one of the causes of duplicates listed in the article on data quality.
- Confirm before deleting. The device should remove data from its buffer only after the receiving side confirms it has been stored.
Messaging patterns can help. Some protocols used with publish subscribe messaging allow different delivery guarantees, such as at least once or at most once, and can keep a session so that missed messages are delivered when a device reconnects. The suitability of any option depends on the system.
Commands sent to offline devices
The same issue applies in the other direction. If someone sends a command or a new setting to a device that is offline, what should happen? One approach is to queue it until the device reconnects. But a command can become stale: an instruction to “turn on the heating” sent hours ago may no longer make sense. Giving commands an expiry time, and having devices report their actual state after reconnecting, avoids surprises. This is part of the broader practice of tracking which devices have applied which settings in managing fleets of connected devices.
Noticing and reporting outages
A system should also know when devices have gone quiet. A heartbeat message sent at regular intervals lets the server detect silence. Distinguishing a device that is offline from one that has failed, and from a network that is down for many devices at once, saves time during diagnosis. Alert rules for this are covered in the discussion of dashboards and alerts. Devices that come back online can include a short report of what happened, such as how long they were disconnected and how many readings were lost.
Test it on purpose
The best way to learn how a design behaves is to break the connection deliberately. Unplug the router, block the server, or move the device out of range, and watch what happens. Check that no data is lost beyond what was planned, that safety functions keep running, and that recovery is orderly. Problems found in a trial are far cheaper than ones discovered in the field.