Why sensor data is rarely clean
Readings from connected devices pass through many steps: a physical sensor, a small processor, a radio link, a network, servers and storage. A problem at any step can leave its mark on the data. The result is that raw sensor streams almost always contain flaws, and analysis built on them can be misleading unless those flaws are noticed first.
Data quality is about whether the data is good enough for its intended use. A rough room temperature log may tolerate some gaps; a record used to prove that medicine stayed within range may not. Decide what the data is for before deciding how much cleaning it deserves.
Common problems
Gaps
Gaps are missing readings. Causes include a dead battery, a dropped network link, a device restart, a server outage or a sensor that failed to take a reading. Some gaps are short and scattered; others last hours or days.
The key point is that a gap is information in itself. Do not quietly fill it with zero, which looks like a real reading. Zero degrees, zero flow and zero pressure are all meaningful values. Mark missing data as missing.
Spikes
A spike is a sudden, isolated reading far from its neighbors. Causes include electrical interference, a loose connection, a brief hardware glitch or a corrupted message. But spikes can also be real: a door opening, a pump starting or a gust of wind. Removing a real event as if it were noise defeats the purpose of monitoring.
Duplicates
Duplicates are the same reading appearing more than once. They often come from retries: a device sends a message, does not get a confirmation in time, and sends it again, even though the first one arrived. Duplicates can inflate totals, such as counts of people or energy use, so they matter most when values are added up. The resend behavior is discussed under offline operation.
Stuck values
A sensor that reports exactly the same number for a long time may be fine, or it may have frozen. Some quantities change constantly in practice, so a perfectly flat line is suspicious.
Drift and bias
Over time a sensor’s readings may slowly move away from the truth, or it may read consistently high or low from the start. These errors are subtle because every reading looks plausible. Calibration and comparison against a trusted reference are the usual remedies, as explained in the article on sensor calibration.
Timestamp problems
Readings arrive out of order, or with timestamps that are wrong because a device clock was never set or has wandered. Late-arriving data, delivered after a link recovers, may carry old timestamps that interleave with newer data. Time handling is also a storage question, discussed in the article on time series data.
Unit and format mix-ups
One batch of devices reports in one unit and another batch in a different unit. A change in software version alters the format of messages. These are less about the sensor and more about consistency, and they often appear right after a configuration change or update.
Impossible values
Readings outside what is physically possible, such as a humidity level above 100 percent or a temperature far below anything the location could produce, point to a fault. The valid range of each sensor is usually listed in its specifications; reading these carefully is covered in the article on sensor specifications.
Basic ways to clean data
Cleaning should be cautious and documented. A few reliable steps:
- Keep the raw data. Store the original readings and apply cleaning to a copy, so that a mistake can be undone and decisions can be reviewed.
- Validate on arrival. Check that required fields exist, that values are numbers, and that they fall within a plausible range for that sensor type. Flag, rather than silently delete, readings that fail.
- Remove exact duplicates. Readings with the same device identifier and the same timestamp can usually be treated as one. Message identifiers help if devices provide them.
- Mark gaps explicitly. Record missing intervals. For short gaps, some analyses use interpolation (estimating between neighbors), but only when the quantity changes smoothly, and the filled values should be marked as estimates.
- Treat spikes carefully. One simple approach compares each reading with a median of its neighbors and flags large deviations. Flagging is safer than deleting. Decide whether to exclude, replace or keep a flagged value based on the use case, and consider whether other nearby sensors saw the same event, which suggests it was real.
- Smooth with care. Averaging over a short window reduces random noise but can blunt true peaks.
- Standardize units and time. Convert everything to agreed units and a single time standard on ingestion.
- Check the heartbeat. Monitor the arrival rate. A sudden drop in the number of readings per hour is itself a data quality signal.
More advanced techniques, such as statistical methods for anomaly detection, build on these basics but are easier to apply once the obvious errors are under control.
A small hypothetical example
Suppose a sensor reports every minute, which would give 1,440 readings per day. If one day contains only 1,300 records, 140 are missing, which is roughly 10 percent. If the data also contains 20 duplicates, the true count of distinct readings is 1,280, not 1,300. Looking at counts like this is a quick first test before looking at the values. The numbers are illustrative only.
Recording what you did
Whatever cleaning is applied, write it down: which rules, which thresholds and when they changed. Conclusions drawn from the data, including those from sensor data analytics, are only as trustworthy as the checks behind them, and a clear record lets others judge how much confidence to place in the results.