What time series data is
A time series is a sequence of values, each tied to a moment in time. A room thermometer that records the temperature every minute produces one. So does a smart meter that reports energy use every fifteen minutes, or a vibration sensor that samples many times a second. The defining feature is that time is the main way to organize and ask about the data: what was the value at this moment, what happened between these two times, how has it changed?
Most connected devices produce this type of data, which is why understanding how it is stored matters for anyone planning a deployment.
What a stored reading looks like
At its simplest, each reading contains a few elements:
- A timestamp: when the measurement was taken.
- An identifier: which device or sensor produced it.
- A value: the measurement itself, with a known unit.
- Optional labels: extra descriptive tags, such as location, room or device type, which are used for filtering and grouping.
Readings are almost always added, not changed. A temperature recorded at 10:01 is a historical fact; it will not be edited later. This “append-only” nature shapes how storage is designed.
Why not just use an ordinary database?
You can store sensor readings in a general-purpose database table, and for small projects that is often fine. But as the volume grows, the pattern of work becomes unusual, and specialized time series storage tends to do better.
Volume and speed. Consider a hypothetical fleet of 10,000 sensors each reporting once per minute. Each sensor produces 1,440 readings a day, so the fleet produces 14,400,000 readings a day. Over a year, a single sensor generates 525,600 readings. These counts are illustrative, but they show how quickly numbers pile up, and many systems record far more often than once a minute. The database must absorb a constant stream of new rows.
Query patterns. Typical questions concern ranges of time (“the last 24 hours,” “last March”), summaries (“the average per hour”) and comparisons across sensors. They rarely ask for a single row by itself.
Time-based organization. Specialized systems commonly divide data into chunks by time period, so that a question about last week reads only the relevant chunks, not the whole history. New data is written to the most recent chunk, and old chunks can be handled separately.
Compression. Time series data is highly regular. Timestamps often arrive at steady intervals, and consecutive values tend to be close to one another. Storing the difference from the previous value or interval, instead of the full value, takes less space. Compression can substantially reduce storage needs, although how much depends on the data.
Built-in time functions. Features such as grouping into time buckets, calculating rates of change and filling gaps are common needs, and dedicated tools offer them directly.
Getting time right
Timestamps deserve care, since the whole structure depends on them.
- Use one standard. Store times in a single reference such as UTC and convert for display, so that daylight saving changes and time zones do not corrupt the order of events.
- Timestamp at measurement. Record when the sensor measured, not when the server received the message. The two differ when a device has been offline and sends buffered data later, as explained in the article on offline operation.
- Check device clocks. Small devices may lack a reliable clock, or their clocks drift. Some systems add a server receive time as well, so that a suspicious device time can be spotted.
- Handle late and out-of-order data. Storage should accept readings that arrive after newer ones.
- Know the resolution. Decide whether seconds suffice or finer precision is needed.
Related issues such as duplicates and gaps are described under data quality, and these are partly handled at write time by rules such as treating the same device and timestamp as one record.
Retention: you cannot keep everything at full detail forever
Because the data keeps growing, a plan for how long to keep it is essential. Retention is the policy that determines what is stored, at what level of detail and for how long. It balances usefulness, cost and, in some cases, legal or privacy considerations (see the discussion of data retention).
A common approach has several layers:
- Raw data for a short period. Keep every reading for recent days or weeks, when detailed investigation is most likely.
- Downsampling for the medium term. Replace raw readings with summaries, such as the minimum, maximum and average per hour. This reduces volume enormously while keeping the shape of the history. Keeping the minimum and maximum, not only the average, preserves information about peaks that an average would hide.
- Coarser summaries for the long term. Daily or monthly values may be enough for trends over years.
- Deletion or archiving. After a set period, data is removed or moved to cheaper, slower storage.
A quick hypothetical check: reducing one-minute readings to hourly summaries means 60 raw readings become one summary record per hour. If you keep three values per summary (minimum, average and maximum), the stored count falls from 60 values to 3 per hour, which is 20 times fewer. The right choices depend on what questions will be asked later. If you might need to investigate a past fault in fine detail, think carefully before discarding the raw data.
Retention rules should be automatic. Manually trimming data tends to be forgotten until storage runs out.
Querying and using the data
Stored readings feed several uses: dashboards (see sensor dashboards), alerts, reports and deeper analysis, including sensor data analytics and machine learning. Queries on recent data should be quick, while long-range queries benefit from pre-computed summaries. Planning those uses early helps determine how often to sample, which labels to attach and how long to keep each level of detail.
Choosing sensibly
Many small projects need nothing more exotic than a well-organized table with an index on device and time. Move to dedicated time series storage when volume, query speed or storage cost starts to become a problem. Whatever the choice, the same habits apply: consistent timestamps, clear units, labels that describe the source, and a written retention plan.