Telemetry usually ends one of two ways. Either you keep every raw reading forever, and storage and query costs grow without end. Or you delete the old data and lose the history. Rolling it up yourself sounds like the middle way, until you hit the maths: an average of averages isn’t the average, and the p99 of sixty one-minute p99s isn’t the hour’s p99. Do it naively and the numbers quietly lie.
RollupDB is an aggregate database in one file. You send it numeric readings and declare the statistics you’ll need. It keeps those statistics in time buckets that get coarser as data ages, seconds to minutes to hours to days, and lets the raw events go. The history stays. The volume shrinks. And because it keeps the pieces that combine correctly, the totals and extremes stay right.
The Pitch in Four Numbers
- 15.4 MB for five years of history from 2.8 million readings, against 207 MB as raw JSON lines.
- +0.37% growth in file size when the same five years had twice as many readings.
- Exact count, sum, min, max and mean after five years of rollups, checked against an outside reference.
- 0 acknowledged writes lost in 15 runs that killed the process mid-write.
How It Works
Readings land in one-second buckets. As data ages, it’s folded into minute, hour and day buckets. Each bucket keeps count, sum, min, max, mean, variance state and a percentile sketch. Those are the mergeable pieces: add two buckets’ counts and sums and you get the right count and sum for both, which is exactly what naive rollups get wrong. You define the statistics for each stream up front, and you can change the definitions later.
Count, sum, min, max and mean are exact. Percentiles come from a KLL sketch, so they’re approximate, and every answer is labelled that way. Ask a question that needs detail the database no longer has, and RollupDB tells you so instead of guessing.
It ships as one binary with no server and no C compiler needed, plus a Go library and a small HTTP service on the same engine. Ask questions in JSON or read-only SQL. Rollups run on their own schedule. A write is acknowledged only once it’s on disk. It also takes Prometheus remote write, and a batch whose commit status is unknown after a dropped connection is counted once, not twice.
Who It’s For
- IoT gateways recording sensor readings, like cold room temperatures, with extremes and percentiles, on small hardware.
- API teams tracking latency and errors per endpoint across a whole year.
- AI product teams tracking token use, cost and response time per model or feature.
- Anyone with a time series where the questions are known in advance.
Limits
This is the trade, stated plainly. Once the raw events are gone, you can’t ask questions you didn’t declare. A breakdown by a field you didn’t keep can’t be rebuilt. If you need to dig through every raw event later, keep them somewhere else as well. Percentiles are approximate by design.
The Four-Question Screening
Every project featured here gets the same four questions, the way you’d screen a startup idea.
1. Where is the money in it?
RollupDB is open source under Apache 2.0, with no paid tier. Observability and telemetry storage is a large market where customers mostly pay by volume, and volume is exactly what RollupDB cuts. The paid paths are a hosted long-term store that sits behind existing monitoring, with Prometheus remote write as the on-ramp; an edge edition for device makers who need years of statistics on a gateway; and support for companies that embed it. A tool that turns 207 MB into 15 MB has an easy story to sell.
2. What real problem does it solve?
Long-term statistics that are either too expensive to keep or wrong. Keeping everything costs a fortune. Deleting loses history. Rolling up by hand produces averages of averages that look fine and aren’t. RollupDB keeps the right pieces so the answers stay correct as the detail fades.
3. The technology in plain words
Think of a weather diary. For today you note every reading. For last month you just keep each day’s high, low, average and how many readings there were. For last year, each week’s. You can still say exactly how many readings you took, the hottest day and the overall average, years later, from a diary that fits in a pocket.
4. Moat and differentiation
Time-series databases store raw points and add downsampling as a feature. RollupDB starts from the other end: it stores only declared statistics, combines them correctly and says plainly what it can no longer answer. In one file, with no server. The exact-totals guarantee, checked against an outside reference after five years of rollups, is the proof point. The moat that can grow is that guarantee plus a footprint small enough for edge hardware, a spot big time-series databases don’t target.
Why It Matters
Most telemetry is kept to answer a handful of known questions. Storing every raw event to answer them is waste. A database that keeps only what you’ll ask, and keeps it right, makes long history cheap.
Follow the Project
Read what an aggregate database keeps when the raw events are gone, start with the quick start, see how it was tested, or get the code from the RollupDB repository on GitHub. RollupDB is listed with the other open-source projects on SoftDevStudio.com, and more featured projects are in the Projects section.