How to Estimate Mango Storage Requirements

Overview

Before deploying Mango by Radix IoT, you need to know how much disk space the time-series data will consume. Underestimating leads to emergency purges; overestimating wastes budget.

Sizing splits into two halves that behave very differently. Record count is arithmetic: you can calculate it exactly, in advance, from point count, each point's logging settings, and retention. Bytes per record is a property of the point value store you are actually running. Mango does not publish a bytes-per-record figure, and in the default Mango NoSQL store the timestamp and file framing are written by its storage library, which compresses its shard files. So the reliable method is: calculate records exactly, then measure bytes on your own installation.

Prerequisites

• A planned or existing point list, with the logging settings you intend to use
• Your intended retention period, and any regulatory floor that applies to specific data
• The point value store the installation uses (by default Mango NoSQL, whose files live under <paths.data>/databases/mangoTSDB)
• Ideally a test or pilot instance you can run a representative subset of points against

Step 1: Calculate Records Exactly

Records per point per day = 86,400 / seconds between logged values
Records per day           = point count × records per point per day
Total records             = records per day × retention days

"Seconds between logged values" is set by each point's Logging type (data point editor, Logging properties tab), not by the poll rate alone:

Worked Example: Small Deployment

Scenario: 10 Modbus devices, 20 points each, polling every 15 seconds, All data logging, 1-year retention.

Points:            10 × 20 = 200 points
Records/day:       200 × 5,760 = 1,152,000
Records/year:      1,152,000 × 365 = 420,480,000

Result: roughly 420 million records held at any one time. Left on the numeric default (Interval, every 1 minute), the same 200 points would hold a quarter of that: 105,120,000.

Worked Example: Medium Deployment

Scenario: 50 devices, 40 points each, All data logging, half polled every 5 seconds and half every minute, 2-year retention.

Fast points:       1,000 points × 17,280 records/day = 17,280,000
Slow points:       1,000 points × 1,440 records/day  =  1,440,000
Total records/day: 18,720,000
Two-year total:    18,720,000 × 730 = 13,665,600,000

Result: around 13.7 billion records at steady state. The 5-second half produces twelve times the records of the 1-minute half, so it dominates the total.

Worked Example: Large Deployment

Scenario: 200 devices, 50 points each, 5-second polling, All data logging, 1-year retention.

Points:            10,000
Records/day:       10,000 × 17,280 = 172,800,000
Records/year:      172,800,000 × 365 = 63,072,000,000

Result: about 63 billion records. At this scale, measure before you buy hardware: a small error in bytes per record becomes a large error in the storage bill.

Step 2: Measure Bytes per Record on Your Own Store

There is no published per-record size that holds across stores, versions and data shapes. Measure it.

To measure:

1. Stand up a test instance, or use a defined subset of the production point list
2. Note the size of the point value store: the NoSQL database size (GB) monitor (an Internal Data Source point), or the size of <paths.data>/databases/mangoTSDB on disk
3. Run a representative set of points at their real logging settings for weeks, not hours: the store compresses its shard files, so an hour of fresh data is not what a year of data costs
4. Note the size again and calculate the growth
5. Divide the growth by the records those points generated over the period (Step 1's arithmetic). That is your bytes per record
6. Multiply by the total record count from Step 1, then add headroom

Re-measure once the system has run long enough for the first purges to delete old values. Until data starts ageing out you are measuring growth, not steady state.

Retention Tier Framework

Not all data needs the same retention. A tiered approach balances storage cost against data availability:

Retention is set in three places, and the most specific one wins (point, then data source, then system):

• Administration > System settings > Purge settings: Purge point values older than (one year on a new install) is the system period. Set it to the operational tier
• A data source's Purge tab: Override system purge settings, for the analytical tier
• A data point's Purge tab: Override data source purge settings, for single exceptions

The compliance tier is usually meta points that store an hourly or daily rollup. They are purged like any other point, so a short system period deletes them too: give their data source its own long override.

Reducing Record Counts

• Use Interval logging where a periodic value is enough. It separates what you store from how fast the source polls, and it is already the default for new numeric points
• Use When point value changes for binary and multistate points. How much it saves depends entirely on how often the signal changes, so measure it
• Add a Tolerance to numeric points on change-based logging, so noise around a steady value is not stored
• Log less often where second-level history is not needed. With All data, 1 minute instead of 5 seconds is exactly twelve times fewer records
• Use purge overrides to keep high-frequency points briefly and summaries longer, subject to the setting above
• Use Do not log for scratch points, such as meta points used only for intermediate calculations. While the overrides setting below is on (the default), the nightly purge deletes all stored values of a point set to Do not log, so do not switch a point whose history you still need

Beyond Time-Series Data

Point values are usually the largest consumer, but not the only one, and the others are not governed by point purge settings:

• Events: each event category has its own purge period on Purge settings, one year by default. Chattering alarms are what make this grow
• Audit trail: Purge audit events older than, one year by default
• Backups: on by default, daily. Configuration and SQL database backups keep 10 files each; the Mango NoSQL backup is incremental and keeps 365 versions. Size these from how many copies you keep
• Logs: ma.log and script.log roll over and are deleted after 30 days, 1,000 files or 1 GB in total, whichever comes first
• Excel report output: finished reports are purged after one month by default (Administration > System settings > Excel reports)
• Heap dumps and files you add to file stores: no retention unless you give them some

Measure these the same way as point values, from the actual installation.

Verification

• ☐ Record count calculated from each point's real Logging type and interval, with change-based points sampled rather than assumed
• ☐ Bytes per record measured on the installed store over weeks, per data type where the point list is mixed (30 bytes per numeric record is the placeholder until then)
• ☐ Estimate re-measured after the first purges, so it reflects steady state rather than initial growth
• ☐ Use data source and data point purge setting overrides (slower) checked, and ma.log read after a purge run, if the plan depends on overrides
• ☐ NoSQL database size (GB), SQL database size (GB) and the partition usable-space monitors (for example NoSQL partition usable space (GB)) collected and trended
• ☐ Backups, logs and report output included in the total
• ☐ Actual usage compared against the estimate monthly, and the estimate revised

Troubleshooting

Summary

Calculate records exactly from each point's logging settings, point count × (86,400 ÷ seconds between logged values) × retention days, remembering that new numeric points log once a minute by default. Then measure bytes per record on your own store over weeks rather than applying a published figure. Add events, audit, backups, logs and report output to the total, confirm that any purge overrides you are counting on are switched on, and trend Mango's database-size monitors so the estimate can be corrected from reality.