How to Plan Data Retention Policies for Compliance and Performance

Overview

Every value that Mango by Radix IoT logs to its historian consumes disk space. Retention policies define how long historical data is kept; purge settings define when and how old data is deleted. Getting them right is the difference between a system whose storage footprint is predictable and one whose growth you find out about from a full disk.

Mango 5.7.4 does not leave you unprotected by default. Point value purge is enabled out of the box, the default retention is one year, and the purge task runs nightly at 3:05 am. Event and audit purge defaults are also one year, per category. What this article is really about is deciding whether those defaults suit your deployment, and overriding them deliberately where they do not.

Prerequisites

• Mango 4.x or 5.x with admin access
• Understanding of your data logging configuration (see How to Configure Point Logging for Optimal Data Collection)
• Knowledge of your organization's data retention requirements (regulatory, operational, or contractual)
• Knowledge of which point value backend your installation uses — the override behaviour below depends on it

Deciding Whether the Defaults Suit You

Rather than assuming the one-year default is wrong, work out what it costs you:

• Query and report latency — longer retention means larger scans for the queries that span it. Measure the queries you actually run.
• Backup size and duration — a larger dataset produces larger backups that take longer and need somewhere to live.
• Restore time — the retention period you choose is also the amount of data you have to restore after a failure.
• Regulatory floor — some data must be kept for a defined period regardless of what it costs. That is a floor, not a target for every point.

System-Wide Purge Settings

Configuration Steps

1. Navigate to Administration → System settings → Purge settings
2. Confirm Enable purge of point values is on. Turning it off means Mango relies on the database's own retention policy instead — only do this when the backend actually has one
3. Set Purge point values older than to your chosen period (default: 1 year)
4. Review the event and audit retention rows below it — each event category has its own period, all defaulting to 1 year
5. Click Save

The nightly task deletes point value records older than the configured period. The same page provides Purge Now Using Purge Settings and Purge All Events for running a purge on demand.

The Override Hierarchy — and Its Precondition

Mango supports three levels of retention: the system default, a data source override, and a data point override. When they are in play, the most specific wins — point override beats data source override, which beats the system default.

They are only in play when per-point purge is active. Two things have to be true:

1. The global setting is on. On the Purge settings page, Use data source and data point purge setting overrides (slower) must be checked. It defaults to on, but it is a real switch and it is frequently turned off for performance.
2. The active point value backend supports it. Some backends handle retention database-wide instead of per series. When per-point purge is not enabled or supported, Mango purges everything against the system default and logs that data source and data point purge overrides have been ignored.

Per-Point Override Purge Settings

Not all data has the same retention requirements. Mango lets you override the system-wide setting on individual points.

1. Navigate to the point's Edit data point page
2. Find the Purge section
3. Enable Override data source purge settings
4. Set the retention period for this specific point
5. Click Save

Per-point settings take precedence over the data source override and the system-wide default — subject to the precondition above.

One behaviour worth knowing: a point whose logging type is None has all of its stored values purged, regardless of any retention period. Mango assumes a point that is not logging should not be holding history.

Per-Data-Source Override Purge Settings

Mango also lets you override retention at the data source level. All points on that source inherit the source purge policy unless a point has its own override.

1. Navigate to the source's Edit data source page
2. Open the Purge section and enable Override system purge settings
3. Set the retention period
4. Click Save

Use data-source overrides when an entire protocol or site shares the same retention requirement — for example, keeping all BACnet points for 90 days while the system default stays at 30.

Three-Tier Retention Strategy

This is a planning framework, not a product feature or a Mango-defined limit. The intervals and periods below are starting points to adapt to your own data and obligations.

Tier 1: Operational / Live Data

Tier 2: Analytical Data

Tier 3: Compliance / Archive

Implementation

1. Confirm Use data source and data point purge setting overrides (slower) is enabled — without it, steps 2 and 3 have no effect
2. Set system-wide purge to Tier 1 retention (e.g., 30 days)
3. Use data source overrides for whole sites or protocols, and per-point overrides for individual Tier 2 points
4. Create rollup meta points for Tier 3 with multi-year retention
5. Verify after the next nightly run that ma.log reports the per-point purge mode, not the database-wide fallback

Sizing Retention Against Real Storage

Record counts are arithmetic and you can calculate them exactly:

Records per point per day = 86,400 / logging interval in seconds
Total records = point count × records per point per day × retention days

A point logging every 10 seconds produces 8,640 records a day; 3,000 such points produce about 25.9 million records a day, or 4.7 billion over 180 days.

Converting records to bytes is where published figures stop being reliable. Per-record storage depends on the point value backend in use, its on-disk format, compression, index structures, and the shape of your data. Mango Core does not define a bytes-per-record figure, and a number taken from someone else's deployment will not transfer to yours.

Measure it instead:

1. Run a representative subset of points at the real logging configuration for a known period — a week is usually enough
2. Record the growth of the point value store on disk over that period
3. Divide by the record count you generated to get your bytes per record
4. Scale that to full point count and full retention, then add headroom
5. Re-measure after the system has been in production long enough for the first purge cycles to run

The one thing you can say confidently without measuring is the direction: halving the logging rate halves the record count, and record count is what drives storage. Model several logging intervals in records first, then price the one you pick.

Backups and Other Growth

Point values are usually the largest single consumer, but they are not the only one. Backup archives, rotated log files, report output, exports, and heap dumps all grow, and none of them are governed by the point purge settings.

• Check the backup retention configured for your installation rather than assuming a default — the SQL and NoSQL backup settings are separate from purge settings, and the available options depend on the modules installed
• Place backup directories on a separate volume from the live database so a runaway backup cannot take the system down
• Give log rotation and report output their own retention
• Before deleting rotated logs or backup archives, confirm the files are complete and inactive and that your retention policy permits it

Monitoring Storage Health

Mango exposes partition and database-size monitors internally. Use the exact monitor for the partition you care about rather than a generic "free space" reading — Mango's home directory, the point value store, the SQL database, and the file store can all live on different volumes:

Total and used-space monitors exist for each partition as well. Trend the database-size monitors over weeks: a flat line after the first full retention period means purge is keeping up, and a line that keeps climbing means it is not.

Also check ma.log after purge runs. Mango logs which purge mode it used and, where the backend can report it, how many point values were deleted.

Common Mistakes to Avoid

Verification

• ☐ Administration → System settings → Purge settings shows purge enabled and your intended retention period
• ☐ Use data source and data point purge setting overrides (slower) matches your intent — checked if you rely on overrides
• ☐ After a nightly run, ma.log states which purge mode was used and reports completion
• ☐ A point with a shorter override has no data older than its override period
• ☐ A point with a longer override still has data older than the system default
• ☐ Event and audit retention periods are set per category to your requirement
• ☐ Partition usable-space and database-size monitors are being collected and trended
• ☐ Measured storage growth is tracking your estimate; the estimate has been revised from measurement at least once
• ☐ Backup, log, and report output have retention of their own

Troubleshooting

Summary

Mango 5.7.4 already purges point values nightly at 3:05 am with a one-year default, so retention planning is about overriding that deliberately rather than rescuing an unprotected system. Set the system default from what your data actually costs on your backend, use data source and point overrides for tiered retention — after confirming the global override setting is enabled and the backend supports it — measure real storage growth instead of relying on published bytes-per-record figures, and monitor the partition and database-size monitors so growth is something you observe rather than discover.