How to Manage Disk Space and Purge Historical Data

Overview

Mango by Radix IoT does not ship wide open on retention. A fresh 5.7.4 install enables point value purge, runs it nightly at 3:05 AM, and keeps one year of history by default. Disk still fills, but it fills for identifiable reasons: purge was turned off, the retention period was extended past what the disk can hold, an override kept far more than expected, or something outside the purge system — backups, logs, exports, heap dumps — grew without anyone managing it.

The consumers that most often matter are time-series point values, backup archives, event messages, and log files. They are not the only things on the partition, so measure your own filesystem rather than assuming these four account for whatever is missing. This guide explains how to monitor each one, configure purge settings, and free space safely when you need to.

Prerequisites

• Mango 4.x or 5.x with admin (superadmin for the SQL console) access
• Shell access to the Mango server for filesystem measurement
• Knowledge of which point value backend your installation uses — it determines how retention is enforced
• An agreed data retention policy, so you know what you are permitted to delete

The Common Disk Consumers

Measuring Before You Plan

Bytes per point value depend on the backend, the data type, and how well values compress, so published per-value figures do not transfer between installations. Size your storage from your own system instead:

1. Work out the record count from the logging configuration — point count multiplied by samples per point per day. See Configure Point Logging for the arithmetic.
2. Measure the current point value store size on disk, wait a representative week, and measure again.
3. Divide the growth by the records added over that week to get a bytes-per-value figure for your deployment, then project forward.

# Measure the point value store and the other consumers
du -sh /opt/mango/databases/ /opt/mango/backup/ /opt/mango/logs/

Mango also exposes NoSQL database size (GB) and SQL database size (GB) internal monitors, which give you the same measurement without a shell.

Configuring Purge Settings

Purge settings control how long historical point values are retained before automatic deletion. They exist at three levels: a system default, an override per data source, and an override per data point.

System-Level Purge (Default for All Points)

1. Go to Administration > System settings > Purge settings
2. Confirm Enable purge of point values (otherwise rely on database retention policy) is ticked
3. Set Purge point values older than — the default is 1 year
4. Confirm the purge schedule; the default cron pattern runs the task at 3:05 AM daily

Mango runs the purge task on that schedule and removes point values older than the retention period. The cutoff is calculated as though the run happened at midnight, so a shifted run time does not shift the retention boundary.

The Override Checkbox

Before configuring any override, look at Use data source and data point purge setting overrides (slower) on the same page.

If you are relying on overrides, verify they are actually taking effect (see Verification below) rather than assuming the configured value is the one in force.

Data Source-Level Purge (Override per Source)

Override the system default for all points belonging to a specific data source:

1. Open the data source’s Edit page
2. Enable Override system purge settings
3. Set a custom period (e.g., 6 months for high-frequency diagnostic data sources)

All data points in this source inherit the override unless they have their own point-level setting.

Data Point-Level Purge (Override per Point)

The most granular control — set retention on individual data points:

1. Open the data point’s Logging properties
2. Enable Override data source purge settings
3. Set the desired retention period

Precedence

When per-point purge is enabled and supported, Mango resolves retention for each point in this order:

1. The data point override, if set
2. otherwise the data source override, if set
3. otherwise the system default

When per-point purge is not enabled or not supported, only the system-wide retention applies.

Freeing Space Safely

When the disk is critically low, work through the options that do not risk your data first:

1. Old backup archives in backup/ — remove them once you have confirmed the file is a completed backup, not one currently being written, and that your retention policy permits it.
2. Rotated log files in logs/ — remove closed, rotated files. Leave the active log alone.
3. Exports, reports, and heap dumps left behind in the data directory.
4. Shorten retention in Purge settings and run a manual purge from Administration > System settings > Purge settings, which deletes through Mango and keeps its indexes and metadata consistent.

Built-In Disk Space Monitoring

Mango publishes internal monitors for partition space, reported in GB. Use them to alert before the disk fills. The relevant monitors are:

Matching total space and used space monitors exist for each partition, along with NoSQL database size (GB) and SQL database size (GB).

1. Locate the internal data source that exposes system monitors on your installation
2. Enable the usable-space point for the partition you care about — pick the one that actually holds the data you are worried about, since these can be different partitions
3. Create an Event Detector with a low-limit threshold (e.g., alarm when usable space drops below 2 GB)
4. Attach an Event Handler (email, Slack, or other) to get notified

This gives your team early warning and time to act before a disk-full condition causes downtime.

Event Message Purge

Event messages (alarms, system events, audit trail entries) accumulate in the SQL database events table and can grow to millions of rows on busy systems.

Configure Event Purge

1. Go to Administration > System settings > Purge settings
2. Set Purge all events older than (e.g., 90 days)
3. Optionally set separate retention per event category — data point, data source, system, publisher, and audit events each have their own setting, as does each alarm level from None through Safety. All default to 1 year

Counting Events with the SQL console

To check how many events are stored:

1. Open Administration > Admin home and click SQL console on the Utilities card. SQL console is superadmin-only and is not in the menu
2. Run:

SELECT COUNT(*) FROM events;

For a breakdown by alarm level:

SELECT alarmLevel, COUNT(*) AS total
FROM events
GROUP BY alarmLevel
ORDER BY total DESC;

Log File Management

Mango rotates its log files automatically via Log4j, and deletes old ma.log and script.log rotations itself. A rotation is deleted once it is 30 days old, or once there are more than 1,000 files or 1 GB of them, whichever comes first (the appender.logfile.delete.* and appender.script.delete.* entries in mango.properties). The three NoSQL logs (iastsdb-corruption, iastsdb-compression and iastsdb-reverse) keep only their last 20 rotations, with no age limit, so they can hold files older than 30 days.

# Check total log directory size
du -sh /opt/mango/logs/

# List rotated logs older than 30 days - review before deleting
find /opt/mango/logs/ -name "*.log.*" -mtime +30 -ls

Review that list before removing anything. Source code cannot tell you whether a given file is still open, shipped to a log aggregator, or required by your retention policy — confirm each of those, then delete.

If logs are growing too fast, change the logging settings in mango.properties. The default log4j2.xml ships inside Mango’s Core and takes its levels, file size and retention from the appender.* entries there; for more control, point the log4j2.configurationFile property at a configuration file of your own. Avoid running DEBUG or TRACE level logging in production — INFO is appropriate for most deployments.

Verification

• Purge settings shows point value purge enabled, with a retention period the disk can actually accommodate
• The nightly purge appears in ma.log at the scheduled time, reporting values deleted
• If you rely on overrides, Use data source and data point purge setting overrides (slower) is enabled and the backend supports per-point deletion
• Pick one point with a short override, query its oldest remaining value after a purge run, and confirm the cutoff matches the override rather than the system default. If it matches the system default, your overrides are not being applied
• Partition usable-space monitors are enabled for the partitions that actually hold your data, with low-limit detectors and a working notification handler
• Measured growth over a week is consistent with the record-count projection for your logging configuration
• No manual deletion has taken place inside the point value store

Troubleshooting

Disk full — Mango will not start

• Free space from the safe consumers first: completed backup archives in backup/, closed rotated logs in logs/, and leftover exports or heap dumps
• Then shorten retention and let Mango purge. Do not hand-delete files from the point value store to recover space

Purge task does not seem to run

• Verify the purge schedule in Purge settings and confirm the configured time has passed
• Confirm point value purge is enabled rather than deferred to a database retention policy
• Check ma.log for purge-related messages or errors

Retention overrides appear to be ignored

• Check Use data source and data point purge setting overrides (slower) on the Purge settings page. When it is off, overrides have no effect
• Confirm the active point value backend supports per-point deletion. Backends that enforce retention database-wide apply only the system-wide period
• Remember the precedence: point override, then data source override, then system default

Disk usage growing despite purge settings

• Data points set to All data logging at high frequency generate enormous volumes, and log every poll even when the value never changes
• Move high-frequency points to an interval-based logging type with a longer period
• Review whether all points truly need history — some diagnostic points can be set to Do not log
• Check the non-purge consumers too: backups, logs, and exports are outside the purge system entirely