Overview
Mango by Radix IoT can monitor itself using an Internal Data Source (the data source type in the picker). That data source exposes internal metrics — JVM memory, CPU load, thread counts, work queue depths, database connection pool counts, partition space, and database sizes. Once these points exist, you can build dashboards, set event detectors, and get alerted when system health degrades — before users notice problems. (The separate Internal metrics page under System status is a live view, not a data source type.)
The point of doing this is not to collect numbers, it is to establish what this system's normal looks like. Almost every useful threshold below is defined relative to your own baseline, because Mango does not define universal alarm levels and a value that is healthy on one deployment is a problem on another.
Prerequisites
- Admin access to Mango, sufficient to create a data source and event detectors
- Working email configuration if you intend to send alert notifications
- Enough retention on the metrics points to hold a useful trend — a few weeks at minimum
- Patience: thresholds set before you have a baseline will either be noise or will miss real problems
Setting Up Internal Metrics
- Navigate to Data integration → Data sources
- In the Select data source list, click New (or the + button on that row)
- Set Data source type to the internal data source type. Its properties section is headed Internal Data Source
- Give it a Name (e.g., "System Health Metrics")
- Set the Update period and its unit. This is how often Mango samples the monitors, and it is a period with a unit selector rather than a plain seconds field — a minute or two is a reasonable starting point for trend data
- Choose a Create points pattern. This is what decides which points you get: it is a selector, not an automatic sweep of everything. All polling metrics creates the three per-data-source polling points for every polling source, which on a large system is a lot of points
- Tick Enabled and click Save
The data source editor's other tabs — Event alarm levels, Purge, Data points, Current alarms, Runtime status — are where you review what was created and how long it is kept. Data points is the one to open first.
Key Metrics to Monitor
The tables below describe what each metric tells you and what pattern is worth acting on. They deliberately do not give universal numeric thresholds — set those from your own baseline once you have one.
JVM Memory
| Metric | What It Means | What to watch for |
|---|---|---|
| JVM used memory (MB) | Heap currently in use. Despite the label, the value is divided by 1024², so the units are really MiB | The shape of the trend across garbage collections, not the instantaneous value |
| JVM maximum available memory (MB) | Maximum heap configured (-Xmx) | Reference value — changes only when you change it |
| JVM free memory (MB) | Maximum minus used. Note this is not Java's own freeMemory(), which reports only the unused part of the currently allocated heap; Mango computes headroom up to -Xmx, which is the more useful number | Same trend, viewed from the other side |
Warning
Do not treat a heap percentage as a sizing rule. A JVM is designed to fill its heap and then collect, so a reading near the top of the heap is expected behaviour and on its own tells you nothing. What matters is the post-GC baseline — the level used memory returns to after each collection. A sawtooth that keeps returning to the same floor is a healthy system, however high the peaks. A floor that creeps upward run after run means something is being retained, and raising -Xmx only delays the symptom while making pauses longer. Mango Core exposes max, used, and free memory; it defines no remediation threshold, and there is no percentage at which “add more heap” is the correct answer without further evidence.
When the post-GC trend does look wrong, widen the investigation before resizing: garbage collection frequency and pause times, CPU, the work queues below, database latency, polling load, query and export volume, and whether the trend started at a particular configuration change.
Threads and Work Queues
| Metric | What It Means | What to watch for |
|---|---|---|
| JVM thread count | Live thread count. JVM daemon thread count, JVM peak thread count and JVM total threads started are separate monitors | A count that climbs and never returns to baseline. The absolute number is only meaningful relative to this system's own history |
| High priority tasks queued / High priority work items currently running | Depth and activity of the high priority work queue. Medium and low have their own pair, and high additionally has Currently scheduled high priority tasks | A queued count that grows and does not drain. Brief spikes during scheduled work are normal |
Queues that drain are doing their job. A queue that grows steadily means work is arriving faster than it completes, which is a capacity or blockage question — not automatically a hardware question.
Warning
The low priority queue monitors are unreliable at 5.7.4. In ServerMonitoringService the low priority block assigns the active monitor twice instead of assigning the waiting monitor, so “Low priority tasks queued” never receives a value and “Low priority tasks currently running” actually reports the queue depth. High and medium priority are computed correctly. Chart and alert on high and medium; treat the low pair as unusable until this is fixed, and do not read a flat “Low priority tasks queued” series as a healthy queue.
Database
| Metric | What It Means | What to watch for |
|---|---|---|
| Active database connections / Idle database connections | Usage of the main connection pool. The read-only pool is reported separately as Active read-only database connections and Idle read-only database connections | Sustained saturation of a pool relative to its configured maximum — db.pool.maxActive (10 by default) and db.read.pool.maxActive (20) |
| SQL database size (GB) | Size of the SQL database | Growth that does not level off after your retention period |
| NoSQL database size (GB) | Size of the point value store | Same — this is the number to trend against your storage estimate |
For query latency, Mango logs slow queries directly rather than exposing an average: set db.useMetrics=true with a db.metricsThreshold in milliseconds and slow queries are logged at WARN. Pick the threshold from what your system normally does.
Disk and Partition Space
| Monitor | What It Reports |
|---|---|
| Mango partition usable space (GB) | Space left on the partition holding Mango's home directory |
| NoSQL partition usable space (GB) | Space left on the partition holding the point value store |
| SQL database partition usable space (GB) | Space left on the partition holding the SQL database. Note the label includes "database", unlike the other three |
| Filestore partition usable space (GB) | Space left on the partition holding the file store |
Total and used-space monitors exist for each of these partitions as well. Alert on the specific partition that matters — these can all be different volumes, and a generic "disk free" reading will not tell you which one is filling.
Polling Performance
| Metric | What It Means | What to watch for |
|---|---|---|
| Poll duration | Time to complete a data source poll — the point's attribute reads "source last poll duration" | Duration approaching or exceeding the configured poll interval, which causes polls to cascade |
| Poll success percentage | The share of this source's scheduled polls that were not aborted, counted since the data source last started. A poll counts as successful as soon as it starts on time, before it talks to the device, so a device that times out or answers with errors does not lower it. Only an aborted poll does: one that would have started more than a whole update period late, or that the scheduler rejected. A source on Use cron is never aborted for lateness. There is no "poll failures" metric | Anything below 100%, which means polls are being aborted. Because the figure is cumulative since the source started, a long-running source moves very slowly when aborts begin, so treat it as a lifetime summary and use Consecutive run polls or the source's Runtime status tab for recent aborts |
| Consecutive run polls | Written only when a poll is aborted: it records how many polls succeeded since the previous abort, or since the source started. It reads -1 until the first abort, and it does not change while polls keep succeeding | Any value other than -1, which means the source has aborted at least once. A run of small values means aborts are recurring close together, even while the percentage still looks acceptable |
Poll duration against poll interval is one of the few comparisons that is meaningful without a baseline, because the interval is a constraint you set rather than an arbitrary number.
These three are created per polling data source, which is why All polling metrics produces so many points: three for every polling source on the system. If a source is removed but its points remain, the attribute column shows a "Missing monitor with id…" message rather than a value.
Building a System Health Dashboard
Favour charts over gauges. A gauge shows an instant; almost every question you will ask of these metrics is about a trend.
- Memory trend chart — used memory over at least 24 hours. This is the primary heap view, because the post-GC floor is only visible over time. A gauge of current heap usage will mislead you
- Work queue chart — queued counts for the high and medium priority queues, so you can see whether they drain. Leave low priority off it until the monitor bug above is fixed
- Thread count trend — with enough history to distinguish a spike from a climb
- Database size trend — SQL and NoSQL database size, against the retention you expect them to level off at
- Partition usable space — one series per partition in use, not a single combined figure
- Polling performance table — data sources with last poll duration and poll success percentage, against their configured intervals
Setting Up Health Alerts
Set thresholds after you have a baseline, not before. The mechanics:
Create the email handler first, because the detector attaches to an existing handler rather than creating one. If you do not have one yet, go to Event Management → Event handlers, click New, set the type to Email and configure its recipients.
Then, for each metric you have baselined:
- Open the data source's Data points tab, open the metric point's More options menu and choose Event detectors (or use the Event detectors link on the point's details page). There is no Event detectors tab: the detectors open in a dialog, Event detectors for point
- Click New (or the + button) on the detector list in that dialog — the row reads "Create event detector"
- Set Event detector type to the one that matches the question you are asking. For a level, that is a high limit
- Set the limit from your observed baseline, pick an Alarm level that reflects what you would actually do about it, and set a Duration so brief normal spikes do not alarm
- Open the detector's Event handlers tab and tick the email handler you created. It is a checklist of existing handlers, not a place to build one
- Click Save
Warning
Choosing a heap detector is different. A plain high limit on used memory will fire on normal pre-collection peaks and train everyone to ignore it. If you alert on heap at all, alert on a level sustained for a duration long enough to span several collections, and treat firing as a prompt to look at the post-GC trend — not as a signal to increase -Xmx.
What to Do When Metrics Trigger Alerts
Each of these has several plausible causes. Use the alert to start the investigation, not to conclude it.
| Alert | First look | Then investigate |
|---|---|---|
| Sustained high heap | Whether the post-GC floor is stable or climbing, and whether anything unusual is running (large exports, heavy dashboard or report use) | GC frequency and pause times, work queues, query and export volume, and when the trend started. Resize only if the evidence says the working set genuinely exceeds the heap |
| Climbing thread count | Whether any data source is stuck or a network path is slow | Which threads are accumulating, and whether the count tracks a specific source, protocol, or scheduled activity |
| Slow queries in the log | Which queries are slow, from the db.useMetrics output | Database server load, disk I/O, dataset size against retention, and whether the same queries were always slow or have degraded |
| Connection pool saturated | Whether the saturation is constant or tied to particular activity | Query duration first — a pool fills because connections are held too long as often as because there are too few. Check the database side before raising the pool maximum |
| Poll success percentage below 100%, or a new Consecutive run polls value | The source's Runtime status tab: Latest aborted poll times, and Poll duration against the update period. These monitors count aborted polls only, so a device answering with errors will not show here; read the source's own events and its points' Point unreliable warnings for that | Whether it is one source or many. One source whose polls outlast its update period needs a longer period or less work per poll; many at once suggests the host, Mango's own load, or a full high priority thread pool (Rejected stats on System status Work items) |
| Partition space falling | Which partition, from the specific monitor | Compare the database-size monitors against partition usage to tell whether point values, backups, logs, or something else entirely is consuming the space |
Baseline Your System
Let the metrics collect for at least a week before setting any threshold, and make sure that week includes your real operating cycle — overnight purges, backup windows, month-end reporting, and whatever else runs on a schedule. Record the normal range for each metric, and set thresholds outside it with a duration attached. Revisit the baseline after significant configuration changes; a threshold tuned to a system that no longer exists is worse than no threshold at all.
Verification
- ☐ The Internal Data Source is enabled and its points are receiving values at the configured interval
- ☐ Metrics points have enough retention to show the trends you need
- ☐ The memory chart shows a full sawtooth, so the post-GC floor is actually visible
- ☐ At least one full operating cycle of baseline data has been collected before thresholds were set
- ☐ Every threshold is written down alongside the observed baseline it was derived from
- ☐ Heap detectors use a sustained duration rather than an instantaneous limit
- ☐ A deliberately triggered test detector produces the expected event and email
- ☐ Partition alerts reference the specific partition monitor, not a generic disk reading
- ☐ Work queue alerts use the high and medium priority monitors, not the low priority pair
- ☐ Alerts have been reviewed after a few weeks and any that only ever produce noise have been retuned or removed
Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
| Internal metrics points have no data | The data source is disabled, or the polling period was never set | The data source's enabled state and last poll time |
| Expected metric points do not exist | Not every monitor is available in every installation — some depend on the modules and backends in use | The point list the data source created, and which modules are installed |
| Heap alerts fire constantly | An instantaneous limit is catching normal pre-collection peaks | Add a duration long enough to span several collections, and re-derive the level from the post-GC floor |
| Heap was increased and the problem came back | The post-GC floor was climbing — more heap postponed the symptom rather than fixing it | Trend post-GC usage across the period before and after the change, and look for what is retaining memory |
| Thresholds produce noise on one system and silence on another | Numbers were copied between deployments instead of derived from each baseline | Re-derive from each system's own observed range |
| No emails arrive when a detector fires | Handler or SMTP configuration, not the metric | That the event is actually being raised, the handler is attached to the detector, and SMTP settings work |
| Trends have gaps | Metrics point retention is shorter than the window you are viewing, or the source was disabled for a period | Retention on the metrics points and the data source's history |
Summary
The Internal Data Source lets Mango monitor itself. Set it up early, chart trends rather than instants, and collect a real baseline before you set a single threshold — Mango exposes the measurements but defines no universal alarm levels, so every threshold worth having comes from your own system's normal range. Be especially careful with heap: read the post-GC trend rather than a percentage, and treat a high reading as a prompt to investigate rather than a reason to resize.