Monitoring Apache Ignite Memory and Disk Usage

Start by creating a dashboard with various memory and disk-usage metrics:

The Ignite keeps all application records and indexes in its memory regions, invisible to Java garbage collectors and fully managed by Ignite. The memory regions are usually known as “off-heap memory.”

Now, you add a widget that displays the off-heap memory usage to the dashboard:

Off-heap memory is the primary storage tier for application records. However, Ignite nodes still use Java heap, mostly for temporary objects that are generated during processing of operations that are executed by the application.

Next, you add a widget that monitors Java heap memory usage to the dashboard:

If Ignite native persistence is configured for your cluster, then a copy of every application record will be persisted on disk.

Perform the following steps to add to the dashboard a widget that reports the size of the disk space that is consumed by the on-disk copies of your application records:

Ignite does not persist changes into storage files immediately, because doing so would require Ignite to perform updates in a random file location, an operation that is quite time-consuming. Instead, all changes are first appended to the WAL and then, later, are checkpointed to the storage files.

Now, you add a widget of the Metrics (table) type that enables you to observe WAL size:

Checkpointing is the process of copying dirty pages from memory to storage files on disk. After the dirty pages are written to disk, Ignite can safely trim the part of the WAL that contained the changes that were checkpointed during the previous checkpointing cycle. It’s prudent to monitor the duration of checkpointing cycles and, if Ignite spends too much time in checkpointing, to perform various disk optimizations.

Now, you create a widget to monitor the duration of the checkpointing process:

Ignite appends changes to the WAL for every data update. However, the operating system (OS) must still fsync the changes to disk to ensure that no data is lost, not even on unexpected node faults. If Ignite or the OS spends a significant amount of time in the fsync stage, then you might need to undertake various WAL-related optimizations.

Finally, you add a widget to the dashboard to monitor the WAL’s fsync stage duration:

Rearrange the widgets of the dashboard by moving them around as you want. In the end, your Storage Usage dashboard should be similar to the following dashboard.

The dashboard that you configured in this part of the tutorial can be displayed on the screens of the room that your team uses to monitor Ignite production clusters. But, what if nobody were in the room when the dashboard began reporting unexpected cluster activity? In the next part of the tutorial, you set up alerting, so you are notified about every event that might affect the stability or performance of the cluster: