Classification: FMOS
Category: System Resource Issue
Context: Warning or Error
Summary
The indicated filesystem is low on disk space. The system may encounter operational issues if there is insufficient storage space.
Description
This warning appears if one or more FMOS filesystems are below the configured threshold of free space. Each filesystem path has a configurable threshold to control when a warning or error message is generated. By default, a warning will be emitted for most filesystems when they reach 75% full.
Impact
The consequences of operating a system without sufficient free space can vary from frequent alerts and warnings to disruption of service and may result in data loss. The sections below describe the specific impact of each path.
System Runtime Directory: /run
The system runtime directory, /run, is critical to the operation of the operating system and all running software. If it becomes too full, the entire system may become unstable. Applications and services may crash or fail to start. Spurious errors from various process are likely.
This filesystem is ephemeral and is not backed by any physical storage medium. It exists in system memory only.
System Configuration Directory: /etc and /run/fmos/etc
The system configuration directory contains configuration files for the operating system, applications, and services. If it becomes too full, it may be impossible to make configuration changes, including changes made automatically during update. Applications and services may crash or fail to start.
Ephemeral Temporary Data Directory: /tmp
The /tmp directory contains temporary data written by applications and services running on the system. If it becomes too full, processes may be not be able to operate normally. Applications and services may crash or fail to start.
Persistent Data Directory: /var
Persistent application and system state are stored in directories located on the filesystem mounted at /var. If it becomes too full, the entire system may become unstable. Applications and services may crash or fail to start. Spurious errors from various processes are likely. Data loss or corruption is very probable.
Log Directories: /var/log and /var/log/audit
Log files created by the operating system, applications, and services are stored under /var/log. If it becomes too full, some applications and services may behave in unexpected ways. Some applications and services may crash or fail to start.
The /var/log/audit directory contains the system audit log. If it becomes too full, audit events may be lost.
System Boot Directory: /boot
The operating system image other files required to boot the system are stored in the /boot volume. If it becomes too full, it may be impossible to update the system to a new version of FMOS.
Resolution
The appropriate process for dealing with low free disk space depends on the filesystem in question.
System Runtime Directory: /run
The system runtime directory should never run low on disk space under normal operating conditions. If an error is emitted for this filesystem, it most likely indicates a larger problem with the system or a running application.
Because the /run filesystem is ephemeral, rebooting the machine will completely erase it. Files stored there will be recreated the next time the machine starts.
If the /run filesystem consistently runs low on space, contact FireMon Support for assistance in troubleshooting and resolving the underlying issue.
System Configuration Directory: /etc and /run/fmos/etc
In general, the size of the system configuration directory will remain relatively constant. A very small filesystem is mounted at this path, as it typically only contains a few plain text configuration files. As such, it is extremely unlikely for this filesystem to run low on space.
Older versions of FMOS allowed users to write arbitrary files to the /etc/fmos/postbackup.d directory. Occasionally, users would mistakenly place very large files in this directory. Check to see that there are no unexpected files in this directory and remove them if possible.
Ephemeral Temporary Data Directory: /tmp
The most common cause for the temporary data directory to run low on space is a user having placed a large file within it. Check for and remove any user-created files stored in this location. Large temporary files should be stored in the /var/tmp directory instead of /tmp.
If there are no user-created files in the temporary directory, there may be files created by applications. The fmos disk-cleanup command may be able to remove these.
Contact FireMon Support if there are files in /tmp that were not created by users or cannot be removed with the FMOS disk cleanup utility.
Persistent Data Directory: /var
The persistent data directory contains all permanent data, including everything stored and manged by the FireMon Security Intelligence Platform software suite. If this directory is full, there are generally two ways to resolve the problem:
-
Increase the available storage capacity
-
Reduce the amount of stored data
Method (1) involves adding additional disks to the system or replacing the current disks with larger ones. For physical machines, using a disk array (RAID (Redundant Array of Independent Disks)), the array configuration will need to be adjusted. For virtual machines and physical machines without a disk array, additional disks can attached to the system. Use the fmos storage autoexpand command to instruct the operating system to make the new space available.
Method (2) may require reducing the amount of data managed by FireMon, either by removing managed devices or reducing the retention period for historical data.
Contact FireMon Support for additional information on how to add storage to the system or reduce the amount of data stored by the system.
Log Directories: /var/log and /var/log/audit
Typically, the log directories will fill up because an application or service is logging messages too quickly. This can be caused by an error condition that needs to be addressed. Alternatively, an application may be incorrectly configured to log too many messages. For example, configuring an application to log all messages at DEBUG level may consume all available space on the log volume in very short order.
Review the messages being logged to files in the log directory to determine the correct course of action.
System Boot Directory: /boot
The contents of system boot directory only change when the operating system is updated. It typically contains one or two operating system images.
Early versions of FMOS (8.26, 9.0, and 9.1) allocated a 4 gigabyte disk partition for the system boot directory. This size is sufficient for all currently-available versions of FMOS, but as the FireMon product grows and gains features and functionality, future versions may require additional space.
FMOS 9.2 and later allocate a 16 gigabyte partition for the system boot directory, however, the size of the partition cannot be changed after installation. Thus, upgrading to FMOS 9.2 or later will not expand system boot directory.
Both 4 GB and 16 GB system boot directory sizes are currently supported and will remain supported in the future. Customers do not need to make any changes to their systems to increase the size of the system boot volume. If a future release of FMOS requires a larger system boot volume, FireMon will provide alternative solutions.
Systems which display the free disk space warning for the system boot volume may require special handling when upgrading to future versions of FMOS. The warning is present as a reminder of this possibility. Customers may wish to silence the warning by changing the alert threshold; FireMon recommends setting the threshold to 10% free.
Customers should avoid installing legacy versions of FMOS when migrating to new physical or virtual machines. FireMon always recommends installing the most recent version of FMOS available when deploying new machines. Doing so has multiple advantages, and will ensure that no additional steps will be necessary during future upgrades.
Comments
0 comments
Article is closed for comments.