Alerts
FW Version: 7611+
Alerts can be generated by system automatically either
EMail alerts
Syslog alerts
SNMP Traps
Alert configuration file is located in
/opt/fmadio/etc/alert.luaBy default all Alert triggers are disabled.
Example Alert.lua
An example alert.lua file is shown below. If the file does not exist please create.
local L =
{
["AlertList"] =
{
LinkState = true,
ByteCache = 1e12,
BytesOverflow = true,
PacketError = true,
PacketDrop = true,
CaptureState = true,
DiskSMART = true,
DiskFreeStore0 = 1e9,
DiskFreeStore1 = 0,
DiskFreeRemote0 = 0,
CPUTemperature = 80,
FANAlert = true,
PSUAlert = true,
Sleep = 60, -- how long to sleep when an alert is triggered. prevents flodding
}
}
return L
Triggers
System has can trigger an a small but well defined list of critical Events. The following is a description and example for each item. Triggers are enabled or disabled in the following part of the configuration file. Each line enables/disabled or puts a threshold on the trigger
Each trigger is described below.
LinkState (Capture Port State)
Monitoring the capture link status is critical to ensure no data is lost. Enabling this option will alert when a capture link goes up or down.
Config
SYSLOG
SNMP
CaptureState
Capture State shows the capture is active or in-active. When using in alert mode it will trigger anytime the capture state changes
Config
SYSLOG
SNMP
Bytes Cached
Bytes Cached indicates how much capture data has been written to SSD, but not written back into long term storage yet. e.g. Its the delta between the capture SSD rate, and the HDD magnetic storage writeback. Trigger on for example 3TB here provides a good indication the HDD writeback process is running too slow for the sustained incoming capture rate.
Config
(example trigger once Cache goes overt 1TB)
SYSLOG
SNMP
BytesOverflow (trigger)
Any time Bytes Over increases an alert is generated. This typically a symptom of capture rates being too high, or HDD writeback too slow (or failing)
Config
SYSLOG
SNMP
PacketError
Counts FCS errors received on the interface. Any time packet error counts changes an alert is generated. Typically occurs when there are Layer1 link stability issues
Config
SYSLOG
SNMP
PacketDrop
Alerts generated when packets are dropped on the capture device.
Config
SYSLOG
SNMP
DiskFreeStore0
When space on /mnt/store0 partition is less than this amount (scientific notation) in bytes. Alerts are generated.
In the below example, an alert is generated when less than 4e9 (4GB) of space is free on /mnt/store0 partition
Config
SYSLOG
SNMP
DiskFreeStore1
When space on /mnt/store1 (scratch analytics workspace) is less than this amount (scientific notation) in bytes an Alert is generated
Config
SYSLOG
SNMP
DiskFreeRemote0
When space on the /mnt/remote0 (typically NFS mount partition) is less than this threshold an Alert is generated
Config
SYSLOG
SNMP
DiskError
Alerts when there is a disk error or RAID error on the device. For example a disk has been lost or HDD RAID redundancy has been reduced.
Config
SYSLOG
SNMP
DiskSMART
Alerts on the total number of disk SMART errors. The value is aggregated across all disks, please check the system log files for more details about which specific disk is having an issue.
Config
SYSLOG
SNMP
Sleep
Minimum number of seconds between alert generation. This is to prevent spamming of alerts due to unexpected system conditions.
SYSLOG Alerts
Alert events are always output to SYSLOG regardless of the other transport modes (email/snmp etc)
SYSLOG logfile is found in
An example syslog alerts as follows.
EMAIL Alerts
Email alerts can be setup as the following, please add the ["Email"] section in the alet configuration file
An example that sends alerts to the address "[email protected]" is shown below.
Email Server
In addition fmadio packet capture system uses msrtp as the email client, it requires smtp configuration file
Example configuration as follows. Please edit to match the email smtp provider
SNMP Broadcast
FW: 7611+
FMADIO devices can operate in SNMP Broadcast mode. In this mode the system will periodically broadcast all SNMP counter values at a fixed time interval to an SNMP target.
SNMP MIB
Latest MIB file is found (last updated 2021/12/25)
Config
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP Broadcast mode only, while SNMP Trap(Alert) mode is disabled. Broadcast frequency is 60e9 nanoseconds, e.g. every 1 minute.
Broadcast and Trap mode can be use simultaneously if required.
Please update ["Target"] = setting to the correct SNMP collector address. Multiple SNMP targets can be specified separated by spaces. For example
Example output in broadcast mode is as follows, from the /mnt/store0/log/monitor_alert.cur logfile
This translates to
Troubleshooting
Logfiles are found /mnt/store0/log/monitor_alert.cur
Verbose mode above can be set to "true" to allow additional logging.
SNMP Trap
FW: 7611+
FMADIO Devices can send SNMP Traps based on the alert triggers described above. This may be preferable to email alerts for infrastructure management.
SNMP MIB
Latest MIB file is found (last updated 2021/12/25)
Config
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP TRAP mode only, SNMP Broadcast mode is disabled. This configuration will only send SNMP TRAP events when a Trigger is alerted.
Please update ["Target"] = setting to the correct SNMP collector address.
Troubleshooting
An easy way to trouble shoot traps is to se the DiskFreeStore0 threshold to a very large number. In this setup the SNMP TRAP event will be constantly generated (every 1 minute).
Logfiles are found in /mnt/store0/log/monitor_alert.cur
Last updated
Was this helpful?