Home

Telemetry Syslog

FMADIO Syslog provides rich telemetry stream for monitoring the health of the system.

Telemetry is provided in JSON format for easy ingestion in monitoring and observability infrastructure. Each syslog moniting log event is described below, these are categorized into subsystems as follows

  • Generic SYSLOG events

  • FMADIO Temperature (Physical Thermal monitoring)

  • FMADIO Time (NTP, PTPv2, PPS)

  • FMADIO Other (uncategorized sensors)

  • FMADIO Power (status of power supply and consumption)

  • FMADIO Capture (monitoring state of the Packet Capture )

  • FMADIO IO (metrics related to Disk IO)

  • FMADIO Link (capture and management link status)

  • FMADIO Fan (physical status of FAN and cooling)

  • FMADIO Disk (status of both capture and OS disks)

  • FMADIO Cat (status of the stream_cat process used for extracting data off the disks)

  • FMADIO Push PCAP (status of the PCAP Push process)

  • FMADIO Push PCAP Split (event indicating each PCAP split completion)

  • FMADIO Alert (Custom onbox system alerts)

FMADIO Telemetry Service

FMADIO Telemetry service is included without charge on all support contracts. It provides a dashboard for each system, example screen shot shown below.

JSON Header

All FMADIO JSON messages have the following header fields

module

This provides a high level granularity on what the event/status is for, these map the sections below.

subsystem

This provides more granular view on what kind of event/status this is for

timestamp

This is the epoch time in seconds, where the FMADIO Monitoring system generated the event. Not the rsyslog time, although under normal situations they should be the same.

ver

This is the Firmware Version, it allows easy backwards compatibility as the system updates and improves as data is ingested into the monitoring system.

Generic SYSLOG

This includes generic syslog messages in free form plain text events. Example shown below

FMADIO Temperature

This provides thermal monitoring of the system to ensure its running within operating ranges.

Example JSON Pretty

FMADIO Time

Provides monitoring of time synchronization

Pretty JSON

FMADIO Other

Miscellaneous other fields

Pretty JSON

FMADIO Power

Provides power status and utilization information

Pretty JSON

FMADIO Capture

Provides status information around the capture process

Pretty JSON

FMADIO IO

Provides status and performance of IO disk/network related systems

Pretty JSON

FMADIO Link

Status information around port link status

Pretty JSON

FMADIO Fan

Status information around fans and cooling

Pretty JSON

FMADIO Disk

Status around the capture and os disk

Pretty JSON

FMADIO Cat

stream_cat is the core FMADIO utility for extracting packets off the storage system. Its used heavily for download, realtime processing and troubleshooting. This provides statistics and state of the currently running stream_cat instances.

Up to 8 simultaniousle stream_cat instances can be run at the same time. cat_*_ is per instance.

Pretty JSON

FMADIO Push PCAP Status

Provides status information of the currently active Push PCAP proceses

Pretty JSON

FMADIO Push PCAP File

Called after completing a PCAP file split event. Helpful to monitoring each and every PCAP split leaving the system

Pretty JSON

FMADIO Alert

The system has capibility to generate system alerts based on configuration files. These alerts can be sent to SYSLOG for ingestion by a monitoring system

Pretty JSON

Telemetry Service Setup

Setting up automatic telemetry as follows

Step 1) Generate unique SSH key

FMADIO devices by default have a pre-installed ssh key. To correctly secure and uniquely identify the system generate your own SSH key as follows.

Using a password less key ensures the automatic setup requires no manual intervention.

Example output per below

Send the above public key (below) to [email protected]

Step 2) Copy to persistent Storage

The SSH public/private keys are on the volatile file system. Copy the keys to the persistent storage.

NOTE: the key is renamed with an fmadio_* prefix. The system copies the keys from this location and renames them in the .ssh/id_rsa .ssh/id_rsa.pub directory during the boot process.

Step 3) Copy the reference boot script

There is a reference boot script located in

Copy this to to the /opt/fmadio/etc/boot.lua file to automatically establish ssh tunnel to the telemetry service.

After copying replace the "username" to the username provided by fmadio support and save the file.

Step 4) Copy the reference rsyslog config

In addition to ssh tunnel setup, rsyslog configuration to forward syslog messages to the SSH tunnel.

Copy the reference config to /opt/fmadio/etc/ directory as follows

No modifications are required.

Step 5) Reboot the system

Reboot the system to check all the above steps are executed correctly

Step 6) Validate

After rebooting log into the Grafana monitoring site with the assigned username and confirm data is being recevied.

Any problems please contact [email protected]

Telemetry via API

In addition to syslog, the systems telemetry information snapshot can be fetched using the JSON API.

Example command:

With the pretty formatted output as follows. This output will always match the above syslog information.

As a raw JSON blob

PreviousAlertsNextApplications

Last updated

Was this helpful?