Victoria Metrics Replaces SNMP

In release 9.3 SNMP and Monit were replaced by the FMOS Health Evaluation System. This system will send email and Syslog messages when health checks return abnormal results. These messages replace the email notifications sent by Monit using SNMP in previous versions of FMOS. With the removal of the SNMP agent, FMOS 9.3 will use collectd to gather performance information and expose it to Victoria Metrics. Victoria Metrics will extract this information from the HTTP service provided by collectd and store them in its time-series database. When the Send Performance Metrics variable is set to true, Victoria Metrics will be configured to send its data to FireMon Support in real-time.

Victoria Metrics is a time series database that is designed to collect, store, and observe metrics. It packs a large feature set into an efficient process. It is written in Go and distributed as a single executable binary file.

• Supports many metrics protocols
  • • Graphite
    • InfluxDB
    • OpenTSDB
    • Prometheus: A metrics collection system and time series database with an HTTP API (FireMon's preferred metric protocol)
    • JSON lines
    • CSV
• Push- or pull-based collection
• Optimized to minimize resource usage
• Prometheus-compatible query API and language. Prometheus is a metrics gathering server that can be connected to a visualization application such as Grafana.
• Can export and import time series data in several formats
• TLS support: HTTP and HTTPS are supported but not both simultaneously

Usage in FMOS

Victoria Metrics replaces Graphite in FMOS as the storage engine for performance metrics. FMOS 9.3 includes Victoria Metrics, configured to store metrics and make them available to you, and optionally FireMon Support. There are a number of different ways metrics are exposed, which can be configured using the FMOS System Configuration Variables and Control Panel:

• Metrics are always stored locally on the FMOS machine itself. FMOS exports the metrics (in Victoria Metrics' native binary format) and includes them in diagnostic packages.
• Optionally, FMOS can be configured to send metrics directly to FireMon Support.
• You can configure FMOS to expose the Victoria Metrics HTTP API to the network, allowing you to issue queries using a metrics monitoring service such as Grafana. Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Scrape Targets

Victoria Metrics (vmagent) can be configured to pull metrics from (scrape) any HTTP resource that exposes data in the Prometheus exposition format.

FMOS configures collectd, using the Write Prometheus plug-in to expose its metrics in this way. Collectd listens on TCP port 9103 (the same port as the Prometheus collectd exporter) and exposes metrics at http://localhost:9103/metrics. This port is never exposed to the network, regardless of the Allow Outside Access setting.

Stand-Alone Mode

By default, every FMOS machine collects its own metrics locally. Victoria Metrics is not accessible over the network.

The Victoria Metrics query API can be made available over the network, allowing you to retrieve metrics directly from FMOS. See Direct Querying below for additional information on this feature.
FMOS diagnostic packages contain all of the time series data in the Victoria Metrics native binary format.

Streaming Mode

In addition to storing metrics locally, FMOS can be configured to send metrics data directly to FireMon Support over the Internet. If the Send Performance Metrics variable is set to true, FMOS will configure the Victoria Metrics agent, vmagent, to scrape metrics and send data to both the local Victoria Metrics server and to FireMon's Victoria Metrics in the cloud.

The same functionality available in stand-alone mode (i.e. diagnostic package exports, and optional network access) is available in streaming mode as well.

In addition to, or instead of, streaming metrics data to FireMon Support, FMOS can be configured to stream Victoria Metrics data to any arbitrary destination. The Send performance metrics to these locations variable can be set to an array of destination HTTP or HTTPS URLs outside of your network. Each entry in this array should point to a Prometheus "remote write" endpoint, such as https://prometheus.mycompany.com/api/v1/write.

Direct Querying

When the Allow outside access to Victoria Metrics variable is set to true, the Victoria Metrics HTTP API will be available on TCP port 8428. This allows you to query the time series data using the Victoria Metrics query language, MetricsQL. MetricsQL is based on and compatible with the Prometheus query language, PromQL.

Although Victoria Metrics supports using TLS to secure the HTTP API, FMOS does not enable this option. In streaming mode, vmagent uses the HTTP API to send the metrics it has scraped to Victoria Metrics. Allowing you to enable TLS could present certificate validation problems that may prevent the vmagent from connecting. By avoiding HTTPS entirely, FMOS ensures that these issues are never present. If you require encryption for all communication with FMOS machines, then you should not enable the Allow Outside Access option. You should investigate using streaming mode to send metrics to your own Prometheus or Victoria Metrics instances.

Ports Used

Configuration Variables

These system configuration variables are set using the CLI of the FMOS server. However, the preferred method of setting is to use the FMOS Control Panel UI.

More Information

FMOS User's Guide: FMOS Performance Metrics
You can view online documentation for:

• Victoria Metrics: https://docs.victoriametrics.com/
• Prometheus: https://prometheus.io/docs/introduction/overview/
• Prometheus with Grafana: https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/
• Prometheus Exposition Format: https://prometheus.io/docs/instrumenting/exposition_formats/
• collectd: https://collectd.org/wiki/index.php/Plugin:Write_Prometheus