Project

General

Profile

Actions
History

Wiki »

Graphing osmo-* KPI with grafana

Ever wanted to render nice grafana graphs / dashboards from the various statistics/counters/KPIs exposed by the various osmocom programs? This is possible via the statsd exporter which is part of the counter / rate_counter infrastructure provided by libosmocore.

This wiki page aspires to provide an introduction on how to collect those statistics to make them available to grafana.

As usual in the Unix/Linux/GNU/FOSS world, there are many possible ways how to achieve this. We are focusing on some examples here. Usually you would only deploy one of the described strategies

osmocom stats reporter -> statsd_exporter -> prometheus

This is a simple configuration which looks like this:

So in this configuration we have
  • various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the statsd_exporter
  • statsd_exporter listening to a local HTTP port waiting for connections from prometheus
  • prometheus connecting via HTTP to statsd_exporter to obtain the statistics in the prometheus-inherent pull semantics
  • prometheus storing the data
  • prometheus listening to a local port, waiting for connections from grafana
  • grafana offering a HTTP web interface, using prometheus as a data source, issuing PromQL queries to prometheus

None of the elements shown in the above graph must run on the same physical or virtual machine. They all interact via IP based protocol.

The general recommendation would be to keep the statsd_exporter close to the data-generating osmo-* programs, as the protocol between is UDP based (can loose packets) and not authenticated or encrypted. So you certainly want to keep that rather local/private. The TCP based protocols between statsd_exporter and prometheus and also between prometheus and grafana can be operated via TLS/HTTPS and hence over public networks, if so desired.

configuration of osmo-*

Please see the stats reporter chapter in the respective osmo-* user manual for details.

In general you would add something like the below snippet to your respective osmo-* program:

stats interval 10
stats reporter statsd
  remote-ip 127.0.0.1
  remote-port 8125
  level global
  no prefix
  enable

This instructs the osmo-* program to report all of its statistics every 10s via UDP to 127.0.0.1:9125

installation / configuration of statsd_exporter

  • obtain the latest release of statsd_exporter from https://github.com/prometheus/statsd_exporter/releases and install it to /opt/statsd_exporter
  • create a statsd_exporter user with useradd -r statsd_exporter
  • deploy a attachment:statsd_exporter.service systemd unit file to /etc/systemd/system/statsd_exporter.service
    • adjust the --web.listen-address and --statsd.listen-udp accordng to your requirements
  • enable + start it via systemctl enable statsd_exporter; systemctl start statsd_exporter
  • make sure by network separation and/or packet filter rule sets that only the IP addresses of your osmo-* processes can access the statsd.isten-udp port, and only the IP address running your prometheus can create inbound TCP connections to the web.listen_address

installation / configuration of prometheus

  • install prometheus (in our example, we just used the debian 11 package via apt install prometheus)
  • configure the retention period in /etc/default/prometheus, e..g. via ARGS="--storage.tsdb.retention.time=1y" for 1 year
  • configure it to scrape statsd_exporter via a snippet like below in /etc/prometheus/prometheus.yml (assuing your statsd_exporter is listening on localhost:9102 for HTTP requests, as per web.listen-address configured above)
      - job_name: 'statsd_exporter'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9102']
  • enable + start it via systemctl enable prometheus; systemctl start prometheus
  • make sure by network separation and/or packet filter rule sets that only the IP addresses hosting your grafana processes can create inbound connections the prometheus port (default: 9090)

installation / configuration of grafana

osmocom stats reporter -> collectd -> prometheus

This is a slightly more complex setup in which collectd is used. This doesn't really provide any known benefits to the setup with statsd_exporter described above, but is useful if you already have existing collectd-based systems running.

So in this configuration we have
  • various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the statsd_exporter
  • collectd listening to a local HTTP port waiting for connections from prometheus
  • prometheus connecting via HTTP to collectd to obtain the statistics in the prometheus-inherent pull semantics
  • prometheus storing the data
  • prometheus listening to a local port, waiting for connections from grafana
  • grafana offering a HTTP web interface, using prometheus as a data source, issuing PromQL queries to prometheus

configuration of osmo-*

see above; identical to the previous case using statsd_exporter.

installation / configuration of collectd

We assume you already have a collectd installation, or simply installed it from your distribution package manager, such as apt install collectd on Debian 11.

  • make sure to enable the Write_Prometheus collectd plugin. This will expose the statistic on a prometheus-compatible HTTP port. Example config:
    LoadPlugin write_prometheus

<Plugin "write_prometheus">
  Port "9103" 
</Plugin>

installation / configuration of prometheus

  • install prometheus (in our example, we just used the debian 11 package via apt install prometheus)
  • configure the retention period in /etc/default/prometheus, e..g. via ARGS="--storage.tsdb.retention.time=1y" for 1 year
  • configure it to scrape collectd via a snippet like below in /etc/prometheus/prometheus.yml (assuing your collectd is listening on localhost:9103 for HTTP requests, as configured above)
      - job_name: 'collectd'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9103']
  • enable + start it via systemctl enable prometheus; systemctl start prometheus
  • make sure by network separation and/or packet filter rule sets that only the IP addresses hosting your grafana processes can create inbound connections the prometheus port (default: 9090)

installation / configuration of grafana

see above; identical to the previous case using statsd_exporter.

Updated by laforge 12 months ago · 5 revisions

Add picture from clipboard (Maximum size: 48.8 MB)