Skip to main content

Custom Metrics

Sending custom metrics from your code

Epsagon supports custom metrics sent by a Prometheus client. By default, your Prometheus is configured to write all the collected metrics to Epsagon.

In order to visualize your custom metrics in Epsagon, all you have to do is make sure those metrics will be collected by Prometheus.

Make sure to first configure your Prometheus to send metrics to Epsagon.

How to send metrics

StatsD metrics#

You can send your StatsD metrics to Prometheus using Prometheus statsd exporter. The exporter translates StatsD metrics to Prometheus metrics via configured mapping rules. The exporter also supports different formats of tagging. You can see below a quick start guide explaining how to install the exporter.

Don't have an existing StatsD environment? see "StatsD environment setup" section.

Prometheus StatsD exporter#

By default, the exporter listens on port 9125 - TCP and UDP.

See the exporter main repository.

Tip: we recommend installing the exporter using this helm chart.

  • If you aren't familiar with Helm then please refer to Helm's documentation to get started.
  1. Add the helm repo as follows:
helm repo add hahow
  1. Install the exporter helm chart:
helm install [RELEASE_NAME] hahow/prometheus-statsd-exporter
  1. Configure Prometheus to scrap metrics from the exporter. For example:
- job_name: 'prometheus-statsd-exporter'    static_configs:      - targets: ['[RELEASE_NAME]-prometheus-statsd-exporter:9102']        honor_labels: true    tls_config:      insecure_skip_verify: true-

For more information about Prometheus scrapping configuration click here.

  • By default, metrics are not deleted from the exporter and exposed to Prometheus forever. In order to change that, make sure to configure the global TTL for metrics expiration to a value bigger than 0 (the default value) at the exporter.

StatsD environment setup#

If you don't have StatsD environment, then you will have to:

  • use a StatsD client within your application to write metrics.
  • Setup a StatsD daemon (optional).

After your environment is setup successfully, you can send your metrics into Prometheus using the Prometheus statsd exporter.

StatsD client#

A StatsD client is used to create & send your custom metrics. The client can either send your metrics directly to a Prometheus StatsD exporter or to a StatsD daemon.

Click here for a list of StatsD client in multiple programming language.

StatsD & Metric labels:

The pure spec of StatsD does NOT support labels. In case you would like to label your StatsD metrics (its not a must), you can do that by formatting the metric name as follows:

<metric_name>#<tag key>=<numeric value>,<tag key>=<numeric value>

There are other StatsD clients that allow you to pass metric labels as a different parameter ("tags") in the relevant API calls - and avoid metric name formatting.

Example (python)#

Let's see an example how to send metrics.

  1. First, we will install a python statsd client.
pip install statsd
  1. Instantiate a new client, and send metrics to your target (Prometheus exporter/StatsD daemon):
import statsdstatsd_client = statsd.StatsClient('<StatsD target>', 8125)statsd_client.incr('myMetricName#firstLabelKey=1,secondLabelKey=22', <some numeric value>)

StatsD daemon (Optional)#

It is recommended (but not mandatory) to use a StatsD daemon in order to aggregate the given StatsD metrics and control the metrics sending rate limit. The daemon allows configuring multiple backend targets to send the metrics to (like Prometheus StatsD exporter, Graphite, and more).

Native Prometheus Instrumentation#

You can send your custom metrics to Prometheus by using a native Prometheus instrumentation client and by either:

  • Prometheus Pushgateway (metrics scrapped by Prometheus)
  • Prometheus scraps the metrics directly from the client

Note that you will also need to take care of manually deleting the metrics from Prometheus Pushgateway on a regular basis as Prometheus Pushgateway keeps exposing the metrics to Prometheus unless explicitly deleted by the client/HTTP request. Therefore, its highly recommend that you opt to use StatsD metrics instead.

  • Use the Prometheus client to create & send your custom metrics.

  • Setup Prometheus Pushgateway and configure the Prometheus client to push metrics to it. You can easily install it while installing Prometheus by setting pushgateway as enabled in the helm chart values. For more information, look here (a link to the section of Prometheus installation).

    Alternatively, use the Prometheus client HTTP server as your push agent (see the [Prometheus clients']( docs).
  • Configure your Prometheus to scrap metrics from your push agent - Prometheus Pushgateway or Prometheus client HTTP server.