You can configure publishers in the publishers configuration section under the top level metrics node or under server.features.observe.observers.metrics. If you do not set up publishers explicitly, Helidon uses an inferred Prometheus publisher for backward compatibility. See this later section for details.

Publishers in Helidon’s Micrometer-based metrics implementation use Micrometer MeterRegistry implementations. For each enabled publisher, Helidon adds the corresponding meter registry to Micrometer’s global registry. This has these important effects:

• Meters which Helidon or your code registers using the Helidon metrics API are registered in all active Micrometer meter registries.

• Each Helidon meter registered has an implementation in every active Micrometer meter registry.

• When Helidon or your code updates a Helidon meter, Micrometer applies the change to every corresponding meter from each active meter registry.

As a result, configuring more than one active meter registry can affect performance.

Configuring an OTLP Publisher

The configuration directly mirrors the Micrometer OtlpMeterRegistry settings so you can control all behavior which Micrometer exposes for the meter registry.

The following example sets up an OTLP publisher to transmit metrics data every 30 seconds.

metrics:
  publishers:         
    otlp:  
      interval: PT30S
      url: 'http://somehost.com:4318/v1/metrics'

• Introduces the configured publishers.
• Configures an OTLP publisher to transmit every 30 seconds to the given endpoint.

Configuring a Prometheus Publisher

Understanding the Inferred Prometheus Publisher

metrics:
  publishers:
    prometheus:
    otlp:
      interval: PT20S

You can write other publishers by following these steps:

1. Choose one of the Micrometer MeterRegistry implementations for the type of publishing you want to support. (for example DatadogMeterRegistry)
2. Create a config blueprint which exposes the meter registry’s settable properties from DatadogConfig.
3. Write a DatadogPublisher class which implements Helidon’s MetricsPublisher for Datadog.
4. Write a DatadogPublisherProvider class which implements Helidon’s MetricsPublisherProvider for your publisher.
5. Advertise your provider so Java service loading can find it, creating a META-INF/services/io.helidon.metrics.spi.PublisherProvider file listing your implementation class.

Look at Helidon’s OTLP publisher blueprint and the related types as an example.

Refer to your publisher in configuration using the config key you set up in the publisher provider.

metrics:
  publishers:
    micrometer-datadog:
      interval: PT15S

When you add the helidon-webserver-observe-metrics dependency to your project, and if you explicitly set up a Prometheus publisher or use the default one, Helidon provides a built-in REST endpoint /observe/metrics which responds with a report of the registered meters and their values.

Clients can request a particular output format from the endpoint.

Clients can also limit the report by specifying the scope as a query parameter in the request URL:

• /observe/metrics?scope=base

• /observe/metrics?scope=vendor

• /observe/metrics?scope=application

Further, clients can narrow down to a specific metric name by adding the name as another query parameter, such as /observe/metrics?scope=application&name=myCount.

curl -s -H 'Accept: text/plain' -X GET http://localhost:8080/observe/metrics

# HELP classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
# TYPE classloader_loadedClasses_count gauge
classloader_loadedClasses_count{scope="base",} 5297.0

See the summary of the OpenMetrics and Prometheus Format for more information.

curl -s -H 'Accept: application/json' -X GET http://localhost:8080/observe/metrics

{
   "base" : {
      "memory.maxHeap" : 3817865216,
      "memory.committedHeap" : 335544320
    }
}

In addition to your application meters, the reports contain other meters of interest such as system and VM information.

The OpenMetrics format and the Prometheus exposition format are very similar in most important respects but are not identical. This brief summary treats them as the same.

The OpenMetrics/Prometheus format represents each meter using three lines of output as summarized in the following table.

The OpenMetrics/Prometheus output converts meter IDs in these ways:

• Names in camel case are converted to "snake case" and dots are converted to underscores.

• Names include any units specified for the meter.

• For percentiles, the ID includes a tag identifying which percentile the line of output describes.

As the earlier example output showed, for a meter with multiple values, such as a timer or a distribution summary, (with, among others, max, mean, and count), the OpenMetrics/Prometheus output reports a "metric family" which includes a separate family member meter for each of the multiple values. The name for each member in the family is derived from the registered name for the meter plus a suffix indicating which one of the meter’s multiple values the line refers to.

The following table summarizes the naming for each meter type.

¹ The OpenMetrics/Prometheus output format reports a timer as a summary with units of seconds.

Unlike OpenMetrics/Prometheus output, which combines the data and the metadata in a single response, you use an HTTP GET request to retrieve metrics JSON data and an OPTIONS request to retrieve metadata in JSON format.

Helidon groups meters in the same scope together in JSON output as shown in the following example.

{
  "application": {  
    "getTimer": {
      "type": "timer",
      "unit": "seconds",
      "description": "Timer for getting the default greeting"
    }
  },
  "vendor": {       
    "requests.count": {
      "type": "counter",
      "description": "Each request (regardless of HTTP method) will increase this counter"
    }
  },
  "base": {         
    "cpu.systemLoadAverage": {
      "type": "gauge",
      "description": "Displays the system load average for the last minute."
    },
    "classloader.loadedClasses.count": {
      "type": "gauge",
      "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
    }
  }
}

• Note the application, vendor, and base sections.

If an HTTP request selects by scope, the output omits the extra level of structure that identifies the scope as shown in the following example.

{
  "cpu.systemLoadAverage": {
    "type": "gauge",
     "description": "Displays the system load average for the last minute."
  },
  "classloader.loadedClasses.count": {
    "type": "gauge",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
  }
}

Understanding the JSON Metrics Data Format

"requests.count": 5

"getTimer": {
  "count": 3,
  "max": 0.0030455,
  "mean": 0.0011060836666666666,
  "elapsedTime": 0.003318251,
  "p0.5": 0.000151552,
  "p0.75": 0.003141632,
  "p0.95": 0.003141632,
  "p0.98": 0.003141632,
  "p0.99": 0.003141632,
  "p0.999": 0.003141632
}

Understanding the JSON Metrics Metadata Format

"requests.count": {
  "type": "counter",
  "description": "Each request (regardless of HTTP method) will increase this counter"
    }

"getTimer": {
  "type": "timer",
  "unit": "seconds",
  "description": "Timer for getting the default greeting"
}

Controlling JSON Timer Output

metrics:
  timers:
    json-units-default: units 

Your application can assign the meter builder’s Meter.Builder baseUnit setting for any meter your application creates. In particular, the Timer.Builder baseUnit method allows code to assign a baseUnit for a timer, passing a Java TimeUnit value. The timer builder also has the String variant of the baseUnit method and enforces that the value corresponds (case-insensitively) to one of the TimeUnit enum values.

Note that, regardless of the baseUnit setting for a Timer, by convention and specification Prometheus output expresses time values in seconds.

By default, the same is true of Helidon’s JSON format: timer values are displayed in seconds regardless of any timer’s baseUnit setting. You can override this as described in the Controlling Timer Output section, in which case the JSON output for each timer reflects its baseUnit setting.

You can choose which endpoints to include in Helidon’s automatic measurements using the auto-http-metrics config section.

The paths section contains zero or more entries, each entry having the following settings:

Helidon decides whether to measure incoming requests as follows:

• If you omit the auto-http-metrics configuration, Helidon measures all endpoints.

• If you specify the auto-http-metrics configuration, by default Helidon does not measure built-in endpoints such as metrics, health, and openapi. You can add items under auto-http-metrics.paths to control more exactly which endpoints to measure.

• If you include the paths section, Helidon checks a request against the path entries in order. A given request matches an entry if its path matches the path pattern and its HTTP method is in the methods list. If there is no methods list for an entry, all HTTP methods match the entry.

• If a request matches an entry, the entry’s enabled setting determines if the request should be measured.

• If a request matches multiple entries, the first match wins.

• If a request matches no entry, it is measured.

The auto-http-metrics.sockets setting controls which sockets are included in the measurements; if not set, Helidon measures requests on all sockets.

server:
  features:
    observe:
      observers:
        metrics:
          auto-http-metrics:
            paths:
              - path: "/greet"              
                methods: ["GET","HEAD"]
              - path: "/greet/{name}"       
                enabled: false
            sockets: ["@default","private"] 

• Measure /greet for only GET and HEAD requests.
• Do not measure the personalized greeting requests.
• Measure only endpoints on the default socket and the socket named private. Endpoints on other sockets (such as if you had an admin socket) are not measured.

The AutoHttpMetricsConfig documentation describes the configuration more fully.

server:
  features:
    observe:
      observers:
        metrics:
          enabled: false

Helidon does not update metrics, and the /observe/metrics endpoints respond with 404.

Enabling Virtual Threads Meters

metrics:
  virtual-threads:
    enabled: true

Controlling Measurements of Pinned Virtual Threads

metrics:
  virtual-threads:
    pinned:
      threshold: PT0.100S

Any time you include the Helidon metrics module in your application, Helidon tracks a basic performance indicator meter: a Counter of all requests received (requests.count)

Helidon SE also includes additional, extended KPI meters which are disabled by default:

• current number of requests in-flight - a Gauge (requests.inFlight) of requests currently being processed

• long-running requests - a Counter (requests.longRunning) measuring the total number of requests which take at least a given amount of time to complete; configurable, defaults to 10000 milliseconds (10 seconds)

• load - a Counter (requests.load) measuring the number of requests worked on (as opposed to received)

• deferred - a Gauge (requests.deferred) measuring delayed request processing (work on a request was delayed after Helidon received the request)

You can enable and control these meters using configuration:

server:
  features:
    observe:
      observers:
        metrics:
          key-performance-indicators:
            extended: true
            long-running:
              threshold-ms: 2000

Your application code uses the Prometheus API to manage metrics. To expose those metrics to clients via a REST endpoint, your code uses the PrometheusSupport interface which Helidon provides.

Your code creates a PrometheusSupport object either using a static factory method (shown in the following example) or by using its Builder.

routing
        .addFeature(PrometheusSupport.create())
        .register("/myapp", new MyService());

This example uses the default Prometheus CollectorRegistry. By default, the PrometheusSupport and exposes its REST endpoint at the path /metrics. Use the builder obtained by PrometheusSupport.builder() to configure a different CollectorRegistry or a different path.