Metrics

Overview

Helidon MP metrics implements the MicroProfile Metrics specification, providing:

• a unified way for MicroProfile servers to export monitoring data telemetry to management agents, and
• a unified Java API which all application programmers can use to register and update metrics to expose telemetry data from their services.
• support for metrics-related annotations.

Learn more about the MicroProfile Metrics specification.

Metrics is one of the Helidon observability features.

metrics.gc-time-type = gauge

Maven Coordinates

To enable metrics, add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
  <groupId>io.helidon.microprofile.metrics</groupId>
  <artifactId>helidon-microprofile-metrics</artifactId>
</dependency>

Adding this dependency packages the full-featured metrics implementation with your service.

Instrumenting Your Service

You add metrics to your service in these ways:

• Annotate bean methods — typically your REST resource endpoint methods (the Java code that receives incoming REST requests); Helidon automatically registers these metrics and updates them when the annotated methods are invoked via CDI.
• Write code which explicitly invokes the metrics API to register metrics, retrieve previously-registered metrics, and update metric values.
• Configure some simple REST.request metrics which Helidon automatically registers and updates for all REST resource endpoints.

Later sections of this document describe how to do each of these.

Metric Types

Helidon supports meters described by the MicroProfile Metrics spec and summarized in the following table:

Types of Metrics

Metric Categories

Helidon distinguishes among scopes, or categories, of metrics as described in the MP metrics specification.

Helidon includes metrics in the built-in scopes described below. Applications often register their own metrics in the application scope but can create their own scopes and register metrics within them.

Built-in metric scopes:

When you add metrics annotations to your service code, Helidon registers the resulting metrics in the application scope.

Metric Registries

A metric registry collects registered metrics of a given scope. Helidon supports one metrics registry for each scope.

When you add code to your service to create a metric programmatically, the code first locates the appropriate registry and then registers the metric with that registry.

Publishing Metrics

Helidon’s Micrometer-based metrics implementation includes these ways of publishing metrics data to external systems:

• Prometheus/OpenMetrics
• OTLP (OpenTelemetry Protocol)

You can configure publishers in the publishers configuration section under the top level metrics node or under server.features.observe.observers.metrics. If you do not set up publishers explicitly, Helidon uses an inferred Prometheus publisher for backward compatibility. See this later section for details.

Publishers in Helidon’s Micrometer-based metrics implementation use Micrometer MeterRegistry implementations. For each enabled publisher, Helidon adds the corresponding meter registry to Micrometer’s global registry. This has these important effects:

• Metrics which Helidon or your code registers using the Helidon metrics API are registered in all active Micrometer meter registries.
• Each Helidon meter registered has an implementation in every active Micrometer meter registry.
• When Helidon or your code updates a Helidon meter, Micrometer applies the change to every corresponding meter from each active meter registry.

As a result, configuring more than one active meter registry can affect performance.

OpenTelemetry Protocol

If you configure an OTLP publisher, Helidon exports metrics data periodically to a backend system you configure.

Configuration options

The configuration directly mirrors the Micrometer OtlpMeterRegistry settings so you can control all behavior which Micrometer exposes for the meter registry.

The following example sets up an OTLP publisher to transmit metrics data every 30 seconds.

metrics:
  publishers:1
    otlp:2
      interval: PT30S
      url: 'http://somehost.com:4318/v1/metrics'

Prometheus Publisher

If you configure a Prometheus publisher or rely on the inferred one, Helidon can make the metrics data available in the Prometheus/OpenMetrics format. (To serve the data at the metrics endpoint in your service, your project must also depend on the Helidon metrics observer component.)

Configuration options

Inferred Publisher

As described earlier, Helidon prepares an inferred Prometheus publisher if you do not set up any publishers.

Note that Helidon uses the inferred publisher only if you add no publishers explicitly, either in the configuration or programmatically. If you specify any publishers explicitly, Helidon uses only the ones you set up.

In particular, Helidon does not use the inferred Prometheus publisher if you create a metrics.publishers section containing only an OTLP publisher.

You can configure other publishers and still have Helidon use the default one by simply adding the prometheus publisher entry. You do not need to specify further settings for it.

Using an OLTP publisher and the default Prometheus publisher:

metrics:
  publishers:
    prometheus:
    otlp:
      interval: PT20S

Additional Publishers

You can write other publishers by following these steps:

1. Choose one of the Micrometer MeterRegistry implementations for the type of publishing you want to support. (for example DatadogMeterRegistry)
2. Create a config blueprint which exposes the meter registry’s settable properties from DatadogConfig.
3. Write a DatadogPublisher class which implements Helidon’s MetricsPublisher for Datadog.
4. Write a DatadogPublisherProvider class which implements Helidon’s MetricsPublisherProvider for your publisher.
5. Advertise your provider so Java service loading can find it, creating a META-INF/services/io.helidon.metrics.spi.PublisherProvider file listing your implementation class.

Look at Helidon’s OTLP publisher blueprint and the related types as an example.

Refer to your publisher in configuration using the config key you set up in the publisher provider.

Example config using a hypothetical Datadog publisher:

metrics:
  publishers:
    micrometer-datadog:
      interval: PT15S

Metrics Endpoint

When you add the metrics dependency to your project, and if you explicitly set up a Prometheus publisher or use the default one, Helidon provides a built-in REST endpoint /metrics which responds with a report of the registered metrics and their values.

Clients can request a particular output format from the endpoint.

Formats for /metrics output

Clients can also limit the report by specifying the scope as a query parameter in the request URL:

• /metrics?scope=base
• /metrics?scope=vendor
• /metrics?scope=application

Further, clients can narrow down to a specific metric name by adding the name as another query parameter, such as /metrics?scope=application&name=myCount.

Example Reporting: Prometheus format:

curl -s -H 'Accept: text/plain' -X GET http://localhost:8080/metrics

# HELP classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
# TYPE classloader_loadedClasses_count gauge
classloader_loadedClasses_count{mp_scope="base",} 5297.0

See the summary of the OpenMetrics and Prometheus Format for more information.

Example Reporting: JSON format:

curl -s -H 'Accept: application/json' -X GET http://localhost:8080/metrics

{
   "base" : {
      "memory.maxHeap" : 3817865216,
      "memory.committedHeap" : 335544320
    }
}

In addition to your application metrics, the reports contain other metrics of interest such as system and VM information.

OpenMetrics and Prometheus Format

The OpenMetrics format and the Prometheus exposition format are very similar in most important respects but are not identical. This brief summary treats them as the same.

The OpenMetrics/Prometheus format represents each metric using three lines of output as summarized in the following table.

The OpenMetrics/Prometheus output converts metric IDs in these ways:

• Names in camel case are converted to "snake case" and dots are converted to underscores.
• Names include any units specified for the metric.
• For percentiles, the ID includes a tag identifying which percentile the line of output describes.

As the earlier example output showed, for a metric with multiple values, such as a timer or a histogram, (with, among others, max, mean, and count), the OpenMetrics/Prometheus output reports a "metric family" which includes a separate family member metric for each of the multiple values. The name for each member in the family is derived from the registered name for the metric plus a suffix indicating which one of the metric’s multiple values the line refers to.

The following table summarizes the naming for each metric type.

¹ The OpenMetrics/Prometheus output format reports a timer as a summary with units of seconds.

JSON Format

Unlike OpenMetrics/Prometheus output, which combines the data and the metadata in a single response, you use an HTTP GET request to retrieve metrics JSON data and an OPTIONS request to retrieve metadata in JSON format.

Helidon groups metrics in the same scope together in JSON output as shown in the following example.

JSON metrics output structured by scope (partial):

{
  "application": {1
    "getTimer": {
      "type": "timer",
      "unit": "seconds",
      "description": "Timer for getting the default greeting"
    }
  },
  "vendor": {1
    "requests.count": {
      "type": "counter",
      "description": "Each request (regardless of HTTP method) will increase this counter"
    }
  },
  "base": {1
    "cpu.systemLoadAverage": {
      "type": "gauge",
      "description": "Displays the system load average for the last minute."
    },
    "classloader.loadedClasses.count": {
      "type": "gauge",
      "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
    }
  }
}

If an HTTP request selects by scope, the output omits the extra level of structure that identifies the scope as shown in the following example.

JSON metrics output for the base scope (partial):

{
  "cpu.systemLoadAverage": {
    "type": "gauge",
     "description": "Displays the system load average for the last minute."
  },
  "classloader.loadedClasses.count": {
    "type": "gauge",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
  }
}

Understanding the JSON Metrics Data Format

The Helidon JSON format expresses each metric as either a single value (for example, a counter) or a structure with multiple values (for example, a timer).

JSON output for a single-valued metric (for example, Counter):

"requests.count": 5

JSON output for a multivalued metric (for example, Timer):

"getTimer": {
  "count": 3,
  "max": 0.0030455,
  "mean": 0.0011060836666666666,
  "elapsedTime": 0.003318251,
  "p0.5": 0.000151552,
  "p0.75": 0.003141632,
  "p0.95": 0.003141632,
  "p0.98": 0.003141632,
  "p0.99": 0.003141632,
  "p0.999": 0.003141632
}

By default, Helidon formats time values contained in JSON output as seconds. You can change this behavior as described below.

Understanding the JSON Metrics Metadata Format

Access the metrics endpoint with an HTTP OPTIONS request and the Accept: application/json header to retrieve metadata in JSON format.

Example Counter metadata:

"requests.count": {
  "type": "counter",
  "description": "Each request (regardless of HTTP method) will increase this counter"
    }

Example Timer metadata:

"getTimer": {
  "type": "timer",
  "unit": "seconds",
  "description": "Timer for getting the default greeting"
}

Generally, the output for a given metric reflects only the metadata that the application or Helidon code explicitly set on that metric.

One exception is that metadata for a timer always includes the unit field. By default, Helidon formats timer data in JSON output as seconds, regardless of any explicit baseUnit setting applied to the timers. But as described below you can change this behavior which can lead to different timers being formatted using different units. Checking the metadata is the only way to know for sure what units Helidon used to express a given timer, so Helidon always includes unit in timer metadata.

Controlling JSON Timer Output

By default, Helidon expresses timer data as seconds.

You can change this using configuration:

metrics.timers.json-units-default=units

• For units specify any valid name for a TimeUnit value (SECONDS, MILLISECONDS, etc.)

If you have configured json-units-default, Helidon formats each timer’s data as follows:

1. If code set baseUnit on the timer, Helidon uses those units for that timer.
2. Otherwise, Helidon uses the default units you configured.

To enable the JSON output behavior from Helidon 3, specify json-units-default as NANOSECONDS.

API

The MicroProfile Metrics API prescribes all the standard interfaces related to metrics. This section summarizes a few key points about using that API and explains some Helidon-specific interfaces.

Annotations

You can very easily instrument your service and refer to registered metrics by annotating methods to be measured and injecting metrics which your code needs to observe.

The MicroProfile Metrics specification describes several metric types you can create using annotations, summarized in the following table:

¹ Place annotations on constructors or methods to measure those specific executables. If you annotate the class instead, Helidon applies that annotation to all constructors and methods which the class declares.

Metric-referencing Annotations

To get a reference to a specific metric, use a metric-referencing annotation in any bean, including your REST resource classes.

You can @Inject a field of the correct type. Helidon uses the MicroProfile Metrics naming conventions to select which specific metric to inject. Use the @Metric annotation to control that selection.

You can also add @Metric on a constructor or method parameter to trigger injection there.

Helidon automatically looks up the metric referenced from any injection site and provides a reference to the metric. Your code then simply invokes methods on the injected metric.

MetricRegistry

To register or look up metrics programmatically, your service code uses the MetricRegistry instance for the scope of interest: base, vendor, application, or a custom scope.

Either of the following techniques gets a MetricRegistry reference. Remember that injection works only if the class is a bean so CDI can inject into it.

• @Inject MetricRegistry, optionally using @RegistryScope to indicate the registry scope.
  Injecting the default MetricRegistry (for the application scope):

  class Example {
  
    @Inject
    private MetricRegistry applicationRegistry;
  }
  

  
  Injecting a non-default MetricRegistry:

  class Example {
  
    @RegistryScope(scope = "myCustomScope")
    @Inject
    private MetricRegistry myCustomRegistry;
  }
  

• Get a Helidon RegistryFactory instance and invoke its getRegistry method.
  Obtain the RegistryFactory using either of the following techniques:
  • @Inject RegistryFactory.
    Getting the RegistryFactory using injection

class InjectExample {

    @Inject
    private RegistryFactory registryFactory;

    private MetricRegistry findRegistry(String scope) {
        return registryFactory.getRegistry(scope);
    }
}

• Invoke the static getInstance() method on the RegistryFactory class.
  Getting the RegistryFactory programmatically

class Example {

    private MetricRegistry findRegistry(String scope) {
        return RegistryFactory.getInstance().getRegistry(scope);
    }
}

Once it has a reference to a MetricRegistry your code can use the reference to register new metrics, look up previously-registered metrics, and remove metrics.

CDI Extensions

You can work with metrics inside your own CDI extensions, but be careful to do so at the correct point in the CDI lifecycle. Configuration can influence how the metrics system behaves, as the configuration section below explains. Your code should work with metrics only after the Helidon metrics system has initialized itself using configuration. One way to accomplish this is to deal with metrics in a method that observes the Helidon RuntimeStart CDI event, which the extension example below illustrates.

Configuration options

To control how the Helidon metrics subsystem behaves, add a metrics section to your META-INF/microprofile-config.properties file.

Deprecated Options

Default Values Specific to Helidon MP:

Metric Type for gc.time

To date Helidon 4 releases have implemented the system-provided metric gc.time as a counter. In fact, a gauge is more suitable for the approximate time the JVM has spent doing garbage collection, and beginning with MicroProfile Metrics 5.1 the TCK relies on gc.time being a gauge.

Helidon 4.4.0-SNAPSHOT continues to use a counter by default to preserve backward compatibility, but you can choose to use a gauge by setting the configuration property metrics.gc-time-type to gauge. You can also set the config property to counter which is the default.

Why should you care? In fact, this distinction might not make a difference for many users. But for others the differences between the programmatic APIs for Counter and Gauge would affect application code that works directly with the gc-time metric. Further, the difference in output particularly in the OpenMetrics/Prometheus format might affect their application or downstream monitoring tools.

The ability to choose the metric type for gc.time is deprecated and is planned for removal in a future major release of Helidon at which time Helidon will always use a gauge.

Metrics Observer

Helidon can make the registered metrics and their current values available externally at an endpoint (/metrics by default). You can control aspects of how Helidon furnishes this information under the server.features.observe.observers.metrics configuration section.

Optional configuration options

Selecting REST Endpoints for Automatic Measurement

You can choose which endpoints to include in Helidon’s automatic measurements using the auto-http-metrics config section.

Configuration options

The paths section contains zero or more entries, each entry having the following settings:

Helidon decides whether to measure incoming requests as follows:

• If you omit the auto-http-metrics configuration, Helidon measures all endpoints.
• If you specify the auto-http-metrics configuration, by default Helidon does not measure built-in endpoints such as metrics, health, and openapi. You can add items under auto-http-metrics.paths to control more exactly which endpoints to measure.
• If you include the paths section, Helidon checks a request against the path entries in order. A given request matches an entry if its path matches the path pattern and its HTTP method is in the methods list. If there is no methods list for an entry, all HTTP methods match the entry.
• If a request matches an entry, the entry’s enabled setting determines if the request should be measured.
• If a request matches multiple entries, the first match wins.
• If a request matches no entry, it is measured.

The auto-http-metrics.sockets setting controls which sockets are included in the measurements; if not set, Helidon measures requests on all sockets.

Including and excluding endpoints from automatic measurement:

server.features.observe.observers.metrics.auto-http-metrics.paths.0.path=/greet1
server.features.observe.observers.metrics.auto-http-metrics.paths.0.methods=GET,HEAD

server.features.observe.observers.metrics.auto-http-metrics.paths.1.path=/greet/{name}2
server.features.observe.observers.metrics.auto-http-metrics.paths.1.enabled=false

server.features.observe.observers.metrics.auto-http-metrics.sockets=@default,private3

The AutoHttpMetricsConfig documentation describes the configuration more fully.

Examples

Helidon MP includes a pre-written example application illustrating enabling/disabling metrics using configuration.

Method-level Annotations

The following example adds a new resource class, GreetingCards, to the Helidon MP QuickStart example. It shows how to use the @Counted annotation to track the number of times the /cards endpoint is called.

Create a new class GreetingCards with the following code:

@Path("/cards")1
@RequestScoped2
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "any-card")3
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Build and run the application:

mvn package
java -jar target/helidon-quickstart-mp.jar

Access the application endpoints:

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "io.helidon.examples.quickstart.mp.GreetingCards.any-card": 2,1
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  }
}

Method-level Metrics

You can also use the @Timed annotation with a method. For the following example. you can just annotate the same method with @Timed. Timers significant information about the measured methods, but at a cost of some overhead and more complicated output.

Note that when using multiple annotations on a method, you must give the metrics different names as shown below, although they do not have to be absolute.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true)1
    @Timed(name = "cardTimer", absolute = true, unit = MetricUnits.MILLISECONDS)2
    public JsonObject anyCard() {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Build and run the application:

mvn package
java -jar target/helidon-quickstart-mp.jar

Access the application endpoints:

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "cardTimer": {
    "count": 2,
    "elapsedTime": 0.002941925,
    "max": 0.002919973,
    "mean": 0.0014709625
  },
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "cardCount": 2
}

Class-level Metrics

You can collect metrics at the class level to aggregate data from all methods in that class using the same metric. The following example introduces a metric to count all card queries. In the following example, the method-level metrics are not needed to aggregate the counts, but they are left in the example to demonstrate the combined output of all three metrics.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards")1
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)2
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)3
    public JsonObject birthdayCard() throws InterruptedException {
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Build and run the application:

mvn package
java -jar target/helidon-quickstart-mp.jar

Access the application endpoints:

curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "birthdayCard": 1,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 1,
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 21
}

Field Level Metrics

Field level metrics can be injected into managed objects, but they need to be updated by the application code. This annotation can be used on fields of type Timer, Counter, and Histogram.

The following example shows how to use a field-level Counter metric to track cache hits.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards")
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @Inject
    @Metric(name = "cacheHits", absolute = true)1
    private Counter cacheHits;

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        updateStats();2
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject birthdayCard() throws InterruptedException {
        updateStats();3
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }

    private void updateStats() {
        if (new Random().nextInt(3) == 1) {
            cacheHits.inc();4
        }
    }
}

Build and run the application, then invoke the following endpoints:

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "birthdayCard": 3,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 2,
  "cacheHits": 2,1
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 5
}

Gauge Metric

The metrics you have tested so far are updated in response to an application REST request, i.e. GET /cards. These metrics can be declared in a request scoped class and Helidon will store the metric in the MetricRegistry, so the value persists across requests. When GET /metrics?scope=application is invoked, Helidon will return the current value of the metric stored in the MetricRegistry.

The Gauge annotation is different from the other metric annotations. The application must provide a method to return the gauge value in an application-scoped class. When GET /metrics?scope=application is invoked, Helidon will call the Gauge method, using the returned value as the value of the gauge as part of the metrics response.

The following example demonstrates how to use a Gauge to track application up-time.

Create a new GreetingCardsAppMetrics class with the following code:

@ApplicationScoped1
public class GreetingCardsAppMetrics {

    private AtomicLong startTime = new AtomicLong(0);2

    public void onStartUp(@Observes @Initialized(ApplicationScoped.class) Object init) {
        startTime = new AtomicLong(System.currentTimeMillis());3
    }

    @Gauge(unit = "TimeSeconds")
    public long appUpTimeSeconds() {
        return Duration.ofMillis(System.currentTimeMillis() - startTime.get()).getSeconds();4
    }
}

Update the GreetingCards class with the following code to simplify the metrics output:

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Build and run the application, then invoke the application metrics endpoint:

curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "io.helidon.examples.quickstart.mp.GreetingCardsAppMetrics.appUpTimeSeconds": 23,1
  "cardCount": 0
}

Working with Metrics in CDI Extensions

You can work with metrics from your own CDI extension by observing the RuntimeStart event.

CDI Extension that works correctly with metrics:

public class MyExtension implements Extension {
    void startup(@Observes @RuntimeStart Object event,1
                 MetricRegistry metricRegistry) {2
        metricRegistry.counter("myCounter");3
    }
}

Helidon does not prevent you from working with metrics earlier than the RuntimeStart event, but, if you do so, then Helidon might ignore certain configuration settings that would otherwise control how metrics behaves.

Instead, consider writing your extension to use earlier lifecycle events (such as ProcessAnnotatedType) to gather and store information about metrics that you want to register. Then your extension’s RuntimeStart observer method would use that stored information to register the metrics you need.

Configuration Example

Metrics configuration is quite extensive and powerful and, therefore, a bit complicated. The rest of this section illustrates some of the most common scenarios:

• Disable metrics entirely.
• Choose whether to report virtual threads metrics.
• Choose whether to collect extended key performance indicator metrics.
• Control REST.request metrics collection.

Disable Metrics Subsystem

Disabling metrics entirely:

metrics.enabled=false

Helidon does not update metrics, and the /metrics endpoints respond with 404.

Configuring Virtual Threads Metrics

Enabling Virtual Threads Metrics

Gathering data to compute the metrics for virtual threads is designed to be as efficient as possible, but doing so still imposes a load on the server and by default Helidon does not report metrics related to virtual threads.

To enable the metrics describing virtual threads include a config setting as shown in the following example.

Enabling virtual thread metrics:

metrics.virtual-threads.enabled = true

Controlling Measurements of Pinned Virtual Threads

Helidon measures pinned virtual threads only when the thread is pinned for a length of time at or above a threshold. Control the threshold as shown in the example below.

Setting virtual thread pinning threshold to 100 ms:

metrics.virtual-threads.pinned.threshold=PT0.100S

The threshold value is a Duration string, such as PT0.100S for 100 milliseconds.

Collecting Basic and Extended Key Performance Indicator (KPI) Metrics

Any time you include the Helidon metrics module in your application, Helidon tracks a basic performance indicator metric: a Counter of all requests received (requests.count)

Helidon MP also includes additional, extended KPI metrics which are disabled by default:

• current number of requests in-flight - a Gauge (requests.inFlight) of requests currently being processed
• long-running requests - a Counter (requests.longRunning) measuring the total number of requests which take at least a given amount of time to complete; configurable, defaults to 10000 milliseconds (10 seconds)
• load - a Counter (requests.load) measuring the number of requests worked on (as opposed to received)
• deferred - a Gauge (requests.deferred) measuring delayed request processing (work on a request was delayed after Helidon received the request)

You can enable and control these metrics using configuration:

Controlling extended KPI metrics:

metrics.key-performance-indicators.extended = true
metrics.key-performance-indicators.long-running.threshold-ms = 2000

Enable REST.request Metrics

Controlling REST request metrics:

metrics.rest-request.enabled=true

Helidon automatically registers and updates Timer metrics for every REST endpoint in your service.

Additional Information

Kubernetes Integration

The following example shows how to integrate the Helidon MP application with Kubernetes.

Stop the application and build the docker image:

docker build -t helidon-metrics-mp .

Create the Kubernetes YAML specification, named metrics.yaml, with the following content:

kind: Service
apiVersion: v1
metadata:
  name: helidon-metrics1
  labels:
    app: helidon-metrics
  annotations:
    prometheus.io/scrape: "true"2
spec:
  type: NodePort
  selector:
    app: helidon-metrics
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-metrics
spec:
  replicas: 13
  selector:
    matchLabels:
      app: helidon-metrics
  template:
    metadata:
      labels:
        app: helidon-metrics
        version: v1
    spec:
      containers:
        - name: helidon-metrics
          image: helidon-metrics-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Create and deploy the application into Kubernetes:

kubectl apply -f ./metrics.yaml

Get the service information:

kubectl get service/helidon-metrics

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-metrics   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s1

Verify the metrics endpoint using port 30116, your port will likely be different:

curl http://localhost:31143/metrics

Prometheus Integration

The metrics service that you just deployed into Kubernetes is already annotated with prometheus.io/scrape:. This will allow Prometheus to discover the service and scrape the metrics. This example shows how to install Prometheus into Kubernetes, then verify that it discovered the Helidon metrics in your application.

Install Prometheus and wait until the pod is ready:

helm install stable/prometheus --name metrics
export POD_NAME=$(kubectl get pods \
  -l "app=prometheus,component=server" \
  -o jsonpath="{.items[0].metadata.name}")
kubectl get pod $POD_NAME

You will see output similar to the following. Repeat the kubectl get pod command until you see 2/2 and Running. This may take up to one minute.

metrics-prometheus-server-5fc5dc86cb-79lk4   2/2     Running   0          46s

Create a port-forward, so you can access the server URL:

kubectl --namespace default port-forward $POD_NAME 7090:9090

Now open your browser and navigate to http://localhost:7090/targets. Search for helidon on the page, and you will see your Helidon application as one of the Prometheus targets.

Final Cleanup

You can now delete the Kubernetes resources that were just created during this example.

Delete the Prometheus Kubernetes resources:

helm delete --purge metrics

Delete the application Kubernetes resources:

kubectl delete -f ./metrics.yaml

Reference

• MicroProfile Metrics specification
• MicroProfile Metrics API
• OpenMetrics format
• Prometheus exposition format