You can configure publishers in the publishers configuration section under the top level metrics node or under server.features.observe.observers.metrics. If you do not set up publishers explicitly, Helidon uses an inferred Prometheus publisher for backward compatibility. See this later section for details.

Publishers in Helidon’s Micrometer-based metrics implementation use Micrometer MeterRegistry implementations. For each enabled publisher, Helidon adds the corresponding meter registry to Micrometer’s global registry. This has these important effects:

• {meters_uc} which Helidon or your code registers using the Helidon metrics API are registered in all active Micrometer meter registries.

• Each Helidon meter registered has an implementation in every active Micrometer meter registry.

• When Helidon or your code updates a Helidon {meter}, Micrometer applies the change to every corresponding {meter} from each active meter registry.

As a result, configuring more than one active meter registry can affect performance.

Configuring an OTLP Publisher

The configuration directly mirrors the Micrometer OtlpMeterRegistry settings so you can control all behavior which Micrometer exposes for the meter registry.

The following example sets up an OTLP publisher to transmit metrics data every 30 seconds.

metrics:
  publishers:         
    otlp:  
      interval: PT30S
      url: 'http://somehost.com:4318/v1/metrics'

• Introduces the configured publishers.
• Configures an OTLP publisher to transmit every 30 seconds to the given endpoint.

Configuring a Prometheus Publisher

Understanding the Inferred Prometheus Publisher

metrics:
  publishers:
    prometheus:
    otlp:
      interval: PT20S

You can write other publishers by following these steps:

1. Choose one of the Micrometer MeterRegistry implementations for the type of publishing you want to support. (for example DatadogMeterRegistry)
2. Create a config blueprint which exposes the meter registry’s settable properties from DatadogConfig.
3. Write a DatadogPublisher class which implements Helidon’s MetricsPublisher for Datadog.
4. Write a DatadogPublisherProvider class which implements Helidon’s MetricsPublisherProvider for your publisher.
5. Advertise your provider so Java service loading can find it, creating a META-INF/services/io.helidon.metrics.spi.PublisherProvider file listing your implementation class.

Look at Helidon’s OTLP publisher blueprint and the related types as an example.

Refer to your publisher in configuration using the config key you set up in the publisher provider.

metrics:
  publishers:
    micrometer-datadog:
      interval: PT15S

When you add the metrics dependency to your project, and if you explicitly set up a Prometheus publisher or use the default one, Helidon provides a built-in REST endpoint /metrics which responds with a report of the registered metrics and their values.

Clients can request a particular output format from the endpoint.

Clients can also limit the report by specifying the scope as a query parameter in the request URL:

• /metrics?scope=base

• /metrics?scope=vendor

• /metrics?scope=application

Further, clients can narrow down to a specific metric name by adding the name as another query parameter, such as /metrics?scope=application&name=myCount.

curl -s -H 'Accept: text/plain' -X GET http://localhost:8080/metrics

# HELP classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
# TYPE classloader_loadedClasses_count gauge
classloader_loadedClasses_count{mp_scope="base",} 5297.0

See the summary of the OpenMetrics and Prometheus Format for more information.

curl -s -H 'Accept: application/json' -X GET http://localhost:8080/metrics

{
   "base" : {
      "memory.maxHeap" : 3817865216,
      "memory.committedHeap" : 335544320
    }
}

In addition to your application metrics, the reports contain other metrics of interest such as system and VM information.

The OpenMetrics format and the Prometheus exposition format are very similar in most important respects but are not identical. This brief summary treats them as the same.

The OpenMetrics/Prometheus format represents each metric using three lines of output as summarized in the following table.

The OpenMetrics/Prometheus output converts metric IDs in these ways:

• Names in camel case are converted to "snake case" and dots are converted to underscores.

• Names include any units specified for the metric.

• For percentiles, the ID includes a tag identifying which percentile the line of output describes.

As the earlier example output showed, for a metric with multiple values, such as a timer or a histogram, (with, among others, max, mean, and count), the OpenMetrics/Prometheus output reports a "metric family" which includes a separate family member metric for each of the multiple values. The name for each member in the family is derived from the registered name for the metric plus a suffix indicating which one of the metric’s multiple values the line refers to.

The following table summarizes the naming for each metric type.

¹ The OpenMetrics/Prometheus output format reports a timer as a summary with units of seconds.

Unlike OpenMetrics/Prometheus output, which combines the data and the metadata in a single response, you use an HTTP GET request to retrieve metrics JSON data and an OPTIONS request to retrieve metadata in JSON format.

Helidon groups metrics in the same scope together in JSON output as shown in the following example.

{
  "application": {  
    "getTimer": {
      "type": "timer",
      "unit": "seconds",
      "description": "Timer for getting the default greeting"
    }
  },
  "vendor": {       
    "requests.count": {
      "type": "counter",
      "description": "Each request (regardless of HTTP method) will increase this counter"
    }
  },
  "base": {         
    "cpu.systemLoadAverage": {
      "type": "gauge",
      "description": "Displays the system load average for the last minute."
    },
    "classloader.loadedClasses.count": {
      "type": "gauge",
      "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
    }
  }
}

• Note the application, vendor, and base sections.

If an HTTP request selects by scope, the output omits the extra level of structure that identifies the scope as shown in the following example.

{
  "cpu.systemLoadAverage": {
    "type": "gauge",
     "description": "Displays the system load average for the last minute."
  },
  "classloader.loadedClasses.count": {
    "type": "gauge",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
  }
}

Understanding the JSON Metrics Data Format

"requests.count": 5

"getTimer": {
  "count": 3,
  "max": 0.0030455,
  "mean": 0.0011060836666666666,
  "elapsedTime": 0.003318251,
  "p0.5": 0.000151552,
  "p0.75": 0.003141632,
  "p0.95": 0.003141632,
  "p0.98": 0.003141632,
  "p0.99": 0.003141632,
  "p0.999": 0.003141632
}

Understanding the JSON Metrics Metadata Format

"requests.count": {
  "type": "counter",
  "description": "Each request (regardless of HTTP method) will increase this counter"
    }

"getTimer": {
  "type": "timer",
  "unit": "seconds",
  "description": "Timer for getting the default greeting"
}

Controlling JSON Timer Output

metrics.timers.json-units-default=units 

The MicroProfile Metrics specification describes several metric types you can create using annotations, summarized in the following table:

¹ Place annotations on constructors or methods to measure those specific executables. If you annotate the class instead, Helidon applies that annotation to all constructors and methods which the class declares.

To get a reference to a specific metric, use a metric-referencing annotation in any bean, including your REST resource classes.

You can @Inject a field of the correct type. Helidon uses the MicroProfile Metrics naming conventions to select which specific metric to inject. Use the @Metric annotation to control that selection.

You can also add @Metric on a constructor or method parameter to trigger injection there.

Helidon automatically looks up the metric referenced from any injection site and provides a reference to the metric. Your code then simply invokes methods on the injected metric.

You can choose which endpoints to include in Helidon’s automatic measurements using the auto-http-metrics config section.

The paths section contains zero or more entries, each entry having the following settings:

Helidon decides whether to measure incoming requests as follows:

• If you omit the auto-http-metrics configuration, Helidon measures all endpoints.

• If you specify the auto-http-metrics configuration, by default Helidon does not measure built-in endpoints such as metrics, health, and openapi. You can add items under auto-http-metrics.paths to control more exactly which endpoints to measure.

• If you include the paths section, Helidon checks a request against the path entries in order. A given request matches an entry if its path matches the path pattern and its HTTP method is in the methods list. If there is no methods list for an entry, all HTTP methods match the entry.

• If a request matches an entry, the entry’s enabled setting determines if the request should be measured.

• If a request matches multiple entries, the first match wins.

• If a request matches no entry, it is measured.

The auto-http-metrics.sockets setting controls which sockets are included in the measurements; if not set, Helidon measures requests on all sockets.

server.features.observe.observers.metrics.auto-http-metrics.paths.0.path=/greet        
server.features.observe.observers.metrics.auto-http-metrics.paths.0.methods=GET,HEAD

server.features.observe.observers.metrics.auto-http-metrics.paths.1.path=/greet/{name} 
server.features.observe.observers.metrics.auto-http-metrics.paths.1.enabled=false

server.features.observe.observers.metrics.auto-http-metrics.sockets=@default,private   

• Measure /greet for only GET and HEAD requests.
• Do not measure the personalized greeting requests.
• Measure only endpoints on the default socket and the socket named private. Endpoints on other sockets (such as if you had an admin socket) are not measured.

The AutoHttpMetricsConfig documentation describes the configuration more fully.

The following example adds a new resource class, GreetingCards, to the Helidon MP QuickStart example. It shows how to use the @Counted annotation to track the number of times the /cards endpoint is called.

@Path("/cards") 
@RequestScoped 
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "any-card")  
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

• This class is annotated with Path which sets the path for this resource as /cards.
• The @RequestScoped annotation defines that this bean is request scoped. The request scope is active only for the duration of one web service invocation, and it is destroyed at the end of that invocation.
• The annotation @Counted will register a Counter metric for this method, creating it if needed. The counter is incremented each time the anyCards method is called. The name attribute is optional.

mvn package
java -jar target/helidon-quickstart-mp.jar

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "io.helidon.examples.quickstart.mp.GreetingCards.any-card": 2, //  
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  }
}

• The any-card count is two, since you invoked the endpoint twice. The other metrics are from the SimpleGreetResource class.

You can also use the @Timed` annotation with a method. For the following example. you can just annotate the same method with @Timed. Timers significant information about the measured methods, but at a cost of some overhead and more complicated output.

Note that when using multiple annotations on a method, you must give the metrics different names as shown below, although they do not have to be absolute.

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true) 
    @Timed(name = "cardTimer", absolute = true, unit = MetricUnits.MILLISECONDS) 
    public JsonObject anyCard() {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

• Specify a custom name for the Counter metric and set absolute=true to remove the path prefix from the name. <2>Add the @Timed annotation to get a Timer metric.

mvn package
java -jar target/helidon-quickstart-mp.jar

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "cardTimer": {
    "count": 2,
    "elapsedTime": 0.002941925,
    "max": 0.002919973,
    "mean": 0.0014709625
  },
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "cardCount": 2
}

You can collect metrics at the class level to aggregate data from all methods in that class using the same metric. The following example introduces a metric to count all card queries. In the following example, the method-level metrics are not needed to aggregate the counts, but they are left in the example to demonstrate the combined output of all three metrics.

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards") 
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true) 
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true) 
    public JsonObject birthdayCard() throws InterruptedException {
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

• This class is now annotated with @Counted, which aggregates count data from all the method that have a Count annotation.
• Use absolute=true to remove path prefix for method-level annotations.
• Add a method with a Counter metric to get birthday cards.

mvn package
java -jar target/helidon-quickstart-mp.jar

curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "birthdayCard": 1,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 1,
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 2 
}

• The totalCards.GreetingCards count is a total of all the method-level Counter metrics. Class level metric names are always fully qualified.

Field level metrics can be injected into managed objects, but they need to be updated by the application code. This annotation can be used on fields of type Timer, Counter, and Histogram.

The following example shows how to use a field-level Counter metric to track cache hits.

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards")
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @Inject
    @Metric(name = "cacheHits", absolute = true) 
    private Counter cacheHits;

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        updateStats(); 
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject birthdayCard() throws InterruptedException {
        updateStats();  
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }

    private void updateStats() {
        if (new Random().nextInt(3) == 1) {
            cacheHits.inc(); 
        }
    }
}

• A Counter metric field, cacheHits, is automatically injected by Helidon.
• Call updateStats() to update the cache hits.
• Call updateStats() to update the cache hits.
• Randomly increment the cacheHits counter.

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "birthdayCard": 3,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 2,
  "cacheHits": 2, 
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 5
}

• The cache was hit two times out of five queries.

The metrics you have tested so far are updated in response to an application REST request, i.e. GET /cards. These metrics can be declared in a request scoped class and Helidon will store the metric in the MetricRegistry, so the value persists across requests. When GET /metrics?scope=application is invoked, Helidon will return the current value of the metric stored in the MetricRegistry.

The Gauge annotation is different from the other metric annotations. The application must provide a method to return the gauge value in an application-scoped class. When GET /metrics?scope=application is invoked, Helidon will call the Gauge method, using the returned value as the value of the gauge as part of the metrics response.

The following example demonstrates how to use a Gauge to track application up-time.

@ApplicationScoped 
public class GreetingCardsAppMetrics {

    private AtomicLong startTime = new AtomicLong(0); 

    public void onStartUp(@Observes @Initialized(ApplicationScoped.class) Object init) {
        startTime = new AtomicLong(System.currentTimeMillis()); 
    }

    @Gauge(unit = "TimeSeconds")
    public long appUpTimeSeconds() {
        return Duration.ofMillis(System.currentTimeMillis() - startTime.get()).getSeconds();  
    }
}

• This managed object must be application scoped to properly register and use the annotated Gauge metric.
• Declare an AtomicLong field to hold the start time of the application.
• Initialize the application start time.
• Return the application appUpTimeSeconds metric, which will be included in the application metrics.

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

{
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "io.helidon.examples.quickstart.mp.GreetingCardsAppMetrics.appUpTimeSeconds": 23, 
  "cardCount": 0
}

• The application has been running for 23 seconds.

You can work with metrics from your own CDI extension by observing the RuntimeStart event.

public class MyExtension implements Extension {
    void startup(@Observes @RuntimeStart Object event,  
                 MetricRegistry metricRegistry) {       
        metricRegistry.counter("myCounter");         
    }
}

• Declares that your observer method responds to the RuntimeStart event. By this time, Helidon has initialized the metrics system.
• Injects a MetricRegistry (the application registry by default).
• Uses the injected registry to register a metric (a counter in this case).

metrics.enabled=false

Helidon does not update metrics, and the /metrics endpoints respond with 404.

Enabling Virtual Threads Metrics

metrics.virtual-threads.enabled = true

Controlling Measurements of Pinned Virtual Threads

metrics.virtual-threads.pinned.threshold=PT0.100S

Any time you include the Helidon metrics module in your application, Helidon tracks a basic performance indicator metric: a Counter of all requests received (requests.count)

Helidon MP also includes additional, extended KPI metrics which are disabled by default:

• current number of requests in-flight - a Gauge (requests.inFlight) of requests currently being processed

• long-running requests - a Counter (requests.longRunning) measuring the total number of requests which take at least a given amount of time to complete; configurable, defaults to 10000 milliseconds (10 seconds)

• load - a Counter (requests.load) measuring the number of requests worked on (as opposed to received)

• deferred - a Gauge (requests.deferred) measuring delayed request processing (work on a request was delayed after Helidon received the request)

You can enable and control these metrics using configuration:

metrics.key-performance-indicators.extended = true
metrics.key-performance-indicators.long-running.threshold-ms = 2000

metrics.rest-request.enabled=true

Helidon automatically registers and updates Timer metrics for every REST endpoint in your service.

The following example shows how to integrate the Helidon MP application with Kubernetes.

docker build -t helidon-metrics-mp .

kind: Service
apiVersion: v1
metadata:
  name: helidon-metrics 
  labels:
    app: helidon-metrics
  annotations:
    prometheus.io/scrape: "true" 
spec:
  type: NodePort
  selector:
    app: helidon-metrics
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-metrics
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: helidon-metrics
  template:
    metadata:
      labels:
        app: helidon-metrics
        version: v1
    spec:
      containers:
        - name: helidon-metrics
          image: helidon-metrics-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

• A service of type NodePort that serves the default routes on port 8080.
• An annotation that will allow Prometheus to discover and scrape the application pod.
• A deployment with one replica of a pod.

kubectl apply -f ./metrics.yaml

kubectl get service/helidon-metrics

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-metrics   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s 

• A service of type NodePort that serves the default routes on port 31143.

curl http://localhost:31143/metrics

The metrics service that you just deployed into Kubernetes is already annotated with prometheus.io/scrape:. This will allow Prometheus to discover the service and scrape the metrics. This example shows how to install Prometheus into Kubernetes, then verify that it discovered the Helidon metrics in your application.

helm install stable/prometheus --name metrics
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl get pod $POD_NAME

You will see output similar to the following. Repeat the kubectl get pod command until you see 2/2 and Running. This may take up to one minute.

metrics-prometheus-server-5fc5dc86cb-79lk4   2/2     Running   0          46s

kubectl --namespace default port-forward $POD_NAME 7090:9090

Now open your browser and navigate to http://localhost:7090/targets. Search for helidon on the page, and you will see your Helidon application as one of the Prometheus targets.

You can now delete the Kubernetes resources that were just created during this example.

helm delete --purge metrics

kubectl delete -f ./metrics.yaml