Metrics in Helidon MP

Overview

Helidon MP metrics implements the MicroProfile Metrics specification, providing:

a unified way for MicroProfile servers to export monitoring data—telemetry—to management agents, and
a unified Java API which all application programmers can use to register and update metrics to expose telemetry data from their services.
support for metrics-related annotations.

Learn more about the MicroProfile Metrics specification.

Metrics is one of the Helidon observability features.

Maven Coordinates

To enable metrics add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
    <groupId>io.helidon.microprofile.metrics</groupId>
    <artifactId>helidon-microprofile-metrics</artifactId>
</dependency>

Copied

Adding this dependency packages the full-featured metrics implementation with your service.

Usage

Instrumenting Your Service

You add metrics to your service in these ways:

Annotate bean methods—typically your REST resource endpoint methods (the Java code that receives incoming REST requests); Helidon automatically registers these metrics and updates them when the annotated methods are invoked via CDI.
Write code which explicitly invokes the metrics API to register metrics, retrieve previously-registered metrics, and update metric values.
Configure some simple REST.request metrics which Helidon automatically registers and updates for all REST resource endpoints.

Later sections of this document describe how to do each of these.

Categorizing Types of Metrics

Helidon distinguishes among scopes, or types, of metrics as described in the MP metrics specification.

Helidon includes metrics in the built-in scopes described below. Applications often register their own metrics in the application scope but can create their own scopes and register metrics within them.

Built-in metric scopes

Built-in Scope	Typical Usage
`base`	OS or Java runtime measurements (available heap, disk space, etc.). Mandated by the MP metrics specification
`vendor`	Implemented by vendors, including the `REST.request` metrics and other key performance indicator measurements (described in later sections).
`application`	Declared via annotations or programmatically registered by your service code.

When you add metrics annotations to your service code, Helidon registers the resulting metrics in the application scope.

Metric Registries

A metric registry collects registered metrics of a given scope. Helidon supports one metrics registry for each scope.

When you add code to your service to create a metric programmatically, the code first locates the appropriate registry and then registers the metric with that registry.

Retrieving Metrics Reports from your Service

When you add the metrics dependency to your project, Helidon automatically provides a built-in REST endpoint /metrics which responds with a report of the registered metrics and their values.

Clients can request a particular output format.

Formats for /metrics output

Format	Requested by
OpenMetrics (Prometheus)	default (`text/plain`)
JSON	Header `Accept: application/json`

Clients can also limit the report by specifying the scope as a query parameter in the request URL:

/metrics?scope=base
/metrics?scope=vendor
/metrics?scope=application

Further, clients can narrow down to a specific metric name by adding the name as another query parameter, such as /metrics?scope=application&name=myCount.

Example Reporting: Prometheus format

curl -s -H 'Accept: text/plain' -X GET http://localhost:8080/metrics

Copied

# TYPE base:classloader_total_loaded_class_count counter
# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
base:classloader_total_loaded_class_count 3157

Copied

Example Reporting: JSON format

curl -s -H 'Accept: application/json' -X GET http://localhost:8080/metrics

Copied

JSON response:

{
   "base" : {
      "memory.maxHeap" : 3817865216,
      "memory.committedHeap" : 335544320
    }
}

Copied

In addition to your application metrics, the reports contain other metrics of interest such as system and VM information.

API

The MicroProfile Metrics API prescribes all the standard interfaces related to metrics. This section summarizes a few key points about using that API and explains some Helidon-specific interfaces.

Metrics Annotations

You can very easily instrument your service and refer to registered metrics by annotating methods to be measured and injecting metrics which your code needs to observe.

Metric-defining Annotations

The MicroProfile Metrics specification describes several metric types you can create using annotations, summarized in the following table:

Metrics Annotations

Annotation	Usage
`@Counted`	Monotonically increasing count of events.
`@Gauge`	Access to a value managed by other code in the service.
`@Timed`	Frequency of invocations and the distribution of how long the invocations take.

Place annotations on constructors or methods to measure those specific executables. If you annotate the class instead, Helidon applies that annotation to all constructors and methods which the class declares.

Metric-referencing Annotations

To get a reference to a specific metric, use a metric-referencing annotation in any bean, including your REST resource classes.

You can @Inject a field of the correct type. Helidon uses the MicroProfile Metrics naming conventions to select which specific metric to inject. Use the @Metric annotation to control that selection.

You can also add @Metric on a constructor or method parameter to trigger injection there.

Helidon automatically looks up the metric referenced from any injection site and provides a reference to the metric. Your code then simply invokes methods on the injected metric.

The `MetricRegistry` API

To register or look up metrics programmatically, your service code uses the MetricRegistry instance for the scope of interest: base, vendor, application, or a custom scope.

Either of the following techniques gets a MetricRegistry reference. Remember that injection works only if the class is a bean so CDI can inject into it.

@Inject MetricRegistry, optionally using @RegistryScope to indicate the registry scope.

Injecting the default MetricRegistry (for the application scope)

class Example {

    @Inject
    private MetricRegistry applicationRegistry;
}

Copied

Injecting a non-default MetricRegistry

class Example {

    @RegistryScope(scope = "myCustomScope")
    @Inject
    private MetricRegistry myCustomRegistry;
}

Copied

Get a Helidon RegistryFactory instance and invoke its getRegistry method.

Obtain the RegistryFactory using either of the following techniques:

@Inject RegistryFactory.

Getting the RegistryFactory using injection

class InjectExample {

    @Inject
    private RegistryFactory registryFactory;

    private MetricRegistry findRegistry(String scope) {
        return registryFactory.getRegistry(scope);
    }
}

Copied

Invoke the static getInstance() method on the RegistryFactory class.

Getting the RegistryFactory programmatically

class Example {

    private MetricRegistry findRegistry(String scope) {
        return RegistryFactory.getInstance().getRegistry(scope);
    }
}

Copied

Once it has a reference to a MetricRegistry your code can use the reference to register new metrics, look up previously-registered metrics, and remove metrics.

Working with Metrics in CDI Extensions

You can work with metrics inside your own CDI extensions, but be careful to do so at the correct point in the CDI lifecycle. Configuration can influence how the metrics system behaves, as the configuration section below explains. Your code should work with metrics only after the Helidon metrics system has initialized itself using configuration. One way to accomplish this is to deal with metrics in a method that observes the Helidon RuntimeStart CDI event, which the extension example below illustrates.

Configuration

To control how the Helidon metrics subsystem behaves, add a metrics section to your META-INF/microprofile-config.properties file.

Type: io.helidon.webserver.observe.metrics.MetricsObserver

This is a standalone configuration type, prefix from configuration root: metrics

This type provides the following service implementations:

io.helidon.webserver.observe.spi.ObserveProvider

Configuration options

Optional configuration options

key	type	default value	description
`app-name`	string		Value for the application tag to be added to each meter ID. @return application tag value
`app-tag-name`	string		Name for the application tag to be added to each meter ID. @return application tag name
`enabled`	boolean	`true`	Whether metrics functionality is enabled. @return if metrics are configured to be enabled
`endpoint`	string	`metrics`
`key-performance-indicators`	KeyPerformanceIndicatorMetricsConfig		Key performance indicator metrics settings. @return key performance indicator metrics settings
`permit-all`	boolean		Whether to allow anybody to access the endpoint. @return whether to permit access to metrics endpoint to anybody, defaults to `true` @see #roles()
`rest-request-enabled`	boolean		Whether automatic REST request metrics should be measured. @return true/false
`roles`	string[]		Hints for role names the user is expected to be in. @return list of hints
`scoping`	ScopingConfig		Settings related to scoping management. @return scoping settings
`tags`	Tag[]		Global tags. @return name/value pairs for global tags

Examples

Helidon MP includes a pre-written example application illustrating enabling/disabling metrics using configuration.

The rest of this section contains other examples of working with metrics:

Example Application Code

Adding Method-level Annotations

The following example adds a new resource class, GreetingCards, to the Helidon MP QuickStart example. It shows how to use the @Counted annotation to track the number of times the /cards endpoint is called.

Create a new class GreetingCards with the following code:

@Path("/cards") 
@RequestScoped 
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "any-card")  
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Copied

This class is annotated with Path which sets the path for this resource as /cards.
The @RequestScoped annotation defines that this bean is request scoped. The request scope is active only for the duration of one web service invocation, and it is destroyed at the end of that invocation.
The annotation @Counted will register a Counter metric for this method, creating it if needed. The counter is incremented each time the anyCards method is called. The name attribute is optional.

Build and run the application

mvn package
java -jar target/helidon-quickstart-mp.jar

Copied

Access the application endpoints

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

Copied

JSON response:

{
  "io.helidon.examples.quickstart.mp.GreetingCards.any-card": 2, //  
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  }
}

Copied

The any-card count is two, since you invoked the endpoint twice. The other metrics are from the SimpleGreetResource class.

Notice the counter name is fully qualified with the class and method names. You can remove the prefix by using the absolute=true field in the @Counted annotation. You must use absolute=false (the default) for class-level annotations.

Additional Method-level Metrics

You can also use the @Timed` annotation with a method. For the following example. you can just annotate the same method with @Timed. Timers significant information about the measured methods, but at a cost of some overhead and more complicated output.

Note that when using multiple annotations on a method, you must give the metrics different names as shown below, although they do not have to be absolute.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true) 
    @Timed(name = "cardTimer", absolute = true, unit = MetricUnits.MILLISECONDS) 
    public JsonObject anyCard() {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Copied

Specify a custom name for the Counter metric and set absolute=true to remove the path prefix from the name. <2>Add the @Timed annotation to get a Timer metric.

Build and run the application

mvn package
java -jar target/helidon-quickstart-mp.jar

Copied

Access the application endpoints

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

Copied

JSON response:

{
  "cardTimer": {
    "count": 2,
    "elapsedTime": 0.002941925,
    "max": 0.002919973,
    "mean": 0.0014709625
  },
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "cardCount": 2
}

Copied

Class-level Metrics

You can collect metrics at the class level to aggregate data from all methods in that class using the same metric. The following example introduces a metric to count all card queries. In the following example, the method-level metrics are not needed to aggregate the counts, but they are left in the example to demonstrate the combined output of all three metrics.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards") 
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true) 
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true) 
    public JsonObject birthdayCard() throws InterruptedException {
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Copied

This class is now annotated with @Counted, which aggregates count data from all the method that have a Count annotation.
Use absolute=true to remove path prefix for method-level annotations.
Add a method with a Counter metric to get birthday cards.

Build and run the application

mvn package
java -jar target/helidon-quickstart-mp.jar

Copied

Access the application endpoints

curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

Copied

JSON response from /metrics?scope=application:

{
  "birthdayCard": 1,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 1,
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 2 
}

Copied

The totalCards.GreetingCards count is a total of all the method-level Counter metrics. Class level metric names are always fully qualified.

Field Level Metrics

Field level metrics can be injected into managed objects, but they need to be updated by the application code. This annotation can be used on fields of type Timer, Counter, and Histogram.

The following example shows how to use a field-level Counter metric to track cache hits.

Update the GreetingCards class with the following code:

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards")
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @Inject
    @Metric(name = "cacheHits", absolute = true) 
    private Counter cacheHits;

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        updateStats(); 
        return createResponse("Here are some random cards ...");
    }

    @Path("/birthday")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(absolute = true)
    public JsonObject birthdayCard() throws InterruptedException {
        updateStats();  
        return createResponse("Here are some birthday cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }

    private void updateStats() {
        if (new Random().nextInt(3) == 1) {
            cacheHits.inc(); 
        }
    }
}

Copied

A Counter metric field, cacheHits, is automatically injected by Helidon.
Call updateStats() to update the cache hits.
Call updateStats() to update the cache hits.
Randomly increment the cacheHits counter.

Build and run the application, then invoke the following endpoints:

curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

Copied

JSON response from /metrics/application:

{
  "birthdayCard": 3,
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "anyCard": 2,
  "cacheHits": 2, 
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 5
}

Copied

The cache was hit two times out of five queries.

Gauge Metric

The metrics you have tested so far are updated in response to an application REST request, i.e. GET /cards. These metrics can be declared in a request scoped class and Helidon will store the metric in the MetricRegistry, so the value persists across requests. When GET /metrics?scope=application is invoked, Helidon will return the current value of the metric stored in the MetricRegistry.

The Gauge annotation is different from the other metric annotations. The application must provide a method to return the gauge value in an application-scoped class. When GET /metrics?scope=application is invoked, Helidon will call the Gauge method, using the returned value as the value of the gauge as part of the metrics response.

The following example demonstrates how to use a Gauge to track application up-time.

Create a new GreetingCardsAppMetrics class with the following code:

@ApplicationScoped 
public class GreetingCardsAppMetrics {

    private AtomicLong startTime = new AtomicLong(0); 

    public void onStartUp(@Observes @Initialized(ApplicationScoped.class) Object init) {
        startTime = new AtomicLong(System.currentTimeMillis()); 
    }

    @Gauge(unit = "TimeSeconds")
    public long appUpTimeSeconds() {
        return Duration.ofMillis(System.currentTimeMillis() - startTime.get()).getSeconds();  
    }
}

Copied

This managed object must be application scoped to properly register and use the annotated Gauge metric.
Declare an AtomicLong field to hold the start time of the application.
Initialize the application start time.
Return the application appUpTimeSeconds metric, which will be included in the application metrics.

Update the GreetingCards class with the following code to simplify the metrics output:

@Path("/cards")
@RequestScoped
public class GreetingCards {

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Counted(name = "cardCount", absolute = true)
    public JsonObject anyCard() throws InterruptedException {
        return createResponse("Here are some random cards ...");
    }

    private JsonObject createResponse(String msg) {
        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

Copied

Build and run the application, then invoke the application metrics endpoint:

curl -H "Accept: application/json"  'http://localhost:8080/metrics?scope=application'

Copied

JSON response from /metrics/application:

{
  "personalizedGets": 0,
  "allGets": {
    "count": 0,
    "elapsedTime": 0,
    "max": 0,
    "mean": 0
  },
  "io.helidon.examples.quickstart.mp.GreetingCardsAppMetrics.appUpTimeSeconds": 23, 
  "cardCount": 0
}

Copied

The application has been running for 23 seconds.

Working with Metrics in CDI Extensions

You can work with metrics from your own CDI extension by observing the RuntimeStart event.

CDI Extension that works correctly with metrics

public class MyExtension implements Extension {
    void startup(@Observes @RuntimeStart Object event,  
                 MetricRegistry metricRegistry) {       
        metricRegistry.counter("myCounter");         
    }
}

Copied

Declares that your observer method responds to the RuntimeStart event. By this time, Helidon has initialized the metrics system.
Injects a MetricRegistry (the application registry by default).
Uses the injected registry to register a metric (a counter in this case).

Helidon does not prevent you from working with metrics earlier than the RuntimeStart event, but, if you do so, then Helidon might ignore certain configuration settings that would otherwise control how metrics behaves. Instead, consider writing your extension to use earlier lifecycle events (such as ProcessAnnotatedType) to gather and store information about metrics that you want to register. Then your extension’s RuntimeStart observer method would use that stored information to register the metrics you need.

Example Configuration

Metrics configuration is quite extensive and powerful and, therefore, a bit complicated. The rest of this section illustrates some of the most common scenarios:

Disable Metrics Subsystem

Disabling metrics entirely

metrics.enabled=false

Copied

Helidon does not update metrics, and the /metrics endpoints respond with 404..

Collecting Basic and Extended Key Performance Indicator (KPI) Metrics

Any time you include the Helidon metrics module in your application, Helidon tracks a basic performance indicator metric: a Counter of all requests received (requests.count)

Helidon MP also includes additional, extended KPI metrics which are disabled by default:

current number of requests in-flight - a Gauge (requests.inFlight) of requests currently being processed
long-running requests - a Counter (requests.longRunning) measuring the total number of requests which take at least a given amount of time to complete; configurable, defaults to 10000 milliseconds (10 seconds)
load - a Counter (requests.load) measuring the number of requests worked on (as opposed to received)
deferred - a Gauge (requests.deferred) measuring delayed request processing (work on a request was delayed after Helidon received the request)

You can enable and control these metrics using configuration:

Controlling extended KPI metrics

metrics.key-performance-indicators.extended = true
metrics.key-performance-indicators.long-running.threshold-ms = 2000

Copied

Enable `REST.request` Metrics

Controlling REST request metrics

metrics.rest-request-enabled=true

Copied

Helidon automatically registers and updates Timer metrics for every REST endpoint in your service.

Additional Information

Integration with Kubernetes and Prometheus

Kubernetes Integration

The following example shows how to integrate the Helidon MP application with Kubernetes.

Stop the application and build the docker image:

docker build -t helidon-metrics-mp .

Copied

Create the Kubernetes YAML specification, named metrics.yaml, with the following content:

kind: Service
apiVersion: v1
metadata:
  name: helidon-metrics 
  labels:
    app: helidon-metrics
  annotations:
    prometheus.io/scrape: "true" 
spec:
  type: NodePort
  selector:
    app: helidon-metrics
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-metrics
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: helidon-metrics
  template:
    metadata:
      labels:
        app: helidon-metrics
        version: v1
    spec:
      containers:
        - name: helidon-metrics
          image: helidon-metrics-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Copied

A service of type NodePort that serves the default routes on port 8080.
An annotation that will allow Prometheus to discover and scrape the application pod.
A deployment with one replica of a pod.

Create and deploy the application into Kubernetes:

kubectl apply -f ./metrics.yaml

Copied

Get the service information:

kubectl get service/helidon-metrics

Copied

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-metrics   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s

Copied

A service of type NodePort that serves the default routes on port 31143.

Verify the metrics endpoint using port 30116, your port will likely be different:

curl http://localhost:31143/metrics

Copied

Leave the application running in Kubernetes since it will be used for Prometheus integration.

Prometheus Integration

The metrics service that you just deployed into Kubernetes is already annotated with prometheus.io/scrape:. This will allow Prometheus to discover the service and scrape the metrics. This example shows how to install Prometheus into Kubernetes, then verify that it discovered the Helidon metrics in your application.

Install Prometheus and wait until the pod is ready:

helm install stable/prometheus --name metrics
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl get pod $POD_NAME

Copied

You will see output similar to the following. Repeat the kubectl get pod command until you see 2/2 and Running. This may take up to one minute.

metrics-prometheus-server-5fc5dc86cb-79lk4   2/2     Running   0          46s

Copied

Create a port-forward, so you can access the server URL:

kubectl --namespace default port-forward $POD_NAME 7090:9090

Copied

Now open your browser and navigate to http://localhost:7090/targets. Search for helidon on the page, and you will see your Helidon application as one of the Prometheus targets.

Final Cleanup

You can now delete the Kubernetes resources that were just created during this example.

Delete the Prometheus Kubernetes resources:

helm delete --purge metrics

Copied

Delete the application Kubernetes resources:

kubectl delete -f ./metrics.yaml

Copied

References

MicroProfile Metrics specification

MicroProfile Metrics API

Contents

Overview

Maven Coordinates

Usage

Instrumenting Your Service

Categorizing Types of Metrics

Metric Registries

Retrieving Metrics Reports from your Service

API

Metrics Annotations

Metric-defining Annotations

Metric-referencing Annotations

The MetricRegistry API

Working with Metrics in CDI Extensions

Configuration

Configuration options

Examples

Example Application Code

Adding Method-level Annotations

Additional Method-level Metrics

Class-level Metrics

Field Level Metrics

Gauge Metric

Working with Metrics in CDI Extensions

Example Configuration

Disable Metrics Subsystem

Collecting Basic and Extended Key Performance Indicator (KPI) Metrics

Enable REST.request Metrics

Additional Information

Integration with Kubernetes and Prometheus

Kubernetes Integration

Prometheus Integration

Final Cleanup

References

The `MetricRegistry` API

Enable `REST.request` Metrics