Observability

Overview

In Helidon 4 all observability features were moved to one logical module: observe. The observability support groups all observe endpoints together under a single context root (the default behavior) /observe.

Maven Coordinates

To enable Helidon Observability, add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe</artifactId>
</dependency>

For Health Observability features:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-health</artifactId>
</dependency>

For Metrics Observability features:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-metrics</artifactId>
</dependency>

For Info Observability features:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-info</artifactId>
</dependency>

For Logging Observability features:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-log</artifactId>
</dependency>

For Configuration Observability features:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-config</artifactId>
</dependency>

Usage

Each provider usually adds a new endpoint (such as health, metrics). This is to have a single easily configurable path for security, proxy etc. purposes, rather than expose multiple "root" endpoints that may collide with the business code.

Discovery

When an application is bootstrapped using ServiceRegistryManager, the observe feature and ObserveProvider instances are discovered from the owning service registry. Providers loaded using ServiceLoader are bridged into the registry when service loader discovery is enabled, which preserves compatibility with existing providers.

When an ObserveFeature is built directly, ObserveProvider instances are discovered using ServiceLoader. In this case, an explicitly configured Observer with the same type as a provider takes precedence so the observer is not duplicated.

For fully manual registry setup, set server.features-discover-services to false to disable automatic server feature discovery, including the observe feature. Set server.features.observe.observers-discover-services to false to disable automatic observer discovery. The application can then register the ObserveFeature and its Observer instances explicitly.

Feature Weight and Endpoint Conflicts

In some ways Helidon treats all types of observers as a single observability feature. In particular, you can use configuration to control the weight of the various Helidon features, and the weight prescribes the order in which Helidon handles routing for those features.

The Helidon-provided feature for processing your application endpoints has weight 100 by default, and the observability feature has default weight 80. This means that Helidon normally prioritizes routing for your application endpoints over the endpoints for the observers such as metrics and health.

This can have unexpected results if your application declares a resource path /{name}. Because Helidon normally prioritizes the routing of your endpoints, Helidon routes requests for /metrics and /health to your /{name} endpoint instead of to the actual metrics and health endpoints.

One way to avoid this is to assign a weight from 101 to 200 to the observe feature in your configuration. Then Helidon prioritizes the routing of the observe feature ahead of routing your application endpoints.

Assign feature weight to control routing in application.yaml:

server:
  features:
    observe:
      weight: 120

Helidon does not enforce the weight range 101-200 for observability, but you should use a value in this range for the observe weight to avoid problems with other features such as security, CORS, and others; their relative ordering is important.

Endpoints

The "Observe" service endpoint can be modified on the ObserveFeature that is registered with routing. The feature endpoint defaults to /observe, and all observers are prefixed with it (see further)

Each observer has customizable endpoints as well, and the result is decided as follows: 1. If the custom endpoint is relative, the result would be under observe endpoint (e.g. for health → /observe/health) 2. If the custom endpoint is absolute, the result would be absolute as well (e.g. for /health → /health)

Configuration Observability

Configuration observability allows reading the current application configuration values. Configuration observability defines the following endpoints:

By default, configuration observability includes values only for keys that match the configured safe-keys regular expression patterns. Values for all other keys are obfuscated with "*" characters. Keys matching the built-in or configured secrets patterns are always obfuscated. Set unsafe-values to true to restore the previous behavior and include all values except those matching secrets patterns.

Health Observability

Health observability allows reading application readiness to serve requests, whether the services are alive. Health observability defines the following endpoints:

For more information, please, check Health documentation.

Information Observability

Info observability allows configuration of custom properties to be available to users. Information observability defines the following endpoints:

Logger Observability

Log observability allows reading and configuring of log levels of various loggers and reading log messages. Logger Observability defines the following endpoints:

Metrics Observability

When you add the metrics dependency to your project, Helidon automatically provides a built-in REST endpoint /observe/metrics which responds with a report of the registered metrics and their values.

Clients can request a particular output format.

Formats for /observe/metrics output:

For more information see Metrics documentation.

Configuration

To customize the endpoint of an observer:

1. Configure a custom endpoint through configuration to modify the ObserveProvider setup (such as server.features.observe.health.endpoint)
2. Configure a custom endpoint through a builder on the specific Observer (HealthObserver.builder().endpoint("myhealth"))

To control the observability features as a whole, add config settings under server.features.observe.

Configuration options

Additional Information

The Observability features are now implemented with HttpFeature and can be registered with HttpRouting.Builder#addFeature(java.util.function.Supplier). Such a feature encapsulates a set of endpoints, services and/or filters.

Feature is similar to HttpService but gives more freedom in setup. Main difference is that a feature can add Filter filters and it cannot be registered on a path (that is left to the discretion of the feature developer).

• Features are not registered immediately - each feature can define a Weight or implement Weighted to order features according to their weight. Higher weighted features are registered first.
• This is to allow ordering of features in a meaningful way (e.g. Context should be first, Tracing second, Security third etc).

Reference

• Metrics documentation.
• Health documentation.