Health Checks

Overview

It’s a good practice to monitor your microservice’s health to ensure that it is available and performs correctly. Applications implement health checks to expose health status that is collected at regular intervals by external tooling, such as orchestrators like Kubernetes. The orchestrator may then take action, such as restarting your application if the health check fails.

A typical health check combines the statuses of all the dependencies that affect availability and the ability to perform correctly:

• Network Latency
• Storage
• Database
• Other Services (used by your application)

Maven Coordinates

To enable Health Checks, add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-health</artifactId>
</dependency>

Optional dependency to use built-in health checks:

<dependency>
  <groupId>io.helidon.health</groupId>
  <artifactId>helidon-health-checks</artifactId>
</dependency>

API

Enabling Health Support (and Built-in Health Checks) in Your Application

The health subsystem is part of the observability support. As a result, your application includes health support by default provided your project meets several conditions:

• Your project depends on the helidon-webserver-observe-health component as described above.
• (Optional) Your project depends on the helidon-health-checks component (if you want the built-in health checks).
• Your code allows the webserver’s automatic feature discovery (enabled by default).
• Your code allows the observe feature’s automatic observer discovery (also enabled by default).

If you disable either type of automatic discovery you can add the observe feature to the webserver explicitly, and you can add the health observer to the observe feature explicitly, customizing the behavior of each programmatically if you wish. You can also use configuration to tailor some of the behavior of the health component (such as changing the URI path from /observe/health to something else).

Writing Custom Health Checks

In many cases, the ability of your application to do its job depends on conditions known only to your application: for example, whether certain external resources such as databases are available. You can create custom health checks which reflect those conditions and add them to the overall health assessment of your application.

A health check is a Java functional interface that returns a new HealthCheckResponse instance each time Helidon queries the health check. Each health check also has a fixed name and a fixed health check type (start-up, liveness, or readiness).

Your code registers a custom health check by invoking a method on Helidon-provided types in one of the following ways:

• Pass the name and type of the health check and a Supplier of a HealthCheckResponse such as a method reference or a lambda expression.
• Pass an instance of a class which implements the HealthCheck interface.

Within an application different techniques might make sense for different custom health checks, depending on the complexity of the logic for computing the status for each check. The various styles are functionally equivalent; for a given custom health check choose the style which enhances the readability and clarity of your code. The examples below, in no particular order, implement the same custom health check functionality in different ways to illustrate.

Option 1: Using a HealthCheckResponse supplier method

If you gather the logic for computing the health check response into a method, then you can use a method reference to register the health check.

Declaring a health check response supplier method:

static HealthCheckResponse slowStartLivenessResponse() {
    long now = System.currentTimeMillis();
    return HealthCheckResponse.builder()
        .detail("time", now)
        .status(now - serverStartTime >= 8000)
        .build();
}

Registering a health check using a method reference:

ObserveFeature observe = ObserveFeature.builder()
    .config(config.get("server.features.observe"))1
    .addObserver(HealthObserver.builder()2
         .useSystemServices(true)3
         .addCheck(Main::slowStartLivenessResponse,4
                   HealthCheckType.LIVENESS,5
                   "live-after-8-seconds")6
         .build())
    .build();

Option 2: Using an in-line lambda expression

If the logic for computing the health check response is fairly simple, express it as an in-line lambda when you register the health check.

Registering a health check using an in-line lambda expression:

ObserveFeature observe = ObserveFeature.builder()
    .config(config.get("server.features.observe"))
    .addObserver(HealthObserver.builder()1
        .useSystemServices(true) // Include Helidon-provided health checks.
        .addCheck(() -> HealthCheckResponse.builder()2
            .status(System.currentTimeMillis() - serverStartTime >= 8000)3
            .detail("time", System.currentTimeMillis())4
            .build(),5
        HealthCheckType.READINESS,6
        "live-after-8-seconds")7
        .build())
    .build();

Note that the logic in the lambda expression runs every time Helidon probes the added health check, so the values passed to status and detail are recomputed every time.

Option 3: Using a HealthCheck Instance

If a custom health check requires a lot of information to compute its health check response, it might be clearest to implement it as a class that implements the HealthCheck interface. Your code instantiates the class with all the information, including references to other data, it might need to compute the response each time Helidon probes it.

This example is not complicated in that way, but it’s useful to illustrate this technique of writing a custom health check.

Declaring a concrete HealthCheck implementation:

/**
 * A custom readiness health check that reports UP 8 seconds after server start-up.
 */
class SlowStartHealthCheck implements HealthCheck {1

    @Override
    public HealthCheckType type() {
        return HealthCheckType.READINESS;2
    }

    @Override
    public HealthCheckResponse call() {
        long now = System.currentTimeMillis();
        return HealthCheckResponse.builder()
                .detail("time", now)3
                .status(now - serverStartTime >= 8000)4
                .build();
    }
}

ObserveFeature observe = ObserveFeature.builder()
    .config(config.get("server.features.observe"))
    .addObserver(HealthObserver.builder()1
        .addCheck(new SlowStartHealthCheck())2
        .build())
    .build();

Adding Observability (including the Custom Health Checks) to Helidon

The code examples above prepare the observe feature instance using the built-in and custom health checks. To activate the health subsystem and other auto-discovered observability subsystems, add that observe instance as a feature to the webserver and start the server.

Register the observe feature with the server and start it:

WebServer server = WebServer.builder()
    .featuresDiscoverServices(false)
    .addFeature(observe)1
    .routing(Main::routing)
    .build()
    .start();

Triggering and Interpreting Health Check Output

Health support in Helidon is part of the observability feature. HealthObserver is a Helidon-provided observability implementation that contains a collection of registered HealthCheck instances and, when queried, invokes the registered health checks and returns a response with a status code representing the overall status of the application.

Health status codes

You control, either using configuration or adding code to your application, whether the HTTP responses to GET requests contain detailed information about each health check. With details enabled, HTTP GET responses include JSON content showing the detailed results of all the health checks which the server executed after receiving the request. With details disabled, HTTP GET responses have no payload. HTTP HEAD requests always return only the status with no payload.

If you add the Helidon health dependency to your pom.xml file, Helidon automatically registers the HelidonObserver service and responds to the default /observe/health endpoint. Further, if you add the built-in health checks dependency, Helidon automatically finds them and adds those checks to the HealthObserver.

Below are parts of health responses which include the custom health check added in the earlier example code. This first response shows the health output within the first eight seconds after start-up. Recall that the custom health check will report DOWN during that time, so the overall health is DOWN and the HTTP response status is 503 Service Unavailable.

Response within 8 seconds: HTTP status 503 (not healthy):

{
  "status": "DOWN",
  "checks": [
    {
      "name": "live-after-8-seconds",
      "status": "DOWN",
      "data": {
        "time": 1701984253071
      }
    }
  ]
}

The next response shows the health output once the server has been running for at least eight seconds. The custom health check now reports UP so the overall health status is also UP now and the HTTP status is 200.

Response after 8 seconds: HTTP status 200:

{
  "status": "UP",
  "checks": [
    {
      "name": "live-after-8-seconds",
      "status": "UP",
      "data": {
        "time": 1701984258292
      }
    }
  ]
}

The following table provides a summary of the Health Check API classes.

Health check API classes

Built-In Health Checks

You can use Helidon-provided health checks to report various common health check statuses:

Simply adding the built-in health check dependency is sufficient to register all the built-in health checks automatically. If you want to use only some of the built-in checks in your application, you can disable automatic discovery of the built-in health checks and register only the ones you want.

The following code adds only selected built-in health checks to your application:

Adding selected built-in health checks:

WebServer server = WebServer.builder()
    .config(config.get("server"))
    .addFeature(ObserveFeature.create(HealthObserver.builder()
        .useSystemServices(false)1
        .addCheck(HealthChecks.deadlockCheck())2
        .addCheck(hc)3
        .details(true)
        .build()))
    .routing(Main::routing)
    .build()
    .start();

You can control the thresholds for built-in health checks in either of two ways:

• Create the health checks individually using their builders instead of using the HealthChecks convenience class. See the table above.
• Using configuration as explained in Configuration.

Kubernetes Probes

• Liveness Probe
• Readiness Probe
• Startup Probe

Probes is the term used by Kubernetes to describe health checks for containers (Kubernetes documentation).

There are three types of probes:

• liveness: Indicates whether the container is running
• readiness: Indicates whether the container is ready to service requests
• startup: Indicates whether the application in the container has started

You can implement probes using the following mechanisms:

1. Running a command inside a container
2. Sending an HTTP request to a container
3. Opening a TCP socket to a container

A microservice exposed to HTTP traffic will typically implement both the liveness probe and the readiness probe using HTTP requests. If the microservice takes a significant time to initialize itself, you can also define a startup probe, in which case Kubernetes does not check liveness or readiness probes until the startup probe returns success.

You can configure several parameters for probes. The following are the most relevant parameters:

Liveness Probe

The liveness probe is used to verify the container has become unresponsive. For example, it can be used to detect deadlocks or analyze heap usage. When Kubernetes gives up on a liveness probe, the corresponding pod is restarted.

We recommend the following:

• Avoid checking dependencies in a liveness probe.
• Set timeoutSeconds to avoid excessive probe failures.
• Acknowledge startup times with initialDelaySeconds.

Readiness Probe

The readiness probe is used to avoid routing requests to the pod until it is ready to accept traffic. When Kubernetes gives up on a readiness probe, the pod is not restarted, traffic is not routed to the pod anymore.

We recommend the following:

• Be conservative when checking shared dependencies.
• Be aggressive when checking local dependencies.
• Set failureThreshold according to periodSeconds in order to accommodate temporary errors.

Startup Probe

The startup probe prevents Kubernetes from prematurely checking the other probes if the application takes a long time to start. Otherwise, Kubernetes might misinterpret a failed liveness or readiness probe and shut down the container when, in fact, the application is still coming up.

Troubleshooting Probes

Failed probes are recorded as events associated with their corresponding pods. The event message contains only the status code.

Get the events of a single pod:

POD_NAME=$(kubectl get pod -l app=acme -o jsonpath='{.items[0].metadata.name}')1
kubectl get event --field-selector involvedObject.name=${POD_NAME}2

Configuration

Built-in health checks can be configured using the config property keys described in this table.

Further, you can suppress one or more health checks by setting the configuration item server.features.observe.observers.health.exclude to a comma-separated list of the health check names you want to exclude. The same table lists the name names for the built-in health checks.

Examples

JSON Response Example

Accessing the Helidon-provided /observe/health endpoint reports the health of your application as shown below:

{
    "status": "UP",
    "checks": [
        {
            "name": "deadlock",
            "status": "UP"
        },
        {
            "name": "diskSpace",
            "status": "UP",
            "data": {
                "free": "211.00 GB",
                "freeBytes": 226563444736,
                "percentFree": "45.31%",
                "total": "465.72 GB",
                "totalBytes": 500068036608
            }
        },
        {
            "name": "heapMemory",
            "status": "UP",
            "data": {
                "free": "215.15 MB",
                "freeBytes": 225600496,
                "max": "3.56 GB",
                "maxBytes": 3817865216,
                "percentFree": "99.17%",
                "total": "245.50 MB",
                "totalBytes": 257425408
            }
        }
    ]
}

Kubernetes Example

This example shows the usage of the Helidon health API in an application that implements health endpoints for the liveness and readiness probes. Note that the application code dissociates the health endpoints from the default routes, so that the health endpoints are not exposed by the service. An example YAML specification is also provided for the Kubernetes service and deployment.

Application code:

ObserveFeature observeFeature = ObserveFeature.builder()
    .addObserver(HealthObserver.builder()
        .useSystemServices(false)
        .endpoint("/health/live")1
        .addChecks(HealthChecks.healthChecks())2
        .build())
    .addObserver(HealthObserver.builder()
        .useSystemServices(false)
        .endpoint("/health/ready")3
        .addCheck(() -> HealthCheckResponse.builder()4
            .status(true)
            .build(),
        HealthCheckType.READINESS,
        "database")
        .build())
    .sockets(List.of("observe"))5
    .build();
WebServer server = WebServer.builder()
    .putSocket("@default", socket -> socket
        .port(8080)6
        .routing(r -> r
            .any((req, res) -> res.send("It works!"))))7
    .addFeature(observeFeature)
    .putSocket("observe", socket -> socket
        .port(8081))8
    .build()
    .start();

kind: Service
apiVersion: v1
metadata:
  name: acme1
  labels:
    app: acme
spec:
  type: NodePort
  selector:
    app: acme
  ports:
  - port: 8080
    targetPort: 8080
    name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: acme2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme
  template:
    metadata:
      name: acme
      labels:
        name: acme
    spec:
      containers:
      - name: acme
        image: acme
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /observe/health/live3
            port: 8081
          initialDelaySeconds: 34
          periodSeconds: 10
          timeoutSeconds: 3
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /observe/health/ready5
            port: 8081
          initialDelaySeconds: 106
          periodSeconds: 30
          timeoutSeconds: 10
---

Additional Information

• Health Checks SE API Javadocs.