Advanced Configuration Topics

Overview

This section discusses several advanced topics related to Helidon configuration.

Advanced Config Sources

Environment Variables Config Source

The config system supports using environment variables as a config source, and is enabled by default. Since environment variable names are normally restricted to alphanumeric characters and underscore, this config source adds aliases that enable setting or overriding config entries with dotted and/or hyphenated keys.

The mapping makes it possible to set or override a config entry with a key of "foo.bar" using an environment variable named "FOO_BAR" and "foo.bar-baz" using "FOO_BAR_dash_BAZ".

One use case for this mapping is config overrides in containers, where passing environment variables directly or via Kubernetes Secrets/ConfigMaps is common. Scripts that solve the mapping problem by explicitly converting variables to system properties can also be simplified.

Aliases are produced for any environment variable name that matches all the following:

1. does not begin or end with a '_' character
2. does not contain "__"
3. contains one or more '_' characters

For each such name, two aliases are added with the names mapped as follows:

1. Replace any "_dash_" or "_DASH_" substrings with "-", e.g. "APP_PAGE_dash_SIZE" becomes "APP_PAGE-SIZE".
2. Replace '_' with '.' and add as an alias, e.g. "APP_GREETING" is added as "APP.GREETING" and "APP_PAGE-SIZE" is added as "APP.PAGE-SIZE". This mapping is added primarily to support mixed case config keys such as "app.someCamelCaseKey".
3. Convert the result of step 2 to lowercase and add as an alias, e.g. "APP.GREETING" is added as "app.greeting" and "APP.PAGE-SIZE" is added as "app.page-size".

Directory Config Source

The config system supports using a file system directory as a config source. Each non-directory file in the directory becomes a config entry: the file name is the key and the contents of that file are used as the corresponding config String value.

The following example shows, for example, one way to load Kubernetes secrets mounted on the pod’s filesystem.

If the directory conf/secrets contains these two files

File secrets/username

jose

File secrets/password

^ery$ecretP&ssword

your application can load this as configuration as follows:

Using directory config source:

Config secrets = Config.builder(
                ConfigSources.directory("conf/secrets"))1
        .disableEnvironmentVariablesSource()2
        .disableSystemPropertiesSource()2
        .build();

assert secrets.get("username")3
        .asString()
        .get()
        .equals("jose");
assert secrets.get("password")4
        .asString()
        .get()
        .equals("^ery$ecretP&ssword");

Remember that your application can process the contents of a given file as configuration. See the config sources section and the ConfigSources.file Javadoc.

In-memory Config Sources

The config system provides several ways to create a Config tree from data already in memory. See the ConfigSources Javadoc for further details.

The numerous variants of the from method construct ConfigSource or Builder<ConfigSource> instances.

Config Subtree

Config anotherConfig = Config.create(classpath("application.conf"));

Config config = Config.create(
    ConfigSources.create(anotherConfig.get("data")));

Properties

Config config = Config.create(
    ConfigSources.create(System.getProperties()).build());

String

Config config = Config.create();
ConfigSources.create("app.greeting = Hi", MediaTypes.create("text", "x-java-properties"));

Map

Config config = Config.create(
    ConfigSources.create(Map.of("app.page-size", "20"))
        .build());1

ad hoc Config Nodes

Config config = Config.create(
    ConfigSources.create(ObjectNode.builder()
        .addList("app.basic-range", ListNode.builder()
             .addValue("-20")
             .addValue("20")
             .build())
    .build()));

Although the examples above use a single source, you can build a single Config from multiple sources.

Handling Key Collisions

Prefixed Config Sources

Sometimes you might want to create a single config tree from multiple sources but in a way that keeps the config from different sources in different subtrees.

The config system lets you assign a prefix to all keys from a given source using the ConfigSources.prefixed method. The following example shows two YAML files as config sources and the code to load each with a different prefix into a single Config tree:

File app.conf:

greeting = "Hello"
page-size = 20
basic-range = [ -20, 20 ]

providers: [
    {
        name = "Provider1"
        class = "this.is.my.Provider1"
    },
    {
        name = "Provider2"
        class = "this.is.my.Provider2"
    }
]

Using prefixed config source:

Config config = Config.create(
    ConfigSources.prefixed("app",1
        classpath("app.conf")),2
    ConfigSources.prefixed("data",3
        classpath("data.conf")));4

assert config.get("app.greeting")5
        .asString()
        .get()
        .equals("Hello");

assert config.get("data.providers.0.name")6
        .asString()
        .get()
        .equals("Provider1");

This technique can be useful, for example, if multiple sources contain keys that might overlap; assigning different prefixes to the keys from different sources gives your application a way to access all config elements distinctly even if their keys would otherwise conflict.

Merging Strategies

When creating config from multiple sources, it is possible that the same key comes from multiple sources. By default, earlier sources in the list have higher priority than later ones. This means that if the same key appears in two or more sources, then the source earlier in the list prevails.

The config system provides the FallbackMergingStrategy which implements the default, "first wins" algorithm. You can write your own implementation of MergingStrategy interface and use it instead to provide a different algorithm.

Composite config source example:

Config config = Config.builder()
        .addSource(file("config-file.properties"))
        .addSource(classpath("application.yaml"))
        .mergingStrategy(MergingStrategy.fallback())1
        .build();

Advanced Config Parsers

Config sources and parsers work together to read and translate configuration data from some external form into the corresponding in-memory config tree.

How Config Chooses Parsers

Although most applications are explicit about the config sources they use in building a Config, the config system often has to figure out what parser to use. It does so by:

1. determining, the best that it can, the media type of the source, and
2. locating a parser that can translate that media type.

Identifying the Media Type

By Inference

Most applications let the config system try to infer the media type of the config source.

By default, config source implementations use the io.helidon.common.media.type.MediaTypes API to infer the source media type from the source, typically (but not always) based on the file type portion of the file path.

Helidon media type module has a predefined set of mappings .properties, .yaml, .json and .conf. To handle other formats you can implement and register your own io.helidon.common.media.type.spi.MediaTypeDetector Java Service implementations. Typically, you would also write and register a config parser to translate that format; see Locating a Parser below.

By Application Directive

Your application can specify what media type to use in interpreting a config source. Use this if your application knows the media type but the system might not be able to infer it correctly, either because no type detector would recognize it or because there might be more than one inferred media type.

Specify mediaType for config source:

Config config = Config.create(classpath("props")1
    .mediaType(MediaTypes.TEXT_PROPERTIES));2

Note that a file type detector could be written to also inspect the contents of the file to infer the media type. The detectors provided by Helidon only inspect the suffix in the name of the file.

Locating a Parser

By Inference from media-type

Each config parser reports which media types it handles. Once the config system has determined a source’s media type, it searches the config parsers associated with the config builder for one that recognizes that media type. It then uses that parser to translate the config in the source into the in-memory config tree.

The application can add one or more parsers to a Config.Builder using the addParser method. This makes the parser available for use by the config sources associated with that builder, but does not directly tie a given parser to a given source. The builder uses media-type matching to select one of the parsers registered with the builder for each source.

If the config system cannot locate a parser that matches the media-type of a source, it throws a ConfigException when trying to prepare the configuration.

By Application Directive

Your application can specify which parser to use for a config source. The AbstractConfigSourceBuilder class exposes the parser method, which accepts the ConfigParser to be used for that source. Several methods on ConfigSources such as classpath, directory, and file return this builder class.

Generally try to rely on media-type matching rather than specifying a given parser for a given source in the application. This keeps your application more flexible, both by insulating it from implementation classes and by letting it easily take advantage of improvements in or alternatives to the parsers available for a given media type.

Specify parser for config source:

Config config = Config.create(classpath("props")1
                                      .parser(ConfigParsers.properties()));2

Parsing a Config Value as Config

A config value node might contain an entire config document in String form, but in a format different from the containing document. Your application can tell the config system to parse such a node as config in a different format and replace the String value node in the original tree with the config tree that results from parsing that String.

In this example, a YAML document contains a JSON document as a leaf.

YAML file with included JSON formatted property:

secrets:
    username: "jose"
    password: "^ery$ecretP&ssword"

app: >1
    {
        "greeting": "Hello",
        "page-size": 20,
        "basic-range": [ -20, 20 ]
    }

Specify Key-to-media-type Mapping

Specify JSON as media type for node:

Config config = Config.create(
    classpath("application.yaml")
        .mediaTypeMapping(key -> {1
            return "app".equals(key.toString())2
                ? Optional.of(MediaTypes.APPLICATION_JSON)
                : Optional.empty();
        }));

assert config.get("secrets.username").asString()3
        .get().equals("jose");
assert config.get("secrets.password").asString()3
        .get().equals("^ery$ecretP&ssword");

assert config.get("app").type() == Type.OBJECT;4

assert config.get("app.greeting")3
        .asString().get().equals("Hello");
assert config.get("app.page-size")3
               .asInt().get() == 20;
assert config.get("app.basic-range.0")3
               .asInt().get() == -20;
assert config.get("app.basic-range.1")3
               .asInt().get() == 20;

Because the function passed to mediaTypeMapping identifies the app node as a JSON document, the config system selects the config parser that is registered with the builder which also handles the JSON media type.

Also, note that the config system replaces the original String value node with an object node resulting from parsing that String value as JSON.

Specify Key-to-parser Mapping

Alternatively, your application could map config keys to the specific parsers you want to use for parsing those keys' values.

Specify JSON formatted property' parser instance:

Config config = Config.create(
    classpath("application.yaml")
        .parserMapping(key -> {1
            return "app".equals(key.toString()) ?2
                Optional.of(HoconConfigParser.create())
                : Optional.empty();
        }));

As before, the config system replaces the value node in the containing config tree with the config tree resulting from the additional parse.

Config Keys with . in name

As described in the hierarchical features section each config node (except the root) has a non-null key.

For example, the following configuration file contains two object nodes with names oracle and oracle.com.

Example application.json with dot character in key:

{
    "oracle" : {
        "com" : true,
        "cz" : false
    },
    "oracle.com" : {
        "secured" : true
    }
}

Working with configuration with dot character in node name:

Config config = Config.create(classpath("application.json"));

// node `oracle`
assert config.get("oracle.com").asBoolean().get() == true;1
assert config.get("oracle").get("com").asBoolean().get() == true;1
assert config.get("oracle.com").type() == Type.VALUE;2
assert config.get("oracle.com").name().equals("com");3
// node `oracle.com`
assert config.get("oracle~1com.secured").asBoolean().get() == true;4
assert config.get(Key.escapeName("oracle.com"))5
               .get("secured").asBoolean().get() == true;
assert config.get(Key.escapeName("oracle.com")).type() == Type.OBJECT;6
assert config.get(Key.escapeName("oracle.com")).name().equals("oracle.com");7

Filters, Overrides, and Token Substitution

When your application retrieves a config value, the config system can transform it before returning the value, according to filters, overrides, and tokens. The config system provides some built-in instances of these you can use, and you can add your own as described in the sections which describe filters and overrides.

Your application can add filters and overrides explicitly to a config builder and the config system by default uses the Java service loader mechanism to locate all available filters and overrides and add them automatically to all config builders (unless your code disables that behavior for a given builder).

Filters

Each filter accepts a key and the value as defined in the source, and returns the value to be used. The filter can leave the value unchanged or alter it, as it sees fit.

The built-in value-resolving filter enables the token substitution described below.

See the ConfigFilter Javadoc for more information.

Overrides

The overrides feature allows you to create an external document containing key/value pairs which replace the value otherwise returned for the name, and then add that document as an override source to a config builder.

There are some key differences between overrides and filters.

• Because overrides are loaded from sources those sources can change while your application runs and so the overrides they that prescribe can change.
• The override document can use wildcards in key expressions.
• Overrides can affect only keys that already exist in the original source; filters can supply values even if the key is absent from the config source.

Each override entry consists of a Java properties-format definition. The key is an expression (which can use wildcards) to match config keys read from the current config sources, and the override value is the new value for any key matching the key expression from that entry. Order is important. The config system tests every key expression/value pair one by one in the order they appear in the overrides sources. Once the config system finds an override entry in which the key expression matches the configuration key, the system returns that entry’s value for the key being processed.

See the

OverrideSource Javadoc for more detail.

Tokens

A token reference is a key token starting with $, optionally enclosed between { and }, i.e. $ref or ${ref}. Even a key composed of more than one token can be referenced in another key, i.e. ${env.ref}.

As an example use case, you can use token references to declare the default values (see resolving-tokens.yaml below), while the references may be resolved in another config source, which identifies a current environment (see env.yaml examples below). You can then use the same overrides for different environments, say test and prod. The configuration in each environment is then overridden with a different values using wildcards (see overrides.properties below).

Initialize Config with Override Definition from overrides.properties file:

Config config = Config.builder()
    .overrides(OverrideSources.file("conf/overrides.properties"))1
    .sources(file("conf/env.yaml"),2
         classpath("resolving-tokens.yaml"))3
    .build();

You can disable key and value token replacement separately as the following example shows.

Disabling Key and Value Token Replacement:

Config config = Config.builder()
    .disableKeyResolving()
    .disableValueResolving()
    // ...
    .build();

Executors for Asynchronous Config Activity

Various parts of the config system work asynchronously:

• polling strategies to detect changes to config sources,
• publishers to notify your application when such changes occur,
• Config instances which subscribe to and respond to change notifications for their underlying sources, and
• retry policies (which might wait between retries).

Each of these uses an executor to perform its work. The config system provides default executors, but your application can specify different ones if necessary.

Executors for Polling Strategy

The method PollingStrategies.regular(Duration) returns builder for polling strategy. This builder provides executor method which your application can invoke, passing a java.util.concurrent.ScheduledExecutorService instance it requires for the polling work. By default, each polling strategy instance uses a separate thread pool executor.

The following example shares the same executor for two different polling strategy instances.

Customize polling strategy executors:

ScheduledExecutorService executor =
    Executors.newScheduledThreadPool(2);1

Config config = Config.create(
    file("conf/dev.properties")
        .pollingStrategy(
            PollingStrategies.regular(Duration.ofSeconds(2))2
                .executor(executor)),3
    file("conf/config.properties")
        .pollingStrategy(
            PollingStrategies.regular(Duration.ofSeconds(5))2
                .executor(executor)));4

Executors for Source Change Events

Recall that when a change watcher detects a change in a source, it informs interested parties of the changes. By default, each Config.Builder arranges for the resulting Config tree to use a shared executor that reuses available threads from a pool, creating new threads as needed. The same executor is used for actually reloading the source.

Your application can invoke the system watcher builder’s executor method to tell the builder to use a different Executor.

Customize config and override sources executors:

ScheduledExecutorService executor =
    Executors.newScheduledThreadPool(2);1

Config config = Config.builder()
    .overrides(OverrideSources
       .file("conf/overrides.properties")
       .changeWatcher(FileSystemWatcher.builder()
          .executor(executor)2
          .build()))
    .sources(file("conf/env.yaml")
    .changeWatcher(FileSystemWatcher.builder()
        .executor(executor)3
        .build()))
    .build();

Retry Policy Custom Executor

You can control which executor a retry policy should use for its work. The RetryPolicies.repeat(int retries) method returns a SimpleRetryPolicy.Builder. Your application can invoke the retry policy builder’s executorService method to specify which ScheduledExecutorService instance it should use to schedule and execute delayed retries. By default, the config system uses a separate thread pool executor for each retry policy instance.

Customize retry policy executors:

ScheduledExecutorService executor =
    Executors.newScheduledThreadPool(2, myThreadFactory);1

Config config = Config.create(
    file("conf/dev.properties")
        .optional()2
        .retryPolicy(RetryPolicies.repeat(2)3
             .executorService(executor)));4