Configuration

• Service configuration structure
  • General
  • Kaa applications
  • Time series configuration
  • Time series definition
  • Time series extraction
    • Timestamp extraction
  • Data samples receiver
  • Time series receiver
  • Data persistence interface
  • NATS
  • Authentication and authorization
  • Management
  • Logging
  • Default

The EPTS loads configuration data from a file (typically backed by a Kubernetes config map). The location of the configuration file is sourced from an environment variable. If the service is unable to retrieve the configuration data, it will not start. Refer to the [configuration][service configuration] for details.

Service configuration structure

The recommended configuration format is YAML. The below sections describe the officially supported configuration options that influence various aspects of the service functionality. The service may support options other than the ones listed below, but those are not a part of the public API and may be changed or deleted at any time.

General

EPTS uses its service [instance name and replica ID][scalability] for communication with other services and load balancing across replicas (for example, see [3/ISM][3/ISM]). You can control the values with the following configuration options:

service:
  instance:
    name: <string>          # Service instance name, should be unique cluster-wide. "epts" by default.
    replica.id: <string>    # Service instance replica ID, should be unique cluster-wide. Random string by default.
  graceful-shutdown: <uint> # Allowed service graceful shutdown period in seconds. 30 seconds by default.

The REST API server is controlled with the following options.

server:
  port: <integer>           # Server port used to expose the REST API at, 80 by default.

Kaa applications

Many Kaa services can be configured for different behavior depending on the application version of the endpoint the processed data relates to. This is called application-specific behavior and is handled in service configurations under kaa.applications.

kaa:
  applications:
    <application 1 name>:
      versions:
        <application 1 version 1 name>:
        <application 1 version 2 name>:  # Multiple application versions can be configured under the "versions" key
          ...
    <application 2 name>:                # Multiple applications can be configured under the "applications" key
      ...

For example:

kaa:
  applications:
    smartKettle:
      versions:
        kettle_v1:
        kettle_v2:
        kettle_v3:

Time series configuration

Time series in EPTS are defined within the scope of their applications.

The configuration consists of two logical parts:

• time series definition specifies the structure of a given time series: which values exist and what their types are;
• time series extraction configuration specifies the rules for extracting time series data points from data samples received from endpoints.

Both time series definition and extraction configuration can be specified within an application. However, only the extraction configuration can be overridden in an application version. The time series definition is the same application-wide for consistency and convenience reasons. If you need a change to the time series structure in a certain application version, that deserves a new time series name.

The full structure of the time series configuration is shown below. We will review it in more detail in the following subsections.

kaa:
  applications:
    <application name>:
      time-series:
        auto-extract: <boolean>                   # Whether to auto-extract all numeric values from data sample. `false` by default.
        timestamp:
          path: <gjson-path>                      # Path of the timestamp field in data samples. Optional.
          format: <timestamp-format>              # Format of the timestamp in data sample. "iso8601" by default.
          fallback-strategy: <fallback-strategy>  # Fallback strategy to use. `fail` by default.
        names:
          <time series name>:
            values:
            - name: <string>          # Time series value name. "value" by default.
              type: <string>          # Time series value type. "number" by default.
              path: <gjson-path>      # Path to the value in the incoming data sample. Same as `name` by default.
      versions:
        <application version name>:
          time-series:                # Time series extraction configuration overridde for an application version.
            auto-extract: <boolean>
            timestamp:
              path: <gjson-path>
              format: <timestamp-format>
              fallback-strategy: <fallback-strategy>
            names:
              <time series name>:     # One of time series names defined at the application level.
                values:
                - name: <string>      # Name of a time series value field. Must be defined at the application level.
                  path: <gjson-path>  # Path to the value in the incoming data sample for the given application version. Required.

Time series definition

Time series structure is defined once per application. The configuration fields that contribute to the definition are:

• kaa.applications.<application name>.time-series.names.<time series name>—the time series name. Must be unique application-wide. Case-insensitive; EPTS will automatically convert configured time series name to lowercase. It is recommended to only use lowercase latin letters and underscore sign _ in time series names (e.g. temperature, ground_speed, etc.).
• kaa.applications.<application name>.time-series.names.<time series name>.values—list of time series values.
• kaa.applications.<application name>.time-series.names.<time series name>.values[].name—time series value name. Must be unique time series-wide. Defaults to “value”. Names “time” and “kaaEndpointID” are reserved and cannot be used.
• kaa.applications.<application name>.time-series.names.<time series name>.values[].type—time series value type. Could be one of “number”, “string”, or “bool”, where “number” is stored as a 64-bit float.

Time series extraction

Upon receiving data samples, EPTS extracts data points according to the time series extraction configuration. Base configuration is defined application-wide and is applied by default, unless you override it for a specific application version. The configuration fields include:

• kaa.applications.<application name>.time-series.auto-extract—enables time series auto-extraction from data samples. If set to true, EPTS will automatically create a time series for each numeric fieled in the data sample JSON (top-level only). This setting can be overriden for an application version using kaa.applications.<application name>.versions.<application version name>.time-series.auto-extract option.
• kaa.applications.<application name>.time-series.names.<time series name>.values[].path—path to the value in the data sample. EPTS uses GJSON for extracting time series values, so you can use the path syntax specified by GJSON. This setting can be overriden for an application version using kaa.applications.<application name>.versions.<application version name>.time-series.names.<time series name>.values[].path option.
• timestamp extraction configuration described in a separate subsection below.

If the processed data sample does not contain a field specified by the path, or the field type does not match the time series specification, EPTS concludes that the data sample does not contain the given time series data and skips the extraction of a data point.

Timestamp extraction

Every time series data point must contain a timestamp. EPTS supports extracting timestamps from received data samples or using the server receipt time. The timestamp extraction can be both defined at the application level under kaa.applications.<application name>.time-series.timestamp, and overriden for an application version under kaa.applications.<application name>.versions.<application version name>.time-series.timestamp. The following fields are supported:

• path—path to the timestamp field in the data sample. Same as with the time series values path, GJSON path syntax is supported. If path is not specified, EPTS does not attempt to extract a timestamp from the data sample, and uses the server receipt time stamp instead.
• format—field format to use for parsing the timestamp located at path. Only used when the path is set. Supported formats include:
  • iso8601—ISO 8601 (default);
  • millis-unix-epoch—integer milliseconds elapsed since Unix epoch;
  • sec-unix-epoch—integer seconds elapsed since Unix epoch.
• fallback-strategy—Defines EPTS behavior when the timestamp parsing fails for any reason (missing field, wrong format). Supported strategies are:
  • server-timestamp—fallback to using the server receipt timestamp;
  • fail—fail the data sample processing and respond back to the DSTP transmitter with an error (default).

Data samples receiver

Use the following options to configure the data samples receiver interface.

kaa:
  dstp.receiver:
    from: <string>        # Name of the data samples transmission service instance EPTS will subscribe to.
    concurrency: <uint>   # The maximum number of concurrent workers, which will consume 13/DSTP messages from message pool. 64 by default.
    queue-length: <uint>  # The maximum amount of messages in the DSTP receiver queue. When the queue is full, new messages will be dropped. 16 * `kaa.dstp.receiver.concurrency` by default.

Time series receiver

Use the following options to configure the time series receiver interface that EPTS uses for time series consumption. You can configure subscription to multiple transmitters and either define specific time series to listen to, or use a “*” wildcard.

kaa:
  tstp.receiver:
    concurrency: <uint>               # The maximum number of concurrent workers, which will consume 14/TSTP messages from message pool. 64 by default.
    queue-length: <uint>              # The maximum amount of messages in the TSTP receiver queue. When the queue is full, new messages will be dropped. 16 * `kaa.tstp.receiver.concurrency` by default.
    from:                             # Map of time series transmission service instance EPTS will subscribe to.
      <transmission service instance name>:
        time-series: <list of string> # Time series names that EPTS will consume. Optional, "*" by default.
      ...

Data persistence interface

EPTS uses InfluxDB for persisting time series data.

kaa:
  influx:
    precision: <string>       # The data points timestamp precision in InfluxDB.
                              # All timestamps stored in the InfluxDB are truncated to the given precision.
                              # `h` (hours), `m` (minutes), `s` (seconds), `ms` (milliseconds), `u` (microseconds), `ns` (nanoseconds).
                              # Defaults to `ns`.

    url: <string>             # InfluxDB URL
    user: <string>            # InfluxDB user (see note below)
    password: <string>        # InfluxDB password (see note below)
    idle-connections: <uint>  # Maximum idle connections in the InfluxDB client HTTP transport pool. 128 by default.
    read:
      chunk.size: <uint>      # InfluxDB request chunk size (number of time series to write to or getting from InfluxDB). 10000 by default.
    write:
      concurrency: <uint>     # The maximum number of concurrent InfluxDB writes; 8 by default.
      queue.size: <uint>      # The maximum length of the DB write queue per application.
                              # Each item in the queue is based on a received DSTP or TSTP message. 1024 by default.

      batch.size: <uint>      # The optimal desired write batch size in data points. 10000 by default.
                              # EPTS uses this configuration option as a recommendation, not as a strict rule.
                              # It will attempt to collect write batches as close to this number as possible without sacrificing the write latency.
                              # Some writes may exceed the configured batch size, which is by design.

      timeout: <uint>         # Write timeout for InfluxDB requests (in ms). Default 5000 ms.

NOTE: For security reasons, username and password must be sourced from the environment variables.

NATS

The below parameters configure EPTS’s connection to NATS. Note that for security reasons NATS username and password are sourced from the environment variables.

nats:
  urls: <comma separated list of URL>  # NATS connection URLs.

Authentication and authorization

EPTS’s REST API security is implemented according to [OAuth2][oauth2] protocol with a UMA profile.

EPTS REST API protection is controlled with the following configuration options:

kaa:
  security:
    enabled: <boolean>      # Controls authentication and authorization on REST API endpoints. False by default.
    issuer: <string>        # URL of the AAA service. Also can be set with KAA_SECURITY_ISSUER environment variable.
    clientID: <string>      # Client ID for making requests to the AAA service. Also can be set with KAA_SECURITY_CLIENTID environment variable.
    clientSecret: <string>  # Client secret for making requests to the AAA service. Also can be set with KAA_SECURITY_CLIENTSECRET environment variable.

Management

To control the EPTS management interface, use the following configuration options.

service.monitoring:
  disabled: <boolean>   # Disables the monitoring interface entirely. False by default (enabled).
  port: <uint>          # TCP port to expose the monitoring server on. 8080 by default.

Logging

EPTS writes logs to stdout. By default, the service only produces logs at startup, shutdown, or in case of errors. To enable debug level logging, use the following option:

service:
  debug: <boolean>    # Enables debug level logging, false by default (disabled).

Keep in mind that enabling debug level logging will produce significantly more log output and degrade the service performance.

Default

Summarizing the above, the default EPTS configuration is as follows. Note that no Kaa applications are defined by default—you have to configure those for any specific Kaa-based solution.

service:
  instance:
    name: "epts"
    replica.id: "<random string generated on boot>"
  monitoring:
    disabled: false
    port: 8080
  debug: false
kaa:
  influx:
    precision: "ns"
    write.timeout: 5000
    read.chunk.size: 10000
    idle-connections: 128