DCX

Data Collection Extension service (DCX) extends the communication capability of Kaa Protocol (1/KP) by implementing Data Collection Protocol (2/DCP). DCX supports this extension protocol to receive endpoint data from a communication service and send it to data receiver services for storage and/or processing.

Interfaces

DCX supports a number of interfaces to perform its functional role. The key supported interfaces are summarized in the following diagram.

DCX interfaces diagram

For inter-service communication, Kaa services mainly use REST APIs and messaging protocols that run over NATS messaging system.

Extension service protocol

DCX implements 2/DCP extension. It uses 4/ESP over NATS to exchange messages with endpoints via communication service.

DCX goes further and expands 2/DCP by consuming single data samples not enclosed into a batching array. When an endpoint sends a single data sample in the data collection payload, it can be sent as follows:

{
  "timestamp":1583422147639
  "temperature":23,
  "humidity":61
}

instead of

[
  {
    "timestamp":1583422147639,
    "temperature":23,
    "humidity":61
  }
]

DCX does not respond to communication service, unless:

  • request processing status is requested by setting non-empty requestID in the Client Data message;
  • DCX encounters a message processing error.

Plain data handling

DCX consumes the plain text data on the separate /plain/${metric-name} extension-specific resource path. The ${metric-name} should be any string that describes the published data. A supported payload is a number or a number with a unit without any delimiters, e.g. 47 or 47%. DCX converts the incoming plain data into the data sample like below.

{
  "${metric-name}-value":${any-number},
  "${metric-name}-unit":"${any-string}"
}

Let’s go through the examples.

  • Value without unit. DCX receives the plain data 2061 with the resource path /plain/altitude. As a data unit the default number will be used. The resulted data sample will be:

    {
      "altitude-value":2061,
      "altitude-unit":"number"
    }
    
  • Value with unit. DCX receives the plain data 2061m with the resource path /plain/altitude. The resulted data sample will be:

    {
      "altitude-value":2061,
      "altitude-unit":"m"
    }
    

Data sample transmission protocol

DCX acts as a data transmitter service in line with 13/DSTP. All received data samples are transferred to the configured data receiver services using 13/DSTP.

Examples of data receiver services in Kaa: Endpoint Time Series service, Kafka Data Collection Adapter.

You can configure Kaa to simultaneously send data to multiple receivers, including your custom ones.

DCX supports receiving acknowledgements from reliable receivers. When such receivers are configured, DCX waits for each of them to confirm a successful receipt of endpoint data before responding back to the communication service.

Enriching data samples with endpoint metadata

DCX can enrich received data samples with a corresponding endpoint metadata. To find out how to enable this function, see the configuration.

Let’s consider an example.

  1. DCX receives the following Client Data message with two data samples:
[
  {
    "temperature":23,
    "humidity":60,
    "co2":870,
    "timestamp":1583422087639
  },
  {
    "temperature":23,
    "humidity":61,
    "co2":865,
    "timestamp":1583422147639
  }
]
  1. Endpoint metadata stored in the Endpoint Register looks like this:
{
  "c02_sensor":{
    "model":"Senseair S8 LP 004-0-0053",
    "accuracy":"± 40 ppm ± 3% of reading"
  },
  "temperature_humidity_sensor":{
    "model":"BME280"
  },
  "location":"kitchen"
}
  1. The output data samples enriched with EP metadata using the default configuration will look like this:
[
  {
    "temperature":23,
    "humidity":60,
    "co2":870,
    "timestamp":1583422087639,
    "~ep-metadata":{
      "c02_sensor":{
        "model":"Senseair S8 LP 004-0-0053",
        "accuracy":"± 40 ppm ± 3% of reading"
      },
      "temperature_humidity_sensor":{
        "model":"BME280"
      },
      "location":"kitchen"
    }
  },
  {
    "temperature":23,
    "humidity":61,
    "co2":865,
    "timestamp":1583422147639,
    "~ep-metadata":{
      "c02_sensor":{
        "model":"Senseair S8 LP 004-0-0053",
        "accuracy":"± 40 ppm ± 3% of reading"
      },
      "temperature_humidity_sensor":{
        "model":"BME280"
      },
      "location":"kitchen"
    }
  }
]

NOTE: Only data samples that are JSON objects can be enriched with an EP metadata.

Endpoint Metadata Management Protocol

DCX acts as an endpoint metadata management client in line with the 19/EPMMP. This interface is used for retrieving EP metadata when the endpoint metadata enrichment feature is enabled. DCX caches endpoint metadata in Redis.

Endpoint Metadata Events

DCX listens to 15/EME protocol events to refresh endpoint metadata cached in Redis.

Tekton integration

DCX is integrated with the Kaa Tekton for centralized application configuration management. It receives configuration update messages from Tekton over 17/SCMP and uses Tekton REST API to retrieve current configs.

See configuration for more information.

Kaa Tenant Manager integration

DCX supports multi-tenancy with each tenant using a separate OAuth 2.0 issuer for authentication, authorization, and resource management. The list of the existing tenants is managed by the Kaa Tenant Manager, which provides REST API for retrieving tenant security configs.

See the security configuration for more details on how to enable multi-tenancy in DCX.

Management interface

DCX exposes an HTTP-based management interface with the following endpoints:

  • GET /health returns 200 OK if the service is up and running properly, and 500 Internal Server Error otherwise. In case of errors, the response payload contains their human-redable descriptions. This endpoint can be used by Kubernetes for liveness and readiness probing.
  • GET /metrics provides service metrics in Prometheus text-based format.