Deployment

All Kaa services, including OTAO, are distributed as Helm charts. You can run these charts using Kubernetes.

Installing OTAO chart on Kubernetes

Preparation

These steps should be done once for your entire Kaa cluster in Kubernetes.

  1. Install Kubernetes.

  2. Install Helm client and Tiller server.

  3. Create a Kaa license secret (remember to put in your Kaa license key file contents and password):

     export HISTCONTROL=ignorespace # Prevent saving your key password in the shell history; note the leading space in the next line
      cat << EOF > /tmp/kaa-licence.yaml
     apiVersion: v1
     data:
       file: < your licence key file contents, base64-encoded >
       password: < your licence key password >
     kind: Secret
     metadata:
       name: license
       type: Opaque
     EOF
     kubectl create -f /tmp/kaa-licence.yaml
    
  4. Specify the image pull secret for the official KaaIoT docker registry. To define this secret, use your KaaID credentials:

     export HISTCONTROL=ignorespace  # Prevent saving your credentials in the shell history; note the leading space in the next line
      export KAAID_EMAIL=<your KaaID email, eg. bob@example.com> KAAID_PASSWORD=<your KaaID password>
     kubectl create secret docker-registry kaaid --docker-server=dev-hub.kaaiot.net --docker-username=$KAAID_EMAIL --docker-email=$KAAID_EMAIL --docker-password=$KAAID_PASSWORD
    
  5. If you want to enable API security in your Kaa cluster (recommended), you should create an auth server backend secret for the service to be able to request PAT tokens.

     export HISTCONTROL=ignorespace # Prevent saving your client ID amd secret in the shell history; note the leading space in the next line
      cat << EOF > /tmp/keycloak-backend-secret.yaml
     apiVersion: v1
     data:
       client-id: < your client-id, base64-encoded >
       client-secret: < your client-secret, base64-encoded >
     kind: Secret
     metadata:
       name: "keycloak-backend"
       labels:
         app.kubernetes.io/name: "kaa-name"
         helm.sh/chart: "kaa-chart-version"
         app.kubernetes.io/instance: "kaa-instance-name"
         app.kubernetes.io/managed-by: "release-service-name"
     EOF
     kubectl create -f /tmp/keycloak-backend-secret.yaml
    
  6. Add the KaaIoT Helm repository:

     helm repo add kaa-museum https://museum.kaaiot.net/
    

Installation

Once you have completed the preparation steps, everything is ready for deploying OTAO on your Kubernetes cluster. To deploy the service, run the following command (observe the reference to the previously created license secret):

helm install --set global.license.secretName=license kaa-museum/otao --name kaa-otao

Check that pods are running:

kubectl get pods

Once the service initialization is complete, you should observe the output similar to the below:

NAME                                          READY   STATUS             RESTARTS   AGE
...
kaa-otao                       1/1     Running            0          2m
...

Chart Requirements

Repository Name Version
@bitnami-pre-2022 mariadb 9.0.1
@kaa service-chart 0.0.88

Chart Values

Key Type Default Description
affinity object {}  
annotations.deployment object {}  
annotations.pod object {}  
aws.useSecret bool false Enables usage of global.s3.existingSecret AWS secret.
config string "" Content for the service config map, automatically mounted as a config file into the pod.
env object {"JAVA_OPTS":{"value":"-XX:MaxRAMPercentage=80.0 -XX:MinRAMPercentage=50.0 -Xverify:none -XX:TieredStopAtLevel=1"}} Defines the environment variables that Kubernetes passes to the service replica.
extraPodSpecs.automountServiceAccountToken bool false  
fullnameOverride string ""  
global.iamcore.enabled bool false  
global.image.pullSecrets list [] List of image pull secret names. Each must be defined as a record with the name field. Overrides image.pullSecrets.
global.kaaIngress.tls.issuerKind string "Issuer"  
global.kaaIngress.tls.issuerName string "letsencrypt-prod"  
global.kaaIngress.tls.selfSigned bool false  
global.keycloak.backend.existingSecret string "-keycloak-backend"  
global.keycloak.enabled string "" Enables API security using the auth provider. Overrides keycloak.enabled.
global.keycloak.privateUrl string ""  
global.keycloak.publicUrl string ""  
global.license.existingSecret string "" Name of the license secret, which must contain two base64-encoded fields: file (license file contents in PKCS #12) and password. Overrides license.secretName.
global.mariadb.existingSecret string "" Name of an existing Kubernetes secret containing MariaDB admin and user passwords (mariadb-root-password and mariadb-password secret keys, respectively).
global.mariadb.url string "" Mariadb URL.
global.monitoring.enabled bool false  
global.nats.url string "" NATS URL.
global.s3.existingSecret string "" Name of the AWS S3 secret, which must contain three base64-encoded fields: awsAccessKeyId, awsAccessKey and awsDefaultRegion.
global.tekton.enabled string "" Enables Tekton integration. When disabled, the service will expect Kaa application configs to be defined in the config map. Overrides tekton.enabled.
global.tekton.url string "" Tekton URL. Overrides tekton.url.
global.tenantManager.baseUrl string "" Tenant manager base URL.
global.tenantManager.enabled string "" Enables multitenancy using the Kaa Tenant Manager. Overrides tenantManager.enabled.
image.pullPolicy string "IfNotPresent" Docker image pull policy.
image.pullSecrets list [] List of image pull secret names. Each must be defined as a record with the name field.
image.repository string "dev-hub.kaaiot.net/core/service//" Docker image repository image URL.
image.tag string "" Docker image tag version to pull and run.
ingress.annotations object {}  
ingress.enabled bool false  
ingress.hosts[0] string "chart-example.local"  
ingress.paths list []  
ingress.tls list []  
mariadb.auth.database string "otao"  
mariadb.auth.existingSecret string "true"  
mariadb.auth.username string "kaa"  
mariadb.enabled bool true Enabled MariaDB dependency. For detailed info refer https://github.com/helm/charts/tree/master/stable/mariadb.
mariadb.init.extraSteps[0] string "mariadb -e \"ALTER DATABASE $NEW_DB_NAME CHARSET utf8mb3 COLLATE = 'utf8mb3_general_ci'\""  
mariadb.primary.persistence.enabled bool true  
mariadb.url string "mariadb://-:3306/" MariaDB URL.
mariadb.urlOverride string "" MariaDB URL. Overrides global.mariadb.url and mariadb.url
metadata.component string "backend"  
metadata.partOf string "kaa"  
monitoring.metrics.export.path string "/prometheus" Specifies REST resource path that exposes metrics.
monitoring.metrics.type string "java"  
monitoring.rules[0].alert string "OTAO connection pool has totally more than 3 leased connections"  
monitoring.rules[0].annotations.message string "Got more then 3 leased pool's connections error on the pod [/]."  
monitoring.rules[0].annotations.runbook string ""  
monitoring.rules[0].expr string "sum(httpcomponents_httpclient_pool_total_connections{httpclient=\"rest-client-connection-pool\", service=~\".*-otao\", state=\"leased\"}) > 3"  
monitoring.rules[0].for string "1m"  
monitoring.rules[0].labels.severity string "error"  
monitoring.rules[1].alert string "OTAO connection pool has more than 1 pending connection"  
monitoring.rules[1].annotations.message string "Got more then 1 pending pool's connection error on the pod [/]."  
monitoring.rules[1].annotations.runbook string ""  
monitoring.rules[1].expr string "sum(httpcomponents_httpclient_pool_total_pending{httpclient=\"rest-client-connection-pool\", service=~\".*-otao\"}) > 1"  
monitoring.rules[1].for string "1m"  
monitoring.rules[1].labels.severity string "error"  
monitoring.rules[2].alert string "OTAO connection pool used all available connections"  
monitoring.rules[2].annotations.message string "Got more then 1 pending pool's connection error on the pod [/]."  
monitoring.rules[2].annotations.runbook string ""  
monitoring.rules[2].expr string "httpclient_pool_connections_per_route_available{host=~\".*keycloak.*\", service=~\".*-otao\"} >= httpcomponents_httpclient_pool_route_max_default{httpclient=\"rest-client-connection-pool\", service=~\".*-otao\"}"  
monitoring.rules[2].for string "1m"  
monitoring.rules[2].labels.severity string "error"  
monitoring.rules[3].alert string "OTAO connection pool has totally more than 75% available connections"  
monitoring.rules[3].annotations.message string "Got more then 75% available pool's connections error on the pod [/]."  
monitoring.rules[3].annotations.runbook string ""  
monitoring.rules[3].expr string "sum(httpcomponents_httpclient_pool_total_connections{httpclient=\"rest-client-connection-pool\", service=~\".*-otao\", state=\"available\"}) > (0.75 * sum(httpcomponents_httpclient_pool_total_max{httpclient=\"rest-client-connection-pool\", service=~\".*-otao\"}))"  
monitoring.rules[3].for string "1m"  
monitoring.rules[3].labels.severity string "error"  
monitoring.rules[4].alert string "OTAO connection pool has routes with half leased connections"  
monitoring.rules[4].annotations.message string "Got warning - appeared connection pool's routes with half leased connections on the pod [/]."  
monitoring.rules[4].annotations.runbook string ""  
monitoring.rules[4].expr string "sum(httpclient_pool_routes_with_half_leased_connections{service=~\".*-otao\"}) > 0"  
monitoring.rules[4].for string "1m"  
monitoring.rules[4].labels.severity string "warning"  
monitoring.rules[5].alert string "OTAO connection pool has routes with all leased connections"  
monitoring.rules[5].annotations.message string "Got warning - appeared connection pool's routes with all leased connections on the pod [/]."  
monitoring.rules[5].annotations.runbook string ""  
monitoring.rules[5].expr string "sum(httpclient_pool_routes_with_max_leased_connections{service=~\".*-otao\"}) > 0"  
monitoring.rules[5].for string "1m"  
monitoring.rules[5].labels.severity string "error"  
monitoring.rules[6].alert string "OTAO connection pool has routes with pending connections"  
monitoring.rules[6].annotations.message string "Got warning - appeared connection pool's routes with pending connections on the pod [/]."  
monitoring.rules[6].annotations.runbook string ""  
monitoring.rules[6].expr string "sum(httpclient_pool_routes_with_pending_connections{service=~\".*-otao\"}) > 0"  
monitoring.rules[6].for string "1m"  
monitoring.rules[6].labels.severity string "error"  
nameOverride string ""  
nodeSelector object {}  
probes.enabled bool true Enables liveness, readiness, and startup probes for containers.
probes.liveness.initialDelaySeconds int 300  
probes.liveness.periodSeconds int 3  
probes.readiness.initialDelaySeconds int 10  
probes.readiness.periodSeconds int 5  
replicaCount int 1 The number of service instance replicas to run.
resources.limits.cpu int 1  
resources.limits.memory string "700Mi"  
resources.requests.cpu string "100m"  
resources.requests.memory string "700Mi"  
runbookUrl string ""  
securityContext object {}  
service.externalIPs list []  
service.loadBalancerIP string ""  
service.port int 80  
service.type string "ClusterIP"  
software.file.storage.s3.bucketName string "" The AWS bucket name that software files will be stored in. Required if the software file storage feature is enabled.
software.file.storage.s3.enabled bool false Enables software files storage backed by AWS S3.
terminationMessagePolicy string "FallbackToLogsOnError" Kubernetes termination message policy.
tolerations list []  
updateStrategy.type string "RollingUpdate" Deployment update strategy.
waitContainers.curl.image string "dev-hub.kaaiot.net/devops/docker/alpine-curl-jq/alpine-curl-jq"  
waitContainers.curl.tag string "0.0.3"  
waitContainers.enabled bool true Wait for dependency services.
waitContainers.timeout int 300  

The keys with no description are standard Kubernetes values. Refer to the official Kubernetes documentation for more information on these.

Environment variables

The table below summarizes the variables supported by the OTAO Docker image and provides default values along with descriptions.

Variable name Default value Description
INSTANCE_NAME otao Service instance name.
APP_CONFIG_PATH /srv/otao/service-config.yml Path to the service configuration YAML file inside container. In case of running in Kubernetes, consider using K8s Volumes for externalization.
KAA_TEKTON_ENABLED false Enables Tekton integration.
KAA_TEKTON_URL http://tekton URL of the Tekton service.
KAA_SECURITY_ENABLED false Enables authentication and authorization on REST API endpoints (inbound and outbound API calls).
KAA_SECURITY_ISSUER_PUBLIC_URL   OAuth 2.0 issuer public URL for the system tenant (“kaa”).
KAA_SECURITY_ISSUER_PRIVATE_URL   OAuth 2.0 issuer private URL for the system tenant (“kaa”).
KAA_SECURITY_CLIENT_ID   Client ID for making requests in the system tenant scope.
KAA_SECURITY_CLIENT_SECRET   Client secret for making requests in the system tenant scope.
KAA_SECURITY_IAMCORE_ENABLED false Enables iamcore integration.
KAA_SECURITY_IAMCORE_API_KEY   iamcore API KEY.
KAA_SECURITY_IAMCORE_URL "https://cloud.iamcore.io" iamcore server URL.
KAA_SECURITY_IAMCORE_APPLICATION "kaa" iamcore application.
KAA_SECURITY_MULTITENANCY_ENABLED false Enables multitenancy via integration with the Kaa Tenant Manager. Only effective when kaa.security.enabled is set to true.
KAA_SECURITY_MULTITENANCY_TENANT_MANAGER_URL "http://tenant-manager" URL of the Kaa Tenant Manager that provides security configurations for tenants.
JMX_ENABLE false Enables JMX monitoring.
JMX_PORT 10500 JMX service port.
JMX_MONITOR_USER_PASSWORD   JMX monitor user password. Required when JMX_ENABLE=true.
JAVA_OPTIONS -Xmx700m Additional parameters for Java process launch.
NATS_USERNAME   Username for connecting to NATS message broker.
NATS_PASSWORD   Password for connecting to NATS message broker.
MARIADB_USERNAME   Username for connecting to MariaDB.
MARIADB_PASSWORD   Password for connecting to MariaDB.
KAA_LICENSE_CERT_PATH /run/license/license.p12 Path to the Kaa platform license certificate file in PKCS #12 format.
KAA_LICENSE_CERT_PASSWORD   License certificate password. Required.