Skip to main content

Embedded Cluster Config

This topic is a reference for the Replicated Embedded Cluster Config custom resource. For more information about Embedded Cluster, see Using Embedded Cluster.

note

Embedded Cluster is in beta. If you are instead looking for information about creating Kubernetes Installers with Replicated kURL, see the Replicated kURL section.

Overview​

To install your application with Embedded Cluster, an Embedded Cluster Config must be created in a release. Embedded Cluster installation artifacts are available only for releases that include an Embedded Cluster Config.

The Embedded Cluster Config lets you define several aspects of the Kubernetes cluster that will be created.

Limitations​

  • The Embedded Cluster Config does not support the use of Go template functions, including KOTS template functions.

For additional property-specific limitations, see the sections below.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  version: 1.17.0+k8s-1.30
  roles:
    controller:
      name: management
      labels:
        management: "true"
    custom:
    - name: app
      labels:
       app: "true"
  extensions:
    helm:
      repositories:
        - name: ingress-nginx
          url: https://kubernetes.github.io/ingress-nginx
      charts:
        - name: ingress-nginx
          chartname: ingress-nginx/ingress-nginx
          namespace: ingress-nginx
          version: "4.8.3"
          values: |
            controller:
              service:
                type: NodePort
                nodePorts:
                  http: "80"
                  https: "443"
              # Known issue: Only use image tags for multi-architecture images.
              # Set digest to empty string to ensure the air gap builder uses
              # single-architecture images.
              image:
                digest: ""
                digestChroot: ""
              admissionWebhooks:
                patch:
                  image:
                    digest: ""

version​

You must specify which version of Embedded Cluster to install. Each version of Embedded Cluster includes particular versions of components like KOTS (Admin Console) and OpenEBS.

For a full list of versions, see the Embedded Cluster releases page in GitHub. It's recommended to keep this version as up to date as possible because Embedded Cluster is changing rapidly.

roles​

You can define node roles in the Embedded Cluster Config. In multi-node clusters, roles are used to determine which nodes run the Kubernetes control plane, and to assign application workloads to particular nodes. One or more roles can be selected and assigned to a node when it is joined to the cluster.

note

Roles are not updated or changed after a node is added. If you need to change a node’s role, reset the node and add it again.

controller​

Controller nodes run the Kubernetes control plane and can also run other workloads, such as application or Replicated KOTS workloads. For this reason, nodes with the controller role are considered controller workers.

All clusters require at least one node with the controller role because a cluster needs a control plane. The first node in a cluster always has the controller role.

For multi-node clusters with high availability, at least three controller nodes are required. If you use more than three controller nodes, it is recommended to always use an odd number of nodes. Using an odd number of nodes ensures that the cluster can make decisions efficiently with quorum calculations. Clusters with an odd number of nodes also avoid split-brain scenarios where the cluster runs as two, independent groups of nodes, resulting in inconsistencies and conflicts. For more information about highly-available clusters, see Options for Highly Available Topology in the Kubernetes documentation.

By default, the controller role is named “controller.” You can customize the name of the controller role with the spec.roles.controller.name field, as shown below:

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  roles:
    controller:
      name: management

custom​

You can optionally define custom roles for any non-controller, or worker, nodes in the cluster. Worker nodes can run any workloads that are deployed to the cluster by Embedded Cluster, such as application or Replicated KOTS workloads. Unlike controller nodes, worker nodes cannot run the Kubernetes control plane.

Custom roles are particularly useful when combined with labels for the purpose of assigning specific workloads to nodes.

Some example use cases for defining custom roles include:

  • Your application has a workload that must run on GPU, so you assign a custom role to a worker node that runs the GPU workload
  • Your application has a database that requires a lot of resources, so you assign a custom role to a worker node that is dedicated to running that workload only

Custom roles are defined with the spec.roles.custom array, as shown in the example below:

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  roles:
    custom:
    - name: app

labels​

Roles can have associated Kubernetes labels that are applied to any node in the cluster that is assigned that role. Labels are useful for tasks like assigning workloads to nodes.

Labels can be defined for the controller role and any custom roles, as shown in the example below:

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  roles:
    controller:
      name: management
      labels:
        management: "true" # Label applied to "management" nodes
    custom:
    - name: app
      labels:
        app: "true" # Label applied to "app" nodes

extensions​

If you need to install Helm charts before your application and as part of the Embedded Cluster itself, you can do this with Helm extensions. One situation where this is useful is if you want to ship an ingress controller, because Embedded Cluster does not yet include one.

Helm extensions are updated when new versions of your application are deployed from the Admin Console. So, for example, you can change the values for a Helm extension from one release to another, and those changes will be applied to the cluster when the new release is deployed.

The format for specifying Helm extensions uses the same k0s Helm extensions format from the k0s configuration. For more information about these fields, see the k0s documentation.

Limitation​

If a Helm extension is removed from the Embedded Cluster Config, the associated Helm chart is not removed from the cluster.

Requirements​

  • The version field is required. Failing to specify a chart version will cause problems for upgrades.

  • If you need to install multiple charts in a particular order, set the order field to a value greater than or equal to 10. Numbers below 10 are reserved for use by Embedded Cluster to deploy things like a storage provider and the Admin Console. If an order is not provided, Helm extensions are installed with order 10.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  extensions:
    helm:
      repositories:
        - name: ingress-nginx
          url: https://kubernetes.github.io/ingress-nginx
      charts:
        - name: ingress-nginx
          chartname: ingress-nginx/ingress-nginx
          namespace: ingress-nginx
          version: "4.8.3"
          values: |
            controller:
              service:
                type: NodePort
                nodePorts:
                  http: "80"
                  https: "443"
              # Known issue: Only use image tags for multi-architecture images.
              # Set digest to empty string to ensure the air gap builder uses
              # single-architecture images.
              image:
                digest: ""
                digestChroot: ""
              admissionWebhooks:
                patch:
                  image:
                    digest: ""

unsupportedOverrides​

important

This feature should be used with caution by advanced users who understand the risks and ramifications of changing the default configuration.

Unsupported overrides allow you to override Embedded Cluster's default configuration, including the k0s config and the Helm values for extensions like KOTS and OpenEBS. This should be used with caution because changes here are untested and can disrupt or break Embedded Clusters. Any issues that are caused by unsupported overrides are not supported.

While they should be used with caution, unsupported overrides are useful if you need to make changes that are not otherwise exposed by Embedded Cluster.

Override the k0s Config​

By default, Embedded Cluster uses a k0s config that is tested and known to work for Embedded Clusters. In some circumstances, you might want to change the k0s config.

For more information on the k0s config, see Configuration options in the k0s documentation.

For example, you can do the following to enable WireGuard-based encryption. Note that other configuration might be necessary. See spec.network.calico in the k0s documentation for more details.

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          network:
            calico:
              wireguard: true

Limtiations​

  • The spec.api and spec.storage keys in the k0s config cannot be changed after installation. Whereas most keys in the k0s config apply to the whole cluster, these two keys are set for each node. Embedded Cluster cannot update these keys on each individual node during updates, so they cannot be changed after installation.

  • Overrides overwrite the corresponding fields in the k0s configuration. They are not merged into Embedded Cluster’s default configuration. When using overrides to override a list, for example, ensure that you include other elements in the list that Embedded Cluster includes by default.

Override the Helm Values for Built-In Extensions​

Embedded Cluster deploys built-in extensions like KOTS and OpenEBS to provide capabilities like storage and application management. These extensions are deployed with Helm, and the Helm values for each can be modified if necessary.

To modify these values, you can use the unsupportedOverrides.builtInExtensions key of the Embedded Cluster Config. Each chart you want to modify is an item in the array. The name key identifies the Helm chart that you want to modify, and the values key is a string where you specify your modified Helm values. Your modified values are merged into the values used by Embedded Cluster.

The following are the built-in extensions available for modification:

  • openebs
  • admin-console
  • velero
  • embedded-cluster-operator

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    builtInExtensions:
    - name: openebs
      values: |
        key: value