Configuring and Customizing Application Images and Their Runtime Environment

Sealos provides a way to patch files at runtime using the Config object. This article will provide detailed instructions on how to do this.

Introduction to the Config Object

The Config is an object that describes how to patch, create, or replace specific files at runtime for a specific application image or all images in the runtime list.

Let's take the case where we need to use a cluster with node-local-dns.

Creating the Cloud Image for the node-local-dns Application

First, we create the cloud image for the node-local-dns application:

$ tree node-local-dns
node-local-dns
├── Kubefile
└── charts
    ├── node-local-dns-1.3.2.tgz
    └── node-local-dns.values.yaml
$ cat node-local-dns/Kubefile
FROM scratch
LABEL VERSION="1.3.2"
COPY . .
CMD ["helm -n kube-system upgrade --install -f charts/node-local-dns.values.yaml node-local-dns charts/node-local-dns-1.3.2.tgz"]
$ cat node-local-dns/charts/node-local-dns.values.yaml
image:
  repository: k8s.tencentcloudcr.com/dns/k8s-dns-node-cache
  pullPolicy: IfNotPresent
  tag: 1.17.4
$ cd node-local-dns && sudo sealos build -t node-local-dns:1.3.2 -f Kubefile .

The chart file can be downloaded from here.

Creating a Clusterfile

Next, we create a Clusterfile:

{{- $kubeDNS := ipAt (default "10.96.0.0/18" .Values.networking.serviceSubnet) 10 -}}

apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  name: default
spec:
  ...
  image:
  - labring/kubernetes:v1.23.8
  - labring/calico:v3.24.1
  - node-local-dns:1.3.2
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  dnsDomain: {{ default "cluster.local" .Values.networking.dnsDomain }}
  serviceSubnet: {{ default "10.96.0.0/18" .Values.networking.serviceSubnet }}
  podSubnet: {{ default "100.64.0.0/17" .Values.networking.podSubnet }}
---
apiVersion: kubelet.config.k8s.io/v1beta1
clusterDNS:
{{- if .Values.localDNS.enabled }}
  - {{ .Values.localDNS.bind }}
{{- else }}
  - {{ $kubeDNS }}
{{- end }}
kind: KubeletConfiguration
---
apiVersion: apps.sealos.io/v1beta1
kind: Config
metadata:
  name: node-local-dns
spec:
  path: charts/node-local-dns.values.yaml
  match: {{ .Values.localDNS.image }}
  strategy: merge
  data: |
    config:
      localDnsIp: {{ .Values.localDNS.bind }}
      zones:
        -
          zone: .:53
          plugins:
            errors: true
            reload: true
            debug: false
            log:
              format: combined
              classes

: all
            cache:
              parameters: 30
              denial:
                {}
                # size: 0
                # ttl: 1
              success:
                {}
                # size: 8192
                # ttl: 30
              prefetch:
                {}
                # amount: 1
                # duration: 10m
                # percentage: 20%
              serve_stale: false
            forward:
              parameters: /etc/resolv.conf
              force_tcp: false
              prefer_udp: false
              policy: "" # random|round_robin|sequential
              max_fails: "" # 10
              expire: "" # 10s
              health_check: "" # 10s
            prometheus: true
            health:
              port: 8080
        - zone: ip6.arpa:53
          plugins:
            errors: true
            reload: true
            debug: false
            log:
              format: combined
              classes: all
            cache:
              parameters: 30
            forward:
              parameters: {{ $kubeDNS }}
              force_tcp: true
            prometheus: true
            health:
              port: 8080
        - zone: in-addr.arpa:53
          plugins:
            errors: true
            reload: true
            debug: false
            log:
              format: combined
              classes: all
            cache:
              parameters: 30
            forward:
              parameters: {{ $kubeDNS }}
              force_tcp: true
            prometheus: true
            health:
              port: 8080
        - zone: {{ printf "%s:53" .Values.networking.dnsDomain }}
          plugins:
            errors: true
            reload: true
            debug: false
            log:
              format: combined
              classes: all
            cache:
              parameters: 30
              denial:
                size: 9984
                ttl: 5
              success:
                size: 9984
                ttl: 30
            forward:
              parameters: {{ $kubeDNS }} # defaults to /etc/resolv.conf
              force_tcp: true
            prometheus: true
            health:
              port: 8080
{{- end }

Using the Config Object

Because the localDNS field in the kubelet configuration needs to match the listening address of node-local-dns, it must be pre-defined in the Clusterfile. At the same time, we want to modify the configuration of node-local-dns flexibly, so we use Config to patch the charts/node-local-dns.values.yaml file before Sealos actually calls the helm command.

Here's a brief introduction to the fields of the Config object:

• metadata.name: This name cannot be duplicated with other names.
• spec.path: The file path in the application image.
• spec.match: Optional. When match is defined, Config is applied to the image that matches it; otherwise, it is applied to all images.
• spec.strategy: Can be merge, insert, append, or override.
  • merge: Only applicable to YAML/JSON files.
  • insert/append: Inserts data into the file.
  • override: Overrides the contents of the file.
• spec.data: The data to be applied.

Run Sealos apply:

sudo sealos apply -f Clusterfile --set localDNS.enabled=true --set localDNS.bind=169.254.20.11 --set localDNS.image=node-local-dns:1.3.2

More Use Cases

We can use this mechanism to customize pre-built images or customize the configuration of a specific component without rebuilding it. For example, adding custom containerd configuration.

Before the initial run, add the Config in the Clusterfile.

---
...
---
apiVersion: apps.sealos.io/v1beta1
kind: Config
metadata:
  name: containerd-config
spec:
  strategy: override
  # only applied to rootfs
  match: labring/kubernetes:v1.23.8
  path: etc/config.toml
  data: |
    version = 2
    root = "/var/lib/containerd"
    state = "/run/containerd"
    oom_score = 0

    [grpc]
      address = "/run/containerd/containerd.sock"
      uid = 0
      gid = 0
      max_recv_message_size = 16777216
      max_send_message_size = 16777216

    [debug]
      address = "/run/containerd/containerd-debug.sock"
      uid = 0
      gid = 0
      level = "warn"

    [timeouts]
      "io.containerd.timeout.shim.cleanup" = "5s"
      "io.containerd.timeout.shim.load" = "5s"
      "io.containerd.timeout.shim.shutdown" = "3s"
      "io.containerd.timeout.task.state" = "2s"

    [plugins]
      [plugins."io.containerd.grpc.v1.cri"]
        sandbox_image = "sealos.hub:5000/pause:3.6"
        max_container_log_line_size = -1
        max_concurrent_downloads = 20
        [plugins."io.containerd.grpc.v1.cri".containerd]
          snapshotter = "overlayfs"
          default_runtime_name = "runc"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
              runtime_type = "io.containerd.runc.v2"
              runtime_engine = ""
              runtime_root = ""
              [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
                SystemdCgroup = true
                __options__
        [plugins."io.containerd.grpc.v1.cri".registry]
          config_path = "/etc/containerd/certs.d"