Insecure Harbor registry with Tanzu Kubernetes Grid+ on vSphere

August 6, 2020 By Corey Dinkens

In searching for ways to use an ‘insecure’ registry with Tanzu Kubernetes Grid, I found a post by William Lam explaining how. I wanted to build on this and figure out what was needed to deploy a customized cluster with tkg-cli, and skip deploying the kind cluster first.

The files located in .tkg/bom/ are the key to this, as they are the templates that tkg-cli uses to bootstrap kind, and deploy the TKG cluster. With the same technique, you can perform additional customization as needed; possibly more on that in the future.

Step 1.

Locate and open the following files (should be in home dir) in editor of choice:

  • .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-dev.yaml
  • .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-prod.yaml

Step 2.

Insert the customized files block below the last line of the preKubeadmCommands section:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
preKubeadmCommands:
- hostname "{{ ds.meta_data.hostname }}"
- echo "::1 ipv6-localhost ipv6-loopback" >/etc/hosts
- echo "127.0.0.1 localhost" >>/etc/hosts
- echo "127.0.0.1 {{ ds.meta_data.hostname }}" >>/etc/hosts
- echo "{{ ds.meta_data.hostname }}" >/etc/hostname
files:
- path: /etc/containerd/config.toml
content: |
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.tkg.vmware.run/pause:3.1"
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]
endpoint = ["http://<fqdn or registry IP>:80"]
preKubeadmCommands: - hostname "{{ ds.meta_data.hostname }}" - echo "::1 ipv6-localhost ipv6-loopback" >/etc/hosts - echo "127.0.0.1 localhost" >>/etc/hosts - echo "127.0.0.1 {{ ds.meta_data.hostname }}" >>/etc/hosts - echo "{{ ds.meta_data.hostname }}" >/etc/hostname files: - path: /etc/containerd/config.toml content: | version = 2 [plugins] [plugins."io.containerd.grpc.v1.cri"] sandbox_image = "registry.tkg.vmware.run/pause:3.1" [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler] runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"] endpoint = ["http://<fqdn or registry IP>:80"]
preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    files:
      - path: /etc/containerd/config.toml
        content: |
          version = 2
          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              sandbox_image = "registry.tkg.vmware.run/pause:3.1"
              [plugins."io.containerd.grpc.v1.cri".containerd]
                default_runtime_name = "runc"
                [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
                    runtime_type = "io.containerd.runc.v2"
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
                    runtime_type = "io.containerd.runc.v2"
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]
                    endpoint = ["http://<fqdn or registry IP>:80"]

Step 3.

Locate and open:

  • .tkg/bom/bom-1.1.2+vmware.1.yaml
    (edit the file that matches the version of tkg being used or version of k8s being deployed. I am using tkg 1.1.2)

Locate the kubeadmConfigSpec section, and insert your customized kindKubeadmConfigSpec immediately after:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
kubeadmConfigSpec:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: registry.tkg.vmware.run
kubernetesVersion: v1.18.3+vmware.1
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.tkg.vmware.run
imageTag: v3.4.3_vmware.5
dns:
type: CoreDNS
imageRepository: registry.tkg.vmware.run
imageTag: v1.6.7_vmware.1
kindKubeadmConfigSpec:
- 'kind: Cluster'
- 'apiVersion: kind.x-k8s.io/v1alpha4'
- 'containerdConfigPatches:'
- '- |-'
- ' [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]'
- ' endpoint = ["https://<fqdn or registry IP>:80"]'
- 'kubeadmConfigPatches:'
- '- |'
- ' apiVersion: kubeadm.k8s.io/v1beta2'
- ' kind: ClusterConfiguration'
- ' imageRepository: registry.tkg.vmware.run'
- ' etcd:'
- ' local:'
- ' imageRepository: registry.tkg.vmware.run'
- ' imageTag: v3.4.3_vmware.5'
- ' dns:'
- ' type: CoreDNS'
- ' imageRepository: registry.tkg.vmware.run'
- ' imageTag: v1.6.7_vmware.1'
kubeadmConfigSpec: apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration imageRepository: registry.tkg.vmware.run kubernetesVersion: v1.18.3+vmware.1 etcd: local: dataDir: /var/lib/etcd imageRepository: registry.tkg.vmware.run imageTag: v3.4.3_vmware.5 dns: type: CoreDNS imageRepository: registry.tkg.vmware.run imageTag: v1.6.7_vmware.1 kindKubeadmConfigSpec: - 'kind: Cluster' - 'apiVersion: kind.x-k8s.io/v1alpha4' - 'containerdConfigPatches:' - '- |-' - ' [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]' - ' endpoint = ["https://<fqdn or registry IP>:80"]' - 'kubeadmConfigPatches:' - '- |' - ' apiVersion: kubeadm.k8s.io/v1beta2' - ' kind: ClusterConfiguration' - ' imageRepository: registry.tkg.vmware.run' - ' etcd:' - ' local:' - ' imageRepository: registry.tkg.vmware.run' - ' imageTag: v3.4.3_vmware.5' - ' dns:' - ' type: CoreDNS' - ' imageRepository: registry.tkg.vmware.run' - ' imageTag: v1.6.7_vmware.1'
kubeadmConfigSpec:
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  imageRepository: registry.tkg.vmware.run
  kubernetesVersion: v1.18.3+vmware.1
  etcd:
    local:
      dataDir: /var/lib/etcd
      imageRepository: registry.tkg.vmware.run
      imageTag: v3.4.3_vmware.5
  dns:
    type: CoreDNS
    imageRepository: registry.tkg.vmware.run
    imageTag: v1.6.7_vmware.1
kindKubeadmConfigSpec:
- 'kind: Cluster'
- 'apiVersion: kind.x-k8s.io/v1alpha4'
- 'containerdConfigPatches:'
- '- |-'
- '  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]'
- '    endpoint = ["https://<fqdn or registry IP>:80"]'
- 'kubeadmConfigPatches:'
- '- |'
- '  apiVersion: kubeadm.k8s.io/v1beta2'
- '  kind: ClusterConfiguration'
- '  imageRepository: registry.tkg.vmware.run'
- '  etcd:'
- '    local:'
- '      imageRepository: registry.tkg.vmware.run'
- '      imageTag: v3.4.3_vmware.5'
- '  dns:'
- '    type: CoreDNS'
- '    imageRepository: registry.tkg.vmware.run'
- '    imageTag: v1.6.7_vmware.1'

Step 4.

Verify config output by doing a dry run:

tkg create cluster –plan {dev|prod|custom} –dry-run

Step 5.

If the dry run succeeded, deploy a TKG cluster using:

tkg create cluster –plan {dev|prod|custom}

(In case you were wondering: creating custom plans)

You can verify that the above steps have successfully patched the node by SSHing into a node, and

 cat /etc/containerd/config.toml
You should see the containerd configuration that you added above

References:

William Lam’s post on TKG + Insecure registry: https://www.virtuallyghetto.com/2020/05/configure-non-secure-harbor-registry-with-tanzu-kubernetes-grid-tkg.html

Erick aka Gubi on custom TKG plans: https://letsdocloud.com/?p=730

Versions Used

vSphere 6.7u3kubectl: 1.18
TKG (Tanzu Kubernetes Grid+): 1.1.2Kubernetes: 1.18.3
VIC (vSphere Intergrated Containers): 1.5.5tkg-cli: 1.1.2