Insecure Harbor registry with Tanzu Kubernetes Grid+ on vSphere

In searching for ways to use an ‘insecure’ registry with Tanzu Kubernetes Grid, I found a post by William Lam explaining how. I wanted to build on this and figure out what was needed to deploy a customized cluster with tkg-cli, and skip deploying the kind cluster first.

The files located in .tkg/bom/ are the key to this, as they are the templates that tkg-cli uses to bootstrap kind, and deploy the TKG cluster. With the same technique, you can perform additional customization as needed; possibly more on that in the future.

Step 1.

Locate and open the following files (should be in home dir) in editor of choice:

• .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-dev.yaml
• .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-prod.yaml

Step 2.

Insert the customized files block below the last line of the preKubeadmCommands section:

preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    files:
      - path: /etc/containerd/config.toml
        content: |
          version = 2
          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              sandbox_image = "registry.tkg.vmware.run/pause:3.1"
              [plugins."io.containerd.grpc.v1.cri".containerd]
                default_runtime_name = "runc"
                [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
                    runtime_type = "io.containerd.runc.v2"
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
                    runtime_type = "io.containerd.runc.v2"
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]
                    endpoint = ["http://<fqdn or registry IP>:80"]

Step 3.

Locate and open:

• .tkg/bom/bom-1.1.2+vmware.1.yaml
  (edit the file that matches the version of tkg being used or version of k8s being deployed. I am using tkg 1.1.2)

Locate the kubeadmConfigSpec section, and insert your customized kindKubeadmConfigSpec immediately after:

kubeadmConfigSpec:
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  imageRepository: registry.tkg.vmware.run
  kubernetesVersion: v1.18.3+vmware.1
  etcd:
    local:
      dataDir: /var/lib/etcd
      imageRepository: registry.tkg.vmware.run
      imageTag: v3.4.3_vmware.5
  dns:
    type: CoreDNS
    imageRepository: registry.tkg.vmware.run
    imageTag: v1.6.7_vmware.1
kindKubeadmConfigSpec:
- 'kind: Cluster'
- 'apiVersion: kind.x-k8s.io/v1alpha4'
- 'containerdConfigPatches:'
- '- |-'
- '  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]'
- '    endpoint = ["https://<fqdn or registry IP>:80"]'
- 'kubeadmConfigPatches:'
- '- |'
- '  apiVersion: kubeadm.k8s.io/v1beta2'
- '  kind: ClusterConfiguration'
- '  imageRepository: registry.tkg.vmware.run'
- '  etcd:'
- '    local:'
- '      imageRepository: registry.tkg.vmware.run'
- '      imageTag: v3.4.3_vmware.5'
- '  dns:'
- '    type: CoreDNS'
- '    imageRepository: registry.tkg.vmware.run'
- '    imageTag: v1.6.7_vmware.1'

Step 4.

Verify config output by doing a dry run:

tkg create cluster –plan {dev|prod|custom} –dry-run

Step 5.

If the dry run succeeded, deploy a TKG cluster using:

tkg create cluster –plan {dev|prod|custom}

(In case you were wondering: creating custom plans)

You can verify that the above steps have successfully patched the node by SSHing into a node, and

 cat /etc/containerd/config.toml

You should see the containerd configuration that you added above

References:

William Lam’s post on TKG + Insecure registry: https://www.virtuallyghetto.com/2020/05/configure-non-secure-harbor-registry-with-tanzu-kubernetes-grid-tkg.html

Erick aka Gubi on custom TKG plans: https://letsdocloud.com/?p=730

Versions Used

vSphere 6.7u3	kubectl: 1.18
TKG (Tanzu Kubernetes Grid+): 1.1.2	Kubernetes: 1.18.3
VIC (vSphere Intergrated Containers): 1.5.5	tkg-cli: 1.1.2

---