In searching for ways to use an ‘insecure’ registry with Tanzu Kubernetes Grid, I found a post by William Lam explaining how. I wanted to build on this and figure out what was needed to deploy a customized cluster with tkg-cli, and skip deploying the kind cluster first.
The files located in .tkg/bom/ are the key to this, as they are the templates that tkg-cli uses to bootstrap kind, and deploy the TKG cluster. With the same technique, you can perform additional customization as needed; possibly more on that in the future.
Step 1.
Locate and open the following files (should be in home dir) in editor of choice:
- .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-dev.yaml
- .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-prod.yaml
Step 2.
Insert the customized files block below the last line of the preKubeadmCommands section:
preKubeadmCommands:
- hostname "{{ ds.meta_data.hostname }}"
- echo "::1 ipv6-localhost ipv6-loopback" >/etc/hosts
- echo "127.0.0.1 localhost" >>/etc/hosts
- echo "127.0.0.1 {{ ds.meta_data.hostname }}" >>/etc/hosts
- echo "{{ ds.meta_data.hostname }}" >/etc/hostname
files:
- path: /etc/containerd/config.toml
content: |
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.tkg.vmware.run/pause:3.1"
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]
endpoint = ["http://<fqdn or registry IP>:80"]
Step 3.
Locate and open:
- .tkg/bom/bom-1.1.2+vmware.1.yaml
(edit the file that matches the version of tkg being used or version of k8s being deployed. I am using tkg 1.1.2)
Locate the kubeadmConfigSpec section, and insert your customized kindKubeadmConfigSpec immediately after:
kubeadmConfigSpec:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: registry.tkg.vmware.run
kubernetesVersion: v1.18.3+vmware.1
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.tkg.vmware.run
imageTag: v3.4.3_vmware.5
dns:
type: CoreDNS
imageRepository: registry.tkg.vmware.run
imageTag: v1.6.7_vmware.1
kindKubeadmConfigSpec:
- 'kind: Cluster'
- 'apiVersion: kind.x-k8s.io/v1alpha4'
- 'containerdConfigPatches:'
- '- |-'
- ' [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]'
- ' endpoint = ["https://<fqdn or registry IP>:80"]'
- 'kubeadmConfigPatches:'
- '- |'
- ' apiVersion: kubeadm.k8s.io/v1beta2'
- ' kind: ClusterConfiguration'
- ' imageRepository: registry.tkg.vmware.run'
- ' etcd:'
- ' local:'
- ' imageRepository: registry.tkg.vmware.run'
- ' imageTag: v3.4.3_vmware.5'
- ' dns:'
- ' type: CoreDNS'
- ' imageRepository: registry.tkg.vmware.run'
- ' imageTag: v1.6.7_vmware.1'
Step 4.
Verify config output by doing a dry run:
tkg create cluster –plan {dev|prod|custom} –dry-runStep 5.
If the dry run succeeded, deploy a TKG cluster using:
tkg create cluster –plan {dev|prod|custom}(In case you were wondering: creating custom plans)
You can verify that the above steps have successfully patched the node by SSHing into a node, and
cat /etc/containerd/config.tomlReferences:
William Lam’s post on TKG + Insecure registry: https://www.virtuallyghetto.com/2020/05/configure-non-secure-harbor-registry-with-tanzu-kubernetes-grid-tkg.html
Erick aka Gubi on custom TKG plans: https://letsdocloud.com/?p=730
Versions Used
| vSphere 6.7u3 | kubectl: 1.18 |
| TKG (Tanzu Kubernetes Grid+): 1.1.2 | Kubernetes: 1.18.3 |
| VIC (vSphere Intergrated Containers): 1.5.5 | tkg-cli: 1.1.2 |