mirror of
https://github.com/outbackdingo/homelab-v.git
synced 2026-08-25 14:53:19 +00:00
feat(vm): Start to use Proxmox
This commit is contained in:
Vegard Hagen
committed by
Vegard Stenhjem Hagen
Vegard Stenhjem Hagen
parent
25692fd11f
commit
d035bec693
@@ -1,27 +1,143 @@
|
||||
# Setup cluster with kubeadm
|
||||
|
||||
Disable swap for kubelet to work properly
|
||||
## Proxmox (optional)
|
||||
|
||||
## Debian 12 – Bookworm
|
||||
|
||||
Enable `sudo` for the user
|
||||
|
||||
```shell
|
||||
swapoff -a
|
||||
~$ su -
|
||||
~# usermod -aG sudo <user>
|
||||
~# apt install sudo
|
||||
~# exit
|
||||
~$ exit
|
||||
`
|
||||
|
||||
Enable `ssh` on server
|
||||
|
||||
```shell
|
||||
sudo apt install openssh-server
|
||||
```
|
||||
|
||||
On client
|
||||
|
||||
```shell
|
||||
ssh-copy-id <user>@<ip>
|
||||
```
|
||||
|
||||
Harden `ssh` server
|
||||
|
||||
```shell
|
||||
echo "PermitRootLogin no" | sudo tee /etc/ssh/sshd_config.d/01-disable-root-login.conf
|
||||
echo "PasswordAuthentication no" | sudo tee /etc/ssh/sshd_config.d/02-disable-password-auth.conf
|
||||
echo "ChallengeResponseAuthentication no" | sudo tee /etc/ssh/sshd_config.d/03-disable-challenge-response-auth.conf
|
||||
echo "UsePAM no" | sudo tee /etc/ssh/sshd_config.d/04-disable-pam.conf
|
||||
sudo systemctl reload ssh
|
||||
```
|
||||
|
||||
## Install prerequisites
|
||||
|
||||
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
|
||||
|
||||
Install vert tools
|
||||
|
||||
```shell
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y apt-transport-https ca-certificates curl
|
||||
sudo apt update
|
||||
sudo apt install -y apt-transport-https ca-certificates curl gpg
|
||||
```
|
||||
|
||||
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
|
||||
Add key and repo
|
||||
|
||||
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
|
||||
```shell
|
||||
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
|
||||
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
|
||||
```
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y containerd conntrack socat kubelet kubeadm kubectl
|
||||
Install kubelet, kubeadm and kubectl
|
||||
|
||||
```shell
|
||||
sudo apt update
|
||||
sudo apt install -y kubelet kubeadm kubectl
|
||||
sudo apt-mark hold kubelet kubeadm kubectl
|
||||
```
|
||||
|
||||
Kubelet ≥ 1.26 requires containerd ≥ 1.6.0.
|
||||
|
||||
```shell
|
||||
sudo apt install -y runc containerd
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
### Disable swap
|
||||
|
||||
Disable swap for kubelet to work properly
|
||||
|
||||
```shell
|
||||
sudo swapoff -a
|
||||
```
|
||||
|
||||
Comment out swap in `/etc/fstab` to disable swap on boot
|
||||
|
||||
```shell
|
||||
sudo sed -e '/swap/ s/^#*/#/' -i /etc/fstab
|
||||
```
|
||||
|
||||
### Forwarding IPv4 and letting iptables see bridged traffic
|
||||
|
||||
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites
|
||||
|
||||
```shell
|
||||
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
|
||||
overlay
|
||||
br_netfilter
|
||||
EOF
|
||||
```
|
||||
|
||||
```shell
|
||||
sudo modprobe overlay
|
||||
sudo modprobe br_netfilter
|
||||
```
|
||||
|
||||
Persist `sysctl` params across reboot
|
||||
|
||||
```shell
|
||||
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
net.ipv4.ip_forward = 1
|
||||
EOF
|
||||
```
|
||||
|
||||
Apply `sysctl` params without reboot
|
||||
|
||||
```shell
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
### containerd cgroups
|
||||
|
||||
Generate default config
|
||||
|
||||
```shell
|
||||
containerd config default | sudo tee /etc/containerd/config.toml
|
||||
```
|
||||
|
||||
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd
|
||||
|
||||
Configure the `systemd` cgroup driver for containerd
|
||||
|
||||
```shell
|
||||
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
|
||||
```
|
||||
|
||||
Restart containerd
|
||||
|
||||
```shell
|
||||
sudo systemctl restart containerd
|
||||
```
|
||||
|
||||
## Initialise cluster
|
||||
|
||||
We are going to use cilium in place of kube-proxy
|
||||
@@ -44,7 +160,7 @@ sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
||||
For remote kubectl copy the config file to local machine
|
||||
|
||||
```shell
|
||||
scp gauss@192.168.1.12:/home/gauss/.kube/config ~/.kube/config
|
||||
scp veh@192.168.1.50:/home/veh/.kube/config ~/.kube/config
|
||||
```
|
||||
|
||||
## (Optional) Remove taint for single node use
|
||||
@@ -108,14 +224,14 @@ to `LoadBalancer` `Service` resources we need to create a `CiliumLoadBalancerIPP
|
||||
Edit the cidr range to fit your network before applying it
|
||||
|
||||
```shell
|
||||
kubectl apply infra/cilium/ip-pool.yaml
|
||||
kubectl apply -f infra/cilium/ip-pool.yaml
|
||||
```
|
||||
|
||||
Next create a `CiliumL2AnnouncementPolicy` to announce the assigned IPs.
|
||||
Leaving the `interfaces` field empty announces on all interfaces.
|
||||
|
||||
```shell
|
||||
kubectl apply infra/cilium/announce.yaml
|
||||
kubectl apply -f infra/cilium/announce.yaml
|
||||
```
|
||||
|
||||
# Sealed Secrets
|
||||
@@ -126,7 +242,7 @@ Used to create encrypted secrets
|
||||
kubectl apply -k infra/sealed-secrets
|
||||
```
|
||||
|
||||
Be sure to store the generated sealed secret key in a safa place!
|
||||
Be sure to store the generated sealed secret key in a safe place!
|
||||
|
||||
```shell
|
||||
kubectl -n kube-system get secrets
|
||||
@@ -134,10 +250,21 @@ kubectl -n kube-system get secrets
|
||||
|
||||
*NB!*: There will be errors if you use my sealed secrets as you (hopefully) don't have the decryption key
|
||||
|
||||
# Gateway API
|
||||
|
||||
```shell
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
|
||||
```
|
||||
|
||||
# Cert-manager
|
||||
|
||||
```shell
|
||||
kubectl kustomize --enable-helm infra/cert-manager | kubectl apply -f -
|
||||
```
|
||||
|
||||
# Traefik
|
||||
|
||||
Remove the `deployment.dnsConfig` from `infra/traefik/values.yaml` and change the `io.cilium/lb-ipam-ips` annotation to
|
||||
a valid IP address for your network.
|
||||
Change the `io.cilium/lb-ipam-ips` annotation in `infra/traefik/values.yaml` to a valid IP address for your network.
|
||||
|
||||
Install Traefik
|
||||
|
||||
@@ -162,14 +289,14 @@ An unsecured test-application `whoami` should be available at [https://test.${DO
|
||||
If you configured `apps/test/whoami/traefik-forward-auth` correctly a secured version should be available
|
||||
at [https://whoami.${DOMAIN}](https://whoami.${DOMAIN}).
|
||||
|
||||
# ArgoCD
|
||||
# Argo CD
|
||||
|
||||
[ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/) is used to bootstrap the rest of the cluster.
|
||||
The cluster uses a combination of Helm and Kustomize to configure infrastructure and applications.
|
||||
For more details read [this blog post](https://blog.stonegarden.dev/articles/2023/09/argocd-kustomize-with-helm/)
|
||||
|
||||
```shell
|
||||
kubectl apply -k infra/argocd
|
||||
kubectl kustomize --enable-helm infra/argocd | kubectl apply -f -
|
||||
```
|
||||
|
||||
Get ArgoCD initial secret by running
|
||||
@@ -210,30 +337,4 @@ kubectl apply -k sets
|
||||
kubectl drain gauss --delete-emptydir-data --force --ignore-daemonsets
|
||||
sudo kubeadm reset
|
||||
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
|
||||
sudo ipvsadm -C
|
||||
```
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
Kubernetes 1.26 requires containerd 1.6.0 or later due to the removal of support for CRI
|
||||
version `v1alpha2` ([link](https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#cri-api-removal)).
|
||||
|
||||
Make sure that `runc` is properly configured in containerd.
|
||||
|
||||
NB: Make sure the correct `containerd` daemon is running.
|
||||
(Check the loaded `containerd` service definition as reported by `systemctl status containerd`)
|
||||
Follow https://github.com/containerd/containerd/blob/main/docs/getting-started.md for further instructions.
|
||||
|
||||
```shell
|
||||
sudo cat /etc/containerd/config.toml
|
||||
```
|
||||
|
||||
```toml
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
|
||||
runtime_path = "/usr/bin/runc"
|
||||
runtime_type = "io.containerd.runc.v2"
|
||||
```
|
||||
|
||||
## Sealed Secrets
|
||||
|
||||
Restart pod after applying master-key.
|
||||
Reference in New Issue
Block a user