gitops/homelab-v
1
0
Fork 0
mirror of https://github.com/outbackdingo/homelab-v.git synced 2026-08-25 14:53:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: Clean up README.md

Browse Source
This commit is contained in:
Vegard Hagen
2024-04-05 23:14:31 +02:00
committed by Vegard Stenhjem Hagen
parent e343d41b85
commit b6b64ab4c4
16 changed files with 792 additions and 883 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+59 -317
View File
@@ -1,340 +1,82 @@
# Setup cluster with kubeadm
<div align="center">
## Proxmox (optional)
<img src="https://raw.githubusercontent.com/vehagn/homelab/main/docs/assets/kubernetes.svg" width="144px" alt="Kubernetes logo"/>
## Debian 12 – Bookworm
# 🪨 Kubernetes Homelab 🏡
Enable `sudo` for the user
</div>
```shell
~$ su -
~# usermod -aG sudo <user>
~# apt install sudo
~# exit
~$ exit
`
---
Enable `ssh` on server
## 📝 Overview
```shell
sudo apt install openssh-server
```
This is the [IaC](https://en.wikipedia.org/wiki/Infrastructure_as_code) configuration for my homelab.
It's mainly powered by [Kubernetes](https://kubernetes.io/) and I do my best to adhere to GitOps practices.
On client
To organise all the configuration I've opted for an approach using Kustomized Helm with Argo CD which I've explained in
more detail [here](https://blog.stonegarden.dev/articles/2023/09/argocd-kustomize-with-helm/).
```shell
ssh-copy-id <user>@<ip>
```
I try to journal my adventures and exploits on my [blog](https://blog.stonegarden.dev) which is hosted by this repo.
Harden `ssh` server
## 🧑‍💻 Getting Started
```shell
echo "PermitRootLogin no" | sudo tee /etc/ssh/sshd_config.d/01-disable-root-login.conf
echo "PasswordAuthentication no" | sudo tee /etc/ssh/sshd_config.d/02-disable-password-auth.conf
echo "ChallengeResponseAuthentication no" | sudo tee /etc/ssh/sshd_config.d/03-disable-challenge-response-auth.conf
echo "UsePAM no" | sudo tee /etc/ssh/sshd_config.d/04-disable-pam.conf
sudo systemctl reload ssh
```
If you're new to Kubernetes I've written a fairly thorough guide
on [Bootstrapping k3s with Cilium](https://blog.stonegarden.dev/articles/2024/02/bootstrapping-k3s-with-cilium/).
In the article I try to guide you from a fresh Debian 12 Bookworm install to a working cluster using
the [k3s](https://k3s.io) flavour of Kubernetes with [Cilium](https://cilium.io) as a [CNI](https://www.cni.dev)
and [IngressController](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/).
## Install prerequisites
I've also written an article on how to get started
with [Kubernetes on Proxmox](https://blog.stonegarden.dev/articles/2024/03/proxmox-k8s-with-cilium/) if virtualisation
is more your thing.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
A third option is the [Quickstart](docs/QUICKSTART.md) in the docs-folder.
Install vert tools
I also have a ["mini-cluster" repo](https://gitlab.com/vehagn/mini-homelab) which might be easier to start understanding
over at GitLab.
```shell
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl gpg
```
## ⚙️ Core Components
Add key and repo
* [Argo CD](https://argo-cd.readthedocs.io/en/stable/): Declarative, GitOps continuous delivery tool for Kubernetes.
* [Cert-manager](https://cert-manager.io/): Cloud native certificate management.
* [Cilium](https://cilium.io/): eBPF-based Networking, Observability, Security.
* [OpenTofu](https://opentofu.org/): The open source infrastructure as code tool.
* [Sealed-secrets](https://github.com/bitnami-labs/sealed-secrets): Encrypt your Secret into a SealedSecret, which is
safe to store - even inside a public repository.
```shell
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
## 📂 Folder Structure
Install kubelet, kubeadm and kubectl
* `apps`: Different applications that I run in the cluster.
* `charts`: Tailor made Helm charts for this cluster.
* `docs`: Supplementary documentation.
* `infra`: Configuration for core infrastructure components
* `machines`: OpenTofu/Terraform configuration. Each sub folder is a physical machine.
* `sets`: Holds Argo CD Applications that points to the `apps` and `infra` folders for automatic Git-syncing.
```shell
sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
## 🖥️ Hardware
Kubelet ≥ 1.26 requires containerd ≥ 1.6.0.
| Name | Device | CPU | RAM | Storage | Purpose |
|--------|---------------------------|-----------------|----------------|------------|---------|
| Gauss | Dell Precision Tower 5810 | Xeon E5-1650 v3 | 64 GB DDR4 ECC | 14 TiB HDD | - |
| Euclid | ASUS ExpertCenter PN42 | Intel N100 | 32 GB DDR4 | - | - |
```shell
sudo apt install -y runc containerd
```
## 🏗️ Work in Progress
## Config
- [ ] Clean up DNS config
- [ ] Renovate for automatic updates
- [ ] Build a NAS for storage
- [ ] Template Gauss
- [ ] Replace Pi Hole with AdGuard Home
- [ ] Use iGPU on Euclid for video transcoding
- [ ] Replace Traefik with Cilium Ingress Controller
- [ ] Cilium mTLS & SPIFFE/SPIRE
### Disable swap
## 👷‍ Future Projects
Disable swap for kubelet to work properly
```shell
sudo swapoff -a
```
Comment out swap in `/etc/fstab` to disable swap on boot
```shell
sudo sed -e '/swap/ s/^#*/#/' -i /etc/fstab
```
### Forwarding IPv4 and letting iptables see bridged traffic
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites
```shell
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
```
```shell
sudo modprobe overlay
sudo modprobe br_netfilter
```
Persist `sysctl` params across reboot
```shell
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
```
Apply `sysctl` params without reboot
```shell
sudo sysctl --system
```
### containerd cgroups
Generate default config
```shell
containerd config default | sudo tee /etc/containerd/config.toml
```
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd
Configure the `systemd` cgroup driver for containerd
```shell
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
```
Restart containerd
```shell
sudo systemctl restart containerd
```
## Initialise cluster
We are going to use cilium in place of kube-proxy
https://docs.cilium.io/en/v1.12/gettingstarted/kubeproxy-free/
```shell
sudo kubeadm init --skip-phases=addon/kube-proxy
```
## Set up kubectl
https://kubernetes.io/docs/tasks/tools/
```shell
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
For remote kubectl copy the config file to local machine
```shell
scp [email protected]:/home/veh/.kube/config ~/.kube/config
```
## (Optional) Remove taint for single node use
Get taints on nodes
```shell
kubectl get nodes -o json | jq '.items[].spec.taints'
```
Remove taint on master node to allow scheduling of all deployments
```shell
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
```
## Install Cilium as CNI (Container Network Interface)
To bootstrap the cluster we can install Cilium using its namesake CLI.
For Linux this can be done by running
```shell
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
```
See the [Cilium official docs](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) for more options.
Next we install Cilium in Kube proxy replacement mode and enable L2 announcements to reply to ARP requests.
To not run into rate limiting while doing L2 announcements we also increase the k8s rate limits.
```shell
cilium install \
--set kubeProxyReplacement=true \
--set l2announcements.enabled=true \
--set externalIPs.enabled=true \
--set k8sClientRateLimit.qps=50 \
--set k8sClientRateLimit.burst=100
```
See [this blog post](https://blog.stonegarden.dev/articles/2023/12/migrating-from-metallb-to-cilium/#l2-announcements)
for more details.
Validate install
```shell
cilium status
```
## Cilium LB IPAM
For [Cilium to act as a load balancer](https://docs.cilium.io/en/stable/network/lb-ipam/) and start assigning IPs
to `LoadBalancer` `Service` resources we need to create a `CiliumLoadBalancerIPPool` with a valid pool.
Edit the cidr range to fit your network before applying it
```shell
kubectl apply -f infra/cilium/ip-pool.yaml
```
Next create a `CiliumL2AnnouncementPolicy` to announce the assigned IPs.
Leaving the `interfaces` field empty announces on all interfaces.
```shell
kubectl apply -f infra/cilium/announce.yaml
```
# Sealed Secrets
Used to create encrypted secrets
```shell
kubectl apply -k infra/sealed-secrets
```
Be sure to store the generated sealed secret key in a safe place!
```shell
kubectl -n kube-system get secrets
```
*NB!*: There will be errors if you use my sealed secrets as you (hopefully) don't have the decryption key
# Gateway API
```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
```
# Cert-manager
```shell
kubectl kustomize --enable-helm infra/cert-manager | kubectl apply -f -
```
# Traefik
Change the `io.cilium/lb-ipam-ips` annotation in `infra/traefik/values.yaml` to a valid IP address for your network.
Install Traefik
```shell
kubectl kustomize --enable-helm infra/traefik | kubectl apply -f -
```
## Port forward Traefik
Port forward Traefik ports in router from 8000 to 80 for http and 4443 to 443 for https.
IP can be found with `kubectl get svc` (it should be the same as the one you gave in the annotation).
# Test-application (Optional)
Deploy a test-application by editing the manifests in `apps/test/whoami` and apply them
```shell
kubectl apply -k apps/test/whoami
```
An unsecured test-application `whoami` should be available at [https://test.${DOMAIN}](https://test.${DOMAIN}).
If you configured `apps/test/whoami/traefik-forward-auth` correctly a secured version should be available
at [https://whoami.${DOMAIN}](https://whoami.${DOMAIN}).
# Argo CD
[ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/) is used to bootstrap the rest of the cluster.
The cluster uses a combination of Helm and Kustomize to configure infrastructure and applications.
For more details read [this blog post](https://blog.stonegarden.dev/articles/2023/09/argocd-kustomize-with-helm/)
```shell
kubectl kustomize --enable-helm infra/argocd | kubectl apply -f -
```
Get ArgoCD initial secret by running
```shell
kubectl -n argocd get secrets argocd-initial-admin-secret -o json | jq -r .data.password | base64 -d
```
# Kubernetes Dashboard
An OIDC (traefik-forward-auth)
protected [Kubernetes Dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) can be
deployed using
```shell
kubectl apply -k infra/dashboard
```
Create a token
```shell
kubectl -n kubernetes-dashboard create token admin-user
```
# ApplicationSets
*NB!*: This will not work before you've changed all the domain names and IP addresses.
Once you've tested everything get the ball rolling with
```shell
kubectl apply -k sets
```
# Cleanup
```shell
kubectl drain gauss --delete-emptydir-data --force --ignore-daemonsets
sudo kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
```
- [ ] Use Talos instead of Debian for Kubernetes
- [ ] Keycloak for auth
- [ ] Dynamic Resource Allocation for GPU
- [ ] Local LLM
- [ ] pfSense
- [ ] Use NetBird or Tailscale
- [ ] Use BGP instead of ARP