diff --git a/COMPONENTS.md b/COMPONENTS.md index d39408c..e0c91a4 100644 --- a/COMPONENTS.md +++ b/COMPONENTS.md @@ -1,10 +1,7 @@ -kubespray -Argo autopilot - CNI: Cilium -LoadBalancer: MetalLB +LoadBalancer: Cilium Ingress: Traefik -Certs: cert-manager +Certs: Cert-manager CD: ArgoCD Monitoring: Prometheus Observability: Grafana diff --git a/QUICKSTART.md b/QUICKSTART.md deleted file mode 100644 index ec43bba..0000000 --- a/QUICKSTART.md +++ /dev/null @@ -1,59 +0,0 @@ -# Kubernetes - -## Disable swap - -```shell -swapoff -a -``` - -## Start Kubernetes - -```shell -sudo kubeadm init -``` - -## Set up kubectl - -```shell -sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config -``` - -## Remove taint for single node use - -```shell -kubectl taint nodes --all node-role.kubernetes.io/control-plane- -``` - -# Cilium - -## Install Cilium as a CNI - -```shell -cilium install -``` - -# MetalLB - -## Install MetalLB for LoadBalancing - -https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml - -```shell -kubectl apply -k infra/metallb -``` - -# Traefik reverse proxy - -https://doc.traefik.io/traefik/v2.9/user-guides/crd-acme/ - -```shell -kubectl kustomize --enable-helm infra/traefik | kubectl apply -f - -``` - -# ArgoCD - -https://argo-cd.readthedocs.io/en/stable/getting_started/ - -```shell -kubectl apply -k infra/argocd -``` \ No newline at end of file diff --git a/README.md b/README.md index d960d1a..3a325e1 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ sudo apt-get update sudo apt-get install -y containerd conntrack socat kubelet kubeadm kubectl ``` -Kubelet 1.26 requires containerd 1.6.0 or later. +Kubelet ≥ 1.26 requires containerd ≥ 1.6.0. ## Initialise cluster @@ -37,13 +37,14 @@ https://kubernetes.io/docs/tasks/tools/ ```shell mkdir -p $HOME/.kube -sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config +sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config +sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` For remote kubectl copy the config file to local machine ```shell -scp veh@192.168.1.12:/home/veh/.kube/config ~/.kube/config +scp gauss@192.168.1.12:/home/gauss/.kube/config ~/.kube/config ``` ## (Optional) Remove taint for single node use @@ -60,18 +61,14 @@ Remove taint on master node to allow scheduling of all deployments kubectl taint nodes --all node-role.kubernetes.io/control-plane- ``` -## Install Cilium as Container Network Interface (CNI) +## Install Cilium as CNI (Container Network Interface) + +To bootstrap the cluster we can install Cilium using its namesake CLI. + +For Linux this can be done by running ```shell -kubectl kustomize --enable-helm infra/cilium | kubectl apply -f - -``` - -https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/ - -Install Cilium CLI - -```shell -CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt) +CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} @@ -80,12 +77,23 @@ sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} ``` -Install Cilium +See the [Cilium official docs](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) for more options. + +Next we install Cilium in Kube proxy replacement mode and enable L2 announcements to reply to ARP requests. +To not run into rate limiting while doing L2 announcements we also increase the k8s rate limits. ```shell -cilium install +cilium install \ + --set kubeProxyReplacement=true \ + --set l2announcements.enabled=true \ + --set externalIPs.enabled=true \ + --set k8sClientRateLimit.qps=50 \ + --set k8sClientRateLimit.burst=100 ``` +See [this blog post](https://blog.stonegarden.dev/articles/2023/12/migrating-from-metallb-to-cilium/#l2-announcements) +for more details. + Validate install ```shell @@ -95,13 +103,16 @@ cilium status ## Cilium LB IPAM For [Cilium to act as a load balancer](https://docs.cilium.io/en/stable/network/lb-ipam/) and start assigning IPs -to `LoadBalancer` services create a viable IP pool, e.g. `192.168.1.128/5`, by creating a `CiliumLoadBalancerIPPool` +to `LoadBalancer` `Service` resources we need to create a `CiliumLoadBalancerIPPool` with a valid pool. + +Edit the cidr range to fit your network before applying it ```shell kubectl apply infra/cilium/ip-pool.yaml ``` -and announce using a `CiliumL2AnnouncementPolicy` +Next create a `CiliumL2AnnouncementPolicy` to announce the assigned IPs. +Leaving the `interfaces` field empty announces on all interfaces. ```shell kubectl apply infra/cilium/announce.yaml @@ -125,6 +136,9 @@ kubectl -n kube-system get secrets # Traefik +Remove the `deployment.dnsConfig` from `infra/traefik/values.yaml` and change the `io.cilium/lb-ipam-ips` annotation to +a valid IP address for your network. + Install Traefik ```shell @@ -134,44 +148,31 @@ kubectl kustomize --enable-helm infra/traefik | kubectl apply -f - ## Port forward Traefik Port forward Traefik ports in router from 8000 to 80 for http and 4443 to 443 for https. -IP can be found with `kubectl get svc`. +IP can be found with `kubectl get svc` (it should be the same as the one you gave in the annotation). -# Test-application +# Test-application (Optional) -## Generate secret - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: traefik-forward-auth-secrets - namespace: whoami -type: Opaque -data: - google-client-id: <...> - google-client-secret: <...> - secret: <...> -``` - -Deploy a test-application by running +Deploy a test-application by editing the manifests in `apps/test/whoami` and apply them ```shell -kubectl apply -k apps/whoami +kubectl apply -k apps/test/whoami ``` An unsecured test-application `whoami` should be available at [https://test.${DOMAIN}](https://test.${DOMAIN}). -If you configured `apps/whoami/traefik-forward-auth` correctly a secured version should be available -at [https://whoami.${DOMAIN}](https://whoami.${DOMAIN}) +If you configured `apps/test/whoami/traefik-forward-auth` correctly a secured version should be available +at [https://whoami.${DOMAIN}](https://whoami.${DOMAIN}). # ArgoCD -[ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/) is configured to bootstrap the rest of the cluster +[ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/) is used to bootstrap the rest of the cluster. +The cluster uses a combination of Helm and Kustomize to configure infrastructure and applications. +For more details read [this blog post](https://blog.stonegarden.dev/articles/2023/09/argocd-kustomize-with-helm/) ```shell kubectl apply -k infra/argocd ``` -Get ArgoCD initial secret +Get ArgoCD initial secret by running ```shell kubectl -n argocd get secrets argocd-initial-admin-secret -o json | jq -r .data.password | base64 -d @@ -195,6 +196,8 @@ kubectl -n kubernetes-dashboard create token admin-user # ApplicationSets +*NB!*: This will not work before you've changed all the domain names and IP addresses. + Once you've tested everything get the ball rolling with ```shell @@ -231,10 +234,6 @@ runtime_path = "/usr/bin/runc" runtime_type = "io.containerd.runc.v2" ``` -## Wrong containerd version - -1.7.x doesn't work? - ## Sealed Secrets -Restart pod after applying master-key \ No newline at end of file +Restart pod after applying master-key. \ No newline at end of file diff --git a/RESOURCES.md b/RESOURCES.md index 91554b7..3212c3e 100644 --- a/RESOURCES.md +++ b/RESOURCES.md @@ -1,10 +1,10 @@ https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ -https://metallb.universe.tf/installation/ https://kubernetes.io/docs/concepts/services-networking/service/ + +https://docs.cilium.io/en/stable/ + +https://github.com/bitnami-labs/sealed-secrets#usage + https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/ -https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started - -https://www.smarthomebeginner.com/traefik-forward-auth-google-oauth-2022/ - -https://github.com/bitnami-labs/sealed-secrets#usage \ No newline at end of file +https://www.smarthomebeginner.com/traefik-forward-auth-google-oauth-2022/ \ No newline at end of file