gitops/homelab-v
1
0
Fork 0
mirror of https://github.com/outbackdingo/homelab-v.git synced 2026-08-25 07:10:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(tofu): refactor tofu-code

Browse Source 
Tidying code and clarifying some sections

Signed-off-by: Vegard Hagen <[email protected]>
This commit is contained in:
Vegard Hagen
2025-04-18 13:56:14 +02:00
parent 266aab34c0
commit e94a97e6f3
22 changed files with 167 additions and 121 deletions
Download Patch File Download Diff File Expand all files Collapse all files
renovate.json
+2 -1
View File
@@ -16,7 +16,8 @@
"\\.tf$",
"\\.tftpl$",
"\\.yaml$",
"\\.sh$"
"\\.sh$",
"\\.tfvars"
],
"matchStrings": [
"(?<currentValue>[\\w+\\.\\-]*)['\",;]*\\s*#\\s?renovate: (?<datasource>\\S+)=(?<depName>\\S+)\\s?(registry=(?<registryUrl>\\S+))?\\s?(versioning=(?<versioning>\\S+))?"
tofu/kubernetes/README.md
+31 -21
View File
@@ -1,11 +1,16 @@
# Kubernetes Tofu
Read [Talos Kubernetes on Proxmox using OpenTofu](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/) for
a more thorough explanation of how everything works.
## Install pre-requisites
1. [tofu](https://opentofu.org/docs/intro/install/)
1. [talosctl](https://www.talos.dev/v1.9/talos-guides/install/talosctl/)
1. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
2. [talosctl](https://www.talos.dev/v1.9/talos-guides/install/talosctl/)
3. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
## Initialize tofu
```shell
tofu init
```
@@ -21,11 +26,12 @@ export TF_VAR_proxmox_api_token="<YOUR_API_TOKEN>"
### Optional External Secrets Manager / Other methods
**Bitwarden Secrets Manager** - Name your secret TF_VAR_proxmox_api_token in bws.
```shell
bws run -- tofu ...
```
Note: By default the shell is sh. Change with --shell if required.
Note: By default, the shell is sh. Change with --shell if required.
## Sealed-secrets
@@ -52,29 +58,33 @@ tofu output -raw talos_config
```
## Upgrading Talos and Kubernetes
[Upgrade](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/#upgrading-the-cluster) talos nodes one by one.
[Upgrade](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/#upgrading-the-cluster) talos nodes one by
one.
1. Set talos_image.auto.tfvars -> image -> update_version to the required update version.
1. Set talos_cluster.auto.tfvars -> talos_cluster_config -> kubernetes_version to the required kubernetes version.
1. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_1 -> update = true and run tofu apply.
1. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_2 -> update = true, leave the previous nodes update = true and run tofu apply.
1. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_3 -> update = true, leave the previous nodes update = true and run tofu apply.
1. ...
1. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_n -> update = true, leave the previous nodes update = true and run tofu apply.
1. After upgrading all nodes, Set talos_image.auto.tfvars -> image -> version to match the update version and set update = false for all nodes.
2. Set talos_cluster.auto.tfvars -> talos_cluster_config -> kubernetes_version to the required kubernetes version.
3. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_1 -> update = true and run tofu apply.
4. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_2 -> update = true, leave the previous nodes update = true and
run tofu apply.
5. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_3 -> update = true, leave the previous nodes update = true and
run tofu apply.
6. ...
7. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_n -> update = true, leave the previous nodes update = true and
run tofu apply.
8. After upgrading all nodes, Set talos_image.auto.tfvars -> image -> version to match the update version and set
update = false for all nodes.
## Upgrading Talos Schematic
1. Create a new schematic file.
1. Same process as above instead of image.version and image.update_version, change image.schematic and image.update_schematic, in talos_image.auto.tfvars.
2. Same process as above instead of `image.version` and `image.update_version`, change `image.schematic` and
`image.update_schematic`, in `talos_image.auto.tfvars`.
## Upgrading Kubernetes Only
## Reuse machine secrets
Dry Run
```shell
sh upgrade-k8s.sh $CONTROLPLANE_NODE_IP --dry-run # For testing
```
Upgrade
```shell
sh upgrade-k8s.sh $CONTROLPLANE_NODE_IP
```
tofu state rm module.talos.talos_machine_secrets.this
tofu import module.talos.talos_machine_secrets.this output/talos-machine-secrets.yaml
tofu apply --refresh=false
```
tofu/kubernetes/REMOTE_BACKEND.md
+12 -8
View File
@@ -1,9 +1,12 @@
## GCS Remote
1. Create a [Service Account](https://cloud.google.com/iam/docs/service-accounts-create) named tofu (after enabling the IAM API if needed). Leave the permissions blank.
1. Create and download the [service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating).
1. Create a GCS bucket for tofu state with public access prevention and versioning as necessary.
1. In the permissions tab of the bucket, give **Storage Object Admin** access to the service account.
1. Copy backend.tf.sample to backend.tf and make necessary changes.
1. Create a [Service Account](https://cloud.google.com/iam/docs/service-accounts-create) named tofu (after enabling the
IAM API if needed). Leave the permissions blank.
2. Create and download the [service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating).
3. Create a GCS bucket for tofu state with public access prevention and versioning as necessary.
4. In the permissions tab of the bucket, give **Storage Object Admin** access to the service account.
5. Copy backend.tf.sample to backend.tf and make necessary changes.
```shell
cp remote_backend.tf.sample remote_backend.tf
```
@@ -16,7 +19,7 @@ Generate the encryption key
python3 -c 'import os;import base64;print(base64.b64encode(os.urandom(32)).decode("utf-8"))'
```
![#f03c15](https://placehold.co/15x15/f03c15/f03c15.png) `Without the encryption key, your state would not be recoverable. Store in a password manager, if not using any kms like bws.`
`Without the encryption key, your state would not be recoverable. Store in a password manager, if not using any kms like bws.`
### Environment variables
@@ -29,10 +32,11 @@ Run tofu init / plan / apply as usual.
### Bitwarden Secrets Manager
Store the downloaded key contents and generated encryption key into GOOGLE_CREDENTIALS and GOOGLE_ENCRYPTION_KEY respectively in bws.
Store the downloaded key contents and generated encryption key into GOOGLE_CREDENTIALS and GOOGLE_ENCRYPTION_KEY
respectively in bws.
Run bws run -- tofu init / plan / apply as usual.
### Beta Notice
![#f03c15](https://placehold.co/15x15/f03c15/f03c15.png) `Please treat this as beta and only use for air-gapped installations as of now. Will remove the beta tag after testing it in due course.`
`Please treat this as beta and only use for air-gapped installations as of now. Will remove the beta tag after testing it in due course.`
tofu/kubernetes/bootstrap/proxmox-csi-plugin/providers.tf
+1 -1
View File
@@ -1,7 +1,7 @@
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
source = "hashicorp/kubernetes"
version = ">=2.32.0"
}
proxmox = {
tofu/kubernetes/bootstrap/proxmox-csi-plugin/variables.tf
+2 -2
View File
@@ -1,7 +1,7 @@
variable "proxmox" {
type = object({
cluster_name = string
endpoint = string
insecure = bool
endpoint = string
insecure = bool
})
}
tofu/kubernetes/bootstrap/sealed-secrets/providers.tf
+1 -1
View File
@@ -1,7 +1,7 @@
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
source = "hashicorp/kubernetes"
version = ">=2.32.0"
}
}
tofu/kubernetes/bootstrap/sealed-secrets/variables.tf
+1 -1
View File
@@ -1,6 +1,6 @@
variable "cert" {
type = object({
certificate_path = string
certificate_path = string
certificate_key_path = string
})
}
tofu/kubernetes/bootstrap/volumes/README.md
-2
View File
@@ -1,5 +1,3 @@
```shell
pvesm alloc local-zfs 8000 vm-8000-app-config 1G
```
tofu/kubernetes/bootstrap/volumes/persistent-volume/providers.tf
+1 -1
View File
@@ -1,7 +1,7 @@
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
source = "hashicorp/kubernetes"
version = ">= 2.31.0"
}
}
tofu/kubernetes/main.tf
+3 -3
View File
@@ -5,9 +5,9 @@ module "talos" {
proxmox = proxmox
}
image = var.talos_image
image = var.talos_image
cluster = var.talos_cluster_config
nodes = var.talos_nodes
nodes = var.talos_nodes
}
module "sealed_secrets" {
@@ -43,5 +43,5 @@ module "volumes" {
}
proxmox_api = var.proxmox
volumes = var.kubernetes_volumes
volumes = var.kubernetes_volumes
}
tofu/kubernetes/output.tf
+31 -1
View File
@@ -1,4 +1,34 @@
resource "local_file" "machine_configs" {
resource "local_file" "talos_machine_secrets" {
content = yamlencode({
cluster = module.talos.machine_secrets.cluster
secrets = module.talos.machine_secrets.secrets
trustdinfo = module.talos.machine_secrets.trustdinfo
certs = {
etcd = {
crt = module.talos.machine_secrets.certs.etcd.cert
key = module.talos.machine_secrets.certs.etcd.key
}
k8s = {
crt = module.talos.machine_secrets.certs.k8s.cert
key = module.talos.machine_secrets.certs.k8s.key
}
k8saggregator = {
crt = module.talos.machine_secrets.certs.k8s_aggregator.cert
key = module.talos.machine_secrets.certs.k8s_aggregator.key
}
k8sserviceaccount = {
key = module.talos.machine_secrets.certs.k8s_serviceaccount.key
}
os = {
crt = module.talos.machine_secrets.certs.os.cert
key = module.talos.machine_secrets.certs.os.key
}
}
})
filename = "output/talos-machine-secrets.yaml"
}
resource "local_file" "talos_machine_configs" {
for_each = module.talos.machine_config
content = each.value.machine_configuration
filename = "output/talos-machine-config-${each.key}.yaml"
tofu/kubernetes/sealed-secrets.auto.tfvars
+1 -1
View File
@@ -1,4 +1,4 @@
sealed_secrets_config = {
certificate_path = "bootstrap/sealed-secrets/certificate/sealed-secrets.crt"
certificate_path = "bootstrap/sealed-secrets/certificate/sealed-secrets.crt"
certificate_key_path = "bootstrap/sealed-secrets/certificate/sealed-secrets.key"
}
tofu/kubernetes/talos/config.tf
+21 -16
View File
@@ -1,24 +1,31 @@
locals {
first_control_plane_node_ip = [for k, v in var.nodes : v.ip if v.machine_type == "controlplane"][0]
kubernetes_endpoint = coalesce(var.cluster.vip, local.first_control_plane_node_ip)
extra_manifests = concat(var.cluster.extra_manifests, [
"https://github.com/kubernetes-sigs/gateway-api/releases/download/${var.cluster.gateway_api_version}/standard-install.yaml",
"https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${var.cluster.gateway_api_version}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
])
}
resource "talos_machine_secrets" "this" {
talos_version = var.cluster.talos_machine_config_version != null ? var.cluster.talos_machine_config_version : var.image.update_version
// Changing talos_version causes trouble as new certs are created
}
data "talos_client_configuration" "this" {
cluster_name = var.cluster.name
client_configuration = talos_machine_secrets.this.client_configuration
nodes = [for k, v in var.nodes : v.ip]
# Don't use vip in talosconfig endpoints
# ref - https://www.talos.dev/v1.9/talos-guides/network/vip/#caveats
endpoints = [for k, v in var.nodes : v.ip if v.machine_type == "controlplane"]
}
resource "terraform_data" "cilium_bootstrap_inline_manifests" {
input = [
{
name = "cilium-bootstrap"
name = "cilium-bootstrap"
contents = file("${path.root}/${var.cluster.cilium.bootstrap_manifest_path}")
},
{
name = "cilium-values"
name = "cilium-values"
contents = yamlencode({
apiVersion = "v1"
kind = "ConfigMap"
@@ -38,8 +45,8 @@ data "talos_machine_configuration" "this" {
for_each = var.nodes
cluster_name = var.cluster.name
# This is the Kubernetes API Server endpoint.
# ref - https://www.talos.dev/v1.9/introduction/prodnotes/#decide-the-kubernetes-endpoint
cluster_endpoint = "https://${var.cluster.endpoint}:6443"
# ref - https://www.talos.dev/latest/introduction/prodnotes/#decide-the-kubernetes-endpoint
cluster_endpoint = "https://${local.kubernetes_endpoint}:6443"
# @formatter:off
talos_version = var.cluster.talos_machine_config_version != null ? var.cluster.talos_machine_config_version : (each.value.update == true ? var.image.update_version : var.image.version)
# @formatter:on
@@ -57,8 +64,8 @@ data "talos_machine_configuration" "this" {
vip = var.cluster.vip
}), each.value.machine_type == "controlplane" ?
templatefile("${path.module}/machine-config/control-plane.yaml.tftpl", {
kubelet = var.cluster.kubelet
extra_manifests = jsonencode(var.cluster.extra_manifests)
kubelet = var.cluster.kubelet
extra_manifests = jsonencode(local.extra_manifests)
api_server = var.cluster.api_server
inline_manifests = jsonencode(terraform_data.cilium_bootstrap_inline_manifests.output)
}) : ""
@@ -79,9 +86,10 @@ resource "talos_machine_configuration_apply" "this" {
resource "talos_machine_bootstrap" "this" {
depends_on = [talos_machine_configuration_apply.this]
# Bootstrap with the first node. VIP not yet available at this stage, so cant use var.cluster.endpoint as it may be set to VIP
# Bootstrap with the first control plane node.
# VIP not yet available at this stage, so can't use var.cluster.vip
# ref - https://www.talos.dev/v1.9/talos-guides/network/vip/#caveats
node = [for k, v in var.nodes : v.ip if v.machine_type == "controlplane"][0]
node = local.first_control_plane_node_ip
client_configuration = talos_machine_secrets.this.client_configuration
}
@@ -105,11 +113,8 @@ resource "talos_cluster_kubeconfig" "this" {
talos_machine_bootstrap.this,
data.talos_cluster_health.this
]
# If using VIP, it should be up by now, but to be safer retrive from one of the nodes
# As mentioned don't use talosctl on vip
# ref - https://www.talos.dev/v1.9/talos-guides/network/vip/#caveats
# In kubeconfig endpoint will be polulated by cluster_endpoint from machine-config
node = [for k, v in var.nodes : v.ip if v.machine_type == "controlplane"][0]
# The kubeconfig endpoint will be populated from the talos_machine_configuration cluster_endpoint
node = local.first_control_plane_node_ip
client_configuration = talos_machine_secrets.this.client_configuration
timeouts = {
read = "1m"
tofu/kubernetes/talos/image.tf
+2 -2
View File
@@ -1,5 +1,5 @@
locals {
version = var.image.version
version = var.image.version
schematic = file("${path.root}/${var.image.schematic_path}")
schematic_id = jsondecode(data.http.schematic_id.response_body)["id"]
@@ -8,7 +8,7 @@ locals {
update_schematic = file("${path.root}/${local.update_schematic_path}")
update_schematic_id = jsondecode(data.http.updated_schematic_id.response_body)["id"]
image_id = "${local.schematic_id}_${local.version}"
image_id = "${local.schematic_id}_${local.version}"
update_image_id = "${local.update_schematic_id}_${local.update_version}"
# Comment the above 2 lines and un-comment the below 2 lines to use the provider schematic ID instead of the HTTP one
tofu/kubernetes/talos/inline-manifests/cilium-install.yaml
+25 -25
View File
@@ -8,9 +8,9 @@ roleRef:
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cilium-install
namespace: kube-system
- kind: ServiceAccount
name: cilium-install
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
@@ -58,28 +58,28 @@ spec:
serviceAccountName: cilium-install
hostNetwork: true
containers:
- name: cilium-install
image: quay.io/cilium/cilium-cli-ci:latest
env:
- name: KUBERNETES_SERVICE_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: KUBERNETES_SERVICE_PORT
value: "6443"
volumeMounts:
- name: values
mountPath: /root/app/values.yaml
subPath: values.yaml
command:
- cilium
- install
- --version=v1.17.3 # renovate: github-releases=cilium/cilium
- --set
- kubeProxyReplacement=true
- --values
- values.yaml
- name: cilium-install
image: quay.io/cilium/cilium-cli-ci:latest
env:
- name: KUBERNETES_SERVICE_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: KUBERNETES_SERVICE_PORT
value: "6443"
volumeMounts:
- name: values
mountPath: /root/app/values.yaml
subPath: values.yaml
command:
- cilium
- install
- --version=v1.17.3 # renovate: github-releases=cilium/cilium
- --set
- kubeProxyReplacement=true
- --values
- values.yaml
volumes:
- name: values
configMap:
tofu/kubernetes/talos/output.tf
+6 -2
View File
@@ -1,3 +1,8 @@
output "machine_secrets" {
value = talos_machine_secrets.this.machine_secrets
sensitive = true
}
output "machine_config" {
value = data.talos_machine_configuration.this
}
@@ -8,7 +13,6 @@ output "client_configuration" {
}
output "kube_config" {
#value = data.talos_cluster_kubeconfig.this
value = talos_cluster_kubeconfig.this
value = talos_cluster_kubeconfig.this
sensitive = true
}
tofu/kubernetes/talos/variables.tf
+5 -5
View File
@@ -15,14 +15,14 @@ variable "image" {
variable "cluster" {
description = "Cluster configuration"
type = object({
name = string
endpoint = string
name = string
vip = optional(string)
gateway = string
gateway = string
subnet_mask = optional(string, "24")
talos_machine_config_version = optional(string)
proxmox_cluster = string
kubernetes_version = string
proxmox_cluster = string
kubernetes_version = string
gateway_api_version = string
extra_manifests = optional(list(string))
kubelet = optional(string)
api_server = optional(string)
tofu/kubernetes/talos/virtual-machines.tf
+5 -5
View File
@@ -54,11 +54,11 @@ resource "proxmox_virtual_environment_vm" "this" {
# Optional DNS Block. Update Nodes with a list value to use.
dynamic "dns" {
for_each = try(each.value.dns, null) != null ? { "enabled" = each.value.dns } : {}
content {
servers = each.value.dns
}
}
for_each = try(each.value.dns, null) != null ? { "enabled" = each.value.dns } : {}
content {
servers = each.value.dns
}
}
ip_config {
ipv4 {
tofu/kubernetes/talos_cluster.auto.tfvars
+7 -13
View File
@@ -1,27 +1,21 @@
talos_cluster_config = {
name = "talos"
# This should point to the vip as below(if nodes on layer 2) or one of the nodes (if nodes not on layer 2)
# Note: Nodes are not on layer 2 if there is a router between them (even a mesh router)
# Not sure how it works if connected to the same router via ethernet (does it act as a switch then???)
name = "talos"
# Only use a VIP if the nodes share a layer 2 network
# Ref: https://www.talos.dev/v1.9/talos-guides/network/vip/#requirements
# Note This is Kubernetes API endpoint. Different from all mentions of Talos endpoints.
endpoint = "192.168.1.102"
vip = "192.168.1.99"
gateway = "192.168.1.1"
vip = "192.168.1.99"
gateway = "192.168.1.1"
# The version of talos features to use in generated machine configuration. Generally the same as image version.
# See https://github.com/siderolabs/terraform-provider-talos/blob/main/docs/data-sources/machine_configuration.md
# Uncomment to use this instead of version from talos_image.
# talos_machine_config_version = "v1.9.2"
proxmox_cluster = "homelab"
proxmox_cluster = "homelab"
kubernetes_version = "1.32.0" # renovate: github-releases=kubernetes/kubernetes
cilium = {
bootstrap_manifest_path = "talos/inline-manifests/cilium-install.yaml"
values_file_path = "../../k8s/infra/network/cilium/values.yaml"
}
extra_manifests = [
"https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml",
"https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.1/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
]
gateway_api_version = "v1.2.1" # renovate: github-releases=kubernetes-sigs/gateway-api
extra_manifests = []
kubelet = <<-EOT
extraArgs:
# Needed for Netbird agent https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#enabling-unsafe-sysctls
tofu/kubernetes/talos_image.auto.tfvars
+1 -1
View File
@@ -1,5 +1,5 @@
talos_image = {
version = "v1.9.2"
version = "v1.9.2"
update_version = "v1.9.3" # renovate: github-releases=siderolabs/talos
schematic_path = "talos/image/schematic.yaml"
# Point this to a new schematic file to update the schematic
tofu/kubernetes/talos_nodes.auto.tfvars
+3 -3
View File
@@ -2,8 +2,7 @@ talos_nodes = {
"ctrl-00" = {
host_node = "abel"
machine_type = "controlplane"
ip = "192.168.1.100"
#dns = ["1.1.1.1", "8.8.8.8"] # Optional Value.
ip = "192.168.1.100"
mac_address = "BC:24:11:2E:C8:00"
vm_id = 800
cpu = 8
@@ -28,13 +27,14 @@ talos_nodes = {
mac_address = "BC:24:11:2E:C8:02"
vm_id = 802
cpu = 4
ram_dedicated = 4096
ram_dedicated = 6144
#update = true
}
# "work-00" = {
# host_node = "abel"
# machine_type = "worker"
# ip = "192.168.1.110"
# dns = ["1.1.1.1", "8.8.8.8"] # Optional Value.
# mac_address = "BC:24:11:2E:A8:00"
# vm_id = 810
# cpu = 8
tofu/kubernetes/variables.tf
+6 -6
View File
@@ -32,14 +32,14 @@ variable "talos_image" {
variable "talos_cluster_config" {
description = "Talos cluster configuration"
type = object({
name = string
endpoint = string
name = string
vip = optional(string)
gateway = string
gateway = string
talos_machine_config_version = optional(string)
proxmox_cluster = string
kubernetes_version = string
extra_manifests = optional(list(string))
proxmox_cluster = string
kubernetes_version = string
gateway_api_version = string
extra_manifests = optional(list(string), [])
kubelet = optional(string)
api_server = optional(string)
cilium = object({