98 lines
2.2 KiB
Markdown
98 lines
2.2 KiB
Markdown
|
---
|
||
|
|
title: What to look out for when setting up a K3S cluster on PIs
|
||
|
|
date:
|
||
|
|
created: 2025-01-07
|
||
|
|
tags:
|
||
|
|
- k8s
|
||
|
|
- k3s
|
||
|
|
- pi
|
||
|
|
---
|
||
|
|
|
||
|
|
<!-- more -->
|
||
|
|
|
||
|
|
## Configure the nodes
|
||
|
|
- static IP
|
||
|
|
- DNS: disable systemd-resolved, or it will conflict with the embeded DNS server of K3S
|
||
|
|
- storage:
|
||
|
|
- nfs-kernel-sever, nfs-common
|
||
|
|
- ZFS
|
||
|
|
|
||
|
|
## Install K3S
|
||
|
|
1st master:
|
||
|
|
`curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --cluster-init`
|
||
|
|
|
||
|
|
Other masters:
|
||
|
|
`curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --server https://<ip or hostname of server1>:6443`
|
||
|
|
|
||
|
|
Workers:
|
||
|
|
`curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - agent --server https://<ip or hostname of server>:6443`
|
||
|
|
|
||
|
|
## Access
|
||
|
|
The kubectl config is located at `/etc/rancher/k3s/k3s.yaml`.
|
||
|
|
|
||
|
|
## Install basic services
|
||
|
|
- keepalived
|
||
|
|
- storage
|
||
|
|
- [NFS](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner)
|
||
|
|
- [ZFS](https://github.com/ccremer/kubernetes-zfs-provisioner)
|
||
|
|
- cert-manager
|
||
|
|
```bash
|
||
|
|
helm repo add jetstack https://charts.jetstack.io
|
||
|
|
helm repo update
|
||
|
|
|
||
|
|
helm install \
|
||
|
|
cert-manager jetstack/cert-manager \
|
||
|
|
--namespace cert-manager \
|
||
|
|
--create-namespace \
|
||
|
|
--set installCRDs=true
|
||
|
|
```
|
||
|
|
|
||
|
|
- DNS: coreDNS
|
||
|
|
- configure traefik
|
||
|
|
```yaml
|
||
|
|
# /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
|
||
|
|
apiVersion: helm.cattle.io/v1
|
||
|
|
kind: HelmChartConfig
|
||
|
|
metadata:
|
||
|
|
name: traefik
|
||
|
|
namespace: kube-system
|
||
|
|
spec:
|
||
|
|
valuesContent: |-
|
||
|
|
additionalArguments:
|
||
|
|
- "--entryPoints.dnsudp.address=:53/udp"
|
||
|
|
- "--entryPoints.dnstcp.address=:53/tcp"
|
||
|
|
...
|
||
|
|
ports:
|
||
|
|
dnsudp:
|
||
|
|
port: 53
|
||
|
|
exposedPort: 53
|
||
|
|
expose:
|
||
|
|
default: true
|
||
|
|
protocol: UDP
|
||
|
|
dnstcp:
|
||
|
|
port: 53
|
||
|
|
exposedPort: 53
|
||
|
|
expose:
|
||
|
|
default: true
|
||
|
|
protocol: TCP
|
||
|
|
...
|
||
|
|
```
|
||
|
|
- install registry
|
||
|
|
- confgure registry
|
||
|
|
```yaml
|
||
|
|
# /etc/rancher/k3s/registries.yaml
|
||
|
|
mirrors:
|
||
|
|
docker.io:
|
||
|
|
endpoint:
|
||
|
|
- https://registry-mirror.example.org/v2
|
||
|
|
configs:
|
||
|
|
registry.example.org:
|
||
|
|
auth:
|
||
|
|
username: username
|
||
|
|
password: password
|
||
|
|
registry-mirror.example.org:
|
||
|
|
auth:
|
||
|
|
username: username
|
||
|
|
password: password
|
||
|
|
```
|