Flannel — default CNI yang digunakan k3s — sederhana, ringan, dan “works out of the box.” Untuk banyak skenario seperti homelab dan cluster kecil, itu sudah cukup. Tapi Flannel tidak fit ke semua kebutuhan networking cluster Kubernetes kita. Salah satunya multi-tenancy: kita butuh pemisahan jaringan yang jelas antar tim/lingkungan (dev/stage/prod), bahkan dengan IP/subnet yang sama (di VPC/logical router yang berbeda).
Kube OVN, berbasis OVN/OVS, menawarkan isolated logical switch/subnet, VPC, floating IP, routing, dan fitur lain yang tidak dimiliki oleh Flannel atau banyak CNI lain. Termasuk: IPAM yang deterministik (static/reservasi IP lebih gampang), EIP/SNAT untuk egress yang konsisten, ACL/policy L3/L4 yang lebih ketat, opsi overlay (Geneve/VXLAN) maupun underlay/VLAN, hingga QoS/bandwidth control dan observabilitas yang lebih rapi.
Kita bisa mengadopsi Kube-OVN di k3s jika membutuhkan fitur-fitur di atas atau ingin cluster ringan dengan rasa VPC.
Setup k3s Without Default CNI
Install k3s tanpa flannel
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy" sh -s
Verifikasi instalasi, seharusnya status node NotReady
kubectl get nodes
Setup Kube OVN
Install Kube OVN
wget https://raw.githubusercontent.com/kubeovn/kube-ovn/release-1.12/dist/images/install.sh
bash install.sh
Jika script install nya return error, bisa kita abaikan. Ini adalah error plugin kubectl-ko saat melakukan health check pada daemonset kube-proxy, yang di mana k3s menjalakan kube-proxy embeded di proses k3s, bukan sebagai daemonset/pod.
Verifikasi instalasi
kubectl get po -A -o wide
test ping cluster ip
root@dama-k3s:~# ping 10.16.0.7
PING 10.16.0.7 (10.16.0.7) 56(84) bytes of data.
64 bytes from 10.16.0.7: icmp_seq=1 ttl=63 time=1.82 ms
^C
--- 10.16.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.821/1.821/1.821/0.000 ms
Membuat Pod dengan static ip
Secara default, saat kita membuat pod akan diberikan dynamic ip dari cni kube-ovn, tapi kita juga bisa mendefinisikan static ip.
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: nginx-static
annotations:
ovn.kubernetes.io/ip_address: "10.16.1.123"
spec:
containers:
- name: nginx
image: docker.io/library/nginx:alpine
Setelah di-apply kita bisa cek ip pod yang sudah dibuat.
Untuk subnet atau range ip default nya adalah 10.16.0.0/16, bisa diubah di script install.sh saat proses instalasi.
Create VPC
Pod Multiple IP
Multus
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
kubectl -n kube-system get pods -l app=multus -w
Fix multus OOM
kubectl -n kube-system patch ds kube-multus-ds --type='json' -p='[
{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"ghcr.io/k8snetworkplumbingwg/multus-cni:stable-thick"},
{"op":"add","path":"/spec/template/spec/containers/0/env","value":[{"name":"MALLOC_ARENA_MAX","value":"2"}]},
{"op":"replace","path":"/spec/template/spec/containers/0/resources","value":{
"requests":{"cpu":"50m","memory":"64Mi"},
"limits":{"cpu":"300m","memory":"256Mi"}
}}
]'
kubectl -n kube-system patch ds kube-multus-ds --type='json' -p='[
{"op":"add","path":"/spec/template/spec/containers/0/env","value":[{"name":"MALLOC_ARENA_MAX","value":"2"}]},
{"op":"replace","path":"/spec/template/spec/containers/0/resources","value":{
"requests":{"cpu":"50m","memory":"64Mi"},
"limits":{"cpu":"500m","memory":"512Mi"}
}}
]'
kubectl -n kube-system rollout status ds/kube-multus-ds
Setup 1 isolated subnet
- setup eternal network
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: ovn-vpc-external-network
spec:
protocol: IPv4
provider: ovn-vpc-external-network.kube-system
cidrBlock: 15.15.15.0/24
gateway: 15.15.15.1 # IP address of the physical gateway
excludeIps:
- 15.15.15.1..15.15.15.15
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: ovn-vpc-external-network
namespace: kube-system
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens3",
"mode": "bridge",
"ipam": {
"type": "kube-ovn",
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
"provider": "ovn-vpc-external-network.kube-system"
}
}'
---
kind: ConfigMap
apiVersion: v1
metadata:
name: ovn-vpc-nat-config
namespace: kube-system
data:
image: docker.io/kubeovn/vpc-nat-gateway:v1.12.35
---
kind: ConfigMap
apiVersion: v1
metadata:
name: ovn-vpc-nat-gw-config
namespace: kube-system
data:
enable-vpc-nat-gw: 'true'
- Subnet user
apiVersion: v1
kind: Namespace
metadata:
name: ns-1001
---
kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
name: test-vpc-1001
spec:
namespaces:
- ns-1001
staticRoutes:
- cidr: 0.0.0.0/0
nextHopIP: 10.0.1.254
policy: policyDst
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
name: net-1001
spec:
vpc: test-vpc-1001
cidrBlock: 10.0.1.0/24
protocol: IPv4
provider: net-1001.ns-1001.ovn
namespaces:
- ns-1001
---
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
name: gw-1001
spec:
vpc: test-vpc-1001
subnet: net-1001
lanIp: 10.0.1.254
selector:
- "kubernetes.io/os: linux"
externalSubnets:
- ovn-vpc-external-network
---
kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
name: eip-1001
spec:
natGwDp: gw-1001
---
kind: IptablesSnatRule
apiVersion: kubeovn.io/v1
metadata:
name: snat-1001
spec:
eip: eip-1001
internalCIDR: 10.0.1.0/24
---
kind: IptablesDnatRule
apiVersion: kubeovn.io/v1
metadata:
name: dnat-1001
spec:
eip: eip-1001
externalPort: '10001'
internalIp: 10.0.1.2
internalPort: '22'
protocol: tcp
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: net-1001
namespace: ns-1001
spec:
config: '{
"cniVersion": "0.3.0",
"type": "kube-ovn",
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
"provider": "net-1001.ns-1001.ovn"
}'
---
apiVersion: v1
kind: Pod
metadata:
namespace: ns-1001
name: nginx
annotations:
ovn.kubernetes.io/ip_address: "10.0.1.2"
spec:
containers:
- name: pod-1001
image: docker.io/library/nginx:alpine
---
apiVersion: v1
kind: Pod
metadata:
namespace: ns-1001
name: ssh
annotations:
ovn.kubernetes.io/ip_address: "10.0.1.253"
#k8s.v1.cni.cncf.io/networks: ns-1002/net-1002
#net-1002.ns-1002.kubernetes.io/ip_address: 10.0.1.50
#ovn.kubernetes.io/subnet: net-1002
#k8s.v1.cni.cncf.io/networks: |
spec:
containers:
- name: ssh
image: alpine:latest
command: ["sh","-c","echo 'Hello from alpine'; uname -a; sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: ubuntu-dind2
namespace: ns-1001
annotations:
ovn.kubernetes.io/ip_address: "10.0.1.3"
spec:
securityContext:
runAsUser: 0
containers:
- name: dind
image: radendi/ubuntu-22-dind
securityContext:
privileged: true # aman karena terkurung micro-VM Kata
allowPrivilegeEscalation: true
env:
- name: DOCKER_TLS_CERTDIR
value: ""
- name: DOCKER_MTU
value: "1450"
tty: true
stdin: true
resources:
requests: { cpu: "1", memory: "1Gi" }
limits: { cpu: "2", memory: "2Gi" }
