Background

First a very basic primer on Kubernetes. This will be overly simplified since I’m only focusing on the parts that are relevant to networking. In other words, this is almost completely wrong, but it should be useful enough.

Kubernetes is a container orchestration system. In other words, a way to distribute multiple instances of multiple workloads over multiple machines. The primary resources we care about are Nodes and Pods. A node is a Linux host that is part of a cluster, and it’s what the workloads run on. A pod is a collection of one or more workloads that run in a namespace. For our purposes, we’ll just focus on the network namespace it runs on.

The namespace is (usually) completely isolated from the host system. To allow pods to talk to each other and the broader network, a Container Network Interface (CNI) plugin is used. In self-managed set-ups this is often Flannel or (more recently) Cilium. The CNI plugin generally works by defining a private network (PodCIDR) per node. When a new pod is created, it will create a virtual Ethernet interface. One end stays in the home namespace, the other is placed inside the pod and gets an IP address from the PodCIDR.

The other main thing the CNI plugin is responsible for is ensuring that pods can talk to each other even if they’re on different nodes. For simplicity, this is usually done with an overlay network (technically the same thing as a VPN, though usually not encrypted), often based on VXLAN. This is what we’re concerned with. While this setup does work pretty much everywhere, there’s a few downsides to it.

The Problem

While this approach works, it has a few downsides. To start with, it necessarily turns each node into a full NAT router. Any packet sent from a pod to outside has to be rewritten to come from the node’s IP address instead. This obfuscates the source. If one workload starts hammering a database outside the cluster for example, they’ll just see connections coming from a node, and you’ll have to try and find out which pod it was coming from. The same thing applies in the other direction. You can expose workloads in multiple ways, like NodePorts where each node listens on a port on its host address and forwards requests to one of the pods associated with it. There’s also solutions like MetalLB which advertise a virtual IP from the nodes.

The problem there is that by default, this will require DNAT. The incoming packet will have its source address rewritten to that of the node that first received it, so the response can be routed back properly. This means the service will no longer know what IP address hit it. You can avoid this by enforcing traffic to stay local, essentially disabling the proxy. The downside there is that you can no longer do load-balancing. With MetalLB in L2 mode this has the consequence that only one node (with usually one pod) will ever receive requests.

The final reason is complexity. Because there’s an overlay, it is no longer obvious how the network works. There’s another layer to debug when things go wrong. The CNI is something you don’t really notice or think about, until it breaks. Then it makes your life much harder than it needs to be.

Going Native

Most CNI plugins allow you to turn off the overlay network. When you do that, you’re now responsible for ensuring the nodes know how to route to each PodCIDR in the cluster. If all nodes are connected to the same switch (people also call this the same L2 network) this is actually quite simple. Each node just needs to route requests destined for a given PodCIDR to the node that PodCIDR belongs to. Cilium can automatically do this for you with the auto-direct-node-routes option. But what if your nodes are not directly connected, like if they’re in different datacentres? That’s where BGP comes in.

Test Setup

To prove this approach, I set up a testing environment using a server at home with libvirt/QEMU. My aim was to set up two routers with one Kubernetes node connected to each, to replicate two datacentres. I chose OpenWRT for the routers and Talos Linux for the nodes. For the routers anything you can run BGP with should work. For the nodes, anything you can run Kubernetes on will do, though for Cilium you’ll want to ensure you have a kernel with eBPF support. If you want to replicate this setup, here’s how you do it. If you already have a cluster with Cilium or you’d prefer to set it up yourself, feel free to skip ahead to the BGP section.

OpenWRT

First, create three isolated networks. One for each DC, and one peering network that connects the two routers. Then create your first router. OpenWRT assumes the first interface is the LAN and the second is the WAN. So your first NIC should be the first DC network. The second should grant you access inside and the router access to the internet. As I was hosting the VMs on my server and accessing them from my laptop, I used a macvtap device. If you’re using the same machine for both, you may want a bridge device instead. The third NIC is your peering network.

To boot it, I went with the fancy way of downloading the kernel and rootfs separately. Go to the downloads for OpenWRT and fetch the kernel and ext4-rootfs image for your target (probably x86-64).

As of writing, you can find the latest release here: OpenWRT 23.05.3 x86-64

Set the VM’s disk to the rootfs image. Enable direct kernel boot, point it at the kernel, and add the kernel args:

root=/dev/vda rootwait

Now start it up. Next, you’ll want to configure the firewall and network. Since this is a test setup and I trust the network, for simplicity I just opened everything. So first we change the default wan and lan zones in /etc/config/firewall:

config zone
    option name 'lan'
    list network 'lan'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    option masq_allow_invalid '1'

config zone
    option name 'wan'
    list network 'wan'
    list network 'wan6'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    option mtu_fix '1'
    option masq '1'
    option masq_allow_invalid '1'

The masq_allow_invalid option is important, because we will be passing traffic that was routed to us from a node. We’ll also want rules for a new peering zone we’ll create, so add the following:

config zone
    option name 'peering'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    list network 'peering'
    option masq_allow_invalid '1'

config forwarding
    option src 'peering'
    option dest 'lan'

config forwarding
    option src 'lan'
    option dest 'peering'

This sets up forwarding between the peering and lan zones. That is traffic coming from a node in DC1 going to a node in DC2. We don’t need to set up forwarding to WAN, since each router will be talking to WAN directly.

Next, in /etc/config/network, I changed the configuration for lan to have the IP address 10.0.0.1/24 from the default of 192.168.0.1/24. This is just to keep things simple, also because my home network is in the 192.168.0.0/16 range. You’ll also want to add the peering interface:

config interface 'peering'
    option proto 'static'
    option device 'eth2'
    list ipaddr '172.16.0.1/24'

This is where we’ll connect the routers. In /etc/config/dhcp you’ll want to make one simple change to add the following to the config dnsmasq section:

    option localservice '0'

This ensures that pods will also be able to make DNS requests. Without this option, dnsmasq will refuse any requests coming from PodCIDRs.

We also want to do BGP. I chose to use Bird 2. So, install it:

opkg install bird2

And then overwrite /etc/bird.conf. I mostly copied this config over from something I had lying around for dn42. It could probably be simplified, but this works. As an aside, I chose to use one ASN for the routers and one for the cluster. You could use more, but don’t use the same ASN everywhere. If you do that, the routes won’t propagate properly, as Bird assumes that all routers within the same AS already have routes to each other.

define OWNAS  = 65000;
define OWNIP  = 172.16.0.1;
define OTHIP  = 172.16.0.2;
define OWNNET = 10.0.0.0/24;
define KUBEAS = 65001;

router id OWNIP;
protocol device {
  scan time 10;
}
protocol kernel {
  scan time 20;
  ipv4 {
    import none;
    export filter {
      if source = RTS_STATIC then reject;
      krt_prefsrc = OWNIP;
      accept;
    };
  };
}
protocol static {
  route OWNNET reject;
  ipv4 {
    import all;
    export none;
  };
}
protocol bgp ibgp_node {
  local as OWNAS;
  neighbor OTHIP as OWNAS;
  direct;
  ipv4 {
    next hop self;
    import all;
    export all;
  };
}

We’ll add the cluster node later, but it first needs an IP address. At this point you can shut down this VM and clone it. Change the first NIC to point to the DC2 network instead of DC1. You should be able to start it fine. At that point, change the LAN in /etc/config/network to 10.0.1.1/24 instead, and swap OWNIP and OTHIP in /etc/bird.conf. Then reboot this VM and restart the other.

Talos Linux

Now we’ll create two more VMs, one control plane node and one worker node. At this point there’s a few utilities you’ll want to install on your local system:

• talosctl
• kubectl
• helm
• cilium-cli

Check your talosctl version and get the appropriate image from the Talos Image Factory with qemu-guest-agent. Fetch the kernel and initramfs from the bottom of the page, and configure a VM with at least 2GB RAM, an empty disk (at least 10GB) and have it boot from the downloaded kernel and initrd. Use the following for kernel args:

talos.platform=metal console=ttyS0 console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512

This VM only needs one NIC, the DC1 network. The VM should come up and get an IP address from the OpenWRT router. You can now shut down this VM and clone it. The clone should again have its NIC switched to DC2.

Now you can restart the first Talos node. Think of a name (I just called it libvirt) and note the IP address of the node. It should be in the 10.0.0.0/24 range, I happened to get 10.0.0.117. Turn that into a HTTPS URL on port 6443. In my example, https://10.0.0.117:6443. This is your cluster-endpoint. We’ll also need to prepare a patch file so it’s ready for Cilium and so that it will install to /dev/vda instead of /dev/sda. Create patch.yaml with the following contents:

cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true
machine:
  install:
    disk: /dev/vda

Then, from your workstation, run:

talosctl gen config <cluster-name> <cluster-endpoint> --config-patch @patch.yaml
export TALOSCONFIG=$PWD/talosconfig

You will want to edit the generated controlplane.yaml and worker.yaml to install to /dev/vda instead of /dev/sda. Now we can install this node as the control plane:

talosctl apply-config --insecure -n <node-ip> --file controlplane.yaml

The --insecure is needed because the node doesn’t have the certificates we generated earlier yet. The node should now reboot. You may want to run the following to make things easier:

talosctl config node <node-ip>
talosctl config endpoint <node-ip>

When the node is running, you’ll have to bootstrap the cluster.

talosctl bootstrap
talosctl kubeconfig

This also sets up kubectl to reach the cluster. The Talos node will hang on phase 18/19 because we currently have no CNI. So we’ll have to install Cilium:

cilium install \
    --helm-set=ipam.mode=kubernetes \
    --helm-set=kubeProxyReplacement=true \
    --helm-set=securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --helm-set=securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --helm-set=cgroup.autoMount.enabled=false \
    --helm-set=cgroup.hostRoot=/sys/fs/cgroup \
    --helm-set=k8sServiceHost=localhost \
    --helm-set=k8sServicePort=7445 \
    --helm-set=bgpControlPlane.enabled=true \
    --helm-set=routingMode=native \
    --helm-set=enableIPv4Masquerade=false

The first set of options are what’s needed to enable Cilium on Talos. The last three set up BGP and native routing. Now we can add the second node. Boot up the second Talos VM and note its IP. Then run:

talosctl apply-config --insecure -n <node-ip> --file worker.yaml

BGP

If you haven’t been following along and you’ve already got a cluster with Cilium, just ensure you have the following Helm values set:

bgpControlPlane.enabled=true
routingMode=native
enableIPv4Masquerade=false

We disable masquerading because the upstream router will take care of it when needed. We need to label our nodes so Cilium knows which router to talk to. Find your node’s name with kubectl get nodes -o wide and label them. The node in 10.0.0.0/24 is dc1, the one in 10.0.1.0/24 is dc2.

kubectl label nodes <node-name> dc=dc1
kubectl label nodes <node-name> dc=dc2

Now we can add the peering policy. Create a file peering.yaml with the following contents:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: dc1
spec:
  nodeSelector:
    matchLabels:
      dc: dc1
  virtualRouters:
  - localASN: 65001
    exportPodCIDR: true
    neighbors:
    - peerAddress: '10.0.0.1/32'
      peerASN: 65000
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: dc2
spec:
  nodeSelector:
    matchLabels:
      dc: dc2
  virtualRouters:
  - localASN: 65001
    exportPodCIDR: true
    neighbors:
    - peerAddress: '10.0.1.1/32'
      peerASN: 65000

And apply it:

kubectl apply -f peering.yaml

It may take a few moments, but after a while cilium bgp routes should show routes being advertised to the routers. At this point, all connectivity should just work, with no overlay.

How this works

The nodes are now advertising their PodCIDRs to the router they’re connected to. That router in turn advertises it to the other router. So if a pod from node-1 tries to reach a pod in node-2, this is what happens:

1. The host (node-1) looks up the target IP address in its routing table. As there is no matching entry, the packet is forwarded to the default gateway. That’s router-1.
2. router-1 does have a route for the target IP address, because it was advertised by router-2. So it forwards the packet to it over the peering network.
3. router-2 has a route advertised by node-2, so it goes there over the lan network.
4. node-2 forwards the packet to the pod.

With more nodes on the same network it is preferable to add auto-direct-node-routes=true to the Cilium configuration so traffic within one network doesn’t go through the router. In addition, you may want to set the PodCIDRs specifically using Cilium Cluster-Pool IPAM, to avoid clobbering the network. At that point, this setup should scale to any amount of nodes or routers.

Diagram

For clarity, I’ve included a diagram of this setup, with routing tables per machine and example routes:

Zigbee Protocol

To understand what’s going on it helps to know a little bit about how Zigbee actually works. This is going to be a gross oversimplification, but it should be enough for our purposes. Zigbee is a wireless mesh protocol that operates in the 2.4GHz band. A network consists of one coordinator, routers and end devices. Generally speaking, any device with a mains connection like a light or a relay will act as a router. Any battery-powered devices like most switches will be end devices. A router can pass messages between devices, an end device can’t.

A coordinator is in charge of adding devices to the network, making sure they’re provisioned with the network key. It’s also what you use to issue or receive commands from outside of the network. This is what lets you schedule timers, or control lights from your phone. Many people will be using a hub of some kind, but you can also buy USB adapters that can turn any computer into a coordinator.

Functionally, a device (router or end device) is divided up into one or more endpoints, and each endpoint has one or more clusters. Endpoints logically divide a single device. A smart power strip may have an endpoint per socket, for example, so they can be addressed individually. Most devices have only one (useful) endpoint.

Clusters

Clusters are where things get interesting. A cluster is a collection of attributes and commands. A light, for example, will have at least an On/Off cluster. This cluster has attributes like OnOff that tell you whether the light is currently on or off, and commands like Off, On and Toggle.

What’s interesting is that a light switch will also have the same clusters. For the light, this would be an input cluster (it can receive commands) but the light switch will have On/Off as an output cluster. When the on button is pressed, the switch will generate an On command from the On/Off cluster.

This can be used to bind devices together. If you bind the On/Off cluster of the switch to a light, then the switch will remember the light’s address, and send On/Off commands to the light when a button on the switch is pressed. This happens without any involvement of the coordinator. Naturally, there’s a limit to how many devices you can link to a given device. We’ll get to that later.

Attempt 0 - IKEA

This part is not especially enlightening, but I want to include it for completion’s sake. When I first got these lights I wasn’t really looking forward to figuring everything out, so I figured I’d just get IKEA’s DIRIGERA hub to start with. I could always replace it later. Unfortunately I could not actually get the thing to work. I had it connected properly, the app found it, but it would not update its firmware or move beyond that point. According to customer service, this is a common issue.

Fortunately the lights can work without any hub at all. You just have to get the remote really close to the light while holding the pair button and it’ll connect directly without a mesh network. Not ideal, but it would do for now. I connected the six spots in the living room to one remote and a couple of bulbs in the kitchen to another, and I’d at least fixed the primary issue.

Attempt 1 - ZHA

By now I’d acquired a Sonoff ZBDongle-E adapter. As I’d been running Home Assistant anyway, it seemed like a logical fit to use ZHA. This worked pretty well. It’s a matter of resetting the bulbs by turning them off and on 6 times until they start flashing, then telling the coordinator to permit joining. A short while later, the light shows up in HA. Same thing with the remote, by using the pairing button. ZHA also supports binding, although the option is hidden away slightly under Zigbee settings.

I bound the lights to my remote one by one, and it all worked fine, right up until the sixth (and last) one. No matter what I did, no matter what order, the sixth to be bound would always fail. It took me quite a while to figure out what was actually going on. Remember when I said there’s a limit to how many devices you can bind to a switch? Well, for this particular switch, it’s 10.

I figured 6 was less than 10, but as it turns out, that’s counting clusters, not devices. It was binding both the OnOff cluster and the Level Control cluster. By the fifth light, it had reached its limit of 10 clusters and refused any new binds.

I did manage to fix this. Using zha-toolkit it’s possible to send a manual bind command with only the OnOff cluster. It’s quite unwieldy to have to copy the address and find the cluster though. It also caused an additional issue where the state of the light wasn’t always reflected in HA, because the command had made it to the light but the updated state of the light didn’t get reported to the coordinator. I fixed this with an automation that polled the lights after receiving a button press on the switch, but it wasn’t pretty. You could also fix this by hand with zha-toolkit, but at this point I was kind of fed up with it. Especially since the binding requires pressing a button on the switch to wake it up each time.

Attempt 2 - Z2M

By now, I’d decided to start over with Zigbee2MQTT. It does require setting up an MQTT server and hooking it up to Home Assistant, but beyond that it’s just as easy to get going. Crucially however, Z2M lets you click each cluster you want to bind. So I could simply select just the OnOff cluster, and it just worked. This didn’t impact the ability to dim the lights by holding the button by the way, which you’d think is what Level Control would do.

Z2M also sets up reporting automatically when binding a cluster. This means it tells the light to send a message to the coordinator whenever its attributes change. That way, it would automatically update the view in HA without any polling needed. I also kind of like decoupling the Zigbee part from the HA part.

I have no idea if ZHA is going to implement these things and render this obsolete, but for now I’m just writing down what I wished I’d known about a month before I got all this set up.