A distributed container overlay network daemon backed by a Raft-replicated store, built from scratch in Go.
Containers running on different hosts cannot talk to each other by default — they live in isolated network namespaces with private IPs that mean nothing outside their host. netd solves this by building a VXLAN overlay network: a virtual flat network stretched across all hosts where every container gets a unique IP that works cluster-wide.
The hard part is not the tunneling — Linux does that for you. The hard part is coordination: every host needs to know which container lives where, which MAC address maps to which IP, and which host owns which subnet. If that information lives in a single database, you have a single point of failure. netd stores all of this in a Raft-replicated state machine embedded directly in the daemon, so the network keeps working as long as a majority of nodes are alive.
This project is built as a deep learning exercise in Go systems programming, distributed consensus, and Linux kernel networking.
Without netd:
Host A Host B
┌─────────────┐ ┌─────────────┐
│ Container │ │ Container │
│ 172.17.0.2 │ ✗ cannot │ 172.17.0.2 │
│ │ reach │ │
└─────────────┘ └─────────────┘
same IP — collision, no routing between hosts
With netd:
Host A Host B
┌──────────────────────┐ ┌──────────────────────┐
│ Container A │ │ Container B │
│ 10.0.1.2 │ │ 10.0.2.3 │
│ │ │ │ │ │
│ veth │ │ veth │
│ │ │ │ │ │
│ vxlan0 (VTEP) ───┼──── UDP/VXLAN ─────┼─── (VTEP) vxlan0 │
└──────────────────────┘ tunnel over LAN └──────────────────────┘
netd agent programs the VTEP rules on each host
Rules come from a Raft-replicated store shared by all hosts
┌─────────────────────────────────────────────────────────────────┐
│ netd daemon │
│ (runs on every host) │
│ │
│ ┌──────────┐ ┌─────────────┐ ┌──────────────────────┐ │
│ │ HTTP │───▶│ Store │───▶│ Raft FSM │ │
│ │ API │ │ (writes │ │ (ContainerRecords, │ │
│ │ /register│ │ go through │ │ SubnetAllocs) │ │
│ │ /deregist│ │ Raft log) │ │ │ │
│ └──────────┘ └─────────────┘ └──────────┬───────────┘ │
│ │ │
│ FSM.Apply() │
│ │ │
│ ┌───────────▼──────────┐ │
│ │ Agent │ │
│ │ (watches FSM state, │ │
│ │ programs kernel) │ │
│ └───────────┬──────────┘ │
│ │ │
└──────────────────────────────────────────────────┼─────────────┘
│ netlink syscalls
┌──────────▼──────────┐
│ Linux kernel │
│ FDB entries │
│ ARP/neigh table │
│ vxlan0 device │
└─────────────────────┘
Raft cluster (3 nodes across 3 hosts):
┌─────────────────┐ AppendEntries ┌─────────────────┐
│ Raft Leader │──────────────────────▶│ Raft Follower │
│ (handles all │ │ (replicates │
│ writes) │──────────────────────▶│ log) │
└─────────────────┘ └─────────────────┘
│
│ AppendEntries
▼
┌─────────────────┐
│ Raft Follower │
│ (replicates │
│ log) │
└─────────────────┘
Write is committed only after majority (2 of 3) ack it.
FSM.Apply() is called on all nodes once committed.
-
Container starts on Host A. The container runtime calls
POST /registeronnetd's HTTP API with the container's IP, MAC, and the host's VTEP IP. -
API writes to the store. The store serializes a
Command{Op: "register", Record: ContainerRecord{...}}as JSON and callsraft.Apply(). -
Raft replicates the log entry. The leader appends it to its log and sends
AppendEntriesRPCs to all followers. Once a majority acknowledges, the entry is committed. -
FSM.Apply() is called on every node. Each node's state machine processes the command and updates its in-memory map of container records.
-
The agent on each host is watching the FSM. When the FSM state changes, the agent computes what VTEP rules are needed and calls into Linux via
netlinkto program them. -
The kernel now knows. The FDB (forwarding database) on each host maps the container's MAC address to the remote host's VTEP IP. The ARP/neighbor table maps the container's IP to its MAC. No ARP flooding needed.
-
Packets route. Container A sends a packet to Container B's IP. The kernel looks up the MAC in the ARP table, looks up the VTEP in the FDB, wraps the frame in a VXLAN/UDP packet, and sends it to Host B. Host B unwraps it and delivers it.
Requires Linux (VXLAN and netlink are Linux-only). Test environment uses 3 VMs or network namespaces.
git clone https://github.com/yourusername/netd
cd netd
go build ./cmd/netd
# Node 1 (leader bootstrap)
sudo ./netd --id node1 --bind 192.168.1.1:7000 --bootstrap
# Node 2
sudo ./netd --id node2 --bind 192.168.1.2:7000 --join 192.168.1.1:7000
# Node 3
sudo ./netd --id node3 --bind 192.168.1.3:7000 --join 192.168.1.1:7000
# Register a container
curl -X POST http://localhost:8080/register \
-d '{"id":"c1","ip":"10.0.1.2","mac":"aa:bb:cc:dd:ee:01","host_ip":"192.168.1.1"}'- The Raft paper (extended version) — read sections 1, 2, 3, 5 first
- Raft interactive visualization — watch elections and log replication live
- hashicorp/raft — the library this project embeds
- flannel-io/flannel — the real-world project this is a learning version of
- VXLAN — Linux Foundation wiki
- vishvananda/netlink — Go bindings for Linux netlink