Monday Boot Sequence: Cluster Running

Date: Monday, May 27, 2025

Location: Somewhere in the damp clay of the Black Swamp, NW Ohio
Status: Extra weekend day. Coffee hot. Logs rolling. No corporate overhead today.

While most folks are sleeping off their barbecues or dreading a Tuesday morning reentry, I’m here firing up my own flavor of madness, a virtual ISP stack that’s finally starting to feel like something alive. This one’s for the lab rats and backroom thinkers: DHCP at scale, simulated clients by the thousands, and routers that know their place in the order of things.

We’ve been poking at this for a while now, but today the gears meshed tighter. Kea is dishing out leases like a government cheese truck in 1982, and the VyOS router on the front line doesn’t even blink.

Here’s what’s working so far.

The VyOS Side: Opening the Floodgates

We carved out a fat /16 subnet, 10.1.0.0/16, with the VyOS edge router sitting comfortably at 10.1.0.1. That’s 65,534 usable addresses for our hungry simulated clients. A basic config like:

set interfaces ethernet eth4 address '10.1.0.1/16'
set interfaces ethernet eth4 description 'ISP Residential DHCP Segment'

The purpose? Build a broad, flat testbed to emulate a wide-open residential service block. No NAT yet, no fancy VLAN splits—just raw lease distribution and churn, like a DOCSIS network with no regard for reality.

Kea Configuration: Memory-Backed Lease Firehose

Here’s the key slice of our working kea-dhcp4.conf:

{
  "Dhcp4": {
    "interfaces-config": {
      "interfaces": [ "enp6s19" ]
    },
    "lease-database": {
      "type": "memfile",
      "persist": true,
      "name": "/var/lib/kea/kea-leases4.csv"
    },
    "subnet4": [
      {
        "subnet": "10.1.0.0/16",
        "interface": "enp6s19",
        "valid-lifetime": 120,
        "pools": [
          { "pool": "10.1.0.100 - 10.1.255.254" }
        ],
        "option-data": [
          { "name": "routers", "data": "10.1.0.1" },
          { "name": "domain-name-servers", "data": "10.0.1.10" }
        ]
      }
    ]
  }
}

The lease lifetime is deliberately short—just 2 minutes—to create high churn. This gives us something like a heat map of lease volatility, which turns out to be gold for systems like GoatWatch, where I’m bucketing activity over time.

Performance Testing: perfdhcp and the Drunken Lease Crawl

We ran perfdhcp in rage mode, 15,000+ simulated clients spamming DISCOVER packets like an army of drunken IoT fridges.

Even after halting the test, Kea kept writing out leases. There’s a lag between when the requests stop and when the in-memory write buffers finish committing to disk. Watching that kea-leases4.csv file grow after the fact was a good reminder of how not-quite-real-time things can still punch through.

Final lease file count from that run? 647,677 entries. Not bad for a throwaway test on a private subnet.

Real Client Emulation: Next Phase Ideas

perfdhcp is fast but dumb. It doesn’t maintain timers or act like a living client. So next up is writing something in Go that spawns 100+ synthetic clients, each maintaining lease state, renewal windows, and randomized behavior. Maybe wrap them in pods or isolate them by namespace—whatever gets me closer to a world where 10,000 leases don’t all come due at the exact same second.

Why This Matters (To Me, Anyway)

I don’t run hobby projects. I run stress tests for reality. If this lab stack bends, breaks, or screams—it’s a sign I’m pushing the right way. And when it doesn’t, I know I’ve got something that scales, something worth handing off to a real ops team one day.

For now, this all stays in the lab—private IP blocks, isolated VMs, zero overlap with production systems. But the design thinking here? This is the skeleton for CodexMCP’s virtual ISP environment. And it all starts with dynamic IPs and the churn they bring.

No tickets to clear today. No Slack to answer. Just logs, loops, and the sound of a stack coming to life.

I’ll post the full test config soon, maybe even package the Go clients once they’re cooking. But for now, just know the engine is running—and the DHCP floodgates are wide open.

-Pauses Are Good, But Don't Think Too Hard
--Bryan